Why I’m Using Mailgun with Geogram
I’m a technology consultant and writer living in Seattle. If you found this tutorial helpful, please consider sharing it on Twitter. You can follow me at @reifman.
Mailgun has published this to their blog: How Geogram built a free group email service using Yii for PHP with MySQL.
This tutorial describes why I chose Mailgun for Geogram and how I implemented its email services with the Mailgun API. I provide some examples of integrating and using Mailgun so that you can see how you might do this in your own application.
What is Geogram?
Geogram provides free email groups for your block, your neighborhood and your favorite places. Users discover groups based on their location (with a bit of HTML5 Geolocation and Google Maps magic). Once they join or create groups on the Geogram website, most of their interaction with our service occurs via email.
In the past, building a scalable email service around processing and sending email was a huge undertaking. In addition to performance issues, managing your server reputation and spam require a great deal of time and specialized expertise (Geogram’s competitor has 55 employees right now). For more background on the complexity of managing your own email services, read: So, You’d Like to Send Some Email (through code).
Why Mailgun?
Geogram is primarily an email-based service so finding a cloud-based solution that could be easily integrated, scale economically and provide reliable support was critical. I also wanted a service that managed inbound email. Mailgun not only processes inbound email but also posts parsed elements of each message to your website in a simple way.
Mailgun is also fast and reliable. Mail sent via Mailgun seems to arrive instantaneously and rarely goes to Junk folders. This can be especially helpful when sending out activation links to new users. There’s virtually no waiting for users or uncertainty for you; after all, you never get a second chance to make a first impression. For these reasons, I chose Mailgun. And, as I studied the service and began working with it, the Mailgun support team exceeded my expectations repeatedly.
Geogram Technology
Geogram uses the open source Yii Framework for PHP with MySQL. Yii is essentially an MVC framework like Ruby on Rails, but with all the simplicity and maturity of PHP. It’s fast, efficient and relatively straightforward. It also includes scaffolding/code generation, active record, transaction support, I18n localization, caching support et al. The documentation, community support and available plugins are also quite good.
In the past, it would have taken me a year to build Geogram and longer to properly mature it. I was new to Yii but still was able to code Geogram in three months (including Mailgun integration) – almost entirely solo. I’m still maturing the service but Yii has been a huge part of my quick success. So, the integration examples provided use Yii, PHP and cURL. They should be understandable to most PHP developers but you may notice bits of Yii-specific code that’s new to you. However, these code examples are generic enough that they should help you understand how you might integrate Mailgun into your own application. Mailgun’s documentation has API examples for using cURL directly or PHP, Ruby, Java, C# and Python.
In addition to the excerpts below, I’ve created a public Github repository (geogram-mailgun-tutorial) with code samples from this tutorial for you to review in a more organized fashion.
Note: This tutorial does not provide you a full working codebase of your own list-based application. It’s just meant to help you understand how to work with Mailgun’s API, how you might approach common challenges to running email-based applications and to ease the process of getting started.
Configuring Mailgun
Before you begin, make sure your server has support for PHP cURL. On Ubuntu Linux, we do this:
sudo apt-get install php5-curl
Mailgun will provide you a secure URL and API key:
We place these in a PHP .ini configuration file outside of the Apache web directory:
mailgun_api_key="key-xx-xxxxxxxxxxxxxxxxxxxx" mailgun_api_url="https://api.mailgun.net/v2"
As part of its startup process, we ask Yii to load these keys into the Yii::app()->params array. Whenever we want to connect to Mailgun, we call this function:
public function configMailgun() {
//set the project identifier based on GET input request variables
$this->_api_key = Yii::app()->params['mailgun']['api_key'];
$this->_api_url = Yii::app()->params['mailgun']['api_url'];
}
We’ve also created a helper function for initiating the most common structure for preparing Mailgun cURL requests – (beware it varies for other API methods):
private function setup_curl($command = 'messages') {
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_USERPWD, 'api:'.Yii::app()->params['mailgun']['api_key']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_URL, Yii::app()->params['mailgun']['api_url'].'/yourdomain.com/'.$command);
return $ch;
}
Sending Mail to Geogram Groups
When a user posts a message via the Geogram website to send to a group, there are a couple of things that we do in sequence which might seem unusual to you.
Rather than using the API to post every message to Mailgun in real time, the first thing we do is store the message to our own database. Not only do we know our database is always available but this also provides fast visual feedback for users.
As the number of messages passing between Geogram and Mailgun scales, having the ability to asynchronously manage posting and receiving messages will be critical.
Here’s a sample database schema migration for our Post table and an example of how Yii’s Active Record implementation saves form data to it.
We run two different background processes that operate on the sent messages. Yii supports running console daemons with full ActiveRecord support. Although, you can also use a CRON job or write a script in your favorite language e.g. Python, NodeJS.
Determining the Recipient List
The first process looks for pending messages and completes pre-processing. During this phase, we determine the exact recipient list.
Some group members have blocked users, others have requested to receive email via daily or weekly digests, some have asked to unsubscribe from a thread, still others have asked to receive no updates via email. Generating the recipient list requires some processing time – so it’s best done in the background. The final recipient list is stored in a second table.
This process takes pending messages in the Outbox and creates records in two related Outbound Mail tables. You can see an example of us processing messages from the initial outbox to the outbound messages table here.
Handing Off Messages to Mailgun for Sending
The second background process manages the actual transfer of the messages from the Outbound Mail tables to Mailgun via cURL.
$items = OutboundMail::model()->pending()->findAll($criteria);
foreach ($items as $i) {
// fetch single message
$message = Yii::app()->db->createCommand()
->select('p.*,q.slug,q.prefix,u.eid')
->from(Yii::app()->getDb()->tablePrefix.'post p')
->where('p.id=:id', array(':id'=>$i['post_id']))
->queryRow();
// get the recipient list
$rxList = Yii::app()->db->createCommand()
->select('*')->from(Yii::app()->getDb()->tablePrefix.'outbound_mail_rx_list o') ->where('delivery_mode = '.Member::DELIVER_EACH,array(':outbound_mail_id'=>$i['id']))->queryAll();
$rx_var ='{';
$to_list = '';
foreach ($rxList as $rx) {
$rx_var = $rx_var.'"'.$rx['email'].'": {"id":'.$rx['id'].'}, ';
$to_list = $to_list.$rx['email'].', ';
}
$rx_var =trim($rx_var,', ').'}';
$mg = new Mailgun();
$fromAddr = getUserFullName($message['author_id']).' <u_'.$message['eid'].'+'.$message['slug'].'@'.Yii::app()->params['mail_domain'].'>';
$result = $mg->send_broadcast_message($fromAddr,$message['subject'],$message['body'].$this->mail_footer.$place_url,$to_list,$rx_var,$i['post_id']);
$mg->recordId($result,$i['id']);
// change status of outbound_mail item to SENT
$outbox = Yii::app()->db->createCommand()->update(Yii::app()->getDb()->tablePrefix.'outbound_mail',array('status'=>OutboundMail::STATUS_SENT,'sent_at'=>new CDbExpression('NOW()')),'id=:id', array(':id'=>$i['id']));
}
The lines above that build $rx_var are constructing a JSON string of Mailgun’s recipient variables. These allow us to send broadcast messages without individually listing each recipient on the To: line.
Here’s how we submit the messages to Mailgun for sending:
public function send_broadcast_message($from='support@yourdomain.com',$subject='',$body='',$recipient_list='',$recipient_vars='',$post_id=0) {
$ch = $this->setup_curl('messages');
curl_setopt($ch,
CURLOPT_POSTFIELDS,
array('from' => $from,
'to' => trim($recipient_list,','),
'subject' => $subject,
'text' => $body,
'recipient-variables' => $recipient_vars ,
'v:post_id' => $post_id)
);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
Receiving messages
One of Mailgun’s powerful value-added features is its processing and parsing of inbound email.
Parsing inbound email successfully into its component parts requires sophisticated code written to specific standards (as well as properly handling inevitable non-standard messages). I’m much happier letting Mailgun write this code and have them post the parsed results to my website.
Furthermore, Mailgun’s spam filtering prevents my web site from having to do the work of spam detection and filtering. For a group-oriented email service like Geogram which uses a catch-all mailbox, this is a great benefit. Their spam filtering prevents me from having to process a large volume of invalid requests.
Setting Up Your Route
Mailgun provides programmable routes for receiving different types of email:
For example, any email to support@geogram.com forwards to our Tender App inbox and any email to jeff@geogram.com forward to my work address.
More importantly, we have a catch-all rule – any email to unknown mailboxes is forwarded to our website as a posted form. In this case, you set up your forwarding address as a URL for your production environment e.g. http://yourdomain/messages/inbound. Mailgun will post the data from incoming messages to that URL.
Receiving and Storing Inbound Posts
Again, for scalability reasons, when Mailgun posts inbound messages to Geogram, we first just store them unprocessed in the database. As mail traffic on Geogram increases, this point of contact must be always available and fast. We simply serialize the POSTed data from Mailgun and store it as a BLOB:
/* Receives posted form from Mailgun with inbound commands
Places data into inbox table in serialized form
*/
public function actionInbound()
{
$mg = new Mailgun;
// verify post made by Mailgun
if ($mg->verifyWebHook($_POST['timestamp'],$_POST['token'],$_POST['signature'])) {
$bundle = serialize($_POST);
$inboxItem = new Inbox;
$inboxItem->addBundle($bundle);
}
}
The verifyWebHook call ensures that the message was POSTed by Mailgun and is not malicious. Here is the code for verifying POSTs:
// this verifies that the post was actually sent from Mailgun and is not malicious
// see http://documentation.mailgun.com/user_manual.html#events-webhooks
public function verifyWebHook($timestamp='', $token='', $signature='') {
// Concatenate timestamp and token values
$combined=$timestamp.$token;
// Encode the resulting string with the HMAC algorithm
// (using your API Key as a key and SHA256 digest mode)
$result= hash_hmac('SHA256', $combined, Yii::app()->params['mailgun']['api_key']);
if ($result == $signature)
return true;
else
return false;
}
Another background process actually looks in our database and processes each message.
Processing Inbound Messages
Inbound messages to Geogram are often new messages or replies to existing messages. We post these to the outbox table and they get sent via the processing described earlier. We receive other kinds of messages which include email commands such as: !private (send a private reply), !leave (resign from this group/place) !favorite (mark this message as a favorite) !spam (mark this message as spam).
Our inbox processing is actually quite detailed but here are a few aspects of it to guide your own implementation.
First, we find all the unprocessed messages starting with the oldest first. This is the framework for our processing loop:
public function process($count=100) {
// processes commands in the inbox
$this->configMailgun();
$criteria = new CDbCriteria();
$criteria->limit = $count;
$criteria->order = "created_at ASC";
$inboxItems = Inbox::model()->pending()->findAll($criteria);
foreach ($inboxItems as $i) {
echo 'inbox item id: '.$i['id'].'
';
$commandRequest=false;
$cmd = unserialize($i['bundle']);
// Lookup sender from $cmd['sender']
$sender_email = $cmd['sender'];
$sender = User::model()->findByAttributes(array('email'=>$sender_email));
if ($sender === null) {
// send alert - we don't know that address
OutboundMail::alertUnknownSender($sender_email);
} else {
// do all of our processing per message (described below)
}
// change status of inbox item
Inbox::model()->updateAll(array( 'status' => self::STATUS_PROCESSED,'processed_at'=> new CDbExpression('NOW()') ), 'id = '.$i['id'] );
// end of loop thru inbox items
}
}
We’re unserializing the POSTed fields from the database, then verifying the sender. If the sender isn’t a registered user, we send them an error message.
Here’s how we look for commands at the top of the message body:
$msgLead = substr($body,0,200);
if (stripos($msgLead,'!nomail')!==false) {
$this->log_item('nomail request'.$sender->id.' '.$sender->email);
// set nomail
$sender->no_mail = true;
$sender->save();
$commandRequest=true;
}
Here’s the basics for how we match the incoming To: address to the known Places in our Geogram world. Basically we’re matching the place part of the To: field to the slugs for Places in our database. We’re checking that the place exists and that the user is a member authorized to post:
// Lookup place from slug / get place_id
$place = Place::model()->findByAttributes(array('slug'=>$slug));
if ($place === null) {
// send alert - we don't know that address
OutboundMail::alertUnknownPlace($sender->email);
} else {
if (!$commandRequest) {
// verify sender is a member of place
if (Place::model()->isMember($sender->id,$place->id))
{
$this->log_item('IsMember - preparing msgpost');
// Prepare message for sending
$msg=new Post;
$msg->subject = $cmd['subject'];
$msg->body = $body;
$msg->status = Post::STATUS_PENDING;
$msg->place_id = $place->id;
$msg->author_id = $sender->id;
$msg->type = $outbound_msg_type;
// submit to the outbox
$msg->save();
} else {
// send alert - you're not a member
OutboundMail::alertNotMember($sender->email,$place);
}
// end of commandRequest false block
}
// end of post / reply to message block
}
Then, we just save the message to the database as if it had been posted from the website. It will get picked up by the background process that handles outbound messages.
Tracking Reply Threads
Replies to emails include a header with a unique message ID identifying the thread of the conversation. When we ask Mailgun to send a message, it returns this unique thread ID to us and we store in the database.
$result = $mg->send_broadcast_message($fromAddr,$message['subject'],$message['body'].$this->mail_footer.$place_url,$to_list,$rx_var,$i['post_id']); $mg->recordId($result,$i['id']);
Here’s where we record the ID in the database:
public function recordId($result,$id) {
// update outbound_mail table with external id
// returned from mailgun send
$resp = json_decode($result);
if (property_exists($resp,'id')) {
OutboundMail::model()->updateAll(array( 'ext_id' => $resp->id ), 'id = '.$id );
}
Later, when we receive messages, we look to see if they are part of an earlier conversation. Here is some example code:
// find thread reference id
$msg_headers=json_decode($cmd['message-headers']);
$reference_exists=false;
foreach ($msg_headers as $header) {
if (strcasecmp($header[0],'References') ==0 ) {
$references = $header[1];
// lookup original post that thread refers to
$thread = OutboundMail::model()->findByAttributes(array('ext_id'=>$references));
if (!empty($thread)) {
$outbound_parent_id=$thread->post_id;
$outbound_msg_type = Post::TYPE_REPLY;
$this->log_item('reply to thread: '.$thread->post_id);
} else {
$outbound_parent_id=0;
}
}
if (strcasecmp($header[0],'In-Reply-To')==0) {
$inreplyto = $header[1];
$this->log_item('replyto: '.$inreplyto);
// lookup reply that inbound msg refers to
$thread = OutboundMail::model()->findByAttributes(array('ext_id'=>$inreplyto));
if (!empty($thread)) {
$outbound_in_reply_to=$thread->post_id;
$this->log_item('reply to: '.$thread->post_id);
$outbound_msg_type = Post::TYPE_REPLY;
} else {
$outbound_in_reply_to = 0;
}
// end reply-to ext-id search
}
// end foreach headers
}
Sending Private messages
One of the most important values / features of Geogram is that we never share our users’ actual email address publicly. When we send messages, each sender is given a unique anonymized FROM address combined with the place name (or slug) they are sending to. e.g. Cynthia B. <u_51b3e170ef1e4+cedarplace@geogram.com>.
If a user replies to this address with !private in the first paragraph, we know the message is meant for private delivery to Cynthia B (user 51b3e170ef1e4). Without the !private command, we send the message to all recipients at cedarplace. Users can also send private messages to each other with just Cynthia B <u_51b3e170ef1e4@geogram.com> and we also allow private messaging through the web interface.
When we receive a request to send a private message, we do the following:
public function sendPrivateMessage($from,$subject,$body,$to) {
$this->init();
$mg = new Mailgun();
// check that either user is not blocked
if (!Block::model()->isBlocked($to->id,$from['id'])) {
// remove !private command if present
$body= ltrim(str_ireplace('!private','',$body)).$this->private_mail_footer;
// anonymize the from address
$fromAddr = getUserFullName($from['id']).' <u_'.$from['eid'].'@'.Yii::app()->params['mail_domain'].'>';
$mg->send_simple_message($to->email,$subject,$body,$fromAddr);
} else {
// send error message
$mg->send_simple_message($from['email'],'Sorry, we\'re not able to deliver that message.','We are unable to deliver private messages to this recipient. If you have any questions, please contact us '.Yii::app()->baseUrl.'/site/contact');
}
}
And, here’s the generic send_simple_message function we use to talk to Mailgun:
public function send_simple_message($to='',$subject='',$body='',$from='') {
if ($from == '')
$from = Yii::app()->params['supportEmail'];
$ch = $this->setup_curl('messages');
curl_setopt($ch, CURLOPT_POSTFIELDS, array('from' => $from,
'to' => $to,
'subject' => $subject,
'text' => $body,
'o:tracking' => false,
));
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
Sending Individual Messages
Sometimes you just want to send individual email messages immediately. When we send new users their activation link, we want do do this without any delay. Or, email from our web-based contact form can just be forwarded to our support queue directly. Obviously, this is easy in Mailgun as well.
The Yii-User module extension uses the standard PHP mail function. We’ve replaced it with a version that sends via Mailgun:
/**
* Send to user mail
*/
public static function sendMail($email,$subject,$message) {
$supportEmail = Yii::app()->params['supportEmail'];
$headers = "MIME-Version: 1.0\r\nFrom: $supportEmail\r\nReply-To: $supportEmail\r\nContent-Type: text/html; charset=utf-8";
$message = wordwrap($message, 70);
$message = str_replace("\n.", "\n..", $message);
$mailgun = new Mailgun();
return $mailgun->php_mail($email,'=?UTF-8?B?'.base64_encode($subject).'?=',$message,$headers);
// old yii user-module standard code
//return mail($email,'=?UTF-8?B?'.base64_encode($subject).'?=',$message,$headers);
}
and
public function php_mail($email ='',$subject='',$message='',$headers='') {
$ch = $this->setup_curl('messages');
curl_setopt($ch, CURLOPT_POSTFIELDS, array('from' => Yii::app()->params['supportEmail'],
'to' => $email,
'subject' => $subject,
'text' => $message,
'o:tracking' => false
));
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
Advantages of Mailgun
I hope this post has given you an idea of how powerful Mailgun and its API are. In summary, here are a few reasons I’ve chosen Mailgun for Geogram:
- It provides a simple API that’s easy to integrate with common platforms
- It manages the complexity of inbound and outbound email for me
- It manages scaling most of the email-side of my application for me
- It manages my server’s email reputation and filters spam
- It provides super fast, reliable delivery – no more wondering if my messages will get through
- And finally, Mailgun provides outstanding technical support
Questions? Criticisms? Feedback?
I’ll be the first to admit that our code can be optimized greatly and that it’s not perfect – it’s a one person startup at this point. But, hopefully, you now have a bit more of a feel of how to work with Mailgun’s API via PHP.
If you liked this tutorial, you can retweet it. If you have any questions, please feel free to post them in the comments and I’ll do my best to respond. You can also follow my blog or @reifman.
Kudos to the Mailgun & Rackspace folks (esp. Michael & Travis) who have been super helpful and supportive in the Geogram construction / launch phase.
What are you waiting for? Invite your neighbors to Geogram and then get coding your awesome Mailgun-integrated application.