Bulk Email Best Practices

sneakyimp

I've been tasked with replicating certain functions of a Joomla bulk mail plugin. Turns out this plugin nearly 50,000 lines of code with no comments. Rather than trying to replicate every single feature, I have decided to just try and hit the big features:
Can send very large numbers of marketing emails via SendGrid with some kind of flood control. I.e., send N emails every M minutes.
Can track delivery stats by user or newsletter or email type of some kind. The plugin has one page that offers these stats for each newsletter sent: email opens, number of emails where someone clicked a link, number of unsubscribe requests, forwarded, sent as html, sent as text, bounces, and failures.
Can tell if users are able to view email as HTML or only as plain text.
track clicks in email
* optimize emails to maximize delivery rates using DKIM or whatever other tools may exist.

Has anyone done this sort of thing before? Can you offers suggestions/tips/warnings?

I've been looking into the SendGrid Web API, Event API, and they also offer SMTP integration, although this appears to have fewer features. From what I can tell, I'm going to need a few things:
means of creating/automating/generating newsletters or email types as an email with an email_id
code to determine recipients for a particular email_id
code to contact SendGrid API and send the emails
some way to track if the email actually gets opened. I'm assuming this is accomplished by putting an IMG tag in the email that actually references a PHP script, once displayed, will tell me the user has viewed the email. Is there any other way? What is best practice here?
a url through which I can route email clicks. I.e., the email links to a url on my site that tracks the click and redirects the user to the link destination.
url to handle unsubscribe requests
How to determine when an email gets forwarded?
code to track how many outgoing emails are sent as html and text.
* code to parse API responses and/or receive Event notifications to track mail delivery failures and bounces.

Any insight you folks might have here would be much appreciated.

Derokorian

sneakyimp;11058961 wrote:
I've been tasked with replicating certain functions of a Joomla bulk mail plugin. Turns out this plugin nearly 50,000 lines of code with no comments. Rather than trying to replicate every single feature, I have decided to just try and hit the big features:
Can send very large numbers of marketing emails via SendGrid with some kind of flood control. I.e., send N emails every M minutes.
Can track delivery stats by user or newsletter or email type of some kind. The plugin has one page that offers these stats for each newsletter sent: email opens, number of emails where someone clicked a link, number of unsubscribe requests, forwarded, sent as html, sent as text, bounces, and failures.
Can tell if users are able to view email as HTML or only as plain text.
track clicks in email
* optimize emails to maximize delivery rates using DKIM or whatever other tools may exist.

Have you considered using a different service, such as MailChimp, which has all of these features? I've recommended its use before, with great resulting success.

sneakyimp

I'm not certain, but I think that SendGrid and MailChimp offer essentially the same features. The client is interested in integrating his website with the service. E.g., all outgoing mail will be sent using the SendGrid API. However the creation of the email content will be handled by his website -- the first task is to replace a cron job that automatically generates a "you might find these links interesting" newsletter on a daily basis. This requires our system to generate the email and its content (which will be different for each recipient) and then our system can hook into SendGrid to send the mail. Client has also made it clear that he'd prefer to host delivery statistics in his own system rather than relying on an API call to ask the mail service for info.

That being said, I think my actual question might be "how much of this functionality must I write and how much is handled by the email service (SendGrid/MailChump)?"

A couple of questions I have:
Does the mail service track delivery rates for non-html (i.e., plaintext) emails? If so, what is the mechanism?
Does the mail service create the DKIM signature or must I formulate this before sending the mail?

dalecosp

Probably the best way to get answers to those questions is to ask them of the email service. Of course, then you're on some salesperson's 'hit list'.

The O.P. is absolutely a "rebuild the wheel" thing. There are several competing companies (we use iContact; others include aWeber, MailChimp, ConstantContact, &c, &c).

An image beacon is fairly standard for detecting opens. It's not a terribly reliable statistic, though, as most MUA's block images by default now until the user allows it (or marks something like "always open images from this sender"). So someone might read your email and not open the photos. Your "opens" number will be lower than reality as a result.

It's mostly impossible to tell when an email gets forwarded by the MUA. Some ESP's include a link in the message, e.g. "Forward to a Friend", which opens a browser window to their site with the email content shown and an address field. That they can track. Now, it's possible to make a guess about forwards, by tracking the IP and a cookie and making a deduction: see this page (Cathoo's answer) about 1/2 way down.

The only way to track an "open" in a plaintext email is to record an "open" along with any link click from that user/email. So that number will be artificially low if people read the mail with a plaintext MUA but don't actually follow any links. I don't know of many stats about how many users are still using plaintext-only MUA's though. Almost every client is HTML capable by default, although I don't know as much about mobile, which is being used by a continually larger group of people ....

That being said, my personal email still gets read with Mutt. In this regard am I not typical.

* code to track how many outgoing emails are sent as html and text.

Both should be sent, and the MUA should decide which to show. You should, if possible, allow the user to set a preference (text vs. HTML), but most won't. If one does, it will usually only be the people like me.

As for automating your message creation and sending, that's kind of a Holy Grail at our shop. I've not found time to do it.

Derokorian

Looking at SendGrid I got the impression it was just a service to manage sending emails, I don't see anything on their site about tracking and statistics. This is a key feature for MailChimp (and one I have relied on in the past). If you can get this information from SendGrid I don't see how from their front end sales area. MailChimp also has API to use.

sneakyimp

dalecosp;11058973 wrote:
Probably the best way to get answers to those questions is to ask them of the email service. Of course, then you're on some salesperson's 'hit list'.

I've been wading through their copious but inconsistent documentation for days. I may have to call 🙁

dalecosp;11058973 wrote:
The O.P. is absolutely a "rebuild the wheel" thing. There are several competing companies (we use iContact; others include aWeber, MailChimp, ConstantContact, &c, &c).

I'm not so sure it's a rebuild-the-wheel request. I want to make use of as much of SendGrid's capabilities as possible to economise. Client is clear that he wants to maintain information within his own system (e.g., generating customized marketing emails based on our database contents, or seeing how many customers have opened the email). It's just not clear to me how much wheel-reinventing I must do and how much I might rely on existing features of SendGrid.

dalecosp;11058973 wrote:
An image beacon is fairly standard for detecting opens. It's not a terribly reliable statistic, though, as most MUA's block images by default now until the user allows it (or marks something like "always open images from this sender"). So someone might read your email and not open the photos. Your "opens" number will be lower than reality as a result.

Image beacon! I knew this kind of thing had to have a name. I've been told by customer support at SendGrid that I will receive event notifications from SendGrid's Event API even if I just send a plain-text email using their Web API. The Event API claims to report Open and Click events for the email. I'm wondering if SendGrid will automagically insert an image beacon or if there's some other magic they are using. I strongly suspect I've been misinformed by SendGrid's customer support guy.

dalecosp;11058973 wrote:
It's mostly impossible to tell when an email gets forwarded by the MUA. Some ESP's include a link in the message, e.g. "Forward to a Friend", which opens a browser window to their site with the email content shown and an address field. That they can track. Now, it's possible to make a guess about forwards, by tracking the IP and a cookie and making a deduction: see this page (Cathoo's answer) about 1/2 way down.

Thanks a lot for this detail.

dalecosp;11058973 wrote:
The only way to track an "open" in a plaintext email is to record an "open" along with any link click from that user/email. So that number will be artificially low if people read the mail with a plaintext MUA but don't actually follow any links. I don't know of many stats about how many users are still using plaintext-only MUA's though. Almost every client is HTML capable by default, although I don't know as much about mobile, which is being used by a continually larger group of people ....

This is one of the very things I've been agonizing over. If I use pine or something (lol) entirely incapable of tracking link clicks or anything like that, I'm guessing that there's no tracking of either clicks or opens. I'm wondering if I must manually prepare my outgoing emails with an image beacon and modified links (pointing to some click-tracking-and-url-forwarding url) or if the SendGrid API will automagically do this. I've signed up for a free account (up to 40,000 emails per month interestingly) and am installing the PHP library (using Composer! AUUUGHHH!) and will see for myself soon enough.

dalecosp;11058973 wrote:
That being said, my personal email still gets read with Mutt. In this regard am I not typical.

Ha! Looks like pine. I'd strongly suggest a look at Thunderbird.

dalecosp;11058973 wrote:
Both should be sent, and the MUA should decide which to show. You should, if possible, allow the user to set a preference (text vs. HTML), but most won't. If one does, it will usually only be the people like me.

Setting HTML and plain text of the same email is something easily done with CodeIgniter. The plugin I'm looking at for this post has an interesting function to strip out the HTML from the HTML message to auto-generate a plain text one (if none is manually provided). I sincerely hope the Sendgrid PHP library has a feature for this that I might rely on.

dalecosp;11058973 wrote:
As for automating your message creation and sending, that's kind of a Holy Grail at our shop. I've not found time to do it.

I know that SendGrid offers some templating features and you can merge customized data into them using the Web API. Still wondering whether we use their templating or whether we maintain email templates in our system and just the web api as a mail relay.

I'm also wondering about DKIM. SendGrid has some docs which I am still wading through. It would appear that there is some account setup/configuration involved but I don't think I'll have to create any DKIM signatures for each email that I construct. Really hoping that stuff is magically constructed by SendGrid for each message after I get the basic configuration set up.

sneakyimp

Derokorian;11058975 wrote:
Looking at SendGrid I got the impression it was just a service to manage sending emails, I don't see anything on their site about tracking and statistics. This is a key feature for MailChimp (and one I have relied on in the past). If you can get this information from SendGrid I don't see how from their front end sales area. MailChimp also has API to use.

I haven't come at SendGrid from their site's home page, but I've seen all kinds of tools like Templates, a Dashboard, Activity Monitoring, Marketing Campaigns, etc.

SerenityPantone

One of the best practices for bulk email that i know will help is to check and review inbox deliverability for web mail companies and major clients.

dalecosp

sneakyimp;11058977 wrote:
Ha! Looks like pine. I'd strongly suggest a look at Thunderbird.

I have little interest in seeing HTML or photos in my personal email, which I access from multiple locations via multiple platforms. I hate web-based stuff too, so SquirrelMail, RoundCube, etc. can also bite me :evilgrin:

@work, they make us use Outlook. I'm OK with that. I just prefer to be anachronistic with my personal email. It keeps me close to my servers, for one thing.

I know that SendGrid offers some templating features and you can merge customized data into them using the Web API. Still wondering whether we use their templating or whether we maintain email templates in our system and just the web api as a mail relay.

We could do "mail merges" with our DB at iContact, but we'd have to put some very significant effort into making sure the data quality was good enough (fields had accurate data) to make a merged message actually have compelling content ....

dalecosp

sneakyimp;11058977 wrote:
Image beacon! I knew this kind of thing had to have a name. I've been told by customer support at SendGrid that I will receive event notifications from SendGrid's Event API even if I just send a plain-text email using their Web API. The Event API claims to report Open and Click events for the email. I'm wondering if SendGrid will automagically insert an image beacon or if there's some other magic they are using. I strongly suspect I've been misinformed by SendGrid's customer support guy.

<snippity>

This is one of the very things I've been agonizing over. If I use pine or something (lol) entirely incapable of tracking link clicks or anything like that, I'm guessing that there's no tracking of either clicks or opens. I'm wondering if I must manually prepare my outgoing emails with an image beacon and modified links (pointing to some click-tracking-and-url-forwarding url) or if the SendGrid API will automagically do this. I've signed up for a free account (up to 40,000 emails per month interestingly) and am installing the PHP library (using Composer! AUUUGHHH!) and will see for myself soon enough.

Just as a side note, I'm working @home today. I opened one of our marketing emails in Mutt, with the Lynx browser embedded via MailCap and .muttrc. So far, our email provider is not showing me as an "open", despite me reading the entire message.

sneakyimp

So I'm wondering if there's any point in creating plaintext versions of emails these days? this article says that html-only emails are a 'red flag' for spam filters. Anyone have thoughts on this?

Derokorian

sneakyimp;11061295 wrote:
So I'm wondering if there's any point in creating plaintext versions of emails these days? this article says that html-only emails are a 'red flag' for spam filters. Anyone have thoughts on this?

I would say I haven't made plaintext versions in years, and as long as I maintain valid SPF records the mail never gets bounced as spam.

dalecosp

Not everyone is "up to speed" on spam filtering. HTML Only is still a flag with most services, but the flag's value has changed and probably would not, by itself, tip you over the threshold to be tagged as SPAM.

That said, I'm still reading some of my email plaintext and have, within the last 90 days, "griped" (mildly) to a sender about their lack of PT content. #old_phart