I know more than a few of our readers like to get a peek at the business behind QS.com, which is why I'm writing about split-testing our newsletter. Here's the Wikipedia definition of it:

A/B testing or split testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates.

Essentially, you select one thing you want to change about your message and leave everything else the same. Then you see which one people like more.

We use Mailchimp, an incredibly powerful and incredibly free newsletter delivery program. It meters just about everything about our newsletter mailings, be they to Insiders only or our general, free mailing list (sign up!).I can tell when people give us a few minutes of their time when they read our content, and I can find out whether Insiders are looking at our article recap links in our periodical Insider-only emails.

Among other things, Mailchimp lets me split-test single variables. This time, it was whether the headline "Make money on Grove of the Burnwillows" would be read by more people than "The biggest engine in Modern..." I used hype-y headlines on both - I like an email to actually tell the reader in the subject line what it's about, instead of being something like "QS Free Newsletter, [Date]." Keep reading to find out which one was better! I think next time I'll try testing a boring headline with a hyped headline and seeing which one our readers prefer. That is, clearly, another thing you can test with A/B tests.

You may have received the email yesterday, or you may have gotten it this morning. I wasn't leaving you out! What I did was schedule Mailchimp to send the "winning" email to the full list. It selected about a quarter of the list and sent it to them. I then set it to send the email that was opened more to the rest of the list. Why did it take a day? With usual newsletters, timeliness isn't that important, so the default is a day. I think I'll explore setting it to half a day or a matter of hours. It's clear from Twitter that people were interested in getting it and discussing it. It should have sent to the full list yesterday afternoon.

I personally like writing our newsletter because I love talking with fans, but it's also an opportunity to show people what our Insider service is about. I detest getting on newsletter lists that just send out ugly pitches and don't entertain at all, so I don't want to inflict that on anyone. With split testing, I can send the most efficient messages to our subscribers and hopefully, encourage some of them to sign up.

It turns out that "Make money on Grove of the Burnwillows" beat the other headline by 3%, which isn't that significant. The other headline, though, had at least 5% more clicks on the internal links (my Survival article and the Insider signup page), so there may be something to that.

I am still finding my legs on the subject of what people like to read about in newsletters. Expect more unlocked Insider articles and format commentary, along with occasional non-sequitors (I just discovered this history documentary I love is online!) or things like decklists that are heating up (wait till you see my Modern Gifts deck).

I'll say one last thing on split testing: we will never split test our up-to-the-minute Insider email alerts. The overwhelming value of those is their timeliness, and I'd never do something to delay them.