What to do When Your Email A/B Test “Fails”

Oh dear… Perhaps we could try a different subject line?

You’ve done the research on your target audience, come up with some killer subject lines and high-quality content you know your subscribers will want to read.

You’ve set up the perfect layout, made sure everything is mobile-friendly, and with great anticipation, sent your email campaign off into the ether…

Why Didn’t My Email A/B Test Get Me More ______?

The reality in email A/B testing (or A/B testing of any kind) is that not every test is a “winner”. By that I mean not every test results in an improvement in the metrics being tracked.

Setting up your A/B tests properly from the start is no guarantee that you will improve your KPIs. However, it does guarantee that no matter the results, you will have actionable data that you can use to improve upon results in the future.

Notice I put “fails” in quotation marks up there in the title — it’s because even a test that doesn’t result in improvement initially can be used to improve in the future.

So, why didn’t your A/B test get you the results you were expecting? There are any number of reasons, these are some of the most common:

• Your hypothesis (setup premise) was incorrect. Maybe you assumed that because your list likes swimwear, they’re also interested in articles about tanning. In reality, your readers might be passionate about preventing sun damage. Another test, this time with articles on sun safety or sunblock, will reveal whether or not this is the case.

• Your subject line and content aren’t aligned. If your subject line promises exciting new information, and the content of your email doesn’t deliver, expect less than stellar results. If you aren’t sure how to set up the content for your email campaign, watch the quick 3-minute video below with the amazingly talented Dr. Flint McGlaughlin of MECLABS to learn the proper structure of email messaging.

• You didn’t design a full conversion funnel. Emails are only one part of a working sales funnel. If you don’t have a landing page designed to match the expectations set by the email then overall conversions are likely to be low.

• You had technical glitches. Maybe your email was too popular and you got so much traffic that it crashed your servers. Maybe your shopping cart solution had an error. A smooth purchase experience builds trust and errors will do exactly the opposite. This is especially the case when new users are attempting their first purchase.

• Something horrible happened. An intern covered your twitter feed with profanity. Your last product launch ended in a recall. Someone has an online petition calling for a boycott of your company. These are extreme examples, but negative factors outside of your email campaign will definitely have an impact on the results you see from subscribers.

Steps to Recover from a “Failed” Email A/B Test

• Do a full postmortem tear-down. Resist the urge to just dump an under-performing A/B test. Take the time to look at why the test flopped, what could be improved, and what should be avoided in the future.

• Review past trends. How does this campaign compare to successful campaigns in the past? If it’s similar but the results are strikingly different, what about your audience has changed?

• Look for new areas to test. A lot of opens but few clicks could indicate you’ve found a winning subject line type, but need to work on the body copy. A high click-to-conversion ratio alongside a poor open rate could indicate that your offer is good, but your messaging needs work.

• Set up a new test. Just because one test didn’t improve your results doesn’t mean the next one won’t! Set up a consistent schedule of tests — with goals and hypothesis in place for each — and you are bound to start seeing your metrics improve.

Have you turned a “failed” A/B test into a “WIN” for your email marketing? Let me know in the comments!