Check out our 2021 Email Marketing Guide that will help you master your email strategy.

Email A/B Testing: Top 6 Tests to Boost Your Campaign Results Today

email a/b testing

Email A/B testing can be one of the simplest, yet most valuable initiatives you can implement for your email marketing program. Different A/B tests focus on improving different metrics, with the ultimate focus on generating more sales from your emails.

When running an A/B test you generally want to focus on one variable at a time leaving all other elements unchanged (multivariate tests require a much deeper discussion). Several beginner elements to test for include:

• Email Subject Lines
• Email Content
• Send Time 

Here are some results from A/B tests we’ve run in the past:

Test 1: Does including the exact offer in the subject line help open rates and sales?
Subject line testing can have a significant impact on your campaign open rates. What’s more, the right subject line shapes the perception of the entire email. This can be done through creating a sense of urgency with your subject lines, like “one day left” or giving your subscribers exclusive offers.

In these tests, we ran subject lines that included the exact offer ($X off or X% off) vs. ones that did not include them.
[table id=15 /]

We took a look on a couple of our clients’ email campaigns, where the subject line had either a sense of urgency, exclusive offer, solving a problem (“For Email VIP’s – Get Our Black Friday Sale Early!”, “How To Style The Sidecar 3 Ways”) vs. a coupon code (“Get 10% Off All Protein Snacks”, “5% Off Plus Free Shipping”). In all cases offering the exact discount in the subject line lead to increase in revenue by 18.3% -100%, however, both the number of clicks and opens dropped on average by 20%-23%, if the subject line didn’t include the sense of urgency or other kinds of incentive. That’s why, as the third example testing suggests, the winning subject line is the one which combines an offer and a sense of urgency, yet is not overloaded (like “All-season flannels. 20% off for today only.”)

Even though the open rates actually seemed to drop when using the exact offer in the subject line, the subsequent revenue generated increases. By putting the exact offer front and center, it may dissuade those with lower buying intent, thus depressing opens, yet those who do open have a higher likelihood of actually purchasing.
Additionally, urgency is a strong catalyst, but it doesn’t stand up very well on its own. Adding urgency doesn’t work if your offer is full of distractions, or if your value proposition is poor. As a result, a sense of urgency and an attractive, clear offer work best together.

Test 2: Which subject line increases engagement: a question or a direct call-to-action
Questions make users think. More specifically, questions make them curious about your email, which can be powerful in getting open. At the same time, it is thought that effective email subject lines should have a direct call to action. So, we decided to run some A/B tests around these.
[table id=16 /]

The subject line in the form of a short, concise question is the winner. For example: “How can we help you next spring?” boosted Open Rate by 46.08% and Click Rate by 344.44% in comparison with “Take our survey & get $50 towards your next order”. A subject line without any CTA’s like “Is 3 months free groceries ok?” likewise got more opens than “3 months free groceries? GO!”.

While a question-form subject line may not always drive the highest click rates or ultimate conversions, it is useful in maximizing open rates. The curiosity generated by the question tends to do well in driving at least that initial open.

Test 3: Is it better to feature multiple products or one product?
Promoting new products with email marketing can be highly effective, but it does take some work to learn how to do it well. Specifically, is it better to promote multiple products or one product in your unsegmented email campaigns?
[table id=17 /]

Test A: one product                                                                             Test B: multiple products
test plus CFG 2                                        test plus CFG 1 

In the specific case, we tested, focusing on a single product can have a sizeable impact on metrics across the board. Of course, product selection is highly important in this case – but depending on your list and audience, showcasing singular, popular products can outperform a wider product showcase of multiple products in some cases.

Promoting one product helps to create an email with easy-on-the-eyes design and short, digestible content. It makes a clear message for the customer because it can grab the client’s attention to one single offer, whereas promoting multiple products can distract from a single, focused CTA.

Test 4: Does including product pricing increase click-through?
How does pricing influence your email click-through? Is it a better idea to hide the pricing or use it as an anchor or even as a value proposition in email campaigns?

We wanted to find out the answer to these questions, so we run two A/B tests on showing vs. hiding product pricing and examined the impact.
[table id=18 /]

Test A: only products                                                                 Test B: products with price
test 4.1                  test 4.2                     

Surprisingly, in both tests, mentioning a price helped to increase the click rate on average by 10%. Moreover, for the second test, the revenue also increased revenue by 20%.

Placing the standard information we all use to make buying decisions (such as price) can actually stimulate your customers to purchase.

For example, one of the main reason why shoppers abandon their shopping carts is because they were presented with unexpected costs (56%). Customers don’t want to seem to like hidden costs, so mentioning prices upfront in your newsletter helps to shape the right perception of your products.

Test 5: What gets more clicks Image-Heavy Emails or Text-Heavy Emails?
Email coding is a fine balance between including images and optimizing for images blocking. We went ahead and ran a test with several subscriber segments for one client to see if Image-Heavy or Text-Heavy emails performed better for them.
[table id=19 /]


Test A: Graphic Email                                                                                 Test B: Text/Button version
test 5.1                                                              test 5.2

We tested a regular email campaign on different groups of subscribers. Some segments saw a boost by 50% & 70% respectively for open & click rates with the text-heavy email. In contrast, for other segments the open rate for the text-heavy version was lower by -26.18%.


The test isn’t completely conclusive. In fact, as we saw –   different segments even within the same email list can respond differently to the image-heavy vs. text-heavy versions.

Test 6: What’s the best time to send email campaigns?
Everyone wants to know, but few actually test it. So, what is the ideal send time? Unfortunately, there’s no single answer. Ideal sending times vary by industry, product, and your specific subscriber list. Here’s what has worked for in this particular case.
[table id=20 /]

Saturday mornings performed worse for engagement rates vs. Friday evening.

Saturdays are generally worse days to send. The other times and days of the week tends to vary for each store.

Based on these A/B testing we’ve conducted, here are our suggestions on how to make your next email campaign really effective:

  • Using subject lines framed as a question can increase open rates.
  • Showing the exact offer in the subject line tends to drive less engagement yet higher revenue.
  • Promoting one product instead of a list of them in one newsletter helps to create an email with a more focused CTA, which can increase engagement.
  • Hiding the price for your products in the email campaigns can decrease click-throughs.
  • Text-heavy emails can work better than image-heavy emails in certain circumstances and to certain segments of a list.
  • The optimal send time varies greatly depending on your store and audience. Test, test, and test to find the best time for you.

Need help with your email marketing? We have helped over 220 companies like yours! 
Book a call with us and we’ll get to the bottom of it!


Top stories from the email marketing world.

No ads. Only news that matters. Sign up.