Email A/B Testing: Top 6 Tests to Boost Your Campaign Results Today

Email A/B Testing: Top 6 Tests to Boost Your Campaign Results Today

email a/b testing

Email A/B Testing: Top 6 Tests to Boost Your Campaign Results Today

Email A/B testing can be one of the simplest, yet most valuable initiatives you can implement for your email marketing program. Different A/B tests focus on improving different metrics, with the ultimate focus on generating more sales from your emails.

When running an A/B test you generally want to focus on one variable at a time leaving all other elements unchanged (multivariate tests require a much deeper discussion). Several beginner elements to test for include:

  • Email Subject Lines
  • Email Content
  • Send Time 

Here are some results from A/B tests we’ve run in the past:

Test 1: Does including the exact offer in the subject line help open rates and sales?

Subject line testing can have a significant impact on your campaign open rates. What’s more, the right subject line shapes perception of the entire email. This can be done through creating a sense of urgency with your subject lines, like “one day left” or giving your subscribers exclusive offers.

In these tests, we ran subject lines that included the exact offer ($X off or X% off) vs. ones that did not include them.

A-GroupB-GroupSend VolumeA-Group StatsB-Group Stats% Difference
Black Friday Sneak Peek:
For Email VIP's - Get Our
Black Friday Sale Early!
15% Off Plus Free Shipping -
Get Our Black Friday Sale Early!
5503Open Rate:20.7%
Click Rate:3.0%
Open Rate:16.7%
Click Rate:2.4%
Open Rate: -19%
Click Rate: -20%
Snack Email Reports:
Protein Snacks this Holiday Season
Get 10% Off All
Protein Snacks
2138Open Rate:30.6%
Click Rate:4.3%
Revenue:$349.70
Open Rate:23.1%
Click Rate:2.8%
Revenue:$413.88
Open Rate: -24.8%
Click Rate: -53.57%
Revenue: +18.35%
How To Style The Sidecar 3 WaysThe Sidecar 3 Ways.
20% Off Updated Classics.
5576Open Rate:29.0%
Click Rate:8.5%
Revenue:$0
Open Rate:28.2%
Click Rate:9.6%
Revenue:$133.6
Open Rate:-2.75%
Click Rate: +12.94%
Revenue: +100%
Hurry! Last Hours To Save
20% Off Best Selling Flannels
All-season flannels.
20% off for today only.
1174Open Rate:9.0%
Click Rate:1.5%
Revenue:$0
Open Rate:11.8%
Click Rate:2.7%
Revenue:$133.60
Open Rate:+31.1%
Click Rate: +80%
Revenue: +100%

Results

We took a look on a couple of our clients’ email campaigns, where the subject line had either a sense of urgency, exclusive offer, solving a problem (“For Email VIP’s – Get Our Black Friday Sale Early!”, “How To Style The Sidecar 3 Ways”) vs. a coupon code (“Get 10% Off All Protein Snacks”, “5% Off Plus Free Shipping”). In all cases offering the exact discount in the subject line lead to increase in revenue by 18.3% -100%, however both the number of clicks and opens dropped on average by 20%-23%, if the subject line didn’t include the sense of urgency or other kind of incentive. That’s why, as the third example testing suggests, the winning subject line is the one which combines an offer and a sense of urgency, yet is not overloaded (like “All-season flannels. 20% off for today only.”)

Interpretation

Even though the open rates actually seemed to drop when using the exact offer in the subject line, the subsequent revenue generated increases. By putting the exact offer front and center, it may dissuade those with lower buying intent, thus depressing opens, yet those who do open have a higher likelihood of actually purchasing.

Additionally, urgency is a strong catalyst, but it doesn’t stand up very well on its own. Adding urgency doesn’t work if your offer is full of distractions, or if your value proposition is poor. As a result, a sense of urgency and an attractive, clear offer work best together.

 

Test 2: Which subject line increases engagement: a question or a direct call-to-action

Questions make users think. More specifically, questions makes them curious about your email, which can be powerful in getting the open. At the same time, it is thought that effective email subject lines should have a direct call to action. So, we decided to run some A/B tests around these.

A-GroupB-GroupSend VolumeChampion StatsChallenger Stats% Difference
Reminder: Take our survey
& get $50 towards your next order
How can we help
you next spring?
236Open Rate:10.2%
Click Rate:0.9%
Open Rate:14.9%
Click Rate:4%
Open Rate:+46.08%
Click Rate:+344.44%
Is 3 months free
groceries ok?
3 months free groceries? GO!4184Open Rate:16.5%
Click Rate: 3.4%
Open Rate:13.6%
Click Rate: 3.6%
Open Rate:-17.6%
Click Rate: +5.88%

Results 

The subject line in the form of a short, concise question is the winner. For example: “How can we help you next spring?” boosted Open Rate by 46.08% and Click Rate by 344.44% in comparison with “Take our survey & get $50 towards your next order”. A subject line without any CTA’s like “Is 3 months free groceries ok?” likewise got more opens than “3 months free groceries? GO!”.

Interpretation

While a question-form subject line may not always drive the highest click rates or ultimate conversions, it is useful in maximizing open rates. The curiosity generated by the question tends to do well in driving at least that initial open.

 

Test 3: Is it better to feature multiple products or one product?

Promoting new products with email marketing can be highly effective, but it does take some work to learn how to do it well. Specifically, is it better to promote multiple products or one product in your unsegmented email campaigns.

A-GroupB-GroupSend VolumeChampion StatsChallenger Stats% Difference
List of ProductsSingle Product Feature10944Open Rate:21.7%
Click Rate:2.6%
Revenue: 0
Open Rate:22.7%
Click Rate: 3.0%
Revenue: $304.99
Open Rate:+4.61%
Click Rate:+15.38%
Revenue: +100%
List of ProductsSingle Product Feature5486Open Rate:19.8%
Click Rate:2.4%
Revenue: 0
Open Rate:22.3%
Click Rate: 3.2%
Revenue: $150.56
Open Rate:+13.16%
Click Rate:+33.3%
Revenue: +100%

Test A: one product                                                                             Test B: multiple products

test plus CFG 2                                        test plus CFG 1 

Results

In the specific case we tested, focusing on a single product can have a sizeable impact on metrics across the board. Of course, product selection is highly important in this case – but depending on your list and audience, showcasing singular, popular products can outperform a wider product showcase of multiple products in some cases.

Interpretation                                                                         

Promoting one product helps to create an email with easy-on-the-eyes design and short, digestible content. It makes a clear message for the customer because it can grab client’s attention to one single offer, whereas promoting multiple products can distract from a single, focused CTA.

 

Test 4: Does including product pricing increase click-throughs?

How does pricing influence your email click-throughs? Is it a better idea to hide the pricing or use it as an anchor or even as a value proposition in email campaigns?

We wanted to find out the answer to these questions, so we run two A/B tests on showing vs. hiding product pricing and examined the impact.

A-GroupB-GroupSend VolumeChampion StatsChallenger Stats% Difference
Without Product PriceWith Product Price7389Open Rate:20.4%
Click Rate:4.3%
Revenue:$27.5
Open Rate:21.3%
Click Rate:4.8%
Revenue:$32.99
Open Rate:+4.41%
Click Rate:+11.63%
Revenue:+19.96%
Without Product PriceWith Product Price10321Open Rate:22.3%
Click Rate:2.1%
Open Rate:22.0%
Click Rate:2.3%
Open Rate:-1.35%
Click Rate:+9.52%

Test A: only products                                                                 Test B: products with price
test 4.1                  test 4.2                     

Results 

Surprisingly, in both tests, mentioning a price helped to increase the click rate on average by 10%. Moreover, for the second test, the revenue also increased revenue by 20%.

Interpretation

Placing the standard information we all use to make buying decisions (such as price) can actually stimulate your customers to purchase.

For example, one of the main reason why shoppers abandon their shopping carts is because they were presented with unexpected costs (56%). Customers don’t want seem to like hidden costs, so mentioning prices upfront in your newsletter helps to shape the right perception of your products.

 

Test 5: What gets more clicks Image-Heavy Emails or Text-Heavy Emails?

Email coding is a fine balance between including images and optimizing for images blocking. We went ahead and ran a test with several subscriber segments for one client to see if Image-Heavy or Text-Heavy emails performed better for them.

A-GroupB-GroupSend VolumeChampion StatsChallenger Stats% Difference
GraphicText1542Open Rate:19.1%
Click Rate:1.9%
Open Rate:24.1%
Click Rate:2.1%
Open Rate:-26.18%
Click Rate: +10.53%
GraphicText3349Open Rate:30.0%
Click Rate:3.8%
Revenue: $128.00
Open Rate:33.9%
Click Rate:5.7%
Revenue: $0
Open Rate:+13%
Click Rate:+50%
Revenue:-100%
GraphicText4370Open Rate:26.6%
Click Rate:2.0%
Open Rate:28.8%
Click Rate:3.4%
Open Rate: +8.27%
Click Rate:+70%
GraphicText1053Open Rate: 33.5%
Click Rate: 5.5%
Revenue: $273.26
Open Rate: 39.7%
Click Rate:6.0%
Revenue: $536.98
Open Rate: +18%
Click Rate: +9.09%
Revenue: +96.51%

Test A: Graphic Email                                                                                 Test B: Text/Button version
test 5.1                                                              test 5.2

Results

We tested a regular email campaign on different groups of subscribers. Some segments saw a boost by 50% & 70% respectively for open & click rates with the text-heavy email. In contrast, for other segments the open rate for the text-heavy version was lower by -26.18%.

Interpretation

The test isn’t completely conclusive. In fact, as we saw –   different segments even within the same email list can respond differently to the image-heavy vs. text-heavy versions.

 

Test 6: What’s the best time to send email campaigns?

Everyone wants to know, but few actually test it. So, what is the ideal send time? Unfortunately, there’s no single answer. Ideal sending times vary by industry, product, and your specific subscriber list. Here’s what have worked for in this particular case.

A-GroupB-GroupSend VolumeChampion StatsChallenger Stats% Difference
Friday 17:00Saturday 11:001587Open Rate:20.2%
Click Rate: 2.2%
Open Rate:19.9%
Click Rate: 1.6%
Open Rate: -1.5%
Click Rate: -27.2%
Wednesday 11:00Wednesday 16:002771Open Rate:30.7%
Click Rate: 4.4%
Open Rate:27.9%
Click Rate: 4.3%
Open Rate: -2.8%
Click Rate: -0.1%
MondayTuesday340Open Rate:32.7%
Click Rate: 4.8%
Open Rate:30.8%
Click Rate: 7.1%
Open Rate: -1.9%
Click Rate: +2.3%
SaturdaySunday5153Open Rate:16.4%
Click Rate: 3.9%
Open Rate:17.9%
Click Rate: 4.5%
Open Rate: +1.5%
Click Rate: +0.6%

Results

Saturday mornings performed worse for engagement rates vs. Friday evening.

Interpretation

Saturdays are generally worse days to send. The other times and days of week tends to vary for each store.

 

Conclusions

Based on these A/B testing we’ve conducted, here are our suggestions on how to make your next email campaign really effective:

  • Using subject lines framed as a question can increase open rates.
  • Showing the exact offer in the subject line tends to drive less engagement yet higher revenue.
  • Promoting one product instead of a list of them in one newsletter helps to create an email with a more focused CTA, which can increase engagement.
  • Hiding the price for your products in the email campaigns can decrease click-throughs.
  • Text-heavy emails can work better than image-heavy emails in certain circumstances and to certain segments of a list.
  • The optimal send time varies greatly depending on your store and audience. Test, test, and test to find the best time for you.

Write a Comment