Always Be Improving: The 5 Rules of A/B Testing

Testing and Optimization


There are two types of marketers in this world: The 2 percent who would happily A/B test all the livelong day, and the 98 percent who emit an audible groan every time the topic gets raised.  The latter group — the vast majority of marketers — complains: “I know I should be testing more, but…”

“But it’s hard.”  “But it’s time-consuming.” “But I already think I know the answers anyway, so why should I bother to test?”

Although a paltry 44 percent of companies currently use split-testing software, I’m here to give you the bad news: testing is crucial. And it can’t just be a one-time event. Even Alec Baldwin agrees — at least in Glengarry Glen Ross. Did you know that his famous “Always Be Closing” speech came from an age-old marketing line?

Always Be Testing the Market

Sadly, that line isn’t catchy enough for a movie. But it’s the gospel of successfully evolving marketing campaigns.

Along with this gospel, there are five more rules of A/B testing that demystify what might seem like a tedious, overwhelming process.

1. Plan ahead 

You’ve created a brilliant email campaign, segmented your lists, and designed a magnetic landing page. You’re all set to go, but in the eleventh hour, someone says, “Hey, maybe we should test this thing.” Collective groan.

No marketer in  her right mind wants to adjust a campaign for A/B testing at the last minute. The best tests take time, so build testing into your process from the beginning, every time.

2. Be willing to learn… slowly

This might sound obvious, but if you’re not willing to learn from your tests, there’s no point in conducting them. The best way to learn from A/B testing is to test one variable within a large population so that you can be confident that your results are accurate across your audience.

If you insist on conducting multivariate testing to “save time,” consider hiring a Ph.D. in statistics to keep track of all the data for you.

Seriously, in a world where you’re always testing, you don’t need to be overambitious. Gleaning one useful bit of information from a campaign can take you much further in your ability to connect with your audience next time.

3. Measure what matters

A lot of marketers simply measure opens and clicks. But opens and clicks aren’t your end goal, are they? I’m guessing you’re looking to convert your audience, so test for what you actually care about.

Improve Your Conversion Rates You’ll want to find a tool that helps you measure performance based on any type of digital activity — for example, how many people who received an email invitation actually went on to register for the event.  Opens and clicks have their place, but measuring based on your high-level goals is where it’s really at.

So, figure out what those exact goals are. Then test for them.

4. Test over time

One of the biggest misperceptions about A/B testing is that it yields quick results. But honing in on immediate numbers ignores a large segment of your audience.

Let’s say you have an automated subscriber email that gets triggered each time someone signs up for your newsletter. Try inserting an A/B test that sends variations of that automated email. However, you’ll still need to test the results over a period of time — say, a month, depending on volume – to see how ongoing signups respond to the two email variations.

The same concept applies to nurturing emails, where every recipient is on a different cadence, and results are tabulated over a long period of time.

5. Always talk about testing

Your entire company can benefit from the insights that testing yields, so talk about your testing. As a marketer, you’re closer to your customer base than almost anyone in the company, so you’re in a prime position to share insight into how your customers think.

And with the cold hard facts of testing at your disposal, ears will perk up when you begin to share what you learn. Make it a regular part of company meetings to impart the wisdom you’ve learned from your testing, and you’ll become the de facto customer expert of your organization.

To put testing into practice, you must first adopt the attitude that everything your marketing team produces is testable. This applies to the subject lines of your emails, the content of your campaigns, the day(s) and time(s) you send messaging, the copy on your landing pages… I could go on and on, but you get the idea. Over time, aim to test every minute facet of your marketing, fine-tuning your strategy, so you’re optimized for utter marketing success.

Basically, if you’re taking the time to create a campaign, take the time to test it. There is always something to test. Never miss an opportunity!