What Is A/B Testing?

A/B testing is the act of running a simultaneous experiment between two or more variants of a page to see which one performs the best.

Imagine, for instance, that you want to test your hypothesis that one headline will generate more leads than another. Sure, you could just make the change and cross your fingers. But what if you’re wrong? Mistakes can get costly.

By sending half your traffic to one version of the page and half to another, you can first gather evidence about which one works best before you commit to the change.

Essentially, A/B testing lets you play scientist—and make decisions based on data about how people actually behave when they hit your page.

A/B Testing Terminology

Variant

Variant is the term for any new versions of a landing page you include in your A/B test. Though you’ll have at least two variants in your A/B test, you can conduct these experiments with as many pages as desired.

Champion

You can think about A/B testing like gladiatorial combat. Two (or more) variants enter, but only one page leaves. This winner (the page with the best conversion performance, typically) is crowned the champion variant.

Challenger

When starting a test, you create new versions (variants) to challenge your existing champion page. These are called challengers. If a challenger outperforms all other variants, it becomes the new champion.

Assigning Traffic Weight in an A/B test

In a typical A/B test, traffic is randomly assigned to each page variant based upon a predetermined weighting. For example, if you are running a test with two page variants, you might split the traffic 50/50 or 60/40. To maintain the integrity of the test, visitors will always see the same variant, even if they return later.

The main factor that decides how much weight you would ascribe to your page variants during a test is timing: whether you’re starting the test with multiple variants at the same time or testing new ideas against an established page.

Starting from Scratch

If you’re starting a new campaign and have several ideas about which direction to take, you can create a variant for each idea.

In this scenario, you’d most likely assign equal weight to each version of the landing page. For two variants, that’d be 50/50. For three, it’d be 33/33/34. And so on. You want to treat them equally and pick a champion as soon as possible. As you have no conversion data on any of the pages, begin your experiment from a position of equality.

Testing Existing Landing Pages

If you have already have a page that you want to try some new ideas out on, it’s usually best to give your new variants a smaller percentage of traffic than the existing champion to mitigate the risk inherent with introducing new ideas.

This will be slower. It’s not recommended that you try to accelerate an A/B test by favoring new variants though, as they’re not guaranteed to perform well. (Remember, A/B testing is all about mitigating risk. Test wisely!)

---

What Should I Test on My Landing Pages?

Most marketing departments rely on a mixture of experience, gut instinct, and personal opinion when it comes to deciding what will work better for their customers. It sometimes works out, but often doesn’t. When you start A/B testing, you should be prepared to throw all the boardroom conjecture out the window: the data (properly interpreted, anyway) doesn’t lie. It’s worth telling your boss this.

There are a certain number of landing page elements that you can focus on in your testing. The different variations and content that goes into the test are up to you, but which one works the best (whether you like it or not) is up to the customers.

Some of the elements you should consider split testing are:

Headlines

Your main headline is usually a succinct rendering of your core value proposition. In other words, it sums up why anyone would want your product or service. There are many approaches you can try when testing your headline:

• Try a longer versus shorter headline
• Express negative or positive emotions
• Ask a question in your headline
• Make a testimonial part of your headline
• Try different Unique Selling Points

Call to Action (CTA)

The call to action is a button that represents your page’s conversion goal. You can test the CTA copy, the design of the button, and its color to see what works best. Try making the button bigger, for example, or make it green for go, blue for link color, orange or red for an emotional reaction.

Hero Shot

A hero shot is the main photo or image that appears above the fold. Ideally, it shows your product or service being used in a real-life context, but how do you know what hero shot will covert for which landing page? Do you go with the smiling couple? Or maybe a close up of the product itself? Experiment and find out.

Forms

Depending on your business, you might need more than just a first name and an email. If you have a particularly strong need for data, try running a test with many variations of your form at different lengths. This way you can make an informed decision about what abandonment rate is acceptable when weighed against the extra data produced.

Copy

Often the biggest factor is long copy versus short copy. Shorter is usually better, but for certain products and markets, detail is important in the decision-making process. You can also try reordering features and benefits, or making your language more or less literal.

There are lots of opinions on what works and what doesn’t, but why not test it and see for yourself?

Layout

Will a CTA on the left outperform one placed on the right? And does that testimonial video do better if you put it at the bottom of the page or the top? Good question. Sometimes changing the layout of a page can have major effects on your conversions.

Are A/B Tests Worth It? A Few Obstacles to Consider

A/B testing your landing pages can be a powerful way to squeeze more conversions (sometimes many more conversions) out of your existing campaigns, increasing your overall return on investment. It’s possible to make mistakes if you’re not careful in setting it up—most commonly, changing more than one element of a page at a time—but with a little reading, you can set yourself up for success.

That said, for small teams and businesses especially, there are a few hurdles that can make A/B testing your pages more challenging:

1. You Need to Wait for Statistical Significance

Imagine you flip a coin in the air. It comes up heads. You flip it a second time. Heads wins again. That’s strange, you think, as you give the coin a final flip. It lands heads up once more.

After three flips, are you ready to conclude that any flipped coin has a 100% chance of landing heads up? (Breaking News: Local Marketer Declares Laws of Probability Are A Sham.)

Probably not. Imagine heading to Vegas thinking a coin flip always comes up heads.

A similar thing happens when you A/B test a landing page. Until you’ve tested your variants with enough visitors to achieve statistical significance, you really shouldn’t apply your learnings. Instead, you need to eliminate as much uncertainty as possible before you decide on a champion variant. How many visitors you need can vary depending on your goals, but it’s typically a high number.

2. You Need Enough Traffic

The need for statistical significance poses another problem for small teams. If you don’t get enough traffic to be confident in your results, you can’t (or shouldn’t) end the A/B test. For smaller businesses, landing pages can take months to achieve the necessary results to draw a single conclusion. And sometimes that conclusion will be that the change you made (changing a button from red to green, for instance) hasn’t impacted your conversion rates at all!

If you’re running a timely marketing campaign, or just want to see results quickly, A/B testing without much traffic can be too slow to be useful. Waiting a year for a 5% conversion lift on a single landing page is unlikely to be appealing and hard to defend. Given that there are manual hassles involved in setting it up too, it won’t be worth your time.

3. It’s a “One-Size-Fits-All” Approach to Optimizing

This issue is one drawback baked into A/B testing: when you crown a champion variant, you’re choosing the version of your page that’s most likely to convert a majority of your visitors. This doesn’t mean that there weren’t other types of visitors who would’ve been more likely to convert on the losing variant. (It’s even possible these neglected visitors are more valuable to your business than the people for whom you’ve optimized.)

By design, A/B testing takes a blunt, “one-size-fits-all” approach to optimizing that’s likely not ideal for anyone. Sure, it can boost raw conversion rates in dramatic ways. But it sometimes lacks the nuance that growth-minded marketers obsessed with segmentation, personalization, and targeting might expect.

A/B Testing Alternatives: Using Smart Traffic

Let’s say you love the idea of optimizing your landing pages for more conversions, but can’t overcome one of the hurdles we’ve just discussed. How do you proceed?

Machine learning, thankfully, can help you improve your conversion rates without the high bar to entry of A/B testing. Using a tool like Unbounce’s Smart Traffic, for instance, lets small teams optimize their landing pages automatically (or, as computer scientists like to say, automagically) by having Artificial Intelligence do the kind of work that a human marketer can’t.

By running contextual bandit testing instead of A/B testing, Smart Traffic allows you to start seeing results in as few as 50 visitors, with an average conversion lift around 30%. There’s never any need to crown a champion because the AI routes each and every visitor to the landing page variant that’s most likely to convert them—based on their own unique context. No more “one-size-fits-all.”

How it works:

• You create one or more variants, changing whatever you’d like. Unlike A/B testing, you’re not limited to just one change at a time—and adding more than one variant doesn’t significantly slow down your time to optimizing. (Here’s a resource about creating landing page variants for Smart Traffic to get your started.)
• Set a conversion goal and turn it on. You decide what counts as a conversion in the Unbounce builder, then turn on Smart Traffic as your preferred optimization method. It starts working right away.
• Smart Traffic optimizes automatically. The beauty of this approach is it is relatively hands’ off. Once Smart Traffic is enabled, it keeps learning and optimizing throughout the life of your campaign.

Because of how easy they make optimizing, AI-powered tools should become a bigger part of your marketing stack. There are still plenty of reasons to choose A/B testing, but Smart Traffic enables even the little guys—or those of us who’re chronically short on time—to take advantage of optimization technology once affordable only by big enterprises.