A/B or Multivariate? How to Choose the Right Type of Test

When it comes to creative testing, the two main areas where people struggle is with what to test and how to test it. If you read my column about the six top elements to test in a PPC ad, you might remember this image of a dissected ad with almost a dozen elements to test:

But those are just elements of your ad. Pair them with elements of your products and you end up with something like the image below, courtesy of DataPop:

So with a list of (almost) endless elements and combinations to test, how do you actually test them? Do you make one change at a time or do you test as many as you can? That’s when we often see the struggle between choosing A/B and multivariate testing.

To answer that question, let’s take a step back to the basics and look at what each type of test entails.

A/B (Univariate)

A/B, or univariate, is a type of test where you only change one variable. You may test different variations of the same variable, but you are still only testing one variable. To do that, it’s important to understand the difference between an element, a variation and variable.

Suppose you want to test the call-to-action (CTA) in your PPC ad and you come up with the following:

• “Try Now”
• “Try it Today”
• “Click to Try it”

In this case, the CTA is the element, the different copy is the variation, and the copy itself is the variable.

So what you’ve created in this case is an A/B/C test where you're testing different variations of the CTA (element), which still falls under the A/B testing umbrella. You’d be surprised by how many people mistake this for a multivariate test. Which brings me to the second type of test.

Multivariate

Multivariate testing, on the other hand, is when you test two or more variables that are nominally independent of one another. If you were to run one on the ad about, for example, you could test removing the dynamic keyword insertion (DKI) from the headline and using the call-to-action “Click to Try it”.

Which Test do You Choose?

Now that you know the difference between the two, it’s time to make a choice.

A/B testing is simpler, allows for more control, and the results are easier to interpret. Think of it as the best way to pick the low-hanging conversion fruit. With A/B testing, it’s easier to test and apply findings to similar ad groups in your account.

Multivariate testing can have a higher impact, but it’s more difficult to isolate the variable that led to a certain outcome. Let’s say that our hypothetical Multivariate test above yielded a 30 percent increase in CTR. Do you attribute this to the new CTA or Headline? Or was it the combination of both? That’s when multivariate can get a little tricky.

Multivariate testing is exciting. Yes, exciting! But it’s also easy to get carried away with wanting to test every possible combination of every variable and variations. Don’t go down that path. I assure you that you neither have the time not traffic to do that.

Think little changes, big differences. That’s what you want to see.

A simple way to decide is by asking yourself the following question: Do you want maximum impact or do you want to understand impact of individual elements?

The Bottom Line?

Very rarely is someone’s sole job just to optimize ads. Because you have to juggle between and manage other aspects of your PPC account, you’ll want to be smart when it comes to testing.

Finding highly influential variations is much more valuable that testing all the potential combinations. That’s because they are easier to isolate and test or apply elsewhere.

Depending on your goals, traffic, and available resources, you should be able to make a decision based on which type of test is a better fit for you. If you aren't testing regularly, start with a simple A/B and then try multivariate as you go along. There is no right or wrong. There is only testing and understanding.

Which type of test do you use more regularly and what kind of results are you seeing from each? Share your thoughts in the comments below.