A 4 Step Multivariate Testing Process That Works

When we talk about website testing, we mostly talk about what to test: unique selling propositions, headlines, calls-to-action, etc. It’s equally important to understand howto test (i.e., the process that produces better results).

The more complex the test, the more important it is to have process to minimize mistakes and maximize results and confidence in the results. Multivariate testing, with its many combinations, creative needs and more demanding implementation, is a great place for larger organizations to implement more rigorous processes around testing.

I suggest a simple four-step process:


Before we dive in, let’s start with some basic vocabulary:

  • Factor: An element of the page to test (headline, image, layout) 
  • Variation: A version of an element that you’re testing (image of a dog vs. cat vs. bird) 
  • Recipe: The combination of the variations

The Pilot Test

The purpose of the pilot test is not to improve conversion rate. Instead, it’s to identify the gaps in your implementation. Often, you’ll find that testing software behaves differently than you anticipated. Or, that some of the variations don’t work together.

Think of a pilot test like a dress rehearsal for your multivariate experiment. Inevitably, something goes wrong. The pilot test gives you the opportunity to QA all of the issues before the actual experiment, thus limiting any potential impact on your results or timeline.

A pilot test should be:

  • Simple: Change very small things, such as punctuation or having a link or not. Again, you’re not trying to improve conversion rate on this first test. 
  • Small in Scope: Only choose two factors and two variations per factor. You only need to verify that all systems are functioning. 
  • Limited: You need to show your test to just enough of your visitors to get enough data to find the weak links. The appropriate percentage depends entirely on your site’s traffic. The higher it is, the lower your test audience can be. Between 5 and 20 percent is a good rule of thumb.

Make sure your developers, creative team, and stakeholders all get a chance to poke and prod the test during the pilot phase. An hour at this stage is worth 5 hours later.

Test Wide

Great testing is based on quantitative (web analytics) and qualitative research (user testing, surveys, etc). That leads to a series of hypotheses about what is going to impact conversion rate and what you should change.

One of the distinct advantages of multivariate testing over A/B testing is that we can more clearly understand which parts of the page impact performance. During the Test Wide phase of the project, we use the data can confirm you which factors on your page impact performance.

Testing wide means that you:

  • Test more factors: Pick a wide variety of types of elements to test. They should be as different as possible. 
  • Test few variations: The goal isn’t to find the perfect headline. It’s to figure out whether the headline actual impacts results at all. That said, big differences are more likely to produce statistically valid results in less time. 
  • Ensure recipes are logical: With multivariate testing, you have to be extra vigilant to ensure that, when combined, the various elements make a cohesive page, regardless of their combination. Most multivariate testing software includes previews for the recipes.

The appropriate number of factors to test depends entirely on your traffic, test setup, current conversion rate, conversion rate improvement. Use this calculator to get a sense of what is reasonable in a doable timeframe.

Test Deep

The result of testing wide is that you know which factors influence conversion. Now, the goal is to determine which variations and recipes maximize that influence. This is Testing Deep.

Before, we had a broad reach with a relatively shallow exploration of creative changes. Now, our test is narrow, but has many more creative variations. The total number of recipes may not change.

Here you should dig into your research to come up with hypotheses you can test with various creative treatments. For example, if you’re testing headlines, testing deep might look like this:

  • Treatment 1 – Urgency: “Order in the Next 48 Hours and Get Free Shipping!” 
  • Treatment 2 – Social Proof: “9 out of 10 Cat Lovers Recommend Happy Cat Food” 
  • Treatment 3 – Discount: “$5 Off Your First Order” 
  • Treatment 4 – Green: “The Only Organic and Vegan Cat Food Approved by Vets”

Testing deep exposes the variations that have the most impact, as well as the recipes that work well together and the interaction effect among elements.

Test Validation

For the truly cautious tester, you can go the extra step to validate all of your findings from testing deep and wide.

In Test Validation, you take the winning recipe from the multivariate test and setup and A/B test of that recipe vs. the control recipe. This adds a layer of statistical confidence in your results before you deploy the new champion recipe to broader audience.

Process Inspires Discipline

There is no doubt that process can sometimes be frustrating when you’re trying to organize a larger team or get projects done quickly. But, it forces attention to detail. That exposes issues before they become problems.

More to the point, testing helps build a culture of data-driven decision making. It gradually removes ego and promotes experimentation and validation. Try this four-step process and let me know what you think.

Related reading

SEO tips tools guides 2018
guide to google analytics terms
SEO travel mistakes to avoid in 2019