In statistical circles, it's a known fact that the Taguchi method is a bad fit for landing page testing. So why to so many marketers use it to test landing pages?

I was under the impression that this is a well-understood fact, at least to anyone who has a solid understanding of basic statistics. Unfortunately, this seems to leave out most landing page testers. So making this statement when I recently spoke on the multivariate testing panel at eMetrics in San Francisco was the equivalent of dropping a hand grenade into the room.

There are two common mathematical approaches in the world of landing page testing: A-B split testing, and parametric multivariate testing. A subset of multivariate testing is known as the design of experiments -- also called a fractional factorial. A common fractional factorial approach is called the Taguchi method.

Some online marketers consider A-B split testing to be kind of wimpy, and endow fractional factorial methods with an almost mythical quality. But they are sadly misguided.

I spend way too much of my time explaining to people that fractional factorial methods are a really bad idea when applied to landing page optimization. Despite this, the illusion persists that this kind of testing is somehow state-of-the-art, when in fact, nothing could be further from the truth.

There's a huge mismatch between the original environment in which fractional factorial testing was developed and how it's usually applied to landing page optimization. It was basically transplanted to online marketing because it's relatively easy for a non-mathematical audience to understand, and not because of its appropriateness or fitness for the task.

I'll explain why in a moment, but if you're interested in going into this in more detail (30 pages worth), you can download this whitepaper: "The Truth About Taguchi."

Landing page testing is composed of two important activities:

1. Deciding what to test and coming up with good ideas.
2. Finding the best solution among your tested alternatives.

People claim to get really good results with fractional factorial multivariate testing, and they credit this to the method they use to analyze the data.

In reality, the improved conversion rates are the result of the great ideas for new landing page elements that go into the test. If your alternative landing pages designs are better then the original, it doesn't really matter what method you use to confirm that. Fractional factorial approaches may actually miss the best version of the landing page in your test and often lead you to a sub-optimal answer.

The principal drawbacks of fractional factorial methods are:

• Very small test sizes.
• Restrictive and inflexible test designs.
• Less accurate estimation of individual variable contributions.
• Drawing the wrong conclusions.
• Inability to consider context and variable interactions.

If you plan on using parametric (i.e. "model building") approaches for landing page testing, you should always use full factorial data collection, regardless of the subsequent analysis you plan to do. It greatly simplifies your test design, and produces better estimates of the main effects.

All parametric methods (including both full factorial and fractional factorial) are also outclassed by newer non-parametric testing methods, such as the SiteTuners TuningEngine

Hopefully, this will set the record straight. If you still have an issue with this, and insist on proclaiming the superiority of fractional factorial methods, tell your statistician to call us and I'll have my chief scientist beat them up properly.

Stumble It! Digg this! Add to del.icio.us

• 17266
  CNET Networks New York, United States
• Display Media Sales Manager
  Become, Inc Mountain View, United States
• 17086
  CNET Networks San Francisco, United States
• 130320
  Dell Nashua, United States
• SEO Manager
  Forex Capital Markets (FXCM) New York, United States