Granularity in Landing Page Optimization

By Tim Ash, Search Engine Watch, Sep 16, 2009

The granularity of your test elements is the level of detail at which you will make changes to your page design. At one extreme, you can use specific and fine localized variations (such as changing button colors or text font sizes). At the other extreme, you can create coarse and fundamental changes that incorporate dozens of smaller individual design alternatives.

It isn't uncommon to completely redesign your whole landing page and test it head-to-head against your original.

In between is a continuum of possible scales at which you can test proposed changes. Sometimes these design alternatives can be nested within one another. For example, you may change the text of the call-to-action button on your form, change all of the text labels on the form input fields as well, or also change the size of the form and its position on the page.

Page Traffic Limits Test Size

The size of your test will be constrained by the traffic to your landing page and its data rate (the number of conversion actions per unit time). This will limit the total number of alternative test elements possible for each particular tuning method.

Changing the granularity of your tests allows you to include all or most of your important ideas while still fitting into a reasonable test size. Reduce your search space size by combining several individual changes into a single larger variable for testing.

My company routinely runs large-scale tests involving millions of possible versions of a landing page. When this kind of test size is available, it makes sense to get very granular on most of the changes you're considering.

With other testing methods or low data rates, you'll be forced to consolidate your test size. At that point, you have to decide if you want to focus on fine granularity changes or combining several of them into larger tuning elements.

The advantage of fine granularity changes is that they're quick and easy to implement. For example, you may want to consider different headlines for your page. It wouldn't take long to come up with some reasonable alternatives, set up a test, and start collecting data.

By continuously running back-to-back fine-granularity tests, you can often make significant conversion improvements. By their nature, these kinds of small incremental tests don't require a lot of work or emotional investment, and are ideal for this kind of champion-challenger continuous testing.

Wholesale page redesigns are sometimes the only option when you want to consider many potential changes but don't have the data rate or time to run a series of finer-granularity tests. Such redesigns are also the only way to deal with landing pages that have low coherency (see the next section).

The main drawback of whole-page redesigns is the time and effort that goes into creating them. Because you don't know if the new design will outperform the original, you're gambling that your larger up-front investment will pay off.

The Page is More than the Sum of Parts

Online marketers have argued that complete redesigns deny them so-called "learnings" about which individual elements contributed the most to the improved performance. I'm highly dubious of this kind of thinking. It's based on the flawed assumption that the individual elements are completely independent of one another, when they're often highly dependent on the context in which they're presented.

There's no inherent advantage to testing fine or coarse granularity changes. In a fine granularity test, one of our business-to-business clients used a "Free Quote Request" headline for their lead capture form. We proposed changing this single element of the landing page to "Instant Quote" and saw the form-fill conversion rate skyrocket by 58 percent.

For another client, we were asked to improve the sign-up rate for a free trial of stock option research software. We concluded that the only viable option, because of the low data rate and fundamental issues with the original page design, was to test complete redesigns of the page.

We tested the two whole-page alternative redesigns against the original in a three-way head-to-head test. Both of the new designs significantly outperformed the original, with the winning one resulting in 75 percent higher revenue per visitor after completion of the free trial.

Granularity doesn't have to be uniform among the elements you test. I frequently devote more attention to key elements of the landing page.

For example, I may have different call-to-action button colors (complementing the rest of the page or contrasting), formats (a button only, or a button with a text link under it), and text (several alternative variations). For less important or visible parts of the page (e.g., the footer), I may only test one alternative that includes several concurrent changes that might improve performance.