The C-level executive -- CEO, CMO, any CxO -- doesn't often get involved in A/B Web site optimization decisions. It's a tactic better left to SEOs, Web site analytics gurus, and statisticians.
What do you do, then, when the CxO asks you to explain how you do A/B conversion testing for an e-commerce site, for example?
The number one SEO challenge during holiday shopping season: explaining to senior management how complex it is to do A/B testing of SEO strategies for large enterprise mega-sites.
Number one SEO challenge: contamination. Number two SEO challenge: bleeding. Number three: moving parts.
Most Common SEO A/B Testing Strategy and Bleeding
The most commonly used strategy for large enterprise A/B testing: take an entire category -- or a sampling of internal Web site pages -- and make a few changes. A small sample of the site will do little to no harm when measuring changes. This type of test will fail in every case, due to what I call the bleeding factor.
Since pages that may be as many as four or five clicks from the home page, rather than one or two clicks, often perform differently, you can never see the true results of your changes. The bleeding effect becomes most apparent when you have a group of pages that you have changed in a positive way, and neighboring pages connected with similar attribute links also boost up in ranking.
The effects of bleeding can be extremely difficult to understand, since it all depends on how relevant the bleed was; this will also vary based on scale and size of the site.
For example, if you make a change to a category such as digital cameras, there will be a few brands that are compatible with other categories, such as Sony. Sony can be a compatible cross theme with televisions, CD players and DVD players, to name a few. Nikon, for example, will not be as compatible with other categories. Thus, the increase is seen with the related Sony categories, but cannot be shared with all other digital camera brands.
SEO A/B Blood Test: Link Building Impact
Another example of bleeding -- and difficult to measure -- is A/B testing's impact on link building. Consider this scenario: if you negotiate a special link with a digital camera manufacturer, and point it to your main digital camera category page, what will be the bleeding effect of that one link?
Let's say this link will come from Nikon. That will help the Nikon sub-category, as well as any of the other digital camera categories. Now your link test results will be contaminated.
An alternative: Nikon links directly to the Nikon sub-category. You'd expect that the uplift would only be felt within the Nikon sub-category, but in fact, any decent site will link back to the top category from the sub-category page, pushing link weight back to the top.
Additionally, the link weight passed to the primary category page would then pass to the other brands marked as a sub-category. Thus, your bleeding effect is felt in other categories as well.
A/B Site Tuning, Duplicate Content, Multiple Brands
Some companies will have multiple brands or sites that either have similar offerings or a different offering altogether. If the offerings are similar, there is a slight chance that accurate A/B testing can occur. However, if the sites are completely different, this type of testing will not be possible.
A simple way of testing would be to change a category in some way and not to change it on the other site. Measure the impact and try and understand the effect.
On the other hand, if there is a lot of duplicate content involved, this test could be considered a failure, since one set of pages may just drop out of the index. If you have a business intelligence department on board, they'll want to have a third site as a way to double check the results.
This can be an effective way to measure if the search engine moved one site up and then took one site down in the process. Inevitably, we come back to the simple conclusion that A/B testing of SEO on large sites will likely drive you insane.
Conversion optimizer vs. User Groups
A/B testing based on groups of users can cause issues with the search engines, since they'll flip pages every time they get spidered (define). Make sure the software package you use can identify a search engine and not try to show different data each time the spider makes a request.
My Best SEO Advice
The best recommendation I can provide: make small changes on entire page types at the same time globally, and measure the impact.
If your site has at least 1 million pages or more -- and at least five pages types -- the impact will not be that bad if it goes south.
This is the most effective way of determining what types of major changes will be most beneficial for your unique site.
Aaron is off this week. Today's column ran earlier on Search Engine Watch.