Lies, Damned Lies, and Statistics in Landing Page Optimization
Another type of sampling bias can be introduced by sequential testing. Even if you run your tests by splitting the available traffic and showing different versions of your site design in parallel, you may still run into biased sampling issues...