The Fallacy of Ad Testing

Test TubesAd copy testing is one of the staples of good SEM agencies. “Test and test often” is a bit of wisdom you’ll hear over and over in the field.

But do you know exactly what you’re testing? Can you really trust the results? In most cases, the answer is no.

I’ll explain using a recent example from one client’s account.

Two ad copies, A1 and A2, were tested using the last 30 days as the time frame.

  • A1 got 8 conversions out of 193 clicks, for a CPA of $19.66 and conversion rate of 4.15 percent.
  • A2 got 10 conversions out of 184 clicks, for a CPA of $14.75 and conversion rate of 5.43 percent.

A2 beat A1 by a whopping 30 percent margin in CPA and conversion rate. With 400 clicks in a 30-day period, the data wasn’t statistically significant, but as marketers we’re often making intuitive decisions on less than ideal sets of data. So A2 was the winner.


We hear this type of talk often in our line of work. Now, this particular example is real data, from a real campaign, but the result is a fallacy.

Why? Because A1 and A2 have the exact same ad copy. See this screenshot:


So there must be other variables at play here. And as all good data scientists know, variables make for unreliable test results.

Ad Copy Testing Variables (the Devil in the Details)

Looking deeper, you’ll note that the above test is actually very rigorous.

  • This ad group only has one exact match keyword.
  • The average position of both ads is 1.1.
  • Both ads are static text, with no DKI.

In most ad testing, people don’t even isolate the keyword. This is the first issue, because a group of keywords, no matter how closely themed they are, will all have different conversion rates. You just don’t know which ads are triggered by which keywords.

Further, each keyword can be triggered by a different search query via broad/phrase matching, so the conversion rate/CPA varies even more depending on what query triggered the ad.

Ad position is another big factor in determining the ad copy performance. If an ad keeps getting triggered by keywords in high ad positions, then the conversion rate may suffer vs. low-position ads, which typically convert better at much lower volume.

When we don’t isolate keyword (exact match), and when ad positions vary, how can we compare the conversion rate and CPA of two ads – especially when even after we isolate all the variance factors, as we did in the above test, we still end up having a big variance on the ad copy performance using the exact same ad copy?

Does Any Ad Test Produce Real Results?

I’m usually highly reluctant to declare ad copy winners. My standard for a real “winner” is that it needs to beat the incumbent by a margin bigger than the incumbent beats itself.

In other words, when the new (A2) was tested against the incumbent (A1), setting up a three-way play should have been considered: A1, A1, A2.

(Note: Always isolate keyword variance by using single keyword exact match; isolate ad position variance by keeping the bid constant; and make sure ads rotate evenly. Then just wait for data to accumulate to a degree of significance.)

Now, look at the data to see if A2 beats both A1s by comfortable margin – and if that margin is bigger than the difference in A1 vs. A1. A winner should win in an obvious way, often by more than 20 to 30 percent margins. If the win isn’t obvious, then you’re probably better off sticking with the incumbent.

Related reading

traffic forecasting customer journey
Making the case for more non-brand funding in paid search
Five things to do on a small digital marketing budget
The fall of ad copy, long live ad copy