Big Data - Has It Blinded Us With Science?

blindedGigaOm's Om Malik recently wrote a great post about how data without soul is meaningless. Essentially, in a world where everything is quantified to the n-th degree, being solely dependent on numbers often obscures our ability to make reasoned, empathetic decisions and tell rich stories with our data.

Om gave an example of Uber's car service, which offers surge pricing in cases of high demand/low supply. Uber's pricing system jacked up the price on its customers who were stranded after Hurricane Sandy tore through New York City. Uber could have used the knowledge of hurricane conditions to give stranded customers a break, but instead, soulless pattern recognition resulted in an erosion in customer confidence.

I got to thinking on how this dogged dependency on the numbers affects the way SEM campaigns are managed day to day.

There is never a scarcity of data in paid search. While that presents incredible opportunities for ongoing optimization, it can also present some distinct challenges for those who want to put everything from bid management, to keyword selection, to ad optimization on data driven auto pilot.

A couple of examples:

Bid Management

By all accounts, in order to manage keyword pricing on large SEM campaigns, some kind of automated bidding system is necessary. However, giving your entire campaign over to automated bidding at all times can have some adverse consequences.

Take, for example, a scenario where a certain brand of products has been selling particularly well because of a sale that creates an attractive price relative to market. At that time, the keywords to represent that brand are performing great relative to target and they are bid up to higher positions.

When the sale ends, bids stay high on those keywords until the model has enough evidence that performance has truly dropped on those terms. A gentler, more human handshake from a merchandiser could have made those bid adjustments more quickly, since they have the knowledge of how much demand shifts for a given product at that higher price point.

Ad Optimization

Search marketing is a channel where every little piece of the campaign can be measured. However, just because something can be quantified doesn't mean it should be – or it least not until every other analysis option has been exhausted.

Case in point is something that I like to call "pixel optimization" within paid search ads: many search marketers run ad tests on things like the punctuation that is used at the end of their ad copy and come up with assertions like "I have proven that the most effective way to drive CTR is through adding an exclamation point vs. a period." It may well be true, at least in some cases in which exclamation points perform better, all else being equal. But I imagine there is much more potential value in testing core ad strategies against one another.

For example, when responding to a product query, is it better to use competitive language vs. scarcity vs. promotional? The results of this experiment would likely yield better results than my punctuation test – the challenge lies in that it takes a more measured and human approach to develop those ad strategies for testing. However, the story you can tell to your client at the end of that process, is much richer and informative as well.

Mistaking Correlation for Causation

I've heard a large number of advertisers say the following to me over the years "I always write promotional copy during a sale because when I write promotional copy, sales go up."

Just because sales copy is correlated with higher sales doesn't mean that the sales copy caused the higher sales. In fact, it is much more likely that the existence of the sale caused the higher revenue than the copy about the sale.

It's easy for an Excel sheet to makes these types of conclusions, and I'm sure that, in many cases, promotional copy does help increase revenue by creating a sense of urgency to click and buy today. However, these things need to be experimented on regularly to determine which areas of the campaign promotional copy actually results in more sales relative to other messages.

The eBay Study

EBay recently announced that it has proven quantitatively that paid search advertising doesn't work and returns only pennies on the dollar back to its business.

Objectively the results of this study are pretty sound. EBay conducted the study with some real academic horsepower behind it. As a result of reviewing these numbers, Ebay has significantly cut back its investment in paid search advertising because the medium wasn't able to immediately return a positive ROI.

While this ultimately might be the right approach for eBay, there is a lot more to be considered here beyond today's dollar return of those paid search clicks including:

  • The competitive landscape that eBay plays in: What would happen if eBay search ads weren't present for a year or more? 
  • The efficacy of eBay's current paid search ads: Are they as effective they could be in driving desired outcomes? 
  • EBay's other consituents – the sellers: Will they now feel that eBay won't support their listings as actively as other channels?

This isn't to say the numbers are lying…the eBay study is definitely groundbreaking and the most comprehensive study I have seen in trying to prove a causal relationship between ad exposure and performance. However, it would be easy to blindly rely on these results to shut down a paid search program that:

  • Could be significantly improved.
  • Likely has more far reaching effects on eBay's business than near term revenue.

Final Thoughts

This isn't a critique of big data in the least (although the term is getting a little stale!). These technologies and techniques are changing the way we do business in SEM and have commoditized a number of the functions that we as SEM practitioners used to spend a lot of time and effort on.

Many times the data produced highlights things that were not apparent to the naked eye, and can spawn a whole new set of inferences and experiments. There is a place for that in every organization doing SEM in this hyper-competitive market.

That said, a dogged reliance on these systems to replace human strategy, ingenuity, and analytic prowess can obscure as much as it enlightens. Not to mention the lack of soul.

About the author

Dave Schwartz is VP of Client Services and Marketing at DataPop, an ad optimization technology company. Dave has been with DataPop since 2009 and was hired by the founders as its first full-time employee in the U.S.

Before joining DataPop, Dave was the Category Director for Local at Yahoo, where he was responsible for driving adoption of the company's local advertising solutions. Prior to Yahoo, Dave was an Engagement Manager for J.D. Power Consulting, working with leading brands such as Toyota and Acura. David holds an AB/MS in economics from Stanford and an MBA from UCLA.