Search Engines As Leeches, The Difference Between Paid & Free Listings & Keyword Price Rises

Jakob Nielsen's just posted a Search Engines as Leeches on the Web article that makes a good point, don't be too search engine dependent. However, he muddles his point by confusing the issue of paid search advertising and free "organic" listings. A closer look at that, plus how "super conversion trackers" and "brand idiots" are likely to keep pressure on keyword prices.

As a reminder, the major search engines give you two main types of listings when you do a search. There are the "organic" or "free" or "natural" listings that they gather from crawling the web. They don't charge for these listings (though Yahoo's paid inclusion program kind of clouds the water over there). These listings are like the editorial content you get at newspapers.

Search engines also carry paid ads. Pay, and you can get listed for terms you want without hoping that it just happens naturally.

Jakob says:

I worry that search engines are sucking out too much of the Web's value, acting as leeches on companies that create the very source materials the search engines index.

That will resonate with many who have long voiced similar concerns that search engines are making tons of money by gathering "content" from sites from across the web to make their listings.

If suddenly every site on the web were to block Google from indexing them, Google would have a crisis in short order. Its main "content" would have gone away, and the ads alone aren't going to keep attracting searchers.

Web site owners have not done that, however. That's because by and large, they've found that search engines drive more traffic to them than they cost in terms of bandwidth of being indexed.

WebmasterWorld has become a classic case study of this. Google and other search engines were banned in November along with "rogue" spiders, because somewhat similar to Jakob's "leech" metaphor, they were seen to have been sucking down more bandwidth than it was worth supporting.

WebmasterWorld founder Brett Tabke was often quoted saying he had the best sleep in months after blocking the spiders. His sleep may have improved, but what to do about the major spiders didn't go away. By the end of December, Brett had done a 180 degree turn and let the major spiders back in.

Until now, WebmasterWorld's been about the only major site I can think of that has tried to block spiders. Craigslist was rumored to have done so, but that wasn't true.

I do believe concerns over spidering are growing, especially as we have more spidering from both the major search engines and from rogues that are out there. Back in October, The lie of distribution--search engines return very little value to news/blog sites yet hog bandwidth and increase server loads from Tom Foremski was an example of this.

As I commented on his blog, it's fair to say that despite grumbles, that the vast majority of site owners do not consider search engines leeches. If they did, they would deleech themselves by blocking spiders. It's not hard to do. A simple change to the robots.txt file will block all the major search spiders. But no on does this, because they want the traffic. Even Jakob's own file isn't blocking Google and gang.

But back to Jakob's point, it turns out he really isn't talking about the "source material" being leeched but instead about the high cost of advertising. Again to his opening statement, with the key part in bold.

I worry that search engines are sucking out too much of the Web's value, acting as leeches on companies that create the very source materials the search engines index.

And the evidence of this?

Paid search confiscates too much of a website's value.

What? Paid search "confiscates" a site's value? Since when did search engines suddenly show up at a web site and demand the owner sign-up for advertising? We've long had rumors that a site that doesn't advertise might find themselves banned with various major search engines. We've even had reports of "monetization targeting" where site owners have found that doing an ad or paid inclusion buy might clear up a spam banning problem. But by and large, there are plenty of web sites that spend nil with search engines on advertising and get plenty of traffic.

In fact, the exhausting, annoying, tiring, boring, you name it regular updates to Google generate plenty of forum fodder that show people aren't spending and getting traffic from search engines. If they weren't, they wouldn't be freaking out any time Google undergoes a major algorithm change that sends rankings dancing and for some, traffic plunging. They wouldn't worry, because they'd have had both a balance of paid and natural search listings that helped them ride out the rough times if there was an issue on the natural side.

Instead, the October 2005 "Jagger" update showed plenty of site owners are still dependent on getting traffic from search engines for free. The Nov. 2003 Google Florida update should have taught many not to be free listing dependent, but clearly they remain this way. And the lessons not to be dependent were in place even before this.

So overall, the issue doesn't involve free listings. Jakob's really concerned about the rising cost in search advertising. Over time, as he's worked with client sites, they've been able to pay more by pushing up conversion rates. But at some point, others catch up and the margins of what his clients can pay is reached. He says:

If your search bid stays the same, your ad will sink off the page as more and more competing sites improve their design enough to afford higher bids. Our site therefore has no choice but to increase its own bid to $7.99 per click if it wants to stay in business.

This simply isn't true unless Jakob's clients are making the mistake of depending solely upon paid search ads to gain customers. If that's the case, yep, you should be looking to diversify. More on this in a moment.

Jakob also says (and the bolding is his):

This is great news for search engines: they can double their income by doing nothing. Just sit and wait for all other websites to improve -- then skim off the increased earnings.

In other words, the search engines get to make more by doing nothing because the advertisers are learning they can afford to pay more. And they sound pretty evil. But rewrite it this way:

This is great news for the Super Bowl: the cost of buying a commercial keeps going up even though they do nothing. Just wait for advertisers to be willing to pay more -- then skim off the increased earnings.

Honestly, perspective like the above is very much in order. Consider:

  • Search advertising has long been undervalued. People still pay less than a dollar for some paid search ads and they obtain clients that will be with them for life. But few advertisers are calculating the lifetime value of search. Jakob doesn't appear to be consider this. The examples he gives are to the purchases made directly from a click on an ad. Do his B2B customers come back to the site directly the next time and buy? If so, the only reason they did so was because they found the site through search in the first place. If you don't factor that lifetime value, then you fail to understand how much you really can afford to pay.
  • Advertisers are getting smarter. Those who better measure conversion know they can pay more and they are. The search engines are to blame for this? They're simply seeing that their undervalued advertising medium is finally growing to what it really is worth. The real person to be upset with is that other advertiser. And the solution is to get smarter.
  • It's an open marketplace. Is Apple is a leech on consumers for overcharging for MP3 players that you can buy from others that work as well if not better but lack the Apple logo? The search engines aren't forcing people to buy ads. If advertisers can't afford to continue paying high prices, they won't. And when they don't, the prices will fall. The exceptions are if the search engine conspire in some way to force purchases (say they really do link buying ads to getting other types of listings) or when they set artificially high minimum bids (Google tried doing this but had to drop many because people weren't willing to pay).

So rather than "despite search engines, websites can make money," as Jakob says, the reality remains that because of search engines, plenty of web sites are making money without spending a dime, pence, euro, yen or whatever on search advertising.

I completely agree that anyone running a web site should heed Jakob's "search engine liberation" advice of alternative ways to promote a web site, such as considering RSS, email newsletters and so on. But this isn't suddenly new advice. Any long-time internet marketer would tell you not to depend just on search engines. Thinking "beyond search engines" has been the core of my basic tips since I put them up back in 1997:

Search engines are a primary way people look for web sites, but they are not the only way. People also find sites through word-of-mouth, traditional advertising, the traditional media, newsgroup postings, web directories and links from other sites. Many times, these alternative forms are far more effective draws than are search engines. The audience you want may be visiting a site that you can partner with or reading a magazine that you've never informed of your site. Do the simple things to best make your site relevant to search engines, then concentrate on the other areas.

It's all a matter of balance. Don't obsess over search engine listings, but don't ignore them, either. Do a variety of online marketing activities -- and do a variety of offline ones, as well. Search -- both paid and free -- is a component of any campaign. But it isn't something you should depend on, any more than you should depend on all television advertising, all print ads, all RSS ads or a strategy of no advertising at all. If you are not diversified, you'll have a weakness that might hit when you least expect it.

To conclude, no one should put all their eggs into any basket, search or otherwise. It's absolutely true that search engines are not the end all be all and that sites can thrive and survive without them. But many sites also can thrive and survive better by incorporating them into a diversified publicity campaign.

Search engines definitely can do more to help those with support on the organic side of things, which is especially needed since webmasters do indeed provide the content that the search engines depend on. The good news is that last year, we saw more changes and developments to give webmasters new tools than ever before.

Finally, ad prices will likely continue to rise, and different advertisers will react in various ways. John Battelle recently pointed at a blog suggesting that FTD might be nearing all it can afford to spend. But just today, we reported on a survey showing four out of five advertisers saying they can still afford to pay more, though the question of whether a plateau is being reached is raised. Then the latest Fathom report on keyword prices saw a continuing "downward spiral," as MediaPost put it. I haven't looked closely at the latest numbers, but the sample is so small (500 terms) that I'm generally wary on depending on it as a foolproof predictor.

From my part, I see two main issues with keyword prices going forward: Super Conversion Trackers & Brand Idiots.

Super Conversion Trackers are those who will indeed track a lifetime value of someone who comes to them from search. They'll understand that the initial purchase may lead to more and more purchases over time and feel comfortable paying multiples above competitors to gain a lead. That will push some out of the bidding. See Most Conversions Happen Offline; You Need To Measure These! for some further thoughts on this.

Brand Idiots are what some marketers think derisively of others who jump into bidding without linking it to a direct ROI target. They can screw up bidding on what seems to be "logical" or "fair" amount that most in the marketplace may assume. But brand idiots are part of that marketplace, and you can expect to see more of them.

Automakers Buy Up 80% Of Ad Space On Car Sites For 2006 from AdAge is a good example of this. It explains how automakers are going more and more online to extend their brands. Edmunds, a car research site, expects to take those brand dollars and buy more search as well as display ads to fuel that desire. That's big brand money that's going to be fueling those buys and putting pressure on others trying to compete.

Big Guys Crowd Out Little Guys in SEM Arena; Some Branding Focused Advertisers Willing to Spend "Whatever" It Takes and C'mon In Brand Owners, The Search Water's Fine has more on these type of moves.

Stuck in a bidding war? How To Get Out Of Bid Wars A Winner? over at our Search Engine Watch Forums may have some helpful advice if you're already tracking and improving conversions as much as possible and getting some brand idiot money is not an option.

Want to comment or discuss? Visit our SEW Forums thread, Search Engine Leeches, Dependency & Losing Perspective.

About the author

Danny Sullivan was the founder and editor of Search Engine Watch from June 1997 until November 2006.

To contact current Search Engine Watch editorial staff, please click here.