The Paid Inclusion Dinosaur

In my first part of this series on paid inclusion, I wrote about how Yahoo sees paid inclusion as a solution to some of the problems crawlers have in gathering content from across the web. I’ll examine that argument more closely later in the series. In this part, I’ll look at the other major reason Yahoo likes paid inclusion: money.

Much of the screen real estate on search engines is unsold — a surprising amount, some will find. To illustrate this, look at the chart below, which shows the number of paid versus free links for a search on “shoes” at several major search engines according to a check I did last week:

Links Yahoo AOL MSN Google Ask
Paid Links






Free Links






Total Links






% Free






An important caveat. I didn’t subtract any paid inclusion URLs that may have been mixed in among the “free” links. Identifying these is a complex task, as I’ll explain more in another part of this series. Despite this, the free figure still serves as a useful starting point.

You’ll see that “pure” Google has one of the lowest percentages of free material on its search results pages. Half of Google’s major outgoing links are devoted to ads.

Of course, most of these paid links at Google reside in the less prominent ad area along the left-side of its results page. That’s why Google doesn’t feel so ad heavy. As for “sell-out” Yahoo, it devotes more of its search results page to editorial content than any of its competitors.

Why not simply have all paid listings and stop giving away traffic away for free? Because doing so might hurt relevancy. You can’t earn at all, if people stop coming to search with you.

Enter the attraction of paid inclusion. Some search engines see it as a way to make money off their editorial results without (hopefully) hurting relevancy. But it’s a gamble, and one that so far remains unproven.

Throw The Dice!

Until Yahoo’s recent move, LookSmart had placed the biggest bet on paid inclusion. My article from 2002, The Bumpy Road To Maximum Monetization, explains how the company sought to earn attractive, recurring income by selling all of its commercial listings on a cost-per-click basis.

Verdict? MSN dumped LookSmart as a search provider last October, saying a big reason was that LookSmart’s relevancy was poor. So there you have it, proof that paid inclusion hurts relevancy!

Were that it was so simple. It’s difficult to be conclusive. For one thing, LookSmart’s limited human-powered database might not have performed as well against a comprehensive crawler-based index, even if paid inclusion weren’t involved. In addition, LookSmart recently fired back that had its US database been tested, rather than its UK one, the results would have been better.

The LookSmart experience is important, because Yahoo has embarked on a somewhat similar path. Its new program puts all paid inclusion listings on a cost-per-click basis. Potentially, that means good pages might disappear, if the owners stop paying. However, Yahoo assures that it will continue to carry the vast majority of its listings for free, including any good pages, regardless of whether they’ve paid.

Another part of this series will look much more closely at this claim and how the Yahoo paid inclusion system operates. For this part, we’ll stay focused on paid inclusion as a revenue source.

Dinosaur Of The “Powered By” Days

Why did I call this part of the series “The Paid Inclusion Dinosaur?” Because paid inclusion evolved as a survival means for companies like Inktomi and LookSmart that “powered” search for others. They needed the income to subsidize the cost of providing results to partners who were pressuring them for better deals due to new competition from Overture and Google.

To understand more, let’s go back in time.

In the 1990s, if you were a portal like MSN, Yahoo or a popular ISP that wanted to offer search, you licensed some or all of your results from a search provider. You paid someone like Inktomi or LookSmart for the right to use their listings. Alternatively, you might have agreed to share the relatively minimal revenue from banner ads that appeared on search results page. The days of cost-per-click paid listings hadn’t yet arrived.

By 2000, things had shifted. Overture was banging on your door, offering you search results in the form of paid listings either to completely replace what you currently had or to be used alongside any existing results. Failing search sites such as took up the offer, along with every healthy search site but Google.

Overture’s activity wasn’t good news for Inktomi and LookSmart, which were expecting to be paid for their results. They both got further bad news in the form of Google.

Google was still playing the traditional “pay me for my results” game to win distribution deals. It had previously won deals with places like the Virgin ISP in the UK and Netscape, but the big win was grabbing Yahoo in 2000. That dumped former Yahoo partner Inktomi out on the street.

Inktomi complained that upstart Google made a better business deal than it could — and as we now know, Yahoo did gain a small stake in Google.

How could players like Inktomi and LookSmart fight back against the competitiveness of Overture and Google? Paid inclusion was their solution. By selling a presence within their editorial results, they gained a new revenue stream to share with partners or to offset the cost of providing partners with results.

Some Paid Inclusion Alternatives

What made sense for Inktomi and LookSmart to do in the past makes no sense for Yahoo to do now. Yahoo operates its own popular search site, powered by its own search technology. It can subsidize the cost of providing “free” editorial results through the ads that it offers, just as Google does. There are simply better ways for it to increase revenue than follow the traditional paid inclusion model — and ways that would avoid the mixed messages paid inclusion brings with it.

How? For one, Yahoo could have reduced the number of editorial listings it shows to 10, rather than the 20 it currently provides by default. That likely would have upped the paid listing clickthrough rate (and revenues), yet Yahoo still would have carried fewer ads than Google.

Alternatively, Yahoo could have increased the number of paid listings it shows at the top of its results page from the current four to more than this. That’s what happens at Ask Jeeves. Ask will show up all the paid listings that its ad partner Google can provide, up to a maximum of 10. These appear above its editorial results.

Of course, there’s a real danger that adding more paid listings “above the fold” could make searchers feel results are too commercial. However, at least the paid listings would still be well marked, and editorial results would still be present on the page.

There are even more creative and valuable ways Yahoo could have made money by partnering with content owners, which I’ll explore later in the series. These include things such as centralized and approved rank checking, clickthrough tracking independent of paid inclusion or express support for crawler issues.

Unfortunately, the current paid inclusion program isn’t creative, doesn’t solve the mixed messages problem, and charging a per click fee for ALL types of paid inclusion was a giant step backwards.

FYI, Yahoo itself may be recognizing this, as I’ve heard rumors the company has been seriously debating dropping the per click component for pages in its basic paid inclusion program. Doing so would solve a variety of problem, as I’ll explore more in a future part of the series. Unfortunately, the decision so far appears to be to hold the course.

Why does the paid dinosaur still live? My pet theory is that Yahoo has plenty of internal paid inclusion cheerleaders who have pushed it along a path perhaps better avoided. Remember, Yahoo is a company that absorbed the people who were tied into running SIX different paid inclusion programs, two types each that were offered by Inktomi, AltaVista and AllTheWeb. Want to save your job? Convince the higher-ups at Yahoo that paid inclusion is a profit center.

I know from an internal contact that there’s been a debate on whether Yahoo should have done paid inclusion at all. Those in favor of it so far have won, arguing that data shows it improves relevance. Perhaps. But there’s no doubt that the decision to go forward with the new-fangled program made things more confusing for consumers and advertisers.

Paid Placement Should Incorporate Paid Inclusion

Ironically, Yahoo’s Overture unit has been saddled with the task of selling confusing paid inclusion alongside a very easy to understand, extremely valuable product — paid placement listings. Unlike paid inclusion, paid placement has real advantages to advertisers and consumers.

Consumers get ads that are clearly labeled and segregated from the editorial results, so they have more faith in their search engine. In addition, these are ads that often are as helpful if not more so than editorial results. As for advertisers, they get guaranteed, controllable placement.

I spoke last year at our Search Engine Strategies conference in San Jose of how I thought paid inclusion would evolve as a subset of paid placement, a tool for creating more targeted ads automatically based on the content of advertisers. The idea has real advantages.

Right now, Overture and Google struggle with how to fill out the “tail” of search queries with ads. These are queries that happen only a few times per month but in aggregate represent a huge amount of unsold inventory.

For example, advertisers focus on high frequency terms such as “shoes,” which Overture reports had over 1,000,000 requests in April 2004. It sells for a top bid of around $0.55 and has 90 advertisers competing for it.

In contrast, “winged track shoes” happened only 27 times last month. No one is bidding on the term. That’s most likely because the low frequency doesn’t make it seem worth the time. Yet someone searching for something so specific might convert better than the more generic “shoes” searcher.

Paid inclusion is a perfect solution for unsold ad inventory like this. Simply tell the advertiser that you’ll spider their site and let pages appear in the paid placement area for a flat, low-cost rate when there is space available. Advertisers can still target the important terms, but they gain easy visibility for other important ones. Yahoo reduces its unsold inventory. Searchers are spared confusion.

Ironically, while Yahoo’s steeped itself in controversy by not integrating paid inclusion as a form of paid placement, Google appears to be heading down that road with a new internal tool to do this exactly.

Don’t Meet, Exceed Disclosure Requirements!

Another revenue building solution Yahoo could have tried would be to segregate its paid inclusion listings entirely. Why not have a section of “Sponsored” paid placement results, followed by something like “Content Partner” listings, then ordinary web results after that? That would help many advertisers see value in doing paid inclusion yet keep things cleaner for the searcher.

Even more radically, why not give paid inclusion listings a ranking boost? After all, Yahoo’s made a big deal that paid inclusion and its other content acquisition programs are supposedly ferreting out the best in content from across the web. So embrace this and reward this content, if you’re proud of it. Again, that gives advertisers value. As for consumers, if the content really is better, then they’d be better served by such promotion.

It’s important to remember that US Federal Trade Commission guidelines don’t forbid paid inclusion listings from being boosted. They simply say that you should somehow provide adequate disclosure of how paid inclusion operates within your service.

Right now, Yahoo appears to be meeting those requirements through a “What’s This?” link that appears at the top of its web search results section, next to the “Top 20 Web Results” heading. The link leads to a basic page that explains how paid inclusion works at Yahoo.

While Yahoo’s doing what’s required, I personally I don’t think just meeting the FTC requirements is sufficient any longer. I’d like Yahoo to take a leadership role and exceed the requirements by somehow clearly labeling paid inclusion listings. It’s a move I think would help improve the somewhat battered image that paid inclusion has earned over the past two years.

What exactly might Yahoo do? Why do it? And what does Yahoo think about such an idea? That’s what I’ll explore in the next part of this series.

Want to comment on this or other search stories? Come test our new Search Engine Watch Forums!

Related reading

The evolution of SEO and the shift from point solutions to platform
How AI is powering real-time SEO research Insights and optimization
SEO case study - How Venngage turned search into their primary lead source