The Evolution Of Paid Inclusion

Paid inclusion has always been a tricky concept to explain, but understanding it is important to both webmasters and searchers, as recent changes have suddenly made paid inclusion commonplace with search engines. For webmasters, do these new programs mean that you won't get listed, unless you pay? And for searchers, do these programs present a threat to the quality of your results?

Paid inclusion programs mean that, in exchange for a payment, a search engine will guarantee to list pages from a web site. These programs typically do not guarantee that the pages will rank well for particular queries, however. Whether a page ranks well still depends on the search engine's underlying relevancy algorithms.

In contrast, paid placement programs do guarantee top listings. In them, if you pay enough, you'll be top ranked for the terms of your choice. Given this, why would anyone bother with paid inclusion?

For one thing, paid placement can be expensive. Getting and keeping a top ranking at paid placement search engine like GoTo involves an ongoing expense, because you pay by the click for any traffic you receive. In contrast, basic paid inclusion programs at crawler-based search engines such as Inktomi operate on a flat-fee basis. You pay once, and then you get all the traffic you can eat, so to speak.

Of course, there's no guarantee you'll get any traffic at all, with paid inclusion. That's why I've likened these programs to playing the "search engine lottery." You are essentially buying additional listings that you might not ordinarily receive. In doing so, just as with a lottery, you may have more chances of coming up than normal for various search terms. If this happens, you may find that you get plenty of quality traffic for less expense than through paid placement programs.

Paid Inclusion At The Crawlers

Inktomi is the oldest crawler-based search engine to offer paid inclusion, having made it available to the general public since November. In June, it was joined by AltaVista, while FAST announced its program would begin in beta this month and be available to the public in September. That means three of the five major crawlers now have these programs -- leaving Excite and Google as the odd ones out.

Of those two, Excite says it expects to launch its own paid inclusion program perhaps later this year, once its parent company Excite@Home has determined how to handle processing of subscription services that might be offered by its portal. That would leave Google standing alone, a position the company is happy to occupy.

"We have no plans for a paid inclusion program. As we've stated in the past, our search results represent our editorial integrity, and we have no plans to alter our automated process, which works very well in gathering information and delivering highly relevant results," said spokesperson Cindy McCaffrey.

That stance gives Google a PR advantage over its rivals, but Google can also afford to do this in part because the company has an alternative stream of revenue, paid placements that it sells on its site. In contrast, competitor Inktomi doesn't operate its own search engine. Instead, it powers portals that are looking to make money, and paid inclusion allows Inktomi the ability to keep them happy with its services by sharing revenue.

To further convince webmasters of the value of paid inclusion, Inktomi plans to roll out new reporting tools later this month to its paid inclusion customers. These will show you exactly how many clicks Inktomi listings brought to your web site, the search terms associated with those clicks and in the future, perhaps even the exact search engines that generated the traffic. FAST says its program will offer similar reporting.

Such reporting is going to be a powerful weapon to convince webmasters that the money they've spent on paid inclusion is worthwhile. In fact, some webmasters would pay just to have traffic reporting itself, independent of being promised that any particular page would be included in an index.

LookSmart's Paid Placement Transformation

Like Inktomi, LookSmart has also faced pressure from advertisers wanting to see the value in paid inclusion. LookSmart's response was to roll out a new version of its program last month that includes a paid placement option.

LookSmart is a human-powered directory, rather than a crawler. In its former "Subsites" paid inclusion program, LookSmart would have its editors do a deeper categorization of a web site than just listing its home page. As a result, people might find the site for products and services where it previously could fail to appear. This benefits the site owner, and it can also benefit the searcher.

Site owners pay a fee to LookSmart for each person who clicks through on their paid inclusion listings, but the old program didn't guaranteed that those listings would be top ranked for any particular terms. The new LookListings program changes this, putting site owners who choose at the top of LookSmart's results, in a special "Featured Sites" area. This area only appears in at places that rebrand LookSmart's results, such as ISPs like Prodigy, as well as on the LookSmart site itself.

Despite this change, it is still possible that you may find sites at LookSmart that are not paid inclusion partners. Plenty of sites that use LookSmart's less expensive paid submission programs (which charge a one-time fee) do get listed and ranked well. There are even non-profit sites that wouldn't have paid anything to be listed which come up, such as the official Red Cross site on a search for "red cross."

The key thing to watch is whether a taste of paid placement revenues will leave LookSmart wanting more. If so, the company could move to a more aggressive paid inclusion model. For example, LookSmart might decide that all of its paid inclusion listings would get a significant ranking boost, in order to increase the number of people who click on them (and thus increase ad revenue generated). To some effect, burying listings that don't provide ongoing revenue in this way could arguably turn LookSmart into a paid placement search engine.

Paid Placement Crawlers?

In a somewhat similar fashion, crawler-based search engines could decide that paid inclusion is no longer an option but instead a requirement for webmasters to get listed. It's a fear I often hear expressed by webmasters. Are the crawlers going to drop free listings, in order to promote their paid inclusion programs?

For example, most web sites probably already have many pages listed in AltaVista. What incentive do they have to enroll in its new paid inclusion program, in particular when most sites probably don't need the fresher update that the program offers? None, really -- especially when you consider the cost of AltaVista's program, over $1,000 per year to get 25 pages listed.

It's possible that AltaVista could scale back its crawling, to push its paid inclusion program. The company says it has no plans to do this, nor has Inktomi taken similar steps to promote its own longer-running program. The most so far that has happened is that the free Add URL services that both AltaVista and Inktomi have operated have become less responsive. That's not really a loss, because the amount of spam submitted through free Add URL pages generally drowns out decent submissions. It's for this reason that Google has never depended much on its own Add URL page. Instead, ignoring Add URL and doing a better job on link crawling benefits webmasters more.

What Does The Future Hold?

Hopefully, we won't see a pull back on crawling or worse, a transformation of carrying only paid inclusion listings. If this happens, then your favorite search engine might suddenly become a Yellow Pages-style products and services guide. That's fine when you want products and services, but users also turn to search engines for other needs. Allowing listings to be dominated by advertiser entries doesn't always serve users best.

What will prevent this? Turning back the clock and eliminate paid inclusion programs isn't the solution. No one wants to return to a situation where submitting to a directory took months to get in or where large web sites could only get a single listing to cover their broad range of products and services. Similarly, paid inclusion has helped sites be assured that high quality pages that sometimes get missed by search engines can be represented. All of these changes benefit both webmasters and searchers.

Instead, the answer will probably be for the search engines to be more creative in how they develop new revenue sources. New services such as clickthrough reporting add value to a paid inclusion program, giving webmasters greater incentive to sign-up. In turn, that takes pressure off the search engines from feeling they need to scale back on indexing in order to force webmasters to pay for traffic.

Pay For Placement

Past articles about paid inclusion programs can be found here, including links to more information about the latest programs mentioned in the article above.

Amid tech slump, more portals sell search engine results to highest bidder
San Francisco Chronicle, June 18, 2001

Looks at the growth of paid placement and other paid participation programs, with lots of quotes from search engine representatives.