A longer, more detailed version of this article is available to
Previously, I've written generally about the many new ways in which search engines are trying to earn money from their search results. This month, we'll take a closer look at one particular method, "paid inclusion," which is now in place at Ask Jeeves, LookSmart and Inktomi.
In pay for inclusion, site owners pay money to guarantee that they will be included in a search engine's listings in greater depth than might ordinarily occur. Paid inclusion does not guarantee that pages will be ranked well for particular search terms. However, sites enrolling in paid inclusion programs are likely to receive greater traffic than those that don't.
To understand this, let's liken search engines to a lottery. When someone searches, it's almost as if a search engine spins a big barrel full of millions of listings, to determine which listings will come up first in its results. In a lottery, the more tickets you have, the more likely it is you'll win something. Similarly, with search engines, the more listings you have, the more likely you'll rank well for various searches.
I can't stress enough that paid inclusion is not equivalent to paid placement programs that guarantee positions. For example, if you want to be number one for a particular search term at GoTo.com, which is a paid placement search engine, you simply agree to pay more money than any other advertiser for the term. In contrast, advertisers are not guaranteed a particular position in paid inclusion systems. They are simply sold more tickets in the search engine lottery, so to speak. That means they may win more often in the ranking game than might ordinarily happen, but they would still need to satisfy all the normal editorial criteria to do so.
Paid Inclusion At Ask Jeeves
Ask Jeeves has experimented with paid inclusion over the past several months, and now it's a standard part of its advertising offerings called "Answer Link." It works two ways. The editorial staff, after creating listings in its usual fashion, may then suggest to the advertising staff that a particular site might be a good prospect for a paid inclusion deal. The advertising staff would then follow up to see if a deal could be signed. Alternatively, an advertiser might approach Ask Jeeves about participating in the program. The editorial staff would then review the site, suggest some types of questions it might be useful in answering, and the advertising side would then complete the deal.
In either case, pages from the partner's site eventually appear as answers to the Ask Jeeves questions that appear at the top of its search results page, in the "I have found answers to the following questions" section. However, the partner can't control exactly what questions it will appear for, Ask Jeeves says. Instead, paid inclusion links will only appear if the search engine's normal ranking systems deem it relevant. In return, Ask Jeeves is paid based on the amount traffic it sends to the partner.
Ask Jeeves stresses that only sites that provide quality information will appear in its results, whether there is paid partnership or not. It also points out that only about 5 percent of its knowledgebase is made up of paid inclusion answers, nor does it plan to greatly increase this percentage. But doesn't favoring some sites in response to money penalize other good ones? Yes, Ask Jeeves responds -- but that doesn't mean the users themselves are penalized.
"For example, say we point to a site that lists stock quotes. A stock quote is a basic piece of information, so if one site has more in depth stock information and quality financial content than another, then we are not opposed to approaching that site to work with us in a paid partnership," said Jonathan Silverman, product manager of Ask.com.
To see paid inclusion in action, try a search for "what should my blood pressure be?" The top link that appears leads to a page from OnHealth, an Ask Jeeves advertiser and one of about 20 "basic knowledge" providers that also include companies such as Ticketmaster-CitySearch, Verizon, GE Financial, ImproveNet, and AllBusiness. Ask Jeeves also has several hundred advertisers for ecommerce topics, such as Sears, Best Buy, Land's End and Garden.com.
The listing has no disclosure that an advertiser is benefiting from it. Ask Jeeves says during recent design testing, flagging paid inclusion links wasn't seen as useful. The company also feels that since such paid inclusion listings will only appear if they meet editorial standards, there's no need to call them out from non-paid listings. However, Ask Jeeves has added a link to its home page called "Editorial Guidelines for Answers" that explains how some answers come from paid partners and that these also meet editorial and ecommerce guidelines.
This is good, but I still think it would help if there was some type of symbol integrated alongside the listings themselves, just to better advise those users who are sensitive about paid programs. Nor should this be exclusive to Ask Jeeves. The entire search engine industry ought to be considering some standard way of labeling such material, so consumers of information have a clear idea of what relationships may be involved.
Paid Inclusion At LookSmart
Over at LookSmart, much attention has been focused recently on the relatively new change where all commercial web sites must pay a fee in order to be considered for listing in the directory. However, LookSmart's "Subsite" paid inclusion program goes far beyond this.
Typically, most web sites might find their home page listed in one or two categories at LookSmart. Large web sites might have further classification, with a few key inside sections listed in appropriate categories. Under Subsites, LookSmart editors do a deep review of a web site, categorizing individual pages they find with suitable content throughout the directory. Ultimately, a site could end up with over a hundred different listings, if not more. The result is that the site will appear in response to a far greater range of queries than if only its home page was listed.
Paid inclusion at LookSmart has similarities to the system at Ask Jeeves. At the request of LookSmart's advertising department, LookSmart editors will review a site and determine appropriate places they feel the site's content could be listed. Only content meeting regular editorial standards is said to be included, and listings aren't guaranteed to appear in response to a particular search. This is especially true for LookSmart, given that it can't control how its many partners rank the information it provides. In return for this work, LookSmart receives a per click fee for each visitor it sends to sites in the program.
While there are concerns that users might miss out finding sites that don't pay, LookSmart notes that its editors spend a significant amount of time searching the web for new sites to add to the directory, independent of its paid inclusion and submission systems. Additionally, the company says its recent acquisition of the Zeal.com community directory is also expected to help it ensure broad representation of valuable not-for-profit and community sites.
LookSmart isn't restrictive in offering the Subsite program to one company in a particular business. Anyone who wishes to pay can be deeply indexed, regardless of whether their competitor is already in the program. "We're wide open so long, as the links meet our editorial standards," said Scott Stanford, LookSmart's vice president of listing services and ecommerce. "We have not signed any deals or exclusives."
The Subsite program was publicly launched in September, with mySimon named as the first advertiser. However, the program has been in beta testing since April of this year, and now approximately 20 large sites are represented, including eBay, through a deal just signed this week.
Try a search for "downloadable software" at LookSmart, and you'll see an example of paid inclusion link from mySimon that appears at the top of the "Reviewed Web Sites" section. You'll also see that, as with Ask Jeeves, there's no disclosure -- nor do LookSmart's partners MSN Search, iWon or AltaVista make any type of disclosure next to the LookSmart Subsite listings they display.
Paid Inclusion At Inktomi
Inktomi's paid inclusion program has only recently gone live through a partnership announced in September with MediaDNA and another announced just this week with Position Technologies, so expect to see it evolve as the program matures and as new partners are announced. FYI, Inktomi's first partnership, announced in July with Network Solutions, isn't expected to go live until January.
At its core, paid inclusion with Inktomi means that site owners pay to be guaranteed that the web pages they select are included in its crawler-based listings and that these pages will be reindexed every 48 hours. See the "Inktomi Debuts Self-Serve Paid Inclusion" article below for more specifics about the service.
That's the end of the guarantees. As with the other paid inclusion programs described, there is no assurance that pages will appear highly ranked for any particular search.
Given that Inktomi crawls the web, its pay for inclusion model is potentially more worrisome to searchers than the ones run by Ask Jeeves and LookSmart. Human-powered directories, by their very nature, have never been inclusive of everything on the web. That's why major search sites using directory information typically back this up with crawler-based results. If the human editors haven't categorized something, then the crawler provides a fall-through.
Because of this, a crawler is automatically expected to be inclusive of everything. Indeed, the reason stories about search engine sizes have continued to attract so much press is that the general public may naturally assume that a crawler will find everything on the web. That's never been, nor dare I say, never will be the case. Nevertheless, until the Inktomi announcement, we've also never had a major crawler say that some sites might be more deeply crawled for reasons other than feeling there was essential content that should be listed.
So what's going to happen as the Inktomi program progresses? Will there be a general degradation of freshness and crawl depth, in order to make the paid inclusion model more attractive to site owners? Definitely not, Inktomi says. Paid inclusion is mainly a way it sees for site owners to share the cost of getting people to their content, plus it makes it possible for Inktomi to list new content from hard to crawl sites that it's never carried before. Rather than a replacement for its regular crawling, paid inclusion is seen as an additional, supplementary system.
"We're going to continue crawling the web much as we have, using the same kind of popularity analysis to build the bulk of our index," said Troy Toman, general manager of Inktomi's search solutions division. "We're not on a path where we'll say were going to remove every site in our index unless they pay. It's really to go more after sites that would wish to be better represented in our index or people who want more timely information from their site made available."
Among the other crawlers, Go says it is readying a paid inclusion system with its own spidered results that may be unveiled in December. Additionally, AltaVista is still determining what services it intends to market to webmasters. Paid inclusion could be one of these, though AltaVista is sending out signals that its first product may allow site owners to enhance their listings with highlighting or even pictures. Notably, Google says it has no such plans for a paid inclusion system, at the moment.
"I think there are some significant philosophical issues," said Google president Sergey Brin. "If someone searches for cancer, and there's a really good cancer site out there, what if you don't have the answer they are looking for because that particular site didn't pay to be in there."
There are certainly some pluses that paid inclusion can offer. It's not unreasonable to expect that extremely large sites might help pick up the costs of making their content available, especially if that in turn helps the searching public in general. Such programs can also make content that's currently unreachable to spiders, such as locked in databases or behind firewalls, easily accessible for the first time. Certainly such programs offer the opportunity for companies to interact with crawlers on a more formal basis, rather than unproductively kept at arms length.
Nevertheless, while paid inclusion programs may impact the editorial quality of search results far less than paid placement, paid inclusion still raises concerns -- whether run through crawler-based search engines like Inktomi or human-powered services such as Ask Jeeves and LookSmart. The test will see whether these pioneers can prove over time that their programs don't hurt the search experience -- or better, actually do improve it.
Ask Jeeves Editorial Guidelines
This page, linked from the Ask Jeeves home page, explains to users how paid partnerships may have a role in the answers Ask Jeeves provides.
LookSmart Earnings Looking Strong
smallcapcenter.com, Nov. 2, 2000
LookSmart's paid submission and paid inclusion programs are a big reason behind its better than expected earnings, which were recently announced. Listing revenues have gone from $600,000 in the first quarter, to $1.6 million in the second, then to $3.3 million in the third quarter just announced. Last year, they were non-existent.
Inktomi Debuts Self-Serve Paid Inclusion
The Search Engine Report, Nov. 3, 2000
More about Inktomi's paid inclusion program and partners can be found here.
Monetizing The Search
The Search Engine Report Sept. 4, 2000
Goes into a range of other programs search engines are experimenting with to earn money and touches on issues relating to pay for inclusion.
Pay For Placement?
Paid inclusion and even paid placement can solve some problems that have plagued search engines. Articles on this page go into more depth about the issues, pro and con. A new chart has also been added as a guide to where significant non-banner advertising components are showing up at different search engines.
A longer, more detailed version of this article is available to