I've been a defender of the concept of paid inclusion in the past and still remain so. There is an advantage to having paid programs that allow site owners to expedite controlled entry into a search engine's database. For instance, getting the home page of a new site listed right away can benefit search engines, search engine users and site owners. Similarly, having the ability to feed search engines content they might typically be unable to access can also be helpful to all.
Unfortunately, paid inclusion can be a difficult concept for search engines to "sell" to site owners, especially for the crawler-based search engines. After all, aren't they supposed to be picking up pages across the web naturally? Why should a site owner pay to make this happen? You should do it, say the search engines on their pages about paid inclusion, because we don't get all the pages, we don't index them quickly, and we don't refresh them on a regular basis.
Such a message might convince site owners that they'd better sign-up for the various paid inclusion programs, but it also works against a completely different message that the search engines want potential portal partners and the general public to believe, that they have fresh, relevant and comprehensive listings of what's on the web.
Let's take a tour of what the crawler-based search engines are saying, to see the mixed message that paid inclusion is producing, starting with the pioneer of paid inclusion, Inktomi.
Inktomi pitches its "Search Submit" program by saying that "traditional search engines" (which would include Inktomi):
"Refresh their indexes less than once a month. Crawl only a portion of your site (usually between 30 to 50 percent) May not fully leverage your metadata."
On the other hand, Inktomi will tell potential partners that it provides:
"Highly relevant search results....Large, regionally specific databases chosen from a master database of two billion URLs....Rapid content refresh cycles."
So which is it? An out of date index that gets only parts of web sites or quality search?
At FAST, we're told that we should consider the paid inclusion program resold by its partners because:
"Typical search engines take as long as 45 days to update their listings. And most listings include only the home page and maybe a couple of additional pages....With FAST PartnerSite, all of your site's key public pages are available."
Yet on the page aimed at potential search partners, FAST says:
"We are committed to providing our users with the largest and freshest catalog on the Net."
So FAST has the largest and freshest catalog on the one hand, while telling site owners that most pages from their sites will be ignored or not refreshed often.
AltaVista explains that its paid inclusion program helps "ends the need to continually submit URLs to AltaVista."
But to potential users of the service, AltaVista says, "AltaVista offers the world's fastest, most comprehensive search service."
So why is the most comprehensive service requiring site owners to continually submit their URLs?
Even new player Teoma has inconsistencies. Here's the Teoma paid inclusion pitch:
"With Site Submit, we guarantee your most important web pages will be included in the database powering Ask.com and Teoma.com. As a member of the Site Submit program, we'll make sure that your latest, best content is always available to our users."
Teoma is the same service that told the press, when it emerged from beta earlier this year:
"Now with Teoma's search methodology, users can feel more confident in the accuracy and relevance of their Web searches."
Shouldn't an accurate and relevant search service already have a site owner's "latest, best content?"
What About Google?
Google alone among the major crawlers does not operate a paid inclusion program. The company has repeatedly said that it thinks such programs are unfair and would alter the quality of search results.
Given the mixed messages above, Google's been very wise in taking this stance. The company can say it offers high quality, comprehensive and fresh search without fear that marketing material from its paid inclusion program will be used to dispute this, as done in the examples above.
Despite this, Google's stand does lead to a different conflict. One of Google's businesses is making company web sites searchable. For instance, if you go to Cisco, you can search through all of Cisco's content using Google.
Google should have as complete as record as one can get of the Cisco site, as well as other sites where it provides site search services. However, this complete record is not made available to those searching the entire web at the Google site itself. Google prevents this, so as not to be accused of running a form of paid inclusion.
This means that Google, whose "mission is to deliver the best search experience on the Internet by making the world's information universally accessible and useful," finds itself in the odd situation of making some of the world's information inaccessible on its own site.
Clean Up The Message
Those who understand how crawlers work know that they do include many of the best pages from across the web. If you create good content, that content will tend to get naturally be included for free.
This doesn't take away from the advantages paid inclusion offers. If you have a product catalog locked inside of a database, paid inclusion offers you the ability to get your content into the crawlers, which typically forgo this type of material.
In addition, you might have pages that you believe are important but which the crawlers haven't picked up, perhaps because their link analysis systems don't deem those pages as important. Paid inclusion allows site owners the ability to get these additional pages included.
Those offering paid inclusion don't need to sell their programs by sending out conflicting messages about what they do. Explain the value without playing on fears. It's more honest with potential customers and less troublesome to potential searchers and search partners.
Pay For Placement?
Past articles on paid inclusion can be found in the Paid Inclusion section of this page.
Inktomi Paid Inclusion Pitch
Inktomi Partner Pitch
FAST Paid Inclusion Pitch
FAST Partners Pitch
AltaVista Paid Inclusion Pitch
AltaVista Users Pitch
Teoma Paid Inclusion Pitch
Teoma Launch Release
Google Company Overview