Inktomi Debuts Self-Serve Paid Inclusion

In a landmark move, Inktomi announced a new partnership with Position Technologies this week that allows site owners to pay for guaranteed inclusion in the Inktomi index. While paid submission (or "pay for consideration") systems have operated at human-powered Yahoo and LookSmart for some time, we've never had a major crawler-based service offer something similar. In addition, paid inclusion means that pages will absolutely be guaranteed to be listed, while in paid submission systems, no such assurances are offered.

"The biggest challenge web site owners face is getting found on the web," said Troy Toman, general manager of Inktomi's search solutions division. "Inktomi Search/Submit is the first time we're allowing Web site owners to publish directly into the Inktomi index, helping them get found faster and stay up-to-date in the database."

The program promises to add any URL submitted to the Inktomi index within 2 days and keep revisiting the URL at that frequency for up to a year. In return, site owners pay on a sliding scale: US $20 for the first URL, $10 for the next 2 to 100 URLs and $6 for any URLs over 100. For a 150 page web site, the costs work out like this:

Page 1: $20
Pages 2-100: $990 (99 pages @ $10 each)
Pages 101-150: $300 (50 pages @ $6 each)
Total: $1,310 per year

The pricing is more expensive than I would have expected. A few dollars per URL, sliding down into less than a dollar for large orders, feels fairer. It's one thing to ask web site owners to help share in the cost of indexing the web, but this is a huge jump for even small sites of 50 pages. In addition, the pricing feels high in contrast to fees charged by Yahoo and LookSmart. At those services, $199 essentially purchases you a listing (if approved) for life, or at least several years. Inktomi charges on an annual basis. True, it has to keep revisiting the pages, but as a crawler-based service, that's something it was always supposed to do. However, the pricing could also change.

"The paid submission program was designed for Web site owners to submit the most relevant pages for entry to their site, not all of their content," said Toman. "This pricing structure is an introductory offer with the new service and we will be adjusting it over the coming months to determine the most equitable price point."

Inktomi says that the new paid inclusion system won't alter the normal crawling that it does. Pages from across the web will continue to be listed for free, especially those that Inktomi deems to be popular or which seem to satisfy queries.

Free Add URL pages that feed into Inktomi, such as the one offered by HotBot, are eventually to be phased out. Inktomi finds that the vast majority of submissions sent via these pages are spam. Other crawler-based services also report the same problem. A small number of people submit an incredibly large number of poor-quality URLs through these pages. By eliminating them and moving to a paid inclusion model, Inktomi (and others) could arguably do a better job of indexing the web.

Such a shutdown won't occur until Inktomi can establish a means for non-commercial sites to submit to the search engine. "We want to make sure we have adequate non-fee submission mechanisms for them," said Troy Toman, general manager of Inktomi's search solutions division. "Our goal is not to charge anybody and everybody. We want submissions from non-profits and non-commercial sites on the web, and we'll wait until those are covered."

Finally, while pages are guaranteed to be listed, there's no guarantee that they will rank well for particular searches. Inktomi will continue to apply its regular relevancy algorithms to paid inclusion pages, just as it would to those it lists for free.

To help you further understand the new program, I thought some Q&A style answers would be helpful. After them, you'll find links to more information about the Inktomi program.

Q. Why would I pay Inktomi to include my pages?

People will have various reasons. Some realize that that more pages they have listed, the more likely they'll generate traffic. If you can earn money off that traffic, then paying for inclusion may be worthwhile.

Others may wish to ensure that Inktomi has the freshest information from their web sites. While Inktomi does try to spider rapidly changing sites more frequently than others, the company still gets complaints from webmasters about stale listings. Inktomi says paid inclusion is the only way it will be able to meet the demand for greater freshness.

"It's one thing to have the technology that knows how to do that [detect changes” and its another thing to build that out for the entire web," said Troy Toman, general manager of Inktomi's search solutions division. "This mechanism allows us to be able to afford to get fresher content."

Q. Why do some of my pages disappear from Inktomi currently? Is this just to get me to sign up for paid inclusion?

Inktomi has operated a system of "trialing" new submissions for over a year. Basically, that means a new page that's picked up may stay in the index for a short time, such as three weeks. By using clickthrough measurements, Inktomi can tell if that page is satisfying queries. If no clicks are detected for a particular listing, then the listing is never being selected by users. In that case, it may be dropped to make room for other pages.

In addition to this, sometimes a spider may have problems accessing a web site. Perhaps there is network traffic. Perhaps the site is down temporarily. Perhaps the spider simply has an internal problem. In any case, it can cause a page to be dropped until the spider can return to visit it. That can take up to a month. With paid inclusion, you are paying to avoid fallout from such problems:

"There are many factors that can exclude a site from being picked up by the spider during a crawl," said Toman. "The paid inclusion program is another tool that web site owners can use to insulate their content against missing a crawl. Web site owners that subscribe to the new service can be confident that their pages will be refreshed every 48 hours."

Q. Inktomi already lists some of my pages. How can I arrange to pay them to add only the ones they don't already list?

You have complete control to submit any URLs you want. The only problem is that there's no way to know which ones Inktomi already feels should naturally be included in its index, even if you don't pay. Given this, you may indeed find you are paying for pages Inktomi might normally list for free.

If you have pages you know generate traffic, then you may want to use the system to guarantee they aren't dropped from the index. Even though you think Inktomi might keep them, it might be worth spending the money to have the reassurance. Position checking tools or log analysis software can help you determine which pages you may wish to protect in this way.

Conversely, you might choose to submit only pages that you've never seen appear in the Inktomi index, trusting that your other pages already listed may be retained.

Q. I have no money and must depend on the free Add URL pages that feed into Inktomi. What's my best strategy?

Inktomi says the best thing to do is to submit to HotBot. You can submit up to 50 pages per day there, and pages MAY appear within three weeks, once submitted. However, the free Add URL situation is very fluid -- it's likely to change in the near future, as Inktomi begins to make changes.

The long term plan is to eliminate free Add URL, except perhaps for non-profit and non-commercial sites. That doesn't mean commercial sites wouldn't get listed if they didn't submit, since Inktomi also crawls the web independently of Add URL. However, it does mean you will probably save yourself time and frustration by recognizing that paid inclusion is the future with Inktomi.

"If you have a commercial reason to need to be in the Inktomi index, it doesn't seem unreasonable that you should have to pay for it," said Toman.

Q. My web site has thousands of URLs. Isn't there a better price I can obtain?

Probably. Inktomi says the pricing it offers through Position Technologies is aimed at those with sites of 1,000 pages or less. For those with larger sites, Inktomi suggests contacting it directly about arranging better bulk pricing.

Q. Will Inktomi really guarantee to list anything I submit?

Inktomi says that it will reject pages it considers spam, but what is spam isn't defined on the paid inclusion signup pages. Typically, the concern is mostly about pages that attempt to mislead users about their content. The signup pages do warn that "illegal" URLs or those infringing copyright will not be accepted and itemizes a few other reasons for page removal. Inktomi will also remove pages for "fraudulent use of the service." What's that? Whatever Inktomi decides it wants it to be -- the company has sole discretion.

Q. Are cloaked pages OK?

As usual, this depends on how you explain cloaking. "We still consider cloaking to be spam," was Toman's initial response to this question, but then he soon qualified this. "Cloaking in and of itself is not necessarily spamming. If someone's putting it up because it helps us find the content, we have a tendency to let those lie. However, if you are doing something to the page that's designed to deceive the search engine, that's not OK."

Q. If pages are guaranteed to be listed, won't this make it easier to reverse engineer Inktomi, as was the case a few year ago with Go (Infoseek), when it had an Instant Add URL mechanism?

Perhaps. Inktomi's ranking system also makes use of off-the-page criteria that webmasters cannot control. Nevertheless, the ability to get listed, make changes and then see if the changes let you rise higher certainly make it easier for those trying to reverse engineer Inktomi's algorithms. Should this prove to be a problem, Inktomi says it may make further changes. "This is sort of the process of discovery when you go in to a new area," Toman said. One idea Inktomi is considering is to allow site owners to also provide meta data about their site, which presumably would be reviewed for accuracy.

Inktomi Paid Inclusion Program

Describes pricing and leads you to an order form

Inktomi Paid Inclusion FAQ

More Q&A directly from Inktomi.

Position Pro

Operated by Position Technologies, Inktomi's new paid inclusion partner, Position Pro has tools especially suited to help those large web sites optimize their pages and submit to crawler-based search engines. However, even small site owners may want to take a look. An updated review is planned for the next newsletter.


Another Inktomi paid inclusion partner, MediaDNA's eLuminator spiders web sites and creates optimized pages for search engines. The eLuminator product is especially designed for sites that have content locked behind password protected areas, as it can make that content visible to search engines without giving it away for free to unregistered users.

Pay For Inclusion Advances
The Search Engine Update, Nov. 3, 2000

Inktomi isn't the only major search service with a paid inclusion program. This article looks at programs run by Ask Jeeves and LookSmart, plus touches on some issues that paid inclusion raises.

Inktomi To Offer Paid Submit Option
The Search Engine Update, Aug. 2, 2000

Network Solutions was the first paid inclusion partner that Inktomi named, and this article provides more details. The Network Solutions offering isn't expected to go live until January.