Inktomi's Free Add URL Penalty

A longer, more detailed version of this article is available to
Search Engine Watch "site subscribers."
Click here to learn more about becoming a site subscriber.

Just after the Search Engine Strategies conference in Dallas last November, a disturbing bit of information popped up at the Webmaster World forums. Search engine optimization consultant Greg Boser, of, shared news that an Inktomi representative had told him at the conference that pages submitted using its free Add URL system were given a ranking penalty.

Boser's news caused debate and disbelief to break out. How could Inktomi do such a thing? Was it all part of a conspiracy to drive people into Inktomi's paid inclusion program? Was this why some people found that the pages submitted via free Add URL suddenly ranked better when instead submitted through Inktomi's paid inclusion system?

Inktomi is indeed penalizing pages submitted via the free Add URL system, the company says. "The free Add URL is very much a magnet for spam and low quality pages, so we do intentionally give those pages a lower ranking," said Michael Palmer, chief technical officer of Inktomi's search services division. The change was made in the middle of last year, Inktomi says.

The confession is disturbing, but it is not as bad as you might think. To understand why, you need to realize that Inktomi is running three separate crawling systems, and your page may be treated differently depending on which one finds it.

First and foremost, Inktomi -- like all other major crawlers -- locates pages across the web independently from Add URL submissions. Instead, its spider follows links. If your page has a link pointing to it, then Inktomi might find the page and include it -- even if you never submitted to the search engine.

Inktomi calls the system above its "main" crawler, but I think "link crawler" is a better name to distinguish it from Inktomi's other crawler components. In addition to the link crawler, Inktomi also has what we can call its "Add URL crawler." Anyone who uses the free Add URL page of a major Inktomi partner, such as HotBot's, is sending their page (in most cases) to Inktomi's Add URL crawler.

Why have both a link crawler and an Add URL crawler? Back in the early days of the web, new pages were constantly being posted online. Add URL pages allowed crawler-based search engines to more quickly locate these new pages for possible inclusion into their listings. There are also some pages that have no links pointing at them. In these cases, crawlers will only find the pages if they are directly submitted.

Unfortunately, the value of using Add URL pages has dwindled for site owners. It used to be that there was a strong correspondence between what you submitted using Add URL and what actually got in. Those days are mostly gone. Now, all major crawlers are far more likely to rely on link crawling than Add URL submissions. Spamming is the reason for this change. The vast majority of submissions to Add URL systems are junk pages not worth listing.

"Since we turned free Add URL on, the quality of the pages that get submitted is relatively low compared to the main crawler," said Palmer.

Changing Add URLs pages into "Suggest URL" pages is hard to complain about, because slavishly adding all new submissions would actually harm search engine listings by flooding them with spam. However, penalizing pages submitted through Add URL, as Inktomi is doing, is a major concern. It is most disturbing because it goes against exactly what Inktomi advises site owners to do, through its partners. For example, HotBot's help pages advise that up to 50 pages per day can be submitted for "immediate addition" and make no mention that submitting would cause pages to be downgraded.

Inktomi also failed to make any mention of free Add URL penalties, when it rolled out its paid inclusion program last November. We were told that free Add URL remained an option for those who did not want to pay. Inktomi advised that the only negative to using free Add URL was that pages submitted through it were not guaranteed to appear in its index. It did not add that they would also be subject to a ranking penalty. This lack of disclosure has caused confusion in the search engine optimization world.

Another concern about penalties is that anyone can submit another person's URL to Inktomi. Potentially, that means that nothing would stop a competitor from submitting your pages and getting them penalized. However, here's where we can move back from the brink and discover that the Add URL penalty isn't as bad as it sounds.

As it turns out, if a page submitted via free Add URL is later found by Inktomi's link crawler, then the penalty is automatically removed. This is because Inktomi has more faith that the page is less likely to be spam, so feels it doesn't need to penalize it from the start.

"If someone submits a page into the free Add URL, and if the main crawler also finds that page, then the normal scoring mechanism takes precedence," said Palmer.

So, if you submit a brand new page, that page starts out at a disadvantage. However, if Inktomi's link crawler discovers it, the disadvantage is removed. Moreover, once the link crawler finds it, then any competitors resubmitting the page shouldn't cause it to drop in rank.

Remember I said there were three crawling systems? The last one is Inktomi paid inclusion crawler. Those using the Inktomi paid inclusion system have their URLs visited by this. The paid pages are not boosted in rank, but neither are they penalized, as with the free Add URL pages. In other words, they are treated exactly the same as if the link crawler found them.

"I think partly because we do charge a little bit for it, people don't submit millions of fake pages," said Palmer. "So, we give those pages normal scoring, just like the main crawler."

No doubt you can expect to see Inktomi eventually kill its free Add URL system. The company says the only reason it hasn't yet done so is to figure out how to establish a new system that only non-profit sites can use. "Before we turn the free one off, we want to solve that, let them request a listing," said Palmer.


Pay For Placement

Past articles about Inktomi's paid inclusion system can be found here.

Ink Plans On Ending Free Submission
Webmaster World, Nov. 12, 2000

Thread from Webmaster World forums where news of the Add URL penalty emerged.

HotBot Add URL page

Some of HotBot's results come from Inktomi, so you can use its free Add URL form to tell Inktomi about your pages -- and that means those that are accepted should eventually show up in the Inktomi-powered portions of MSN Search and at other Inktomi-powered search engines.

Inktomi Paid Inclusion Program via Position Tech

Signups for Inktomi's paid inclusion program are a taken here.

Inktomi Paid Inclusion Program via WebGravity

Inktomi named WebGravity its new European partner for its paid inclusion program, and orders can be placed here.

A longer, more detailed version of this article is available to
Search Engine Watch "site subscribers."
Click here to learn more about becoming a site subscriber.