Inktomi's Free Add URL Penalty

Just after the Search Engine Strategies conference in Dallas last November, a disturbing bit of information popped up at the Webmaster World forums. Search engine optimization consultant Greg Boser, of, shared news that an Inktomi representative had told him at the conference that pages submitted using its free Add URL system were given a ranking penalty.

Boser's news caused debate and disbelief to break out. How could Inktomi do such a thing? Was it all part of a conspiracy to drive people into Inktomi's paid inclusion program? Was this why some people found that the pages submitted via free Add URL suddenly ranked better when instead submitted through Inktomi's paid inclusion system?

Inktomi is indeed penalizing pages submitted via the free Add URL system, the company says. "The free Add URL is very much a magnet for spam and low quality pages, so we do intentionally give those pages a lower ranking," said Michael Palmer, chief technical officer of Inktomi's search services division.

The confession is disturbing, but it is not as bad as you might think. To understand why, you need to realize that Inktomi is running three separate crawling systems, and your page may be treated differently depending on which one finds it.

First and foremost, Inktomi -- like all other major crawlers -- locates pages across the web independently from Add URL submissions. Instead, its spider follow links. If your page has a link pointing to it, then Inktomi might find the page and include it even if you never submitted to the search engine.

Inktomi calls the system above its "main" crawler, but I think "link crawler" is a better name to distinguish it from Inktomi's other crawler components. In addition to the link crawler, Inktomi also has what we can call its "Add URL crawler." Anyone who uses the free Add URL page of a major Inktomi partner, such as HotBot's, is sending their page (in most cases) to Inktomi's Add URL crawler.

Why have both a link crawler and an Add URL crawler? Back in the early days of the web, new pages were constantly being posted online. Add URL pages allowed crawler-based search engines to more quickly locate these new pages for possible inclusion into their listings. There are also some pages that have no links pointing at them. In these cases, crawlers will only find the pages if they are directly submitted.

Unfortunately, the value of using Add URL pages has dwindled for site owners. It used to be that there was a strong correspondence between what you submitted using Add URL and what actually got in. Those days are mostly gone. Now, the major crawlers are far more likely to rely on link crawling than Add URL submissions. Spamming is the reason for this change. The vast majority of submissions to Add URL systems are junk pages not worth listing.

"Since we turned free Add URL on, the quality of the pages that get submitted is relatively low compared to the main crawler," said Palmer.

The same is true for every other major service you talk to. For instance, AltaVista is often quoted for saying at a past Search Engine Strategies conference that 95 percent of the submissions it receives are spam. Of course, you need to understand that this does not mean 95 percent of those using the Add URL page are spammers. Not at all. What happens is that a relatively small number of people submit a very large number of spam pages. Filtering out the relatively small amount of good from large volume of bad is a huge challenge.

Because of this, we have the growing disconnect between what's submitted and what gets crawled. Excite is a great example here. For well over two years, submissions to its Add URL system have been considered more as " suggestions" rather than URLs that actively get included. Instead, Excite's crawler is far more likely to visit a site and select pages it wants to list independently of the Add URL suggestions.

Excite's not alone. FAST, Google and Go are other major crawler that operate in a similar fashion. Indeed, of the major crawlers, only Inktomi and AltaVista have continued to keep a close connection to what's submitted and what gets listed -- and recent complaints about AltaVista sluggishness suggest that its Add URL system is the next to go into "suggestion" mode.

Changing Add URLs pages into "Suggest URL" pages is hard to complain about, because slavishly adding all new submissions would actually harm search engine listings by flooding them with spam. However, penalizing pages submitted through Add URL, as Inktomi is doing, is a major concern. It is most disturbing because it goes against exactly what Inktomi advises site owners to do, through its partners. For example, HotBot's help pages advise that up to 50 pages per day can be submitted for "immediate addition" and make no mention that submitting would cause pages to be downgraded.

Inktomi also failed to make any mention of free Add URL penalties, when it rolled out its paid inclusion program last November. We were told that free Add URL remained an option for those who did not want to pay. Inktomi advised that the only negative to using free Add URL was that pages submitted through it were not guaranteed to appear in its index. It did not add that they would also be subject to a ranking penalty. This lack of disclosure has caused confusion in the search engine optimization world, as will be explained further below.

Another concern about penalties is that anyone can submit another person's URL to Inktomi. Potentially, that means that nothing would stop a competitor from submitting your pages and getting them penalized. However, here's where we can move back from the brink and discover that the Add URL penalty isn't as bad as it sounds.

As it turns out, if a page submitted via free Add URL is later found by Inktomi's link crawler, then the penalty is automatically removed. This is because Inktomi has more faith that the page is less likely to be spam, so feels it doesn't need to penalize it from the start.

"If someone submits a page into the free Add URL, and if the main crawler also finds that page, then the normal scoring mechanism takes precedence," said Palmer.

So, if you submit a brand new page, that page starts out at a disadvantage. However, if Inktomi's link crawler discovers it, the disadvantage is removed. Moreover, once the link crawler finds it, then any competitors resubmitting the page shouldn't cause it to drop in rank.

Remember I said there were three crawling systems? The last one is Inktomi paid inclusion crawler. Those using the Inktomi paid inclusion system have their URLs visited by this. The paid pages are not boosted in rank, but neither are they penalized, as with the free Add URL pages. In other words, they are treated exactly the same as if the link crawler found them.

"I think partly because we do charge a little bit for it, people don't submit millions of fake pages," said Palmer. "So, we give those pages normal scoring, just like the main crawler."

Now let's look at how all three crawling systems may cause different people to have completely different experiences, when submitting to Inktomi.

Person 1 is a search engine optimizer who creates highly targeted doorway pages. These pages have never ranked well when submitted using free Add URL, but when submitted using the paid inclusion system, they suddenly perform great. Conclusion? Paid inclusion equals ranking boost. Reality? What has happened is that these pages were never visited by the link crawler, so the page penalty imposed by free Add URL kept them down. But, when submitted through the paid inclusion system, the penalty is dropped, and the pages performed better.

Person 2 runs a small site with great content about a particular subject. They submit using the free Add URL system, find their pages were picked up and indeed rank well. Conclusion? Free Add URL works fine, and all those people complaining about it don't know what they are talking about. Reality? Maybe free Add URL did get them in, and the content of their pages and other important factors were good enough to overcome the submission penalty. Or, perhaps the link crawler also found their pages before free Add URL added it. Given this, they had no penalty imposed. That, and other factors, may have helped their pages rank well.

Person 3 runs a large corporate site and submits 10 pages through Inktomi's paid system. They find no noticeable rank improvements. Conclusion? Paid inclusion has no relationship to rank boosting. Reality? The pages may already be listed by the link crawler, so being picked up by the paid system would have no impact, as there was no submission penalty to remove.

Person 4 also runs a large corporate site. They submit 200 pages and find traffic definitely increases. Conclusion? Paid inclusion is worth the money. Reality? Chances are, some pages were never listed by Inktomi, and having greater representation generated more ancillary traffic. Particular ranks weren't improved, but the traffic still does better overall.

Four different people, four different reactions. As you can see, just going by another person's comments doesn't necessarily mean you'll have the same results. Also, remember that beyond submission system, other factors such as the content of your pages, link analysis and clickthrough come into play, to determine how your page will rank.

By the way, penalties have been assigned to Add URL submissions for some time, according to Inktomi. "This has been our policy for some time," Palmer said, explaining it wasn't done in conjunction with the release of the paid inclusion system. I'm following up to get a more exact date on when the change was made.

So, how to proceed with Inktomi? If your pages aren't listed at all, submitting via the free Add URL page is better than nothing, Inktomi advises. Your submissions may get included for a short period, as Inktomi likes to "trial" new pages to see if they overcome the submission penalty and still manage to rank well. If so, they may be retained. Expect it to take up to 60 days for your pages to get listed when using free Add URL, and remember that being listed is not guaranteed.

Even better is to build links to your site from other sites. If Inktomi's crawler encounters links to you from a variety of sites, you are more likely to get visited. Also ensure you do a good job of linking between your own pages. That ensures that Inktomi's crawler can find your other pages from however it initially enters your site. A site map can be helpful here. It is simply a list of all the pages in your site. If a crawler comes to it, then it easily learns of other pages it might wish to visit. Your human visitors may appreciate a site map, also.

If you're the type going the doorway page route, remember that these pages typically don't have links pointing at them. Thus, the crawler has no chance of finding them unless you submit using free Add URL or create some links to these pages. Having a "hallway" page is one concept doorway page users are advised to follow. This is simply a list of all the doorway pages in your site. You submit it to the crawler, then the crawler -- if it reads the hallway page -- MAY decide to index some of the doorway pages on the list.

Hallway pages are essentially site maps of just your doorway pages. You wouldn't put a hallway page out for your human visitors to see, because you don't want to send them back to the various "entrances" to your site.

Ultimately, your fastest and only guaranteed route is to use paid submission. If you must have a page listed, then pay the money, and you'll be in within two days.

No doubt you can expect to see Inktomi eventually kill its free Add URL system. The company says the only reason it hasn't yet done so is to figure out how to establish a new system that only non-profit sites can use. "Before we turn the free one off, we want to solve that, let them request a listing," said Palmer.


Inktomi Debuts Self-Serve Paid Inclusion
The Search Engine Update, Nov. 3, 2000

More details about Inktomi's paid inclusion system.

Paid Inclusion At Search Engines Gains Ground
The Search Engine Update, Nov. 3, 2000

Paid inclusion doesn't guarantee particular rankings, but greater representation can increase ancillary traffic, by letting you win the "search engine lottery" more often.

HotBot Add URL page

Some of HotBot's results come from Inktomi, so you can use its free Add URL form to tell Inktomi about your pages -- and that means those that are accepted should eventually show up in the Inktomi-powered portions of MSN Search and at other Inktomi-powered search engines. There's a 50 page per day, per site limit -- and I'm also reconfirming that these guidelines haven't changed.

Ink Plans On Ending Free Submission
Webmaster World, Nov. 12, 2000

Thread from Webmaster World forums where news of the Add URL penalty emerged.

Search Engine Talk

Other places like Webmaster World where search engines are discussed.