The Search Engine Update, January 16, 2001, Number 93


About The Update

The Search Engine Update is a twice-monthly update of search engine news. It is available only to Search Engine Watch "site subscribers." Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.


In This Issue

+ Site & Conference News
+ Inktomi's Free Add URL Penalty
+ Interesting Search Engine Articles
+ List Info (Subscribing/Unsubscribing)


Site & Conference News

Hello Everyone--

I'm not quite done with the Search Engine Watch awards, but I do have a date: January 22. The results will be posted online then. Watch the What's New page for a link.

What's New

The next Search Engine Strategies conference is coming to London on February 15. I'll be presenting and moderating sessions at the conference that features experts on search engine marketing issues and panelists from the various major search engines themselves. Services confirmed to participate include AltaVista, Google, GoTo, Inktomi, Lycos Europe, MSN Search, Netscape/The Open Directory and Yahoo

There will be a special presentation on regional and language issues, of interest to those based in Europe or who need to be listed with European search engines. We will also be having roundtable sessions where European search engine marketing experts will answer questions about advanced issues. More information can be found via the URL below:

Search Engine Strategies 2001 - London

We now also have the dates for our first two day event, which features a day for searcher issues, in addition to the topics aimed at web marketers. It will be March 20 and 21, in Boston, at the Marriott Copley hotel. An agenda and more details for the session should be available later this month. To be notified when it is ready, just leave your email address at the form below:

Search Engine Conferences


Inktomi's Free Add URL Penalty

Just after the Search Engine Strategies conference in Dallas last November, a disturbing bit of information popped up at the Webmaster World forums. Search engine optimization consultant Greg Boser, of, shared news that an Inktomi representative had told him at the conference that pages submitted using its free Add URL system were given a ranking penalty.

Boser's news caused debate and disbelief to break out. How could Inktomi do such a thing? Was it all part of a conspiracy to drive people into Inktomi's paid inclusion program? Was this why some people found that the pages submitted via free Add URL suddenly ranked better when instead submitted through Inktomi's paid inclusion system?

Inktomi is indeed penalizing pages submitted via the free Add URL system, the company says. "The free Add URL is very much a magnet for spam and low quality pages, so we do intentionally give those pages a lower ranking," said Michael Palmer, chief technical officer of Inktomi's search services division.

The confession is disturbing, but it is not as bad as you might think. To understand why, you need to realize that Inktomi is running three separate crawling systems, and your page may be treated differently depending on which one finds it.

First and foremost, Inktomi -- like all other major crawlers -- locates pages across the web independently from Add URL submissions. Instead, its spider follow links. If your page has a link pointing to it, then Inktomi might find the page and include it even if you never submitted to the search engine.

Inktomi calls the system above its "main" crawler, but I think "link crawler" is a better name to distinguish it from Inktomi's other crawler components. In addition to the link crawler, Inktomi also has what we can call its "Add URL crawler." Anyone who uses the free Add URL page of a major Inktomi partner, such as HotBot's, is sending their page (in most cases) to Inktomi's Add URL crawler.

Why have both a link crawler and an Add URL crawler? Back in the early days of the web, new pages were constantly being posted online. Add URL pages allowed crawler-based search engines to more quickly locate these new pages for possible inclusion into their listings. There are also some pages that have no links pointing at them. In these cases, crawlers will only find the pages if they are directly submitted.

Unfortunately, the value of using Add URL pages has dwindled for site owners. It used to be that there was a strong correspondence between what you submitted using Add URL and what actually got in. Those days are mostly gone. Now, the major crawlers are far more likely to rely on link crawling than Add URL submissions. Spamming is the reason for this change. The vast majority of submissions to Add URL systems are junk pages not worth listing.

"Since we turned free Add URL on, the quality of the pages that get submitted is relatively low compared to the main crawler," said Palmer.

The same is true for every other major service you talk to. For instance, AltaVista is often quoted for saying at a past Search Engine Strategies conference that 95 percent of the submissions it receives are spam. Of course, you need to understand that this does not mean 95 percent of those using the Add URL page are spammers. Not at all. What happens is that a relatively small number of people submit a very large number of spam pages. Filtering out the relatively small amount of good from large volume of bad is a huge challenge.

Because of this, we have the growing disconnect between what's submitted and what gets crawled. Excite is a great example here. For well over two years, submissions to its Add URL system have been considered more as "suggestions" rather than URLs that actively get included. Instead, Excite's crawler is far more likely to visit a site and select pages it wants to list independently of the Add URL suggestions.

Excite's not alone. FAST, Google and Go are other major crawler that operate in a similar fashion. Indeed, of the major crawlers, only Inktomi and AltaVista have continued to keep a close connection to what's submitted and what gets listed -- and recent complaints about AltaVista sluggishness suggest that its Add URL system is the next to go into "suggestion" mode.

Changing Add URLs pages into "Suggest URL" pages is hard to complain about, because slavishly adding all new submissions would actually harm search engine listings by flooding them with spam. However, penalizing pages submitted through Add URL, as Inktomi is doing, is a major concern. It is most disturbing because it goes against exactly what Inktomi advises site owners to do, through its partners. For example, HotBot's help pages advise that up to 50 pages per day can be submitted for "immediate addition" and make no mention that submitting would cause pages to be downgraded.

Inktomi also failed to make any mention of free Add URL penalties, when it rolled out its paid inclusion program last November. We were told that free Add URL remained an option for those who did not want to pay. Inktomi advised that the only negative to using free Add URL was that pages submitted through it were not guaranteed to appear in its index. It did not add that they would also be subject to a ranking penalty. This lack of disclosure has caused confusion in the search engine optimization world, as will be explained further below.

Another concern about penalties is that anyone can submit another person's URL to Inktomi. Potentially, that means that nothing would stop a competitor from submitting your pages and getting them penalized. However, here's where we can move back from the brink and discover that the Add URL penalty isn't as bad as it sounds.

As it turns out, if a page submitted via free Add URL is later found by Inktomi's link crawler, then the penalty is automatically removed. This is because Inktomi has more faith that the page is less likely to be spam, so feels it doesn't need to penalize it from the start.

"If someone submits a page into the free Add URL, and if the main crawler also finds that page, then the normal scoring mechanism takes precedence," said Palmer.

So, if you submit a brand new page, that page starts out at a disadvantage. However, if Inktomi's link crawler discovers it, the disadvantage is removed. Moreover, once the link crawler finds it, then any competitors resubmitting the page shouldn't cause it to drop in rank.

Remember I said there were three crawling systems? The last one is Inktomi paid inclusion crawler. Those using the Inktomi paid inclusion system have their URLs visited by this. The paid pages are not boosted in rank, but neither are they penalized, as with the free Add URL pages. In other words, they are treated exactly the same as if the link crawler found them.

"I think partly because we do charge a little bit for it, people don't submit millions of fake pages," said Palmer. "So, we give those pages normal scoring, just like the main crawler."

Now let's look at how all three crawling systems may cause different people to have completely different experiences, when submitting to Inktomi.

Person 1 is a search engine optimizer who creates highly targeted doorway pages. These pages have never ranked well when submitted using free Add URL, but when submitted using the paid inclusion system, they suddenly perform great. Conclusion? Paid inclusion equals ranking boost. Reality? What has happened is that these pages were never visited by the link crawler, so the page penalty imposed by free Add URL kept them down. But, when submitted through the paid inclusion system, the penalty is dropped, and the pages performed better.

Person 2 runs a small site with great content about a particular subject. They submit using the free Add URL system, find their pages were picked up and indeed rank well. Conclusion? Free Add URL works fine, and all those people complaining about it don't know what they are talking about. Reality? Maybe free Add URL did get them in, and the content of their pages and other important factors were good enough to overcome the submission penalty. Or, perhaps the link crawler also found their pages before free Add URL added it. Given this, they had no penalty imposed. That, and other factors, may have helped their pages rank well.

Person 3 runs a large corporate site and submits 10 pages through Inktomi's paid system. They find no noticeable rank improvements. Conclusion? Paid inclusion has no relationship to rank boosting. Reality? The pages may already be listed by the link crawler, so being picked up by the paid system would have no impact, as there was no submission penalty to remove.

Person 4 also runs a large corporate site. They submit 200 pages and find traffic definitely increases. Conclusion? Paid inclusion is worth the money. Reality? Chances are, some pages were never listed by Inktomi, and having greater representation generated more ancillary traffic. Particular ranks weren't improved, but the traffic still does better overall.

Four different people, four different reactions. As you can see, just going by another person's comments doesn't necessarily mean you'll have the same results. Also, remember that beyond submission system, other factors such as the content of your pages, link analysis and clickthrough come into play, to determine how your page will rank.

By the way, penalties have been assigned to Add URL submissions for some time, according to Inktomi. "This has been our policy for some time," Palmer said, explaining it wasn't done in conjunction with the release of the paid inclusion system. I'm following up to get a more exact date on when the change was made.

So, how to proceed with Inktomi? If your pages aren't listed at all, submitting via the free Add URL page is better than nothing, Inktomi advises. Your submissions may get included for a short period, as Inktomi likes to "trial" new pages to see if they overcome the submission penalty and still manage to rank well. If so, they may be retained. Expect it to take up to 60 days for your pages to get listed when using free Add URL, and remember that being listed is not guaranteed.

Even better is to build links to your site from other sites. If Inktomi's crawler encounters links to you from a variety of sites, you are more likely to get visited. Also ensure you do a good job of linking between your own pages. That ensures that Inktomi's crawler can find your other pages from however it initially enters your site. A site map can be helpful here. It is simply a list of all the pages in your site. If a crawler comes to it, then it easily learns of other pages it might wish to visit. Your human visitors may appreciate a site map, also.

If you're the type going the doorway page route, remember that these pages typically don't have links pointing at them. Thus, the crawler has no chance of finding them unless you submit using free Add URL or create some links to these pages. Having a "hallway" page is one concept doorway page users are advised to follow. This is simply a list of all the doorway pages in your site. You submit it to the crawler, then the crawler -- if it reads the hallway page -- MAY decide to index some of the doorway pages on the list.

Hallway pages are essentially site maps of just your doorway pages. You wouldn't put a hallway page out for your human visitors to see, because you don't want to send them back to the various "entrances" to your site.

Ultimately, your fastest and only guaranteed route is to use paid submission. If you must have a page listed, then pay the money, and you'll be in within two days.

No doubt you can expect to see Inktomi eventually kill its free Add URL system. The company says the only reason it hasn't yet done so is to figure out how to establish a new system that only non-profit sites can use. "Before we turn the free one off, we want to solve that, let them request a listing," said Palmer.


Inktomi Debuts Self-Serve Paid Inclusion
The Search Engine Update, Nov. 3, 2000

More details about Inktomi's paid inclusion system.

Paid Inclusion At Search Engines Gains Ground
The Search Engine Update, Nov. 3, 2000

Paid inclusion doesn't guarantee particular rankings, but greater representation can increase ancillary traffic, by letting you win the "search engine lottery" more often.

HotBot Add URL page

Some of HotBot's results come from Inktomi, so you can use its free Add URL form to tell Inktomi about your pages -- and that means those that are accepted should eventually show up in the Inktomi-powered portions of MSN Search and at other Inktomi-powered search engines. There's a 50 page per day, per site limit -- and I'm also reconfirming that these guidelines haven't changed.

Ink Plans On Ending Free Submission
Webmaster World, Nov. 12, 2000

Thread from Webmaster World forums where news of the Add URL penalty emerged.

Search Engine Talk

Other places like Webmaster World where search engines are discussed.

Password Finder

Search Engine Articles

Cache at the End of His Rainbow
Wired News, Jan. 15, 2001,1367,41065-2,00.html

How Google's caching feature is being used by people to recover missing web sites.


Dont Be Shy, LadiesGoogle Him! Check Out His Search Engine First
New York Observer, Jan. 15, 2001

Forget "Do You Yahoo?" The question on the dating scene is, "Have you been Googled?" In what sounds like it should be an episode of "Sex And The City," Deborah Schoeneman of the New York Observer explains how checking up on potential dating partners using Google is all the rage. What's your search appeal?


Tough Times for Data Robots
New York Times, Jan. 12, 2001

Crawlers face another legal roadblock, due to a injunction last month preventing Verio from running a robot to gather potential customer leads from's publicly-accessible WHOIS database.


LookSmart Pains On 4Q Revision, Jan. 11, 2001,2198,3531_558671,00.html

LookSmart is cutting its staff by a third, due to dropping ad revenues, and it is consolidating operations to focus more tightly around a different type of "ad" product -- selling listings within its directory.


Yahoo Meets Street Expectations, Cuts 2001 Forecasts
Internet Advertising Report, Jan. 10, 2001,,12_557361,00.html

Expect the next year to be rough, Yahoo tells investors -- but it also says that efforts to diversify revenue away from ads will help it thrive going forward. Ads make up 90 percent of Yahoo's income.


No IPO for AltaVista...Again, Jan. 10, 2001,2198,3531_557401,00.html

Forget the IPO plans. With dotcom stocks in the dumps, AltaVista has dropped plans to go public for the near future.


Resuscitating Dead Links
ClickZ, Jan. 4, 2001

Don't get mad that a search engine continues to list dead links to your site. Instead, make use of that traffic! Pay particular attention to producing a custom 404 page. It's an essential component of making your site more usable.


Search terms of endearment, Jan. 3, 2001

Recap of the top search terms of 2000, from Yahoo, Lycos and AltaVista. Britney Spears is both the queen of teen pop and of search engine queries.


Customization Options for Web Searching
Online, Jan. 2001

Guide to customizing how you search, for several major search engines.


Is Yahoo's New Business Express Program Right For You?, January 2001

If you are rejected after submitting through Yahoo's Business Express system, you are entitled to one appeal. However, found that the appeal did no good. Interesting account of how a site could get rejected, for no apparent good reason. In this case, the fixation seemed to be that a portion of the home page didn't load properly in Netscape 4.76. This is despite the small market share of Netscape in general and that version, in particular.


Hit Me, My Lovely
Modern Humorist, Jan. 2001

Short mystery incorporating the top 50 searched for words from Lycos in 2000.

List Info

How do I unsubscribe?
+ Follow the instructions at the very end of this email.

How do I subscribe?
The Search Engine Update is only available to paid subscribers of the Search Engine Watch web site. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at:

How do I see past issues?
Follow the links at:

Is there an HTML version?
Yes, but not via email. View it online at:

How do I change my address?
+ Send a message to

I need human help with my subscription!
+ Send a message to DO NOT send messages regarding list management or site subscription issues to Danny Sullivan. He does not deal with these directly.

I have feedback about an article!
+ I'd love to hear it. Use the form at