The Search Engine Report, Sept. 3, 1999, Number 34

Search Engine Watch

September 3, 1999 - Number 34

By Danny Sullivan
Editor, Search Engine Watch

About The Report

The Search Engine Report is a monthly newsletter that covers developments with search engines and changes to the Search Engine Watch web site,

The report has 73,000 subscribers. You may pass this newsletter on to others, as long either part is sent in its entirety.

If you enjoy this newsletter, consider showing your support by becoming a subscriber of the Search Engine Watch web site. It provides you with some extra benefits and access to some exclusive materials and articles. Details can be found at:

Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.

In This Issue

+ General Notes
+ Excite Enlarging Index, Partnered With LookSmart
+ New MSN Search, AOL Search Available in Beta
+ Lycos, Copernic Release Search Tools
+ GovSearch To Keep Going
+ Search Boxes That Pay
+ Search Engine Articles
+ Subscribing/Unsubscribing Info

General Notes

Hello Everyone--

First, in a bit of non-site news, I'm very pleased to announce the Search Engine Strategies '99 conference. I'm organizing this first-of-its-kind conference in partnership with Search Engine Watch's parent,

Held on November 18 in San Francisco, it will be a one-day conference with sessions entirely about search engine marketing issues. If you know nothing about search engines, I'll be leading off with a Back to Basics session. Then there will be sessions devoted to meta tags, submission issues, dealing with directories and other topics. We've assembled a wide-range of experts to talk about these topics, and it will be an exciting and informative day. The final session is a "Meet The Search Engines" panel, where site owners can put those perplexing questions of how they are listed directly to the search engines themselves.

The conference agenda has just gone up via the URL below, and we'll be tidying up the information over the information, adding speakers bios, etc. over the next few days.

Search Engine Strategies '99

In search engine news, there are lots of interesting things floating on my radar screen to write about, but I'm keeping this edition of the newsletter relatively light so I can finish some updating throughout the web site itself.

I've just posted the latest results to the Company Name Test. In the search engine category, HotBot and Google got perfect scores for delivering users to official company sites in the test. As for directories, Netscape Search and MSN Search take top honors. Honorable mentions also go out to Go, Ask Jeeves, Yahoo and Snap -- they all did very well. A link can be found from the What's New page.

This marks the resumption of regular relevancy tests that I have planned. I hope to do a different one every month -- but depending on how much news there is, I may slip this to every two months. Once I've done enough continuing tests, I'll also begin a cumulative total for the different services.

I also expect to post the latest Media Metrix and NetRatings figures shortly, along with a number of other changes this month. Keep an eye on What's New, and you'll know when these go up.

Additionally, expect a new home page for Search Engine Watch to go up within a few days. There won't be any dramatic changes, but I am doing a slight organization of the site's content and using new section names that I think will be more descriptive to new visitors.

What's New

Search Engine News

Excite Enlarging Index, Partnered With LookSmart

In August, Excite began the first phase of an ambitious plan to enlarge its search index to 250 million web pages and improve the relevancy of its search results. The search engine also debuted new LookSmart-powered directory listings.

Under its new indexing system, which has been in the works for the past year and a half, Excite plans to visit 500 million or more pages across the web on a regular basis. It will then retain only those pages that it determines are most popular, or which offer the best quality information, or which seem to satisfy the queries its users make.

This "visit many, keep some" approach is how Excite hopes to expand its index coverage without simultaneously overwhelming users with irrelevant or off-topic documents.

"We don't think just adding more content will do the job for us," said Kris Carpenter, Excite's Director of Search Products. "We view that as our number one challenge, understanding what's out there and producing that top quality content in the first two pages of results."

Excite is using a number of "off-the-page" criteria to determine both which pages to retain in its index and how to rank those pages in response to queries. By off-the-page, I mean factors that are not tied to what's on the page itself.

For instance, search engines have traditionally ranked pages by criteria such as where and how often search terms appear in them. Since these factors happen "on-the-page," webmasters could make changes to their pages to try and increase rankings.

In contrast, off-the-page criteria are those not directly in a webmaster's control. A good example is link popularity. It is very difficult for a webmaster to try an outwit a good system that uses link popularity as a ranking criteria. That's because such a system leverages information from across the web, which a single webmaster cannot control.

Excite has long made use of link popularity, and that criteria is now being given heavier weight in its new system. Some have also noticed that Excite has been measuring clickthrough from its results. Carpenter said the Excite has experimented with using this data to influence rankings, but that it is not currently being used as part of its relevancy system.

Excite is also using another set of off-the-page information that I can't disclose publicly. I can say that it is unique among the major search engines in using this type of information, and that it would seemingly offer yet another way of getting the best information to the top of search results lists. Of course, the proof will be if relevancy actually does improve in the long term.

Each of these off-the-page criteria are weighted differently, but term frequency and location still come into play. In general, the mixture should work to reward sites with good content or that at least somehow distinguish themselves online.

One big plus to the expanded Excite index will be that good pages should no longer suddenly disappear from the service for no apparent reason. This problem has plagued Excite over the past year. It would constantly drop pages out of its index to make room for new finds. As a result, webmasters with good representation in Excite might suddenly find all their pages gone. Similarly, this had an adverse impact on searchers, because pages that were satisfying their queries one week might no longer be present the next. With the new system, pages that are deemed popular or high quality in some way should be retained.

So when does all this happen? Excite says it is currently at about 113 million web pages indexed, and that they will increase their volume of pages indexed by, on average, a rate of over a million pages per day. It is also introducing a new system meant to revisit pages based on how often they change, in order to keep the entire index as fresh as possible.

In addition to crawling the web, Excite has also maintained a human-compiled directory of web sites. As at Yahoo, this is where sites have been reviewed by editors and organized into categories. A new deal struck in August means that this web directory will now be produced by LookSmart. In fact, LookSmart's information has already be integrated into Excite.

Just like at Yahoo, you can access the directory by selecting a main category from the Excite home page. You'll find them just under the search box. These links take you into one of Excite's "channels," which are filled with information beyond just web site listings.

On the left-hand side of each channel page, you'll see a box called "Directory" filled with topics related to that channel. For instance, in Excite's Lifestyle channel, the first topic in the directory box is "Beauty & Fashion." By selecting this topic, you'll then be shown a list of Beauty & Fashion web sites.

Only a few top sites will automatically be displayed for any topic. To see more, click on the "More Web Sites" link. You'll also see that as you drill down, even more topics will be revealed.

A faster way to get to relevant directory listings is just to do a search at Excite. If Excite finds any categories that match, it will display them in the search results under the heading of "Directory."

A couple of other Excite notes. A new Adult Content filter was introduced earlier this year. You'll find it on the advanced search page. It has to be enabled each time you do a search, unlike filtering options offered by AltaVista, Go and Lycos. A more permanent solution may appear later this year. Filtering is done by a combination of looking for the presence of certain words at the time a page is spidered and through the use of a site block list.

Excite is also offering the ability to search by language. As with other services doing this, language determination is made by looking for the presence of certain words unique to a particular language. You'll find this option on the Advanced Search page.

I also wanted to take a moment and briefly provide an update on Excite's two other search properties, WebCrawler and Magellan.

Magellan is now essentially a stripped-down version of Excite's directory listings and search index. Magellan's home page features the directory -- click on a topic, and you'll get web sites and only web sites -- no channel bells and whistles as you might get at Excite. Do a search, and your query goes against about two million pages from the Excite index, which are predominately site home pages. Magellan also uses Excite's ranking algorithms, so for popular queries, you may get the same results as at Excite.

Magellan also used to feature the ability to view "green light" web sites; however, this kid-friendly feature is temporarily gone. A replacement should appear by end of the year, Excite says.

WebCrawler is similar to Magellan in being a lighter-version of Excite. It also presents directory information, and web searching also goes against only two million page from the entire Excite index. However, the service has much more personality than Magellan, plus it does have expanded channel content that Magellan lacks. Additionally, WebCrawler uses a much different ranking system than Excite, so expect to see differences if comparing the two.

In the future, both services may have their web search ability expanded to tap into about 3 and 5 million pages from the Excite index.


Excite Advanced Search

Click on the words "Advanced Search" on this page to get complete options, including the adult content filter.

Excite WebCrawler

Excite Magellan


Kids Search Engines

Listing of services offering kid-friendly searches


New MSN Search, AOL Search Available in Beta

Two major portals are readying new versions of their search services. Both MSN and AOL have made unveiled their next-generation search offerings at beta sites that have just gone live.

The new MSN Search nicely integrates information from RealNames, the LookSmart directory and AltaVista. You will also find an option to search at Direct Hit to the left of the search results screen. It should take over from the current Inktomi-powered service in mid-September.

The new AOL Search service will be the successor to the current AOL NetFind. A version that works internally for AOL members will blend matching content from within AOL and from across the web into one results screen. AOL's internal keywording system will also be integrated into the system. The external version that will be accessible via the web won't list content only available to AOL members. Both versions will offer Inktomi-powered web-wide searching, but they will also be presenting Netscape's Open Directory Project's categorized listings, as well.

As both beta sites have just gone live, I'm holding off on longer reviews for the next newsletter. Meanwhile, the curious can explore them via the URLs below. Remember -- both of these services are still in beta, so they are going to be some rough edges.

MSN Search Beta

MSN Search

AOL Search Preview

AOL NetFind


Lycos, Copernic Release Search Tools

A new utility from Lycos allows those using Internet Explorer 4 or 5 to highlight any words on a web page, then right click and send the words as a query to the Lycos search engine. Installation takes only a few seconds.

Lycos See More

Copernic has released versions of its desktop meta search tool for Mac users. A free version provides basic search abilities, including the ability to query the major search engines, while a fee version includes the ability to perform over 100 specialty searches.



GovSearch To Keep Going

Earlier this year, Northern Light launched a new government search engine in partnership with the US Department of Commerce's National Technical Information Service. Apparently, the NTIS is now to be closed -- but Northern Light says it will continue to run the government search engine on its own.



Search Boxes That Pay

Lycos and Direct Hit have launched new affiliate programs that pay web site owners for putting up search boxes. Site owners can earn 2 cents per search at Lycos-owned HotBot, or 3 cents if they link to other Lycos-powered features. Direct Hit is offering 3 cents per search A similar program has also run for some time at

Lycos Affiliate Program

Direct Hit Affiliate Program Search On Your Site

Search Engine Articles

AltaVista Reaches Record Users With Free Internet Access
AtlaVista Press Release, Aug. 25, 1999

AltaVista's new free Internet access service has already drawn 225,000 subscribers in its first two weeks.


The Quality of Researchers Searches of the ERIC Database
Education Policy Analysis Archives, Aug. 25, 1999

Typical users of an in-house database were found rarely go beyond the first page of hits and to examine only about 3 to 4 matches, in contrast to advanced searchers who would look longer and harder.


The Art of Backward Searching
About Web Search Guide, Aug. 24, 1999

How to find your site's popularity on search engines or find related pages to one you like.


Browser multitasks as "Internet Desktop", Aug. 23, 1999,4,0-40835,00.html

Lycos is partnering with NeoPlanet to produce a browser that integrates portal services like instant message into the software.


Q & A: Jeff Bennett, Lycos V.P. of E-Commerce, Aug. 20, 1999,1281,3551_187021,00.html

Lycos VP discussed where the company is going in e-tailing and e-commerce.


Compaq, CMGI finalize Alta Vista deal, Aug. 19, 1999,4,40692,00.html

It's official -- CMGI now has the controlling stake in AltaVista.


Are "registered user" figures worth anything?, Aug. 16, 1999,4,40538,00.html

A look at how much you can trust the registered user numbers quoted by portals.


AltaVista's International Mirrors
EContent, August. 1999

Covers differences between AltaVista's mirror sites.


The Battle of the Browser Sidekicks
PC World, July 28, 1999,1510,12021,00.html

A review of add-ons that bring content from Yahoo and Excite to your desktop.


How do I unsubscribe?
+ Use the form at or follow the instructions at the very end of this email.

How do I subscribe?
+ Use the form at or send a blank email message to

How do I see past issues?
+ Follow the links at

Is there an HTML version?
+ Yes, but not via email. View it online at

I didn't get Part 1 or 2. Can you resend it?
+ No, but you can view the entire issue online, via the link above.

How do I change my address?
+ Unsubscribe your old one, then subscribe the new one, using the links above.

I need human help with a list issue!
+ Write to DO NOT send messages regarding list management issues to Danny Sullivan. He does not deal with these.

I have feedback about an article!
+ I'd love to hear it. Use the form at

This newsletter is Copyright (c) Corp, 1999