The Search Engine Report March 5, 2001 - Number 52

March 5, 2001 - Number 52

By Danny Sullivan
Editor, Search Engine Watch
Copyright (c) 2001 corporation

About The Report

The Search Engine Report is a monthly newsletter that covers developments with search engines and changes to the Search Engine Watch web site,

The report has over 160,000 subscribers. You may pass this newsletter on to others, as long either part is sent in its entirety.

Did you know that there's a longer, more in-depth version of this newsletter?
The twice-monthly "Search Engine Update" newsletter is just one of the many benefits
available to Search Engine Watch members Learn more about the advantages
to becoming a member at this page:

Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.


In This Issue

+ Site News
+ Conference News
+ Blending Vertical Results & Other AltaVista Improvements
+ Being Search Boxed To Death
-- (full story online, link provided)
+ Network Solutions Launches Inktomi Paid Inclusion
+ New ProFusion Site Offers Better View Of Invisible Information
+ GoTo Increases Prices, Appears At iWon
+ Top Search Terms Of 2000
-- (full story online, link provided)
+ Will P2P Search Replace Search Engines?
+ LookSmart Submission Data Was Left Vulnerable
-- (full story online, link provided)
+ Google Acquires Deja Newsgroup Service
-- (full story online by Tuesday, link provided)
+ Interesting Search Engine Articles
+ List Info (Subscribing/Unsubscribing)


Site News

Hello Everyone--

Within the web site, I've completely updated both the Media Metrix and NetRatings search engine ratings pages. I expect to get the trend chart back on the Media Metrix page later this month, but you can still see ups-and-downs on the NetRatings page. It shows that Yahoo, MSN and Google have had noticeable gains in popularity over the past few months. Most other players have shown an erosion of their audiences, with AltaVista in particular seeing a significant plunge. Links to both pages can be found via What's New, below:

Search Engine Watch What's New


Conference News

Search Engine Strategies is coming to Boston on March 20 and 21. Day 1 is "Promoters Day," which features experts on search engine marketing issues, roundtables on advanced search engine optimization issues and panels with representatives from various major search engines themselves. Day 2 is "Searchers Day," which features two tracks designed to help Internet searchers better understand how to use the search tools available to them. There is also a special promoters track that repeats the advanced roundtables held on Day 1.

Sponsorship and exhibiting opportunities are still available, and you can contact Frank Fazio Jr,, for more information. If you are an accredited member of the press interested in attending, please contact Mary Ann Boland at

Search companies and services confirmed to speak include, AltaVista, Ask Jeeves, Atomz, CompletePlanet, EasyAsk, Excite, FAST, Google, GoTo, Inktomi, InfoSpace (Metacrawler/Dogpile), Intelliseek, iWon, Looksmart, Lycos, Mondo, Moreover, MSN Search, Netscape/The Open Directory, Northern Light, Oingo, Sandy Bay and Quiver.

You can attend either day of interest or receive a substantial discount to attend both. More details and agenda can be found below.

Search Engine Strategies


Blending Vertical Results & Other AltaVista Improvements

Over the past month, AltaVista has been busy making some under-the-hood changes to improve its search results. In particular, the service has upgraded its shopping search and is "blending" links that lead to its shopping search engine for appropriate queries into its main results. How searches themselves are processed has also been changed.

Do a search for product oriented information, such as "dvd players," "inline skates" or "computers" and you'll see the new shopping search links appear. They look similar to AltaVista's regular numbered listings and appear just before these. They usually begin "Compare Prices and Features on..." followed by your search terms, and the words "Shop Smart" also appear next to the link. Selecting the shopping link takes you to the AltaVista shopping search engine, where you can obtain product pricing from over 600 online web merchants.

"We've learned that 20 to 25 percent [of searches” are shopping related searches, but the average user may not know how to get to our shopping vertical," said Ganon Giguiere, senior director of search verticals at AltaVista. By integrating the links to shopping search in the main results, AltaVista hopes to better serve its users.

Naturally, getting people into the shopping search area also benefits AltaVista. Some of the merchants pay AltaVista to be included, but the "vast majority" don't, Giguiere said. Most major shopping search engines do have some type of payment model with merchants, and as long as there's a wide variety of major, reputable companies, I don't think users need to fear these deals.

I'm pleased to see the shopping links integrated as they are, and I'll be looking forward to watching how AltaVista integrates other links to new verticals that it plans to launch in the near future. For the most part, users would benefit by finding better specialty search tools. It's simply been a big challenge to figure out how best to direct them, as I've covered more in the separate "Being Search Boxed To Death" article (see below in the newsletter).

For the most part, I'd rather one or two targeted vertical links replace all the "dumb" links that appear in response to any query. For instance, search for anything on AltaVista, and you'll always get "Extend Your Search" links at the bottom of the page that include whatever you searched for, even if it is absurd that you would do this -- a search for "korean war" makes this area suggest things like "Shop the web for korean war" and "Searching for korean war? Find it at" Nor is AltaVista alone in having these dumb links. Removing them would eliminate clutter from the result page and ought to increase usability.

AltaVista's shopping search service lets you compare prices, but I'd also like to see product information added. It would be great to see the service expanded to pick up product reviews and consumer information from key sites, which would help it be more than an online mall.

If you like the shopping search service, it remains accessible also from its own home page, plus the service can be customized to turn off product pictures, increase the number of results and place the "Shop Near You" section first on the page.

Shop Near You is completely new. This is information that comes from "brick and mortar" stores near your home. You give AltaVista your zip code, and it pulls back matches powered by information from This can be especially helpful if you prefer to use AltaVista Shopping to do online research, then find a local vendor with a "real" store that you can visit. In addition, auction listings from uBid for products can also be found.

I noticed some bugs with the shopping link integration. Try a search for "pc computers," for example, and rather than a generic "Compare Prices" style link, you'll get a link to a specific product. I don't think the user experience is as good, in these cases. "Cordless mouse" was another example, but fortunately, I didn't see many other instances.

The next vertical search product to appear will be news. Giguiere said will contain stories from many leading news providers. Finance, real estate and travel verticals are also planned. In all these cases, links to the appropriate vertical service will be integrated as with shopping into the main results page, appearing in response to appropriate queries.

Don't forget that AltaVista already has a variety of vertical search tools that you can reach by using the "tabs" above the search box on the results page. Within the AltaVista Tools area, you can also access specialty search features that let you search against US educational sites or US government sites.

Beyond the vertical links, AltaVista has upgraded how it processes search requests, being more expansive with what it returns. Here's a look at the new "query reformulation" changes.

Single word queries are easy -- AltaVista will return any pages it knows of that contain that particular word.

For multiword queries of between two to four terms in length, AltaVista continues to perform automatic phrase detection. This means that it looks to see if there are any recognized phrases that match a dictionary of about 500,000 phrases it maintains. If your search terms appear in the phrase dictionary, then AltaVista automatically translates your request into a phrase search. For example, if you search for:

new york

the request will be turned into a phrase search behind the scenes, even though you didn't specifically request this by placing quotation marks around the terms.

AltaVista has done automatic phrase searching like this since November 1998. However, in a twist that began in February, it now goes beyond providing only exact phrase matches and will also return pages that contain all of the words in your query, even if they aren't contained in an exact phrase. For instance, consider a search for

new york stock exchange

AltaVista will look for pages that match the exact phrase, but then it also finds pages that have all four words on them, even if they don't occur in that exact order.

Before this change, if you had searched for an exact phrase and there were no pages or few pages with that exact phrase, you would have come up with no results or only a few matches. With this change, AltaVista better ensures that its automatic phrase detection, which is helpful in many cases, doesn't leave users without results in some situations.

"The advantage of not removing documents without the phrase is that sometimes the exact phrase isn't exactly how the page describes the concept. For example, the user might be looking for information on Clinton's cybersecurity proposal. The user enters: 'president bill clinton cybersecurity,' said Vaughn Rhodes, senior director of product marketing for AltaVista. "None of the documents discussing the proposal use that specific phrase [so would be missed the old way”. The new AltaVista technique correctly returns those documents."

Finally, in situations where there are five or more words, AltaVista continues to do phrase detection, and then it will seek pages that have ANY of the words on them, rather than ALL of them.

"When users enter a small number of query concepts [such as two to four words”, they are usually looking for documents that have all of the terms in them. However, when large numbers of concepts are used [five or more”, users tend to be in a 'find stuff like what I describe here' mode, in which they don't necessarily require that every term they enter is present," Rhodes said.

AltaVista clearly hopes the changes will mean better results for average web users, and most people probably will be fine letting AltaVista do the driving, so to speak. However, advanced searchers may still prefer to control AltaVista themselves by using power commands such as the + symbol or Boolean operators. AltaVista says that if such commands are used, they'll take precedence over the internal logic it tries to follow. That hasn't been the case in the past, and I haven't had a chance to look closely to see if this is indeed happening. In addition, AltaVista's power and advanced search pages have none of the behind-the-scenes processing happening.

"The query document selection technique changes apply only to the main search box, not to Advanced or Power Search. Users of those interfaces won't see any changes. In addition, if users enter into the main search box the syntax elements that were used in the previous technique [such as +, -, or quotes”, we will automatically fall back to the previous methodology," Rhodes said.

Related to search processing, another relatively new feature is the ability to search within results at AltaVista. After performing a search, just check the "Search Within Results" box under the search box on the results page. Then you can do a new search just against the first set of results retrieved.

Also, be aware that if you get to the bottom of the page and want to see more results, you might accidentally select the "More Sponsored Listings" option, if you aren't careful. That brings up more advertisements from GoTo. Instead, if you want more editorial listings, you need to select any of the "Results Pages" numbers or the "Next" link. These appear near the bottom of the results page and just above the "Sponsored Listings" heading.

In yet more under-the-hood changes at AltaVista, the company says that it now has in excess of 500 million web pages indexed, up from its previous 350 million and in line with the full-text indexes other major crawlers such as Inktomi, Google and FAST.

A longer version this article is available to Search Engine Watch members. Learn more about becoming a member at


AltaVista Shopping

AltaVista Tools

Access to the AltaVista education and government search engines can be found here, on the left-side of the page. The Power Search page is also listed with them.

Being Search Boxed To Death
The Search Engine Report, March 5, 2001

Overview of how search blending such as that being done at AltaVista may improve the search experience for users.


Being Search Boxed To Death

General purpose search engines are wonderful tools. You can search for entertainment, news, sports, company and many other types of information and still often find what you are looking for. Like a Swiss Army Knife, general purpose search engines often can do many different jobs. Nevertheless, your results might be better if you turn to a vertical tool. This may be the year that the general purpose search engines finally figure out a way to get the right vertical tools into the hands of their users. They've tried before, but some new efforts might succeed where we've previously had failure. More about these efforts, via the story below:

Being Search Boxed To Death
The Search Engine Report, March 5, 2001


Network Solutions Launches Inktomi Paid Inclusion

Network Solutions was Inktomi's first paid inclusion partner, announced way back in the middle of last year. Finally, that partnership has gone live, with Network Solutions now offering guaranteed inclusion in the Inktomi web index through its web site.

Inktomi has two other self-serve paid inclusion partners: US-based Position Technologies, which began offering the service in November, and Europe-based WebGravity, which began its inclusion program in January.

All offer the same basic service: inclusion of submitted URLs into the Inktomi web index within two days and revisiting those URLs on a weekly basis, for up to a year.

At Network Solutions, paid inclusion costs $30 for the first URL, then $15 for each URL after that, with up to 100 URLs allowed per order. The Network Solution program allows the list of URLs to be changed at any time. So, if you decide three months into the program that you want a completely different set of URLs included, you could change these from those you initially submitted without incurring an extra fee.

Will Position Tech and WebGravity be able to offer similar flexibility in the future? Probably, but exactly when is not certain. Inktomi says that it wants to see how the experiment with URL swapping works at Network Solutions, then ultimately expects that all self-service paid inclusion resellers will have standard offerings.

The deal between Inktomi and Network Solutions is also going to expose the paid inclusion system to a larger audience that ever before, because Network Solutions will be offering cobranded programs for portals to use while also letting its 65,000 affiliates receive fees for referrals to the sign-up form at the Network Solutions site itself.

Cobranded programs mean that portals that make use of Inktomi data, such as iWon or MSN Search, might offer paid inclusion into the Inktomi database from Add URL pages within their own sites, maintaining their own branding and sharing revenues with Network Solutions, which will administrate the submissions behind the scenes.

"That's kind of what we were bringing to the table, the ability to distribute this through the portal channels," said Mike Cornell, product manager of value added services and products at Network Solutions.

A longer version this article is available to Search Engine Watch members. Learn more about becoming a member at

Network Solutions: Inktomi Search/Submit

Information about paid inclusion via Network Solutions can be found here.

Inktomi Search/Submit Partners

Links to all of Inktomi's various partners offering self-serve paid inclusion can be found here.

Pay For Placement?

Links to past articles about Inktomi paid inclusion can be found here.


New ProFusion Site Offers Better View Of Invisible Information

Intelliseek has launched a new version its ProFusion meta search site that also combines easier access to "invisible web" information. The combination of comprehensive web search results combined with the ability to target topic-specific search resources make the developing service one of great appeal to searchers.

After certain types of searches -- especially those involving medical terms, company names, geographical locations, celebrities and other data that ProFusion believes may available in "invisible web" areas -- you'll see the page topped with a "search assistant." This is a person holding a magnifying glass, with the words "ProFusion Recommends" above. Next to the person are links designed to route you into search results from topic-specific or "vertical" search services.

For instance, if you search for "pregnancy," you'll see suggestions such as "Health Tips: Try your search for pregnancy within Health Tips." Selecting this link would bring up a new set of search results. These come from specialty sites that deal with health issues, such as,, Medline Plus and the American Medical Association.

Some information at these sites might not be available to ordinary search engines, because it comes out of databases and is only retrieved if you used a special search form when visiting the sites. Hence the concept of "invisible web." This is information that might not be visible to an ordinary search engine. However, ProFusion is specifically designed to pass along your automatically query to appropriate invisible web resources and find the information that might otherwise be missed.

In addition to routing you toward invisible web resources, ProFusion also continues to offer more ordinary search listings. For instance, the results page will often begin with any matching "Web Guides" information, which are category lists from LookSmart's human editors. For example, select the "Pregnancy, Birth and Newborns" link after a search for "pregnancy," and you'll have access to resources on that topic, which have all been reviewed.

The "Web Search Engines" section presents search results pulled back from major search engines from across the web, including AltaVista, MSN Search and Yahoo. Google is not included.

Unfortunately, there is no way to control exactly which services are queried when doing an ordinary search. Instead, if you want this control, you must use the advanced search page (use the link below the search box on the home page). The settings you choose on the advanced search page will be remembered, if you always start from it. But if you do so, you lose the search assistant suggestions to invisible web content and the Web Guide links.

Every listing in the results is followed by a "page alert" link. Select this link, and you'll be notified each week of any changes to the page. Similarly, the small "Set Search Alerts" link at the top right of the Web Search Engines portion of the results page will rerun your search each week and notify you via email if there are new or different results.

ProFusion's automatic suggestions of invisible web resources are good, but I think the real power lies in browsing topics from its home page. That's because the suggestion list at ProFusion is still growing, so I think you could miss out on resources you'd more likely find if you drill down the category levels.

For example, say you want historical information on electrical consumption in California, which has been plagued by brownouts and shortages over the past year. If I search for that -- "historical information on electrical consumption in California" -- Profusion comes back with no search assistants pointing me toward invisible web resources. However, if browse from the home page into "Government," then I discover the option to search against invisible web resources with information about commerce, US state and local government information and energy resources. All of these sound promising.

Promising, yes -- successful, no. As it turns out, the results for that query were very poor. In contrast, the Web Search results that ProFusion found for the query were excellent, bringing up a very good document from the California Department of Energy. This just goes to show ordinary search engines can and still do bring back plenty of good information, despite concerns that they may miss invisible web content.

This isn't to knock the work Profusion has done. There are great applications for invisible web searching, and as Profusion develops over time, its suggestions within the ordinary search results should improve.

In the meantime, do consider browsing topics from the home page, if your ordinary results don't seem to work. The ability to hit patent databases, music reviews, legal resources and other great specialty sources can all be found. Not only can you locate them, which was always possible through Intelliseek's web site, but you can also search against several of them in a category at the same time.

It would be nice if it were possible to search for matching invisible web categories more easily from ProFusion. For example, search for "census" at, and you are shown multiple resources related to the term. At ProFusion, you have to search for "census," then scroll to the bottom of the results page and go through the extra step of selecting the "Invisible Web Sites" link at the bottom of the page. Even this fails to show you which invisible web sites match the query. Instead, it just runs your query against whatever invisible web sites ProFusion considers relevant.

A longer version this article is available to Search Engine Watch members. Learn more about becoming a member at

ProFusion Beta

The site is expected to lose its beta status in the next six weeks or so and replace the existing ProFusion service, below.


The older version of ProFusion remains live but offers only meta searching abilities.

Intelliseek's existing catalog of invisible web resources, but it doesn't integrate meta search, as does the ProFusion beta.


BrightPlanet's catalog of invisible web or deep web resources.

I can search the 'invisible Web.' Here's how you can, too
ZDNet, Feb. 8, 2001,10738,2683349,00.html

Another review of the new ProFusion service.

Search Links: Invisible Web

Further resources and information about the invisible web.


GoTo Increases Prices, Appears At iWon

The days of 1 cent clicks at GoTo's US service are gone, made extinct by a price increase put into place on March 1, which raised the minimum bid to 5 cents. A new $20 minimum charge per month has also been implemented. GoTo advertisers learned of the changes in an email sent to them on the day the price changes took effect.

"Over the last 2 quarters, we have signed a number of big traffic deals with premier web sites which has resulted in even more targeted traffic to your site. These traffic deals have increased our costs and as a result we're implementing a modest price increase," the letter from GoTo began.

Reaction to the changes at the Search Engines Forum discussion area was mostly negative, with only a few seeing the change as having no impact on them or being positive in some way. GoTo said it has received several hundred emails about the increases, almost all expressing displeasure at the raise in minimum bid price. In contrast, the $20 minimum monthly hasn't attracted great concern, the company said.

Higher prices and minimum spend amounts should generate more revenue for the service, assuming they make up for any losses from customers abandoning it due to the hikes. GoTo says that about 25 percent of its clicks are from 1 cent listings, so the potential for customer dissatisfaction is broad. However, as of last Friday, only one official loss was reported.

"We had one advertiser I'm told that pulled his business. My understanding is that he was spending less $5 per month," Ted Meisel, GoTo's president and CEO.

Ironically, losing that type of low-spending customer may actually generate revenue for GoTo by saving money. The service must review each listing submitted, to ensure that it meets minimum relevancy standards. It's under even more pressure to ensure that these listings are somehow related to bidded terms now that its listings are distributed to major partners such as AOL Search, the company says. The time to do this for the customer who spends $5 per month on 1 cent listings and $5,000 per month on $1 listings is the same.

"There's a certain minimum cost in serving a business," said Meisel. "We're not asking for a listing [setup” fee. We're just asking for you to spend a certain amount of money with GoTo."

In other GoTo news, its seven top paid listings are now appearing at iWon, another major search site. The deal hasn't yet been announced publicly, but the links can be found at iWon on the right-hand side of the screen, under the heading, "Partner Search Results."

A longer version this article is available to Search Engine Watch members. Learn more about becoming a member at


Frequently Asked Questions for Advertisers

More information from GoTo on minimum bids.

GoTo Price Increase
Search Engine Forums, March 1, 2001

Mostly negative reaction to the GoTo price hikes.

Paid Listing Search Engines

If you are after traffic, any type of traffic, for 5 cents or less, then some of the pay for placement services here are worth looking at. However, none of them have the distribution on major search engines to match what GoTo has. However, several of them are carried by meta search services.


Top Search Terms Of 2000

I know -- the New Year has come and gone, but there were a number of good stories and resources I've saved up on top search terms from last year that I wanted to share. Also, search engines are starting to be more forthcoming with what people are searching for, which is great data for anyone who is interested in what's popular on the web. A roundup can be found below:

Top Search Terms Of 2000
The Search Engine Report, March 5, 2001


Will P2P Search Replace Search Engines?

Will Napster-style peer-to-peer searching mean an end to search engines? Despite the continuing hype, I doubt this will be a replacement for web-wide searching. There are strong advantages to a centralized system, not the least in terms of dealing with spam. I do think P2P has promise for intranet solutions, especially in situations where you have trusted data that you want to find on a few hundred or thousands of machines. But P2P for full-text search against a billion or more URLs? Unlikely. Below are two new articles, pro and con on P2P search, plus my last article on this topic.

Search project prepares to challenge Google, Feb. 26, 2001

Pandango is an interesting idea. By examining what we visit, it would then select the more popular documents in response to a search. This is akin to the bookmark-based search engines that were all the rage this time last year but which have failed to take off. It's also somewhat similar to the Alexa "Related Pages" technology already built into Internet Explorer and Netscape Navigator but which I've yet to encounter many people using. Aside from the big disadvantage of requiring users to download software, there could also be privacy concerns. Long story detailing the project. Despite the company's suggestion they are the "third generation" of web searching, when Pandango launches, it will really be a first generation service. The technology may be a move in a new direction, but until it has been tried and tested against a web-wide audience, it can't really grab that claim to be the next rung on the evolutionary ladder of web search.

P2P Goes in Search of 'Doogle'
Wired, Feb. 16, 2001,1282,41850,00.html

P2P is unlikely to replace centralized web search engines, is the conclusion from this major P2P conference.

More Than Just Music Search
The Search Engine Report, June 2, 2000

Napster has obviously changed since this was written, but the contrast and comparison of centralized web search to the potential of P2P search remains relevant.


LookSmart Submission Data Was Left Vulnerable

If you've never submitted to LookSmart, don't worry about this story. If you did submit, there's a very small chance someone might have obtained your phone number and email address. This is because information in LookSmart's submission queue was been left open to public, giving access to some details about those who submitted sites to the service. Credit card data, however, was not exposed.

LookSmart Submission Data Was Left Vulnerable
The Search Engine Report, March 5, 2001


Google Acquires Deja Newsgroup Service

Google purchased the Deja newsgroup archives last month and is now running them within its own site, which you will find here: Despite saving the service, Google came under criticism from hardcore Deja users upset about lost functionality during the transition period. I'll bring you a round up of articles on this topic and more details from Google itself by Tuesday, at the URL below. It will also have short details on the new Google country-specific editions that are emerging and the i-mode service for wireless web users in Japan.

Google Acquires Deja Newsgroup Service, March 6, 2001

Search Engine Articles

Revving Up the Search Engines to Keep the E-Aisles Clear
New York Times, Feb. 28, 2001

A look at the challenges in improving search at shopping sites.

Ask Jeeves eyes large enterprises
InfoWorld, Feb. 23, 2001

Ask Jeeves is to place its knowledge management tools in software form, so that enterprises can maintain their own answer databases.

FirstGov Web portal finds a home in new administration
Government Executive Magazine, Feb. 23, 2001

Update on development work at the US government's search site, FirstGov.

Bidder's Edge pushes Web site over cliff, Feb. 15, 2001

The auction meta search service sought that fought against eBay and lost is now closing.

Search results becoming more commercial
San Jose Mercury News, Feb. 15, 2001

Review of trend for paid listings at search engines, with comments from different services.

List Info

How do I unsubscribe?
+ Use the form at or follow the instructions at the very end of this email.

How do I subscribe?
+ Use the form at or send a blank email message to

How do I see past issues?
+ Follow the links at

Is there an HTML version?
+ Yes, but not via email. View it online at

I didn't get Part 1 or 2. Can you resend it?
+ No, but you can view the entire issue online, via the link above.

How do I change my address?
+ Unsubscribe your old one, then subscribe the new one, using the links above.

I need human help with a list issue!
+ Write to DO NOT send messages regarding list management issues to Danny Sullivan. He does not deal with these.

I have feedback about an article!
+ I'd love to hear it. Use the form at