The Search Engine Update, March 5, 2001, Number 96


About The Update

The Search Engine Update is a twice-monthly update of search engine news. It is available only to Search Engine Watch members. Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.


In This Issue

+ Conference News
+ Blending Vertical Results & Other AltaVista Improvements
+ Being Search Boxed To Death
-- (full story online, link provided)
+ Network Solutions Launches Inktomi Paid Inclusion
-- (much expanded and updated from last newsletter)
+ New ProFusion Site Offers Better View Of Invisible Information
+ GoTo Increases Prices, Appears At iWon
+ Yahoo Loses International Directors & Other Search Financial News
-- (full story online, link provided)
+ Top Search Terms Of 2000
-- (full story online, link provided)
+ Will P2P Search Replace Search Engines?
+ LookSmart Submission Data Was Left Vulnerable
-- (full story online, link provided)
+ Google Acquires Deja Newsgroup Service
-- (full story online by Tuesday, link provided)
+ Bits & Pieces
-- Yahoo Sponsored Listings Warnin
-- Frames Support Improves
-- Excite's "Zoom In"
+ Interesting Search Engine Articles
+ List Info (Subscribing/Unsubscribing)


Hello Everyone--

It's a big newsletter, this issue. The AltaVista piece is especially long. There have been many substantial changes within the service, so I wanted to document much of what is happening there. I've also greatly expanded the information about Network Solutions and its offering of paid inclusion into Inktomi from what was mentioned briefly in the last newsletter. Be sure to review this, if you are considering the Inktomi paid inclusion program, so that you can choose the right partner.


Conference News

Search Engine Strategies is coming to Boston on March 20 and 21. Day 1 is "Promoters Day," which features experts on search engine marketing issues, roundtables on advanced search engine optimization issues and panels with representatives from various major search engines themselves. Day 2 is "Searchers Day," which features two tracks designed to help Internet searchers better understand how to use the search tools available to them. There is also a special promoters track that repeats the advanced roundtables held on Day 1.

Sponsorship and exhibiting opportunities are still available, and you can contact Frank Fazio Jr,, for more information. If you are an accredited member of the press interested in attending, please contact Mary Ann Boland at

Search companies and services confirmed to speak include, AltaVista, Ask Jeeves, Atomz, CompletePlanet, EasyAsk, Excite, FAST, Google, GoTo, Inktomi, InfoSpace (Metacrawler/Dogpile), Intelliseek, iWon, Looksmart, Lycos, Mondo, Moreover, MSN Search, Netscape/The Open Directory, Northern Light, Oingo, Sandy Bay and Quiver.

You can attend either day of interest or receive a substantial discount to attend both. More details and agenda can be found below.

Search Engine Strategies


Blending Vertical Results & Other AltaVista Improvements

Over the past month, AltaVista has been busy making some under-the-hood changes to improve its search results. In particular, the service has upgraded its shopping search and is "blending" links that lead to its shopping search engine for appropriate queries into its main results. How searches themselves are processed has also been changed.

Do a search for product oriented information, such as "dvd players," "inline skates" or "computers" and you'll see the new shopping search links appear. They look similar to AltaVista's regular numbered listings and appear just before these. They usually begin "Compare Prices and Features on..." followed by your search terms, and the words "Shop Smart" also appear next to the link. Selecting the shopping link takes you to the AltaVista shopping search engine, where you can obtain product pricing from over 600 online web merchants.

"We've learned that 20 to 25 percent [of searches” are shopping related searches, but the average user may not know how to get to our shopping vertical," said Ganon Giguiere, senior director of search verticals at AltaVista. By integrating the links to shopping search in the main results, AltaVista hopes to better serve its users.

Naturally, getting people into the shopping search area also benefits AltaVista. Some of the merchants pay AltaVista to be included, but the "vast majority" don't, Giguiere said. Most major shopping search engines do have some type of payment model with merchants, and as long as there's a wide variety of major, reputable companies, I don't think users need to fear these deals.

I'm pleased to see the shopping links integrated as they are, and I'll be looking forward to watching how AltaVista integrates other links to new verticals that it plans to launch in the near future. For the most part, users would benefit by finding better specialty search tools. It's simply been a big challenge to figure out how best to direct them, as I've covered more in the separate "Being Search Boxed To Death" article (see below in the newsletter).

For the most part, I'd rather one or two targeted vertical links replace all the "dumb" links that appear in response to any query. For instance, search for anything on AltaVista, and you'll always get "Extend Your Search" links at the bottom of the page that include whatever you searched for, even if it is absurd that you would do this -- a search for "korean war" makes this area suggest things like "Shop the web for korean war" and "Searching for korean war? Find it at" Nor is AltaVista alone in having these dumb links. Removing them would eliminate clutter from the result page and ought to increase usability.

AltaVista's shopping search service lets you compare prices, but I'd also like to see product information added. It would be great to see the service expanded to pick up product reviews and consumer information from key sites, which would help it be more than an online mall.

AltaVista gathers the pricing information for its shopping service though a system it calls "scraping," which means its shopping team evaluates a web site, understands how products are listed and prices displayed, then configures its spiders to pull back the information for use in the service. It can also take product feeds from merchants, especially those who it has relationships with.

If you'd like to be one of the included merchants, it's essential that your site first offer quality products, look reputable, BE reputable and offer features such as a return policy. If you meet the prequalifications, then AltaVista will then negotiate a deal.

"We only charge for qualified traffic. We apply a conversion rate and get into a cost per customer acquired model that works for them," said Giguiere.

If you like the shopping search service, it remains accessible also from its own home page, plus the service can be customized to turn off product pictures, increase the number of results and place the "Shop Near You" section first on the page.

Shop Near You is completely new. This is information that comes from "brick and mortar" stores near your home. You give AltaVista your zip code, and it pulls back matches powered by information from This can be especially helpful if you prefer to use AltaVista Shopping to do online research, then find a local vendor with a "real" store that you can visit. In addition, auction listings from uBid for products can also be found.

I noticed some bugs with the shopping link integration. Try a search for "pc computers," for example, and rather than a generic "Compare Prices" style link, you'll get a link to a specific product. I don't think the user experience is as good, in these cases. "Cordless mouse" was another example, but fortunately, I didn't see many other instances.

The next vertical search product to appear will be news. Giguiere said will contain stories from many leading news providers. Finance, real estate and travel verticals are also planned. In all these cases, links to the appropriate vertical service will be integrated as with shopping into the main results page, appearing in response to appropriate queries.

Don't forget that AltaVista already has a variety of vertical search tools that you can reach by using the "tabs" above the search box on the results page. Within the AltaVista Tools area, you can also access specialty search features that let you search against US educational sites or US government sites.

Beyond the vertical links, AltaVista has upgraded how it processes search requests, being more expansive with what it returns. Here's a look at the new "query reformulation" changes.

Single word queries are easy -- AltaVista will return any pages it knows of that contain that particular word.

For multiword queries of between two to four terms in length, AltaVista continues to perform automatic phrase detection. This means that it looks to see if there are any recognized phrases that match a dictionary of about 500,000 phrases it maintains. These have been culled by analyzing actual web pages in the AltaVista index. If your search terms appear in the phrase dictionary, then AltaVista automatically translates your request into a phrase search. For example, if you search for:

new york

the request will be turned into a phrase search behind the scenes, even though you didn't specifically request this by placing quotation marks around the terms.

AltaVista has done automatic phrase searching like this since November 1998. However, in a twist that began in February, it now goes beyond providing only exact phrase matches and will also return pages that contain all of the words in your query, even if they aren't contained in an exact phrase. For instance, consider a search for

new york stock exchange

AltaVista will look for pages that match the exact phrase, but then it also finds pages that have all four words on them, even if they don't occur in that exact order.

Before this change, if you had searched for an exact phrase and there were no pages or few pages with that exact phrase, you would have come up with no results or only a few matches. With this change, AltaVista better ensures that its automatic phrase detection, which is helpful in many cases, doesn't leave users without results in some situations.

"The advantage of not removing documents without the phrase is that sometimes the exact phrase isn't exactly how the page describes the concept. For example, the user might be looking for information on Clinton's cybersecurity proposal. The user enters: 'president bill clinton cybersecurity,' said Vaughn Rhodes, senior director of product marketing for AltaVista. "None of the documents discussing the proposal use that specific phrase [so would be missed the old way”. The new AltaVista technique correctly returns those documents."

Finally, in situations where there are five or more words, AltaVista continues to do phrase detection, and then it will seek pages that have ANY of the words on them, rather than ALL of them.

"When users enter a small number of query concepts [such as two to four words”, they are usually looking for documents that have all of the terms in them. However, when large numbers of concepts are used [five or more”, users tend to be in a 'find stuff like what I describe here' mode, in which they don't necessarily require that every term they enter is present," Rhodes said.

This is all pretty tricky stuff. Most of the other major search engines operate simply on an ANY or ALL basis. They either find ANY of the words you request (also called OR processing) or they find ALL of them (also called AND processing). They don't try to detect phrases and alter their ANY or ALL behavior based on word count.

So far, I've covered changes to what AltaVista retrieves. However, it also is looking at the search request differently to help rank the pages it recovers. Remember how phrase detection in the past meant that only a subset of pages with those exact phrases would be retrieved? Now, even pages without the exact phrases will be listed. Webmasters, this means that your pages have a chance of appearing in response to some queries at AltaVista when they wouldn't before, because you didn't use an exact phrase on your page that someone had searched for (either knowingly or because AltaVista automatically performed phrase detection).

However, the ranking algorithm has been tweaked to help ensure that those pages with exact phrases do rise to the top of the results. It's also true generally that after this, pages with ALL of the search terms will be listed, then pages with ANY of the terms. Nevertheless, this general system still can be influenced by AltaVista's other ranking factors.

"It is not entirely accurate that we first list pages with matching phrases, then with all search terms, then with any matches. This ranking scheme is true in a general sense, but the ranking algorithm actually uses a number of different factors to determine rank. If a document with a partial match has a higher connectivity [link analysis/popularity” or page quality measurement than one that matches the exact phrase, it can be ranked higher," Rhodes said.

AltaVista clearly hopes the changes will mean better results for average web users, and most people probably will be fine letting AltaVista do the driving, so to speak. However, advanced searchers may still prefer to control AltaVista themselves by using power commands such as the + symbol or Boolean operators. AltaVista says that if such commands are used, they'll take precedence over the internal logic it tries to follow. That hasn't been the case in the past, and I haven't had a chance to look closely to see if this is indeed happening. In addition, AltaVista's power and advanced search pages have none of the behind-the-scenes processing happening.

"The query document selection technique changes apply only to the main search box, not to Advanced or Power Search. Users of those interfaces won't see any changes. In addition, if users enter into the main search box the syntax elements that were used in the previous technique [such as +, -, or quotes”, we will automatically fall back to the previous methodology," Rhodes said.

By the way, AltaVista's "word count" area at the bottom of the page used to be a good way to determine exactly how AltaVista processed your query. No longer. "The 'word count' area at the bottom of the page shows in effect how the documents are being ranked, but does not necessarily show what words were used in selection," Rhodes said. The "AltaVista's Automatic Phrase Searching" article below explains more about how word count used to operate.

Related to search processing, another relatively new feature is the ability to search within results at AltaVista. After performing a search, just check the "Search Within Results" box under the search box on the results page. Then you can do a new search just against the first set of results retrieved.

Also, be aware that if you get to the bottom of the page and want to see more results, you might accidentally select the "More Sponsored Listings" option, if you aren't careful. That brings up more advertisements from GoTo. Instead, if you want more editorial listings, you need to select any of the "Results Pages" numbers or the "Next" link. These appear near the bottom of the results page and just above the "Sponsored Listings" heading. Webmasters, this also means that even if you aren't in the top three from GoTo, you might find some traffic from AltaVista via your paid links, due to accidental or even intentional clicks in the sponsored area.

Leaving behind GoTo links, here's an update on some LookSmart-AltaVista integration:

Some webmasters noticed that for a time, AltaVista was using LookSmart descriptions for their sites. AltaVista says that its contract with LookSmart no longer allows it to do this. Inktomi does continue to use LookSmart descriptions.

Also, I reported in the past that you might find sites ordered differently in AltaVista's LookSmart-powered directory than at LookSmart itself, when comparing the same categories at both places. For example, look at this at AltaVista:

versus the same category at LookSmart:

AltaVista says the order is mainly related to when sites are added to LookSmart -- presumably, newer sites are coming first. However, this is going to change.

LookSmart now supplies in their feed to us a sorting mechanism which can allow us to sort the categories and web sites according to editor-determined order of relevance. We anticipate implementing this feature in the near future," said Josh Trapp, analyst relations specialist with AltaVista.

By the way, for all those who have asked, AltaVista says it refreshes its listings from LookSmart each day.

In yet more under-the-hood changes at AltaVista, the company says that it now has in excess of 500 million web pages indexed, up from its previous 350 million and in line with the full-text indexes other major crawlers such as Inktomi, Google and FAST. AltaVista generally hopes to revisit these pages at least once per month, if not sooner.

"Our goal for the new process, which we expect to be done by end of month, is to initially revisit every page at least monthly, with an intelligent scheme to revisit top pages or frequently changing pages much more often. However, we expect to quickly improve on that 30 day time frame. The real goal is not really stated as how often we'll revisit every page on the Internet, but probably more accurately stated as having an index that is as close as possible to what is actually on the Internet right now. Some pages are known to never change, so we'd really not need to revisit them. Other pages change daily, so having a two week schedule for those pages would be inadequate," Rhodes said.

In terms of how AltaVista will add brand new pages, things are being revised. For years, the service has operated where any page directly submitted to it would tend to appear within the index in a day or two, assuming the page wasn't flagged as spam and that too many submissions from the same site weren't received in the same day. However, for the past few months, this historical dependability has gone away.

"Normally, we have added URL submissions weekly. Recently, we have incurred a backlog. We expect to deal with the URL submissions we have on hand in the near future and are in the process of putting in place a more streamlined submission system," said Trapp.

What the new system is remains to be determined. AltaVista has long-hinted that paid inclusion is in the works, so don't be surprised to see this emerge. In the meantime, AltaVista says it remains fine to submit around 5 pages per day per web site via its Add URL page, and with luck, you should see these pages appear within a week, if the service goes back to its most recent schedule. Don't forget, the search engine will still gather pages from your site as a consequence of its own crawling independent of the submission queue.


AltaVista Shopping

AltaVista Tools

Access to the AltaVista education and government search engines can be found here, on the left-side of the page. The Power Search page is also listed with them.

AltaVista Shopping Advertising Form

Use this form to have AltaVista consider your site for its shopping area. The form assumes that you want to establish a paid relationship. AltaVista also says that sites are considered for inclusion simply on an editorial basis but has yet to explain how to submit your site for consideration if you do not wish to pay. I'll let you know if I get more details on this.

Being Search Boxed To Death
The Search Engine Report, March 5, 2001

Overview of how search blending such as that being done at AltaVista may improve the search experience for users.

AltaVista's Automatic Phrase Searching
The Search Engine Report, February 4, 1999

Describes how the word count feature used to work.


Being Search Boxed To Death

General purpose search engines are wonderful tools. You can search for entertainment, news, sports, company and many other types of information and still often find what you are looking for. Like a Swiss Army Knife, general purpose search engines often can do many different jobs. Nevertheless, your results might be better if you turn to a vertical tool. This may be the year that the general purpose search engines finally figure out a way to get the right vertical tools into the hands of their users. They've tried before, but some new efforts might succeed where we've previously had failure. More about these efforts, via the story below:

Being Search Boxed To Death
The Search Engine Report, March 5, 2001


Network Solutions Launches Inktomi Paid Inclusion

Network Solutions was Inktomi's first paid inclusion partner, announced way back in the middle of last year. Finally, that partnership has gone live, with Network Solutions now offering guaranteed inclusion in the Inktomi web index through its web site.

Inktomi has two other self-serve paid inclusion partners: US-based Position Technologies, which began offering the service in November, and Europe-based WebGravity, which began its inclusion program in January.

All offer the same basic service: inclusion of submitted URLs into the Inktomi web index within two days and revisiting those URLs on a weekly basis, for up to a year.

At Network Solutions, paid inclusion costs $30 for the first URL, then $15 for each URL after that, with up to 100 URLs allowed per order. The Network Solution program allows the list of URLs to be changed at any time. So, if you decide three months into the program that you want a completely different set of URLs included, you could change these from those you initially submitted without incurring an extra fee. In addition, URLs from more than one domain can be on the same order.

The flexibility that Network Solutions allows is great, but you are paying more to have it. At Position Tech, pricing is $10 for each additional URL, not the $15 that Network Solutions charges. Which offers the best value? It depends on your exact circumstances.

Let's say you have 10 different web sites, and you want the home page from each site listed. With Position Tech, each site will need its own account, which also means you'll have to pay the higher rate of $30 for the first page submitted to each account. Overall, you'll pay $300 to get all the home pages included.

In contrast, Network Solutions allows you to include URLs from different web sites (technically, different domains) in the same account. That means you pay the higher rate of $30 for the first page only once, then $15 for each additional page. Overall, you'll pay $165 to get all your pages listed -- almost half the Position Tech price.

Now let's say you have 50 URLs you want included. You submit them all via Position Tech, paying $30 for the first URL and $10 each for the rest of them, $520 total. To do the same thing at Network Solutions would cost $765, because each extra URL is $15 there. Clearly, Position Tech is the cheaper alternative!

Or is it? Let's assume you monitor your web site logs and discover that of the 50 URLs you submitted, only 10 of them are getting any measurable traffic that can be attributed to Inktomi partners. That means you are paying for 40 URLs but not recovering their cost in traffic. Solution? You'll need to see if you can optimize those URLs and get better results, which will take time.

In contrast, the program at Network Solutions would let you drop those 40 URLs and swap in 40 new ones. It may be that some of these will just naturally rank well, saving you from having to perform optimization work. As a result, the extra money spent at Network Solutions for flexibility might be worth it.

Here's my suggestion, at the moment. If you have many URLs from the same site you absolutely know must be included with Inktomi, then go with Position Tech -- it's cheaper, and you don't need to pay the premium that Network Solutions charges for flexibility. You can also have up to 1,000 URLs in the same account, versus the 100 URL limit per account at Network Solution (note: you can have multiple accounts at Network Solutions, however).

In contrast, go with Network Solutions if you aren't certain which pages might bring you traffic. Also, Network Solutions is certainly the leading choice if you want to include pages from multiple web sites in the same order. Then you'll get a real savings, as described above.

What about Inktomi's other self-serve partner, WebGravity? Everything I've described about Position Tech is applicable to WebGravity with one key exception -- WebGravity has pricing in UK pounds. Check the exchange rates to see if your local currency, pounds, dollars, euros or whatever, works out that it is cheaper to go with WebGravity.

Also keep in mind that you can mix and match accounts. You could have some of your URLs submitted via Position Tech and others submitted via Network Solutions. Ultimately, you get the same basic service regardless. However, do NOT submit the same URL to more than one Inktomi paid inclusion partner. Doing so is simply a waste of money.

Will Position Tech and WebGravity be able to offer flexibility in the future? Probably, but exactly when is not certain. Inktomi says that it wants to see how the experiment with URL swapping works at Network Solutions, then ultimately expects that all self-service paid inclusion resellers will have standard offerings.

Inktomi also has two other paid inclusion partners, MediaDNA and Moreover. Neither offer self-service inclusion. Instead, they provide custom solutions. MediaDNA is especially targeted at those with content locked behind password protected areas, while Moreover works with news content providers. Inktomi makes Moreover's news content available to its portal partners who want it, such as iWon. If you have news-oriented material, an inclusion deal with Moreover may help ensure that this is placed out there.

Got an order of 1,000 pages or more? Then you may want to approach Inktomi itself, as the company also works directly large site owners to work out tailored solutions.

The deal between Inktomi and Network Solutions is also going to expose the paid inclusion system to a larger audience that ever before, because Network Solutions will be offering cobranded programs for portals to use while also letting its 65,000 affiliates receive fees for referrals to the sign-up form at the Network Solutions site itself.

Cobranded programs mean that portals that make use of Inktomi data, such as iWon or MSN Search, might offer paid inclusion into the Inktomi database from Add URL pages within their own sites, maintaining their own branding and sharing revenues with Network Solutions, which will administrate the submissions behind the scenes.

"That's kind of what we were bringing to the table, the ability to distribute this through the portal channels," said Mike Cornell, product manager of value added services and products at Network Solutions.

LookSmart has run similar cobranded programs for its paid submission service with partners such as AltaVista, Excite and MSN Search, which also use some LookSmart information. Since many site owners look to the search engines' own Add URL pages as a first step in getting listed, offering paid inclusion on these pages is a wise distribution strategy.

Network Solution affiliates will also be able to post links to the paid inclusion form at the Network Solutions site and receive 20 percent of any sales. Search engine optimization companies or even anyone considering paid inclusion with Network Solutions will want to take advantage of this opportunity. It means that you can get back 20 percent of any order you place -- even if it is for yourself. Simply sign up for the program, then install the affiliate code on a page within your web site. Next, click through from that page to Network Solutions and place your order. That should then register the sale as one generating an affiliate fee for you.

A few last issues, Inktomi-wise:

Some people noticed that in mid-February, their paid inclusion pages were dropped, then restored to the index but didn't rank as well. Inktomi confirmed that there were some algorithm anomalies during that time that could have caused the ranking drops. Inktomi also said that its scoring system has recently updated, which could help explain any continuing problems. In addition, Inktomi is rolling out new relevancy and crawling improvements over the coming weeks, the company says, so expect some more bumps along the way. I'll report on the new changes in the future, once they have stabilized.

In the FAQ about doorways pages and paid inclusion at Network Solutions, the text says that "Inktomi will allow one doorway per destination." Specifically, this means that you can have one doorway page leading to any one particular content page at your web site.

In other words, you can have more than one doorway page leading to different parts of a web SITE. However, you can not have more than one doorway page leading to any single web PAGE within the site.

Finally, have you been trying to locate whether Inktomi lists any particular URL from your web site? This has always been difficult, because Inktomi has lacked a "url:" style command. However, now there's a solution -- preface the URL with "originurl:" to see if that particular page is listed. For example:


brings up that particular page on generic engineering within the Greenpeace site. Note that at HotBot, you'll need to search on this and then maybe click to the second page of results, which are powered by Inktomi, to get the correct response, since the first page may still come from Direct Hit.

Network Solutions: Inktomi Search/Submit

Information about paid inclusion via Network Solutions can be found here.

Inktomi Search/Submit Partners

Links to all of Inktomi's various partners offering self-serve paid inclusion can be found here.

How Inktomi Works

Links to past articles about Inktomi paid inclusion can be found here, including reviews of the MediaDNA and Position Tech service. I'm also working to integrate information from all the past articles into the page itself, and that should hopefully be finished soon.

Network Solutions Affiliate Program

Sign-up to become an Network Solutions affiliate here.

Moreover Inks Inktomi Deal
Interactive Week, Dec. 11, 2000,4164,2662997,00.html\

More details on the Moreover-Inktomi partnership and Moreover's new enterprise offerings.


New ProFusion Site Offers Better View Of Invisible Information

Intelliseek has launched a new beta version of its ProFusion meta search site that also combines easier access to "invisible web" information. The combination of comprehensive web search results combined with the ability to target topic-specific search resources make the developing service one of great appeal to searchers.

After certain types of searches -- especially those involving medical terms, company names, geographical locations, celebrities and other data that ProFusion believes may available in "invisible web" areas -- you'll see the page topped with a "search assistant." This is a person holding a magnifying glass, with the words "ProFusion Recommends" above. Next to the person are links designed to route you into search results from topic-specific or "vertical" search services.

For instance, if you search for "pregnancy," you'll see suggestions such as "Health Tips: Try your search for pregnancy within Health Tips." Selecting this link would bring up a new set of search results. These come from specialty sites that deal with health issues, such as,, Medline Plus and the American Medical Association.

Some information at these sites might not be available to ordinary search engines, because it comes out of databases and is only retrieved if you used a special search form when visiting the sites. Hence the concept of "invisible web." This is information that might not be visible to an ordinary search engine. However, ProFusion is specifically designed to pass along your automatically query to appropriate invisible web resources and find the information that might otherwise be missed.

In addition to routing you toward invisible web resources, ProFusion also continues to offer more ordinary search listings. For instance, the results page will often begin with any matching "Web Guides" information, which are category lists from LookSmart's human editors. For example, select the "Pregnancy, Birth and Newborns" link after a search for "pregnancy," and you'll have access to resources on that topic, which have all been reviewed.

The "Web Search Engines" section presents search results pulled back from major search engines from across the web, including AltaVista, MSN Search and Yahoo. Google is not included.

Unfortunately, there is no way to control exactly which services are queried when doing an ordinary search. Instead, if you want this control, you must use the advanced search page (use the link below the search box on the home page). The settings you choose on the advanced search page will be remembered, if you always start from it. But if you do so, you lose the search assistant suggestions to invisible web content and the Web Guide links.

Every listing in the results is followed by a "page alert" link. Select this link, and you'll be notified each week of any changes to the page. Similarly, the small "Set Search Alerts" link at the top right of the Web Search Engines portion of the results page will rerun your search each week and notify you via email if there are new or different results.

ProFusion's automatic suggestions of invisible web resources are good, but I think the real power lies in browsing topics from its home page. That's because the suggestion list at ProFusion is still growing, so I think you could miss out on resources you'd more likely find if you drill down the category levels.

For example, say you want historical information on electrical consumption in California, which has been plagued by brownouts and shortages over the past year. If I search for that -- "historical information on electrical consumption in California" -- Profusion comes back with no search assistants pointing me toward invisible web resources. However, if browse from the home page into "Government," then I discover the option to search against invisible web resources with information about commerce, US state and local government information and energy resources. All of these sound promising.

Promising, yes -- successful, no. As it turns out, the results for that query were very poor. In contrast, the Web Search results that ProFusion found for the query were excellent, bringing up a very good document from the California Department of Energy. This just goes to show ordinary search engines can and still do bring back plenty of good information, despite concerns that they may miss invisible web content.

This isn't to knock the work Profusion has done. There are great applications for invisible web searching, and as Profusion develops over time, its suggestions within the ordinary search results should improve.

In the meantime, do consider browsing topics from the home page, if your ordinary results don't seem to work. The ability to hit patent databases, music reviews, legal resources and other great specialty sources can all be found. Not only can you locate them, which was always possible through Intelliseek's web site, but you can also search against several of them in a category at the same time.

It would be nice if it were possible to search for matching invisible web categories more easily from ProFusion. For example, search for "census" at, and you are shown multiple resources related to the term. At ProFusion, you have to search for "census," then scroll to the bottom of the results page and go through the extra step of selecting the "Invisible Web Sites" link at the bottom of the page. Even this fails to show you which invisible web sites match the query. Instead, it just runs your query against whatever invisible web sites ProFusion considers relevant.

While the ProFusion site is oriented toward consumers, but Intelliseek really sees it as a way to show companies how it can create custom vertical and invisible web search solutions for their own research needs. About half a dozen Fortune 500 companies are currently using custom versions of Intelliseek's technology, in deals that will be announced publicly in the future, the company says.

"Rather than being the B2C site that we were trying to monetize before, it's no longer the case," said Sundar Kadayam, Intelliseek's chief technology officer. "Nevertheless, we found we have a reasonably loyal user base, so with ProFusion, we can give them the latest and greatest technology and not annoy them with banners."

Banners may not be present, but sponsored links to occasionally appear on the results page. Up to two ads appear near the top of the results, in reverse boxes and are clearly marked as "Sponsored Link." Try a search for "casino," "movies" or "cars" to see them appear. I didn't hear back from Intelliseek on how sites can appear in these before the newsletter went out, so I'll pass along details in the next issue.

Finally, if you have a resource that you think should be included in Profusion, consider submitting via the Add URL page at, below.

ProFusion Beta

The site is expected to lose its beta status in the next six weeks or so and replace the existing ProFusion service, below.


The older version of ProFusion remains live but offers only meta searching abilities.

Intelliseek's existing catalog of invisible web resources, but it doesn't integrate meta search, as does the ProFusion beta.


BrightPlanet's catalog of invisible web or deep web resources.

I can search the 'invisible Web.' Here's how you can, too
ZDNet, Feb. 8, 2001,10738,2683349,00.html

Another review of the new ProFusion service.

Search Links: Invisible Web

Further resources and information about the invisible web.


More about Intelliseek's enterprise solutions can be found here.

Invisible Web: Suggest A Site

Use this form to suggest your site as an invisible web resource for and ProFusion.


GoTo Increases Prices, Appears At iWon

The days of 1 cent clicks at GoTo's US service are gone, made extinct by a price increase put into place on March 1, which raised the minimum bid to 5 cents. A new $20 minimum charge per month has also been implemented. GoTo advertisers learned of the changes in an email sent to them on the day the price changes took effect.

"Over the last 2 quarters, we have signed a number of big traffic deals with premier web sites which has resulted in even more targeted traffic to your site. These traffic deals have increased our costs and as a result we're implementing a modest price increase," the letter from GoTo began.

Reaction to the changes at the Search Engines Forum discussion area was mostly negative, with only a few seeing the change as having no impact on them or being positive in some way. GoTo said it has received several hundred emails about the increases, almost all expressing displeasure at the raise in minimum bid price. In contrast, the $20 minimum monthly hasn't attracted great concern, the company said.

Higher prices and minimum spend amounts should generate more revenue for the service, assuming they make up for any losses from customers abandoning it due to the hikes. GoTo says that about 25 percent of its clicks are from 1 cent listings, so the potential for customer dissatisfaction is broad. However, as of last Friday, only one official loss was reported.

"We had one advertiser I'm told that pulled his business. My understanding is that he was spending less $5 per month," Ted Meisel, GoTo's president and CEO.

Ironically, losing that type of low-spending customer may actually generate revenue for GoTo by saving money. The service must review each listing submitted, to ensure that it meets minimum relevancy standards. It's under even more pressure to ensure that these listings are somehow related to bidded terms now that its listings are distributed to major partners such as AOL Search, the company says. The time to do this for the customer who spends $5 per month on 1 cent listings and $5,000 per month on $1 listings is the same.

"There's a certain minimum cost in serving a business," said Meisel. "We're not asking for a listing [setup” fee. We're just asking for you to spend a certain amount of money with GoTo."

Those with existing bids of 5 cents or less will be allowed to keep them through September 1 of this year. However, any change in the bid amount will cause the bid to lose its protected "grandfathered" status. In other words, if you were in position 2 with a 1 cent bid and there was someone in position 1 with a 2 cent bid, you might wish to bid 3 cents to jump to the top. However, GoTo wouldn't allow this. Instead, you'd have to start off with a bid of at least 5 cents.

Grandfather status is not lost if you simply want to change a listing's title or description. Also, existing accounts are exempted from the $20 per month minimum spend through September 1.

With new accounts, if you don't spend at least $20 per month in clickthrough charges, then GoTo will charge the difference to meet the minimum spend. For example, say your clicks cost you $7.50 in a particular month. GoTo would then charge the difference, $12.50, so that you would be billed $20 in total. If your clicks cost $20 or above, then there are no additional charges. Also, if your account is inactive for an entire month (i.e., you don't have any active bids), there will be no minimum charge.

In other GoTo news, its seven top paid listings are now appearing at iWon, another major search site. The deal hasn't yet been announced publicly, but the links can be found at iWon on the right-hand side of the screen, under the heading, "Partner Search Results."

GoTo also launched the concept of "Premium Listings" last month to help advertisers more easily identify the positions likely to bring them more traffic because they appear on many distribution partners. Premium Listings are simply the top three bids for any search term. They are said to get twice as many clicks as GoTo's other listings (4 and below), which isn't surprising given that these other listings don't appear in places such as AOL Search and AltaVista.

Finally, the UK service has debuted a UK-specific search term suggestion tool. It can be found from within the GoTo DirectTraffic Centre, off the Add Listings tab. You can also reach it directly via this URL: The tool shows top terms as derived from queries conducted on GoTo partner sites, such as Freeserve and Ask Jeeves UK.

FYI, the UK service has had a minimum bid of 5 pence per click (roughly 7.5 US cents) since it was launched at the end of last year. There is currently no minimum monthly spend, but given the US change, this could come in the future. Should that happen, then existing accounts at the UK service might be exempted from a monthly spend for several months as has happened with the US site.

To protect yourself, you might then consider opening an account at GoTo UK now, if you've already been considering this. There is a 10 pound account sign-up fee, but that has been waived through the end of this month.



Frequently Asked Questions for Advertisers

More information from GoTo on minimum bids and Premium Listings.

GoTo Price Increase
Search Engine Forums, March 1, 2001

Mostly negative reaction to the GoTo price hikes.

Paid Listing Search Engines

If you are after traffic, any type of traffic, for 5 cents or less, then some of the pay for placement services here are worth looking at. However, none of them have the distribution on major search engines to match what GoTo has. However, several of them are carried by meta search services.


Yahoo Loses International Directors & Other Search Financial News

The heads of Yahoo Europe, Asia, Canada, Korea and Yahoo's China operations have all resigned recently. Yahoo CEO Tim Koogle told the Financial Times of London last week that these were "lifestyle choices" rather than signs of problems with Yahoo's international operations. Some analysts disagree. A recap of stories on the topic, along with some other interesting stories related to search engine finances, can be found via the URL below.

Yahoo Loses International Directors & Other Search Financial News
The Search Engine Update, March 5, 2001

Password Finder


Top Search Terms Of 2000

I know -- the New Year has come and gone, but there were a number of good stories and resources I've saved up on top search terms from last year that I wanted to share. Also, search engines are starting to be more forthcoming with what people are searching for, which is great data for anyone who is interested in what's popular on the web. A roundup can be found below:

Top Search Terms Of 2000
The Search Engine Report, March 5, 2001


Will P2P Search Replace Search Engines?

Will Napster-style peer-to-peer searching mean an end to search engines? Despite the continuing hype, I doubt this will be a replacement for web-wide searching. There are strong advantages to a centralized system, not the least in terms of dealing with spam. I do think P2P has promise for intranet solutions, especially in situations where you have trusted data that you want to find on a few hundred or thousands of machines. But P2P for full-text search against a billion or more URLs? Unlikely. Below are two new articles, pro and con on P2P search, plus my last article on this topic.

Search project prepares to challenge Google, Feb. 26, 2001

Pandango is an interesting idea. By examining what we visit, it would then select the more popular documents in response to a search. This is akin to the bookmark-based search engines that were all the rage this time last year but which have failed to take off. It's also somewhat similar to the Alexa "Related Pages" technology already built into Internet Explorer and Netscape Navigator but which I've yet to encounter many people using. Aside from the big disadvantage of requiring users to download software, there could also be privacy concerns. Long story detailing the project. Despite the company's suggestion they are the "third generation" of web searching, when Pandango launches, it will really be a first generation service. The technology may be a move in a new direction, but until it has been tried and tested against a web-wide audience, it can't really grab that claim to be the next rung on the evolutionary ladder of web search.

P2P Goes in Search of 'Doogle'
Wired, Feb. 16, 2001

P2P is unlikely to replace centralized web search engines, is the conclusion from this major P2P conference.

More Than Just Music Search
The Search Engine Report, June 2, 2000

Napster has obviously changed since this was written, but the contrast and comparison of centralized web search to the potential of P2P search remains relevant.


LookSmart Submission Data Was Left Vulnerable

If you've never submitted to LookSmart, don't worry about this story. If you did submit, there's a very small chance someone might have obtained your phone number and email address. This is because information in LookSmart's submission queue was been left open to public, giving access to some details about those who submitted sites to the service. Credit card data, however, was not exposed.

LookSmart Submission Data Was Left Vulnerable
The Search Engine Report, March 5, 2001


Google Acquires Deja Newsgroup Service

Google purchased the Deja newsgroup archives last month and is now running them within its own site, which you will find here: Despite saving the service, Google came under criticism from hardcore Deja users upset about lost functionality during the transition period. I'll bring you a round up of articles on this topic and more details from Google itself by Tuesday, at the URL below. It will also have short details on the new Google country-specific editions that are emerging and the i-mode service for wireless web users in Japan.

Google Acquires Deja Newsgroup Service, March 6, 2001


Bits & Pieces: Yahoo Sponsored Listings Warning, Frames Support Improves, Excite's "Zoom In"

+ I heard recently from someone who found that after applying for a Yahoo Sponsored Listing, his regular listing was altered -- causing him to drop for some key phrases. I'll be following up on this, but be forewarned -- applying to the program may mean that your regular listing gets reviewed and changed in ways you may not like.

+ Google is now supporting frame links, and I believe Excite is also doing so (I'll reconfirm this shortly). If so, that means a majority of the major search engines do support frames for the first time. Be aware that they will still index your frames pages out of "context," so the restoration workaround described on may still be helpful.

+ Watch for a new search refinement feature coming from Excite this month. Called "Zoom In," it lets you select related search terms to refine your query from a pop-up box.

Search Engine Articles

Revving Up the Search Engines to Keep the E-Aisles Clear
New York Times, Feb. 28, 2001

A look at the challenges in improving search at shopping sites.

RealNames upgrade stalls refunds, Feb. 26, 2001

If you requested a refund on a RealNames purchase, you might still be waiting due to technical difficulties. Problems are supposed to be resolved by this week.

Ask Jeeves eyes large enterprises
InfoWorld, Feb. 23, 2001
Ask Jeeves is to place its knowledge management tools in software form, so that enterprises can maintain their own answer databases.

FirstGov Web portal finds a home in new administration
Government Executive Magazine, Feb. 23, 2001

Update on development work at the US government's search site, FirstGov.

Portals fight over 18- to 24-year-olds
Upside, Feb. 16, 2001

To all those who want to know what demographics different portals appeal to, here are some figures on the youth market, and how the major portals are targeting them.

Bidder's Edge pushes Web site over cliff, Feb. 15, 2001

The auction meta search service sought that fought against eBay and lost is now closing.

Search results becoming more commercial
San Jose Mercury News, Feb. 15, 2001

Review of trend for paid listings at search engines, with comments from different services.

List Info

How do I unsubscribe?
+ Follow the instructions at the very end of this email.

How do I subscribe?
The Search Engine Update is only available to paid members of the Search Engine Watch web site. If you are not a member and somehow are receiving a copy of the newsletter, learn how to become a member at:

How do I see past issues?
Follow the links at:

Is there an HTML version?
Yes, but not via email. View it online at:

How do I change my address?
+ Send a message to

I need human help with my subscription!
+ Send a message to DO NOT send messages regarding list management or membership issues to Danny Sullivan. He does not deal with these directly.

I have feedback about an article!
+ I'd love to hear it. Use the form at