The Search Engine Update, Jan. 9, 1998, Number 20

About The Update

The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch,

Please note that long URLs may break into two lines in some mail readers. Please cut and paste, should this occur.

In This Issue

+ Offline Articles
+ Site Updates
+ Alexa: Searching Serendipity And More
+ Lycos Adds New Features, Reorganizes Suggested Links
+ Mining Company Illustrates Yahoo Limits
+ GeoCities Lands Partnership With Second Search Engine
+ HotBot Breaks 110 Million Pages Indexed
+ AltaVista Adds People, Business Search
+ WebCrawler Changes Algorithm
+ Search Engine Articles

Offline Articles

News Robot Leads To Linking, Indexing Dispute
The Search Engine Report, Jan. 9, 1998

News search engine News Index crawled The Sunday Times web site, and the paper complained. The situation was resolved peacefully, but the article highlights the possible importance of the robots.txt file where a case might go to court.

Fake Yahoo "Award" Message Scam, Virus Scare
The Search Engine Report, Jan. 9, 1998

In Dec. 1997, some users were told they been exposed to a virus (not true), while 100 or so were victims of a scam involving the Yahoo name.

Site Updates

The following is new in the main site:

News Search Engines

A list of sites devoted to crawling news sources, to provide timely results for current events searching.

Search Engine News

Alexa: Searching Serendipity And More

I've been meaning to write about Alexa for some time. Numerous people had messaged me raving about how wonderful it was, and I finally found time to install it during December. Now you can add me to its many fans.

After installing Alexa, it appears as a small toolbar that floats over your browser. There are two main areas, "Where You Are" and "Where To Go Next."

The Where You Are menu provides a wealth of data. When you connect to a web site, Alexa displays key information about that site within the menu, and you can drill down to get even more information. All of it is incredibly useful.

Alexa immediately displays who owns the site and how well it is rated, in terms of overall traffic. The ownership information is pulled from InterNIC records, so only .com, .org, .edu and .net sites are reported. Efforts are being made to expand this to international registries. Traffic data is determined by analyzing requests found from key Internet caches.

The traffic data alone makes this a gem for any web marketer. People are dying for decent figures about how popular different web sites are. Alexa can give you a traffic estimate for all sorts of web sites, big and small. When building reciprocal links, it's a perfect tool to sort out which sites are particularly important.

Click next to the window, you're shown even more information in a drop down box. The site ownership data is expanded, to provide a street address and phone number, if listed. You're also shown how Alexa users have voted for the site, plus a Yahoo Internet Life review rating and a RSAC rating, if either is available.

Voting is somewhat helpful. If you like a site, you vote for it. Don't like it? Vote against. Alexa records the votes and shows them to all users. You can vote once per site. This doesn't seem to be the most popular feature, since most sites have only a small number of votes, either way. But as a general guide, it's interesting to see.

Click in the drop down box, and Alexa really shines. You're taken to an expanded page of data collected about the site, complete with helpful explanations. This includes information on site freshness and responsiveness, which Alexa determines when it spiders the site.

To see this page without Alexa, simply enter:

where you replace SITE with the web address of the site. Don't include the http://, and keep in mind that at the moment, sites may be listed with or without www. Be sure to check both. For example, Yahoo has two pages:

All this information can really help you determine the quality of a web site, something useful when anyone can put up an attractive site yet not be so attractive behind the scenes.

To illustrate this, Alexa CEO and co-founder Brewster Kahle, used a search engine to display some travel sites. Clicking through to one site, there were pitches for low airfares. The Alexa data showed the site as belonging to an individual with a PO Box, and Alexa voters overwhelmingly did not like it. That information can be helpful in deciding whether to do business with an online merchant.

Alexa wants to increase partnerships such as those announced last month with Yahoo Internet Life and VeriSign to greatly increase the amount of data displayed on the site information pages. He sees this compilation of trustworthy data as crucial in helping users go forth on to the web with confidence.

"As the number of sites escalates, I think its going to get harder and harder to know, "Is this someplace I want to be," Kahle said. "If we don't have a good way to answer that for people, they'll stick with only a few sites."

Alexa's also a tool of discovery. Its "Where To Go Next." window displays links that it thinks are related to the page you are visiting. I found it to be consistently on target.

The links that appear come out of Alexa watching how people move between sites. It analyzes the relationship to determine appropriate related links. Some human assistance is also involved.

You can add to this list, but the process of "one way" and "two way" linking is somewhat confusing. Alexa hopes to make this easier in the future.

Alexa emerged from the Internet Archive project, an effort to crawl the web to preserve past pages. "Snapshots" of the web are taken every few months, and one of the toolbar options provides the ability to tap the archive and retrieve old or missing web pages.

For example, if you visit Excite and click on the archive button, you can see the home page from several months ago. Those without Alexa can do this for any page by entering:

Replace SITE with the address you are looking for, without http://. For Excite, it would be:

Alexa also offers the ability to search the Encyclopedia Britannica Online, though this may change to the Britannica Internet Guide, which lists web sites. That would be very useful.

Alexa is a free service, supported by ads that appear seamlessly in the various places on the toolbar. I never found them a problem.

Alexa's still in its initial release, so there are some refinements needed. One bug is that Alexa kills the ability to open a new window from a link with Internet Explorer 4. This is supposed to be fixed in the next release in late January.

I've never been a fan of things cluttering my browser real estate, but Alexa's data is so good that I'll tolerate it. Nevertheless, it would be great for either Microsoft, Netscape or both to integrate Alexa functionality directly into their browsers -- let's see who gets there first.

Alexa is available for Windows 95 and Windows NT. Windows 3.1, Macintosh and Unix versions are planned.


Alexa Index Request Form

If your site reflects no data, Alexa may not have crawled it. Use this page to be added.


Lycos Adds New Features, Reorganizes Suggested Links

Lycos has added some additional search enhancements, including reorganizing the way it handles suggested links.

Last month, I wrote of how the service was ranking its own "Web Guides" first in response to queries such as "entertainment." I also noted one case where Microsoft's CarPoint site was being ranked first, in response to "autos."

At the time, Lycos said it was still experimenting with how to handle these types of preferential listings. There are some good reasons for using them, though the CarPoint link might create concerns of a mixture of editorial and advertising content. See last month's article (link below) for more details on these issues.

Now when you search, links to relevant Lycos web guides appear in the "Related Topics" section of the results page, not in the actual search results.

For example, a search for "travel" lists the Lycos Travel Web Guide. Additional links ask if you wish to perform specialty searches on the topic, such as to find pictures, or see matching sites in the Lycos Top 5% directory.

A link to Barnes and Noble's book sales database also appears in this area, similar to how retailer links appear in other search engines. For example, a search for "tom clancy" brings up results on Yahoo, accompanied by a small box prompting you to find related books at Amazon.

The key is that these listings are delineated from the actual search results. That's what was so unusual with the CarPoint listing last month. It was a paid placement appearing directly in the search results, something that has not happened with any search engine since Open Text's experiment in mid-1996.

It's important to keep in mind that Lycos did say they were experimenting with how to handle this, and true to their word, at the public launch of the new changes at Internet World last month, that link was moved out of the regular search results.

Now, the link appears at the bottom of the "Related Topics" with a bullseye symbol and the words "BULLSEYE!" next to it. It also appears in the suggested links above this, though Lycos said this is a glitch that will be corrected.

"We definitely do not want to put paid things under the related topics," said Rajive Mathur, search manager for Lycos. "The non-bullseye links under "Related Topics" are not ad-supported, fee-based links."

The bullseye links, which Lycos calls "pop-up text," will be available on any keyword to any advertiser within the next week, Mathur said. At the moment, only Microsoft is running them.

In other changes, Lycos is running a small "Browse Topics" box that appears on the left-hand side of results pages. This box is meant to suggest other searches related to the original query. For example, someone searching simply for "travel" is prompted with a suggestion to search for "travel information" or "travel agents." The result is to bring people who search too generally closer to what they may be looking for.

At the bottom of the search results is a new "Refine Your Search" box. This allows you to search within a set of results, a completely new feature for Lycos. For example, if you had searched for "travel agents," entering "newport beach" would sift through the results to find pages that also contain those words. That can increase your odds of finding an agent in that city.

Below the Refine box is a "More Ways to Search" section, with three links: to search for your query within page titles, within a URL or within a particular web site.

You can also go directly to these specialty searches by clicking on "Search Features" from the home page or via the address below. The URL search will be of particular interest to webmasters wishing to check how their various pages are listed. Previously, the only way to do this was to query each page individually, using the Lycos Check URL feature.

The search within a site feature is also handy for those who wish to find a page at a site that does not have its own internal search engine

Unfortunately, the URL search and site search features aren't working perfectly. Some pages that are listed may not be found, and sometimes the searches aren't processed at all. Lycos is looking to correct the problems.

There are a few other changes that pre-date this latest round, but are worth mentioning in the context of the new additions.

Lycos has continued to add items to the drop down box that appears next to the search box. By default, it will cause you to search for web pages. However, you can choose to search for weather information, Reuters News and other options. "Personal Homepages" is a search of free web space provided to GeoCities homesteaders.

On the front page, "Advance Search" takes you to the Lycos Pro interface. This allows you to have much more control over the searches you wish to perform. By default, the HTML version loads. Here, new "match strength" options were added last October.

"Good match" looks for your first term before your second term. "Near match" tries to find terms with 25 words of each other, in any order, the same as using the NEAR command. "Close match" does a near search, but the terms have to appear in the order entered. "Strong match" finds pages where the terms are adjacent to each other.

Other options are phrase search, which looks for the terms as a phrase, natural language, which treats operators such as OR as regular words, not commands. Finally, you can search for all or any terms.

The small Power Panel tab brings up an HTML version of the Lycos Pro Power Panel, which provides controls over the actual ranking algorithm. The Java version also remains. Just look for the links, to use it.

Finally, in the next week or so, expect to see Lycos debut two new search features: Sort By Site and More Like This.


Lycos Search Features Page

Lycos Pro: Advanced Search Page

Lycos Adds Predetermined Listings
The Search Engine Report, Dec. 4, 1997

Lycos Launches Lycos Pro
The Search Engine Report, July 2, 1997

More information on using the Lycos Pro Power Panel


Mining Company Illustrates Yahoo Limits

As a general rule, Yahoo lists web sites under only one or two categories. By limiting listings, Yahoo hopes to ensure that its users are not overwhelmed by seeing the same sites everywhere within the service.

Like all rules, this one gets broken. Sometimes Yahoo editors feel a particular section of a web site deserves its own listing. Sometimes they think that a site deserves to be in several categories.

The Mining Company has struggled unsuccessfully to get Yahoo to break its rule for the many sites that it operates. The situation illustrates problems that Yahoo, webmasters and searchers all face in the effort to classify the web.

The Mining Company launched with great fanfare in April 1997. It was widely heralded as a new navigation service, a competitor to Yahoo and other search services. Its own press releases touted the fact it would help people find what they were looking for better than search engines.

In reality, this is a bad description of the service. With a search engine, you'd typically search for a topic, then "exit" to one of the destinations listed. In contrast, the Mining Company is a collection of destinations. There are over 500 Mining Company sites, ranging from Mark Twain to Southern Cooking, each run by a different "guide." The sites are not just a collection of links. Most of them are full of information on their topics.

The guides behind each of the sites are attracted to publishing via the Mining Company for a variety of reasons. The service provides hosting, easy to use templates, chat areas and other functions, as well as handling ad sales. That's a relief for many people. They could run a site on their own, and some of them did so before the Mining Company existed. But by joining up with the Mining Company, they can concentrate on their content.

As you might expect, guides began submitting their Mining Company sites to Yahoo shortly after launch. They were told that Yahoo would not list them because their sites were really "subpage submissions." The main listing for the Mining Company would have to suffice for all the sites, Yahoo told them.

"For large, substantive, comprehensive domains like yours, we continually struggle with when to break out the individual parts and separately list them. We'll continue to think about that in your case, but for the foreseeable future we really believe that by providing a prominent, well-described listing for your main hub page, we strike the best balance between serving our users and managing our limited resources," the Mining Company was informed, by Srinija Srinivasan, who oversees the listing process.

The Mining Company is not the first site to encounter this problem. Many web publishers have sites that simply cannot be properly classified with two categories and 25 words. But the Mining Company is a useful for examining the impact of this problem on search engine users, because of the overall excellence of their sites.

It's fair to say that many of the Mining Company's sites are the best on the Internet for their particular topics, as witnessed by the many awards individual sites have won. In some cases, they may be the only comprehensive sites on particular topics. But the Mining Company's "hub" listing does not, and cannot, encompass them at all.

To see this in action, consider what the guide producing the Mining Company's Christianity site was told:

"The URL you submitted is one part of a larger site that already exists in Yahoo After reviewing both the submitted URL and the existing URL that encompasses it, we've determined that the existing listing is adequate for users to find your site."

The Mining Company is listed under two categories, and Yahoo has not provided it with a description, feeling the context of the categories summarizes the site well enough. Let's examine that hub listing:

Business and Economy: Companies: Internet Services: Search and Navigation
Mining Company

Computers and Internet: Internet: World Wide Web: Searching the Web: Web Directories
Mining Company

Now imagine someone comes to Yahoo and searches for "christianity." Since that term is not associated with the listing, the Mining Company site will not appear. Nor will even the categories be listed, as the word does not appear in them. That means that Yahoo visitors will have no chance of discovering the Mining Company's Christianity site.

Another example is the Mining Company's Personal Web Pages site, one that I know well. It provides advice on getting the most out of the free web space offered by various vendors. There aren't many sites like it. A search for "free web pages" in Yahoo won't find it. You discover places that offer free space, but no sites are listed with advice on the many things to be aware of when using free web space.

So aren't users missing out because of this policy, with the Mining Company specifically, and perhaps with other sites generally? Yahoo's Srinivasan responds:

"While I agree that for the user searching in Yahoo for "christianity," it's perhaps less than ideal that they won't see the Mining Company's Christianity site in our Christianity category, [but” that same user also won't see Excite's Christianity Channel or Infoseek's Christianity category.

We simply can't re-index these sites within our own index; it would be practically infeasible and the user experience would suffer. I don't think the user wants to go to Yahoo's basketball category, see links to Excite's basketball area, Infoseek's basketball area, the Mining Company's basketball area, etc., then go to Yahoo's philosophy area and see those same analogous links again...just as we wouldn't expect Excite or the Mining Company to point to every analogous Yahoo category from every one of their categories."

In part, Yahoo is considering the fact that there are literally thousands of sites out there that may wish to have sublistings. There's fear of a "slippery slope" if they become too liberal. "If we start, where do we stop," Srinivasan said.

To this, the Mining Company notes that there are plenty of sites that are listed beyond the main hub.

"There are many examples of other sites with multiple Yahoo category listings, including: ZDNet, The Rough Guides, Argus Clearinghouse, Citysearch, ESPN Sportzone, TV Guide, GeoCities, all the TV networks’ TV show sites, Excite’s various services, ISP-produced features and home pages and many, many others," wrote Jacob Levich, the Mining Company's Director of Guide Mentoring, in a letter asking Yahoo to reconsider its decision.

Srinivasan offers various explanations, not the least being that Yahoo constantly makes judgement calls. With the Mining Company, the consistent branding throughout the site gave it the nudge toward a central hub-only listing, while with GeoCities pages are radically different from each other, without GeoCities branding throughout (though this was before GeoCities recently implemented pop-up ads that appear when personal pages are accessed).

"The Mining Company is not an invisible ISP. They are a very visible support network," Srinivasan said. This is consistent with what she in response to Levich's letter: "The design and interface of these sites clearly indicates that they're each a part of a larger whole, with the consistent navigation frames and look and feel running throughout. We think this is analogous to many other cases where we choose not to break up the sites."

Srinivasan pointed out to me another prominent site where sub-sites are not listed, for exactly the same reason. She didn't want the company publicly named, but the example is similar with the Mining Company's situation.

Of course, many of the ZDNet sub-sites listed in Yahoo have common branding but still get in. MSN is similar. Its Car Talk site, Encarta site, CarPoint site and numerous others are listed. They do look different, though all feature a prominent drop-down box allowing navigation between MSN sites.

Then there's Yahoo itself. It has numerous sublinks to its own content, from Yahoo Classifieds, to Sports, to Lottery Results, each regional guide and more. Shouldn't there simply be a main hub listing for the service?

"It's unabashedly our guide," Srinivasan said, explaining that Yahoo would be dumb not to promote itself. But it's not just self-interest, she adds. "If we're going to bother to make a whole sports area, we had better think that it's the best sports area out there."

In the end, the situation points out the delicate balancing act that Yahoo's editors face when they consider how to list sites. They have some general rules, but unlike a library, there is no classification system set in stone. Instead, the consensus of the various Yahoo surfers acts as a living, ever-changing system.

"There are no perfect solutions to this (or at least if there are, we haven't figured them out yet!), there are just partial solutions that are practically feasible, competitively sound, and maximally beneficial to the user. All of these factors get weighed in, and we try to decide how to make the call," Srinivasan said.

For search engine users, that means that Yahoo should be one of several search tools used. By using several services, you may find some important sites that might otherwise be missed.

For web publishers, it's another illustration that depending on Yahoo for traffic is a dangerous strategy. You may not get in. If you do, you may not be classified as you think you should. Descriptions and categories are sometimes revised, causing sites to lose traffic. Smart sites learn there's "life after Yahoo" and embark on link building and other Internet publicity efforts.

For the Mining Company, and in particular for its guides, it's unfortunate that the decision has gone against them, at the moment. They aren't being judged on their content. Some sites were listed independently before joining the Mining Company. Now, as part of the organization, they don't qualify. With so many exceptions being made elsewhere, it's somewhat surprising Yahoo feels the Mining Company's sites should be excluded -- especially as they readily agree to their quality.

It's important to remember, however, that the guides have gained the efforts of an organization that can advertise itself and its sites in other ways. The Mining Company can, and has, gone well beyond Yahoo to build up its traffic.


Mining Company


GeoCities Lands Partnership With Second Search Engine

Late last year, GeoCities and Lycos began a partnership promoting each others sites with reciprocal services. Now GeoCities has received a $5 million investment from Yahoo, along with another cross-promotion agreement.

Under the Yahoo deal, announced Jan. 5, those who have become "members" of Yahoo by signing up for free email, chat and other services will now also receive free personal web pages in GeoCities. In turn, new and existing GeoCities members will receive Yahoo membership benefits.

Yahoo will also be promoted throughout GeoCities, with links to appropriate categories and service. This won't affect the Lycos deal, however.

"GeoCities will continue to build on its relationship with Lycos, including providing access to the Lycos-created search catalog of GeoCities as presently integrated throughout both services," according to the GeoCities announcement.

In the Lycos deal, GeoCities visitors can either search the site or the web via a search box that taps into Lycos. The Lycos logo is associated with the box. Additionally, visitors to the Lycos service can choose to search "Personal Homepages," which is really a search specifically of GeoCities free web pages.

Yahoo Press Release

GeoCities Press Release

Softbank, Yahoo fund GeoCities, Jan. 5, 1998,4,17791,00.html

Search Engine Notes

HotBot Breaks 110 Million Pages Indexed

HotBot announced on Dec. 11 that it now indexes more than 110 million web pages, making it the largest search engine. AltaVista remains a close second, at 100 million.

HotBot last increased its size to 80 million pages in September. It has increased its size again to keep up with the growth of the web, estimating there were 175 million web documents as of Dec. 1997.



AltaVista Adds People, Business Search

AltaVista partnered with Switchboard on Dec. 29 to provide people and business searching directly from the AltaVista service. To access the new features, select from the small tabs directly above the search box.



WebCrawler Changes Algorithm

You may have noticed that WebCrawler's search results have become strangely similar to Excite's. In response to some queries, many of the same documents may appear in the top listings on both search engines.

This is due to a change in December, when WebCrawler stopped using retrieval technology from PLS and shifted over to using Excite's own technology.

Excite and WebCrawler continue to have their own spiders and indexes, but the extremely small size of the WebCrawler collection helped the similarities.

"Because WebCrawler is only two million URLs, there’s a tremendous amount of overlap of what’s the most relevant results for the user," said search product manager Kris Carpenter.

Over the next few weeks, work will be done to help restore WebCrawler to its old self, Carpenter said.

Also, Excite's index has not been updated as regularly as normal, due to some ongoing work. As a result, some sites submitted in December may not yet be listed.

Excite says they should appear within the next week or so. If not, try resubmitting. If you still see no appearance after about three weeks, get in contact via Excite's feedback address.



Search Engine Articles

It may be 'Yikes!' for Yahoo
MSNBC, Dec. 1997

An analyst explains why he thinks Yahoo's stock is overvalued and due for a fall. One note: he misses out Yahoo-competitor Lycos recently showing a profit, when calling Yahoo a search engine rarity for not losing money.

Niche play pays off for Yahoo
MSNBC, Dec. 24, 1997

Excellent stats on how much traffic Yahoo's "niche" or channel-oriented areas such as news and classifieds generate in comparison to leading sites devoted to these topics. Answer: a lot. Also interesting quote from Excite on how channel activity is now outweighing search-only traffic.

Search engines, online services meld in 1997, Jan. 6, 1998,4,17804,00.html

A recap of how AOL, MSN, search engines and others have all moved toward a common service model.

Sponsor Messages

Below are sponsor messages that ran in this month's issue of the Search Engine Report, which may be of interest to Search Engine Update readers.


When a prospect does a search for a keyword related to your products or services, do you appear in the top 10 or does your competition? Submitting alone does nothing to insure good visibility. WebPosition is the first software product to monitor and analyze your search positions in the top search engines. WebPosition has been compared to similar "services" on the web and has been overwhelmingly voted the best and most accurate tool for search position management.

With WebPosition you'll know your exact positions for an unlimited number of keywords. You'll know if you drop in rank. You'll know when a search engine FINALLY indexes you. You'll know when you've been dropped from an engine. You'll even know how you rank in relation to your competition!

Try WebPosition yourself for FREE at:


Who is visiting your site? Where are they coming from? What keywords are they searching for at search engines?

Maximized Software FlashStats can help you answer these questions, and provide more information about your web site. FlashStats log analysis combines affordability with speed, ease of use and comprehensive information.

All major log formats are supported, and 12 different reports can be generated from any browser, anywhere.

FlashStats is an ideal solution for ISPs needing to provide statistics for their customers; webmasters at small- to medium-sized sites; users needing to analyze log files provided by their ISP; and intranet administrators needing to provide secure multi-user access to reports.

Maximized Software is currently offering the subscribers of Search Engine Report a 15% discount on FlashStats. Download your free 30 day trial copy at today!

More detailed product information can be found at:

Contact Maximized Software directly at (888) 629-7638,


======= Highly Recommended by Experienced Media Planners ========

The first legitimate search engine position improvement system.

Example Search Engine Positions "Before and After"

You do nothing but handle point-of-sale for unique visitors.
We code, submit and maintain your listings in high relevancy
with hands-off your website.

Watch your search engine positions improve after start-up.
No risk, pay only for actual visitors generated.
You'll be pleased with results.

ClientDirect performs search engine optimization like no other!
Complimentary 15,000 click-thru's with
Annual Corporate On-line Media Package.

================== ==================

End Notes

To unsubscribe from this list, send a message to, with the following in the subject field:

unsubscribe searchupdate

If you have problems, just send a message to

This newsletter is Copyright (c) Mecklermedia. It should not be distributed. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at: