The Search Engine Report October 4, 1999 - Number 35

October 4, 1999 - Number 35

By Danny Sullivan
Editor, Search Engine Watch

About The Report

The Search Engine Report is a monthly newsletter that covers developments with search engines and changes to the Search Engine Watch web site,

The report has 78,000 subscribers. You may pass this newsletter on to others, as long either part is sent in its entirety.

If you enjoy this newsletter, consider showing your support by becoming a subscriber of the Search Engine Watch web site. It provides you with some extra benefits and access to some exclusive materials and articles. Details can be found at:

Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.

In This Issue

+ General Notes
+ FTC Steps In To Stop Spamming
+ Improved MSN Search Launches
+ New AOL Search Unveiled
+ Google Gets Out Of Beta
+ Go Guides Goes Live
+ Northern Light Launches Ads, Alerts Service
+ New Search Lookup Tool Available
+ Lycos 50 Features Our Searches
+ Search Engine Articles
+ Subscribing/Unsubscribing Info

General Notes

Hello Everyone--

September was a pretty busy month for search engine news, as you'll see from this issue. Nevertheless, I've been doing updates throughout the site and will be continuing to do so though this month.

The Media Metrix Search Engine Ratings page has been updated, and I expect to update the NetRatings page shortly. Additionally, I hope to have two new pages with ratings data from other sources posted later this month.

Looking at the Media Metrix home share numbers, you'll see that Yahoo hit its highest share ever and stands well above its nearest rivals, with the exception of MSN. That service has seen an incredible rise and is closing the gap on Yahoo rapidly. Bear in mind that these are figures for, which will include non-search MSN properties.

Go, Excite and Lycos continue to show slight declines, while AltaVista looks likely to finally stop its drop. Meanwhile, LookSmart, Snap and HotBot have firmly established themselves as consistent top draws and are threatening to overtake AltaVista's popularity. And when you look at the work and home combined figures, GoTo now draws a significant share, while answer service Ask Jeeves and meta search Go2Net are climbing in the ranks.

I've also added two new pages that explain how to use RealNames links and Direct Hit data at the search engines that offer them. Additionally, a new GVU Survey page has been added, which shows how people look for web sites. The site's home page has also been redesigned, and a guide to the changes and the other new pages mentioned can be found on What's New.

What's New

For those thinking of attending the Search Engine Strategies '99 conference in November, keep in mind that the registration deadline is Oct. 15 to receive the "Early Bird" discount. You'll also find updated speaker and panel information at the conference web site. We're going to have panelists from many of the major search services, as well as experts discussing all aspects of how search engines interact with your sites.

Search Engine Strategies '99

Search Engine News

FTC Steps In To Stop Spamming

There's a new sheriff in town, with the initials FTC. In September, the US Federal Trade Commission made a landmark move in establishing some jurisdiction over search engine results.

By now, you may have seen the headlines of how the FTC stopped an alleged scheme to mislead consumers to porn sites via search engines. That's a big deal, which we'll look at below, but the real story is that this is the first time a government agency has ever asserted regulatory authority over search engine listings.

"This is definitely a case of the FTC stepping in to take control," said Stephen Cohen, lead attorney on the recent case.

It's a good move. For too long, there's been an almost "anything goes" attitude when it comes to manipulating search engines. While the services themselves do much to combat spam, I still receive constant complaints from readers that see it get through, even after contacting the search engines about it.

The FTC's action is a big deterrent against the worst type of spam, that which attempts to mislead users. Those contemplating this type of behavior can no longer assume they'll get away with it because the search engines are too busy to care, or because victimized site owners can't undertake expensive copyright or trademark infringement lawsuits. The FTC cares, and it has now shown a willingness to take action on behalf of consumers.

In this case, the FTC accuses Portugal resident Carlos Pereira of misdirecting consumers to porn sites when they instead expected to reach sites about recipes, movies, children's songs, automobiles and many other non-sexual topics. Also named was the company running the sites that received traffic from Pereira's work, WTFRC.

The FTC says that Pereira would make copies of other people's web pages, submit them to the search engines, then benefit if these pages ranked well for particular search terms. For instance, the case cites an example where the top ranked site for "kids Internet games" at AltaVista appeared to be a page from the Mining Company (now Selecting this page instead took users to an adult web site. Additionally, using the browser's Back button or trying to close the browser caused new windows to open displaying additional porn sites, something the FTC has labeled "mouse trapping."

The FTC has also labeled the act of taking someone else's web pages in the manner described above as "pagejacking." It's been a fairly common brute-force method used by spammers for some time. They find a page that ranks well, make a copy, then place the copy on their own servers. By adding meta refresh tags or using JavaScript, they may also jump visitors who select the page to other pages. Using cloaking software, spammers can even prevent visitors from seeing that they are using someone else's pages.

Ironically, it's a tactic that's losing value as search engines continue to apply other criteria for ranking pages beyond the words that appear on the page itself. Taking a high-ranking page and putting it on another server is no guarantee of gaining the same positioning. Nevertheless, use enough pages, and you'll generate some traffic and maybe even gain some key spots. And according to the FTC, the number of pages estimated involved in this case was huge: 25 million.

Some press reports have given the impression that the victimized web sites were "hacked" and "hijacked" so that anyone trying to reach them was automatically redirected to a porn site. That's not the case. The victimized sites continued to operate as exactly before. No one hacked into them or the search engines. Instead, pagejackers hope to capture visitors by misleading them into thinking they are getting something they want. This only happens if the visitors take the bait planted in the search engine results.

It's all the more reason for search engine users who wish to protect themselves to look closely at the URLs listed for each result. For instance, here's one of the pagejacking examples from AltaVista cited in the case:

Kids Internet Games - Home Page
THE starting place for exploring Kids Internet Games, from your Mining Co....

Notice the term "anal" used in the URL. That's a dead giveaway that this is a site that has nothing to do with kids. Also, spammers often use numbers for their pages, such as 141 in this case. Or course, perfectly legitimate sites also use numbers. Nevertheless, by looking at the URL in addition to the page title and description, you may protect yourself from unpleasant surprises.

This isn't the first legal action involving search engine results, just the first filed by a government agency. Most search engine related suits have involved companies upset with others for using their trademarks. Similarly, the victimized sites in this case could have filed their own lawsuits claiming copyright infringement, since their pages have apparently been used without permission.

In contrast, the FTC suit seeks to protect consumers, not intellectual property rights. That fits in with its charter. Those concerned about possible trademark or copyright infringement relating to search engines shouldn't expect the FTC to step in on their behalf, Cohen said. Nor is the FTC planning to police all types of spam.

"We have no intention of regulating content. That's not was this case was about. This case was about lying to consumers about what you were selling. People thought they were going to pie recipes, movie reviews, folk songs," Cohen said. "That was the misrepresentation, and when consumers clicked on that, they were taken to an adult site, not what they intended. This is where the unfairness comes in. People couldn't get out."

Cohen said that FTC does want to hear from consumers who feel mislead by listings in search engines, especially in cases similar to this one. However, he said consumers should first try contacting the search engines, to get them to take action.

The search engines themselves also need to step up responsiveness, in this area. Cohen said one of the victimized parties involved in the suit complained about AltaVista's slow response in the matter, and I know from the feedback I receive that this is fairly typical for the industry as a whole.

"I think the search engines really need to help. We can't police all this ourselves," Cohen said.

In turn, AltaVista marketing director Tracy Roberts said the FTC action will help her service better deal with spam.

"With the FTC drawing a line in the sand about what is acceptable and what's not, that gives us a way to push back," Roberts said. "We spend a lot of time and energy and resources combating spam. This is a big draw on our resources, and I'm really happy there are deterrents."

The FTC case was filed on Sept. 13, and the US District Court for the Eastern District of Virginia granted a restraining order against the defendants a week later. That gave the FTC the ability to close the defendants' web sites by getting their domain names suspended. In what appears to be a first, the court allowed the restraining order to be served via email. The case is also notable for crossing national boundaries. Authorities in Australia and Portugal worked with the United States to take action.

FTC Halts Internet Highjacking Scam
FTC Press Release, Sept. 22, 1999

Details of FTC v. Carlos Pereira

Plenty of nitty-gritty details about the case. Especially look at the comments at the bottom of the Motion for Temporary Restraining Order to see how this adversely impacted consumers.

FTC moves against Internet scam
MSNBC, Sept. 22, 1999

More details of the FTC's action.

Scam diverts surfers to porn sites
MSNBC, May 18, 1999

The article that first attracted the FTC's attention to the case.

Search engines can be tricked into serving up porn sites
MSNBC, Aug. 18, 1999

Another article about stolen pages which also become part of the FTC suit.

Bait-And-Switch Gets Attention
Search Engine Report, June 1999

Short piece I did talking about how cloaking complicates finding pagejackers. Note -- cloaking was not involved in this case.

Search Engines and Legal Issues

Links and information about meta tag lawsuits and other disputes involving search engines.

Kid Search Engines

This page lists search engines with filters designed to screen out porn sites. However, filtering wouldn't have helped in this case. The pages involved contained no pornographic terms, so the filters wouldn't have caught them.


Improved MSN Search Launches

Microsoft officially launched its new version of MSN Search in late September, and the improvements leapfrog the service forward into being a compelling choice for those searching the web.

Previously, MSN Search was simply Microsoft's name slapped on top of Inktomi's search results. Now MSN Search takes data from multiple sources, applies its own unique ranking mechanisms, then blends answers into a well-organized results page that proves you can be more than the sum of your parts.

After performing a search, the results page will usually be topped by a "Featured Sites" section. This is where MSN Search is placing its top picks for a search -- look for "star trek," for instance, and a link to the official Star Trek web site tops the list. MSN Search also draws upon the RealNames database to guide users to official company web sites. For example, a search for "united airlines" pops a link that says "Go directly to the United Airlines site."

You'll also find MSN's own content appearing in the Featured Sites section, along with occasional paid links to MSN sponsor sites. I see nothing to worry about here -- search engines have long used this type of page topper area for promoting sponsors. MSN Search is simply unique in actually disclosing it via the small "about" link to the right of this section.

Next, you'll probably encounter the "Web Directory Topics" area for many of your searches. Here, MSN Search is making available up to four categories of web sites that it thinks are related to your search. They've made a conscious choice to limit the options to the best four.

"We think the directory is really cool, but often you are overwhelmed with the directory results," said Bill Bliss, general manager of MSN Search, explaining the reasoning.

Take a close look at how the categories are presented. You'll see that you can navigate to precise areas of the directory beyond the main categories shown. For instance, here's a category presented for "star trek:"

TV Shows : Sci-Fi & Fantasy : Star Trek, Voyager, Next Generation, Original Series

Each word is a separate link. Selecting "Star Trek" would take you straight into the Star Trek category, while choosing "TV Shows" would take you into the higher-level TV category. Additionally, notice where the first comma begins. This separates the main category from its "cousins." So in this case, you are shown subcategories related to the main topic of "star trek."

If you choose a category link, you'll be shown a list of web sites for that topic. These listings have all been compiled by the editors at LookSmart. You can also choose to browse categories via the MSN Search home page.

MSN Search also tries to provide a variety of different categories for when a search term might have multiple meanings. For instance, look for "dolphins," and you'll be shown categories for both the marine mammal and the Miami football team.

Below the directory categories comes the "Web Directory Sites" area, where you are shown top sites from the directory itself. One thing that MSN Search is doing to improve the relevancy of sites listed is to actually review the results for the most popular queries, then apply some human intervention to ensure good sites make it to the top.

Additionally, MSN Search is employing a thesaurus-approach to its retrieval. Search for "cars," and it will also find "automobile." Even better, look for "geneology," which is spelled incorrectly, and MSN Search will also search for the correct "genealogy" spelling plus other common misspellings of the word.

Similarly, MSN Search works to interpret URL as words. MSN Search estimates that 10 to 15 percent of searches contain URLs, such as "" MSN Search will equate this to a search for Hotmail itself, and thus you see a link for Hotmail in the Featured Sites area.

After the Web Directory Sites comes the "AltaVista Sites" section. These are matching web pages retrieved from the AltaVista search engine, which now takes over from Inktomi as MSN Search's crawler-based partner. One thing that MSN Search loses in the transition is the page clustering that Inktomi provided. For instance, search for "collecting beanie babies," and you'll see how pages from one site can crowd out others. With page clustering, something AltaVista sorely needs, only a single page from the site would have been presented. That would allow greater representation in the results.

In some cases, only AltaVista results will be presented. This happens if MSN Search feels AltaVista is providing better information that can be retrieved from the directory.

Direct Hit also has a presence, but it is a minor one. After you do a search, look to the left-hand side of the page. You'll see an option to view Direct Hit's "Top 10 Most Popular Sites" for your search.

So there you have it -- a completely revamped service. It's one that Microsoft has so much faith in that as of Oct. 1, it stopped promoting other search services on the main home page. Instead, the search box now queries MSN Search and only MSN Search.

"Where putting our money where our mouth is and betting the farm on MSN Search," said Bliss. Of course, those looking to access other services can still find them by selecting the More Searches option from the MSN Search home page. Plus, Internet Explorer users still have options to other places. But the move clearly shows MSN deciding that search is not just a commodity. They don't send their users to a variety of free email services -- instead, it's Microsoft-owned Hotmail that's promoted. Similarly, visitors are no longer to be sent as easily to competing search services.

Need some more proof of their conviction? Try search for "Yahoo." MSN Search will direct you there, but it also pops up a separate window telling you "Why MSN Search beats Yahoo."

MSN Search

Microsoft moves to revive MSN, Sept. 23, 1999

More details about the future of the MSN portal site here and in the story below.

MSN Finds Focus - Finally
Inter@active Week, Sept. 20, 1999,4164,2336631,00.html

A Yard Sale in Redmond
Industry Standard, Sept. 10, 1999,1449,6288,00.html

Could Microsoft be planning to tighten its focus by selling off sites like MSN CarPoint? A look at possible transactions.


New AOL Search Unveiled

AOL retired its longstanding AOL NetFind service in favor of the new AOL Search in September, though the official launch of AOL Search actually comes later this week. That's to coincide with the release of AOL 5, where AOL Search will serve a major role in helping AOL users find things within their service and across the web with a single search.

I'm holding off on a longer review until after the AOL 5 launch, so expect it in the next issue. In the meantime, non-AOL members can access the service via the URL below.

AOL Search


Google Gets Out Of Beta

Google finally lost its "beta" moniker on September 21, with the new site featuring a cleaner look and a new "GoogleScout" related pages feature.

When you do a search, you'll see a little GoogleScout link appear at the end of each listing. Clicking on this brings up a list of other sites Google feels are similar to the site you selected.

Google says it also plans an index expansion to 200 million web pages, in the coming weeks. And for those statistic-loving types, the service say that it now averages about 65 searches per second at peak times and 3.5 million searches per day.



Go Guides Goes Live

Go's new Go Guides directory went live in September. Similar to the Open Directory, Go Guides makes use of volunteers to catalog the web.

The system has been in beta testing since July, and Go Guide's product manager Ramsey Ksar says small changes have since been made to make it work better for editors. In particular, a new "Please Edit" button allows those reviewing submissions to send them back to the original guide for minor corrections.

"People were declining submissions, even if there were little typos," Ksar said. "With the Please Edit system, the reviewer sends a message to original person to correct the entry. It's done miracles for the system."


Go Guides

Sign-up from here.

Go Beta Tests User-Assisted Directory

More information about how the Go Guide system works.


Northern Light Launches Ads, Alerts Service

Northern Light launched its first advertising campaign in September, with ads appearing in print and on television and radio. The service also gave its home page a facelift and introduced a new "alert" service for searchers. Search Alerts notifies users via email when there are new finds for their registered search topics.

Northern Light

Northern Light Search Alerts


New Search Lookup Tool Available

GuruNet, still in beta testing, lets you highlight words in a document and then send them to an online dictionary or thesaurus. You can also send the words as a query to one of several search engines. It includes an ability to view and suggest related links for a site, but little of this actually works yet. Additional information is offered, as appropriate, such as stock charts and news headlines.

The main downside to GuruNet is that it requires a 700K download. Nor is the program integrated into your browser, though I didn't mind this as much with other standalone search utilities. Overall, GuruNet puts an amazing amount of quality information at your fingertips.



A mature product that offers similar functionality to GuruNet and far more search functionality. It also directly integrates into your browser. Internet Explorer users can even download it in less than a minute.


Allows you to "jump" directly to major companies by entering their names, stock ticker symbols or trademarks. More importantly, it provides easy access to a wealth of data about these companies: financial data, stock quotes, management teams, SEC filings and more. Use it directly online or download a software companion for Windows 95/98/NT.

Webopedia Personal Search Tool

Easy access to technical definitions from within your browser, and no download time to speak of to install it.

Search Utilities

Like GuruNet? Check out other search tools that are available.


Lycos 50 Features Our Searches

Forget live search displays. Lycos has realized that all our searches can make for great content, if assembled correctly. The Lycos 50 shows top searches each week, plus glimpses at what's up-and-coming from the query logs, along with other features. Extremely well done -- make it a regular visit!

Lycos Top 50

Search Engine Articles

ClickZ, Sept. 28, 1999

Excite has instituted a new auction system for banner ad buys. Good tips on how to cope.

AltaVista to climb new heights with IPO, Sept. 27, 1999

AltaVista's IPO is now scheduled for next year, and the service has a relaunch scheduled for this month.

The Perfect Search
Newsweek, Sept. 27, 1999

Nice review of current thinking in search.

Internet icons on parade?
Salon, Sept. 24, 1999

Ask Jeeves will be in the next Macy's Thanksgiving Day parade, and Salon pictures it as the start of a net-related character rush in future years.

A Search Site for Search Sites Is Accused of Trespassing
New York Times, Sept. 24, 1999

A meta search service for online shoppers taps into another meta search service for shoppers, and a lawsuit results.

Bertelsmann plans IPO for merged search engines
Bloomberg, Sept. 23, 1999

Bertelsmann plans to merge Lycos Europe with the popular German Fireball search engine and take them public next year.

Best Search Sites on the Web
PC Magazine, Sept. 20, 1999

Yahoo, Northern Light and HotBot take top honors out of the 15 services reviewed.

Web Portals
PC Magazine, Sept. 6, 1999,6755,2327792,00.html

This review looked specifically at portal features. Yahoo came in tops again, with Excite as runner-up.

Infoseek executive arrested on solicitation charges
Bloomberg, Sept. 18, 1999

A top Infoseek exec is charged with trying to solicit sex with a minor via the Internet.

How kept a top idea guy
Red Herring, Sept. 8, 1999

Why did Amazon want Alexa? Still no specifics, but somewhere in all that data is a better way to improve electronic commerce, Amazon hopes.

10 Questions About Meta Data, Sept. 7, 1999

Comprehensive introduction to using meta tags designed for search engines and for other uses.

On the Internet, there is such thing as free labor, Sept. 7, 1999

A look at labor issues involved with using volunteers to build directories.

Yahoo Two-Faced on Spam
Wired, Aug. 27, 1999

Interesting story where Yahoo's advice to Yahoo Store owners to gain links conflicted with its Yahoo Mail guidelines. However, I wouldn't consider true the other suggestion that Yahoo's tips regarding search engines were meant to encourage search engine spamming.


How do I unsubscribe?
+ Use the form at or follow the instructions at the very end of this email.

How do I subscribe?
+ Use the form at or send a blank email message to

How do I see past issues?
+ Follow the links at

Is there an HTML version?
+ Yes, but not via email. View it online at

I didn't get Part 1 or 2. Can you resend it?
+ No, but you can view the entire issue online, via the link above.

How do I change my address?
+ Unsubscribe your old one, then subscribe the new one, using the links above.

I need human help with a list issue!
+ Write to DO NOT send messages regarding list management issues to Danny Sullivan. He does not deal with these.

I have feedback about an article!
+ I'd love to hear it. Use the form at

This newsletter is Copyright (c) Corp, 1999