THE SEARCH ENGINE UPDATE
October 4, 1999 - Number 62
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch, http://searchenginewatch.com/.
Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
In This Issue
+ General Notes
+ FTC Steps In To Stop Spamming
+ Improved MSN Search Launches
+ New AOL Search Unveiled
+ Google Gets Out Of Beta
+ Go Guides Goes Live
+ Northern Light Launches Ads, Alerts Service
+ New Search Lookup Tool Available
+ AltaVista's Oddities Explained
+ Lycos 50 Features Our Searches
+ Search Engine Articles
+ Subscribing/Unsubscribing Info
September was a pretty busy month for search engine news, as you'll see from this issue. Nevertheless, I've been doing updates throughout the site and will be continuing to do so though this month.
The Media Metrix Search Engine Ratings page has been updated, and I expect to update the NetRatings page shortly. Additionally, I hope to have two new pages with ratings data from other sources posted later this month.
Looking at the Media Metrix home share numbers, you'll see that Yahoo hit its highest share ever and stands well above its nearest rivals, with the exception of MSN. That service has seen an incredible rise and is closing the gap on Yahoo rapidly. Bear in mind that these are figures for MSN.com, which will include non-search MSN properties.
Go, Excite and Lycos continue to show slight declines, while AltaVista looks likely to finally stop its drop. Meanwhile, LookSmart, Snap and HotBot have firmly established themselves as consistent top draws and are threatening to overtake AltaVista's popularity. And when you look at the work and home combined figures, GoTo now draws a significant share, while answer service Ask Jeeves and meta search Go2Net are climbing in the ranks.
I've also added two new pages that explain how to use RealNames links and Direct Hit data at the search engines that offer them. Additionally, a new GVU Survey page has been added, which shows how people look for web sites.
Links to all of these pages can be found via the What's New page, below:
Within the Subscribers-Only Area, I've revised the pages for Go and Ask Jeeves. Numerous other minor updates have also been made in the area. Especially of note is a link to the Search Engine Matrix discussion area added to the Search Engine Talk page and new resource for updated spider agent names and IP addresses that was added to the bottom of the More SpiderSpotting page. Links to all these pages can be found via the Subscribers-Only What's New page, below:
Subscribers-Only Area What's New Page
For those thinking of attending the Search Engine Strategies '99 conference in November, keep in mind that the registration deadline is Oct. 15 to receive the "Early Bird" discount. You'll also find updated speaker and panel information at the conference web site. We're going to have panelists from many of the major search services, as well as experts discussing all aspects of how search engines interact with your sites.
Search Engine Strategies '99
Search Engine News
FTC Steps In To Stop Spamming
There's a new sheriff in town, with the initials FTC. In September, the US Federal Trade Commission made a landmark move in establishing some jurisdiction over search engine results.
By now, you may have seen the headlines of how the FTC stopped an alleged scheme to mislead consumers to porn sites via search engines. That's a big deal, which we'll look at below, but the real story is that this is the first time a government agency has ever asserted regulatory authority over search engine listings.
"This is definitely a case of the FTC stepping in to take control," said Stephen Cohen, lead attorney on the recent case.
It's a good move. For too long, there's been an almost "anything goes" attitude when it comes to manipulating search engines. While the services themselves do much to combat spam, I still receive constant complaints from readers that see it get through, even after contacting the search engines about it.
The FTC's action is a big deterrent against the worst type of spam, that which attempts to mislead users. Those contemplating this type of behavior can no longer assume they'll get away with it because the search engines are too busy to care, or because victimized site owners can't undertake expensive copyright or trademark infringement lawsuits. The FTC cares, and it has now shown a willingness to take action on behalf of consumers.
In this case, the FTC accuses Portugal resident Carlos Pereira of misdirecting consumers to porn sites when they instead expected to reach sites about recipes, movies, children's songs, automobiles and many other non-sexual topics. Also named was the company running the sites that received traffic from Pereira's work, WTFRC.
The FTC says that Pereira would make copies of other people's web pages, submit them to the search engines, then benefit if these pages ranked well for particular search terms. For instance, the case cites an example where the top ranked site for "kids Internet games" at AltaVista appeared to be a page from the Mining Company (now About.com). Selecting this page instead took users to an adult web site. Additionally, using the browser's Back button or trying to close the browser caused new windows to open displaying additional porn sites, something the FTC has labeled "mouse trapping."
Ironically, it's a tactic that's losing value as search engines continue to apply other criteria for ranking pages beyond the words that appear on the page itself. Taking a high-ranking page and putting it on another server is no guarantee of gaining the same positioning. Nevertheless, use enough pages, and you'll generate some traffic and maybe even gain some key spots. And according to the FTC, the number of pages estimated involved in this case was huge: 25 million.
Some press reports have given the impression that the victimized web sites were "hacked" and "hijacked" so that anyone trying to reach them was automatically redirected to a porn site. That's not the case. The victimized sites continued to operate as exactly before. No one hacked into them or the search engines. Instead, pagejackers hope to capture visitors by misleading them into thinking they are getting something they want. This only happens if the visitors take the bait planted in the search engine results.
It's all the more reason for search engine users who wish to protect themselves to look closely at the URLs listed for each result. For instance, here's one of the pagejacking examples from AltaVista cited in the case:
Kids Internet Games - Home Page
THE starting place for exploring Kids Internet Games, from your Mining Co....
Notice the term "anal" used in the URL. That's a dead giveaway that this is a site that has nothing to do with kids. Also, spammers often use numbers for their pages, such as 141 in this case. Or course, perfectly legitimate sites also use numbers. Nevertheless, by looking at the URL in addition to the page title and description, you may protect yourself from unpleasant surprises.
This isn't the first legal action involving search engine results, just the first filed by a government agency. Most search engine related suits have involved companies upset with others for using their trademarks. Similarly, the victimized sites in this case could have filed their own lawsuits claiming copyright infringement, since their pages have apparently been used without permission.
In contrast, the FTC suit seeks to protect consumers, not intellectual property rights. That fits in with its charter. Those concerned about possible trademark or copyright infringement relating to search engines shouldn't expect the FTC to step in on their behalf, Cohen said. Nor is the FTC planning to police all types of spam.
"We have no intention of regulating content. That's not was this case was about. This case was about lying to consumers about what you were selling. People thought they were going to pie recipes, movie reviews, folk songs," Cohen said. "That was the misrepresentation, and when consumers clicked on that, they were taken to an adult site, not what they intended. This is where the unfairness comes in. People couldn't get out."
Explaining further, he said, "We probably aren't going to do anything about the people who put Pamela Anderson Lee in their meta tags 50 times to get higher in the search results. I don't think that's what we are about, though we don't like that practice," Cohen said. "We're concerned about totally irrelevant search results, such as filling meta tags with typical search terms, when intending to sell adult content."
Cohen said that FTC does want to hear from consumers who feel mislead by listings in search engines, especially in cases similar to this one. However, he said consumers should first try contacting the search engines, to get them to take action.
The search engines themselves also need to step up responsiveness, in this area. Cohen said one of the victimized parties involved in the suit complained about AltaVista's slow response in the matter, and I know from the feedback I receive that this is fairly typical for the industry as a whole.
"I think the search engines really need to help. We can't police all this ourselves," Cohen said.
In turn, AltaVista marketing director Tracy Roberts said the FTC action will help her service better deal with spam.
"With the FTC drawing a line in the sand about what is acceptable and what's not, that gives us a way to push back," Roberts said. "We spend a lot of time and energy and resources combating spam. This is a big draw on our resources, and I'm really happy there are deterrents."
The FTC case was filed on Sept. 13, and the US District Court for the Eastern District of Virginia granted a restraining order against the defendants a week later. That gave the FTC the ability to close the defendants' web sites by getting their domain names suspended. In what appears to be a first, the court allowed the restraining order to be served via email. The case is also notable for crossing national boundaries. Authorities in Australia and Portugal worked with the United States to take action.
There are a number of lessons for site owners to take away from this case:
1) Ensure that any company you use to generate search engine related traffic is not misleading users. Remember, both the promotional person and the company receiving the traffic got named in this suit. Take a hard look at exactly how they are helping consumers reach your site.
2) Similarly, ensure any promotional company you hire is not using other people's web pages or even portions of other people's web pages. In the latter tactic, companies sometimes take portions of high ranking pages and combine them all into a single page. Both tactics could lead you to be named in trademark or copyright infringement suits. Similarly, this is clearly a tactic the FTC wouldn't like.
3) If you do your own promotional work, obviously follow the tips described in 1 and 2.
FTC Halts Internet Highjacking Scam
FTC Press Release, Sept. 22, 1999
Details of FTC v. Carlos Pereira
Plenty of nitty-gritty details about the case. Especially look at the comments at the bottom of the Motion for Temporary Restraining Order to see how this adversely impacted consumers.
FTC moves against Internet scam
MSNBC, Sept. 22, 1999
More details of the FTC's action.
Scam diverts surfers to porn sites
MSNBC, May 18, 1999
The article that first attracted the FTC's attention to the case.
Search engines can be tricked into serving up porn sites
MSNBC, Aug. 18, 1999
Another article about stolen pages which also become part of the FTC suit.
Bait-And-Switch Gets Attention
Search Engine Report, June 1999
Short piece I did talking about how cloaking complicates finding pagejackers. Note -- cloaking was not involved in this case.
Search Engines and Legal Issues
Links and information about meta tag lawsuits and other disputes involving search engines.
Kid Search Engines
This page lists search engines with filters designed to screen out porn sites. However, filtering wouldn't have helped in this case. The pages involved contained no pornographic terms, so the filters wouldn't have caught them.
Improved MSN Search Launches
Microsoft officially launched its new version of MSN Search in late September, and the improvements leapfrog the service forward into being a compelling choice for those searching the web.
Previously, MSN Search was simply Microsoft's name slapped on top of Inktomi's search results. Now MSN Search takes data from multiple sources, applies its own unique ranking mechanisms, then blends answers into a well-organized results page that proves you can be more than the sum of your parts.
After performing a search, the results page will usually be topped by a "Featured Sites" section. This is where MSN Search is placing its top picks for a search -- look for "star trek," for instance, and a link to the official Star Trek web site tops the list. MSN Search also draws upon the RealNames database to guide users to official company web sites. For example, a search for "united airlines" pops a link that says "Go directly to the United Airlines site."
You'll also find MSN's own content appearing in the Featured Sites section, along with occasional paid links to MSN sponsor sites. I see nothing to worry about here -- search engines have long used this type of page topper area for promoting sponsors. MSN Search is simply unique in actually disclosing it via the small "about" link to the right of this section.
Next, you'll probably encounter the "Web Directory Topics" area for many of your searches. Here, MSN Search is making available up to four categories of web sites that it thinks are related to your search. They've made a conscious choice to limit the options to the best four.
"We think the directory is really cool, but often you are overwhelmed with the directory results," said Bill Bliss, general manager of MSN Search, explaining the reasoning.
Take a close look at how the categories are presented. You'll see that you can navigate to precise areas of the directory beyond the main categories shown. For instance, here's a category presented for "star trek:"
TV Shows : Sci-Fi & Fantasy : Star Trek, Voyager, Next Generation, Original Series
Each word is a separate link. Selecting "Star Trek" would take you straight into the Star Trek category, while choosing "TV Shows" would take you into the higher-level TV category. Additionally, notice where the first comma begins. This separates the main category from its "cousins." So in this case, you are shown subcategories related to the main topic of "star trek."
If you choose a category link, you'll be shown a list of web sites for that topic. These listings have all been compiled by the editors at LookSmart. You can also choose to browse categories via the MSN Search home page.
MSN Search also tries to provide a variety of different categories for when a search term might have multiple meanings. For instance, look for "dolphins," and you'll be shown categories for both the marine mammal and the Miami football team.
Below the directory categories comes the "Web Directory Sites" area, where you are shown top sites from the directory itself. One thing that MSN Search is doing to improve the relevancy of sites listed is to actually review the results for the most popular queries, then apply some human intervention to ensure good sites make it to the top.
Additionally, MSN Search is employing a thesaurus-approach to its retrieval. Search for "cars," and it will also find "automobile." Even better, look for "geneology," which is spelled incorrectly, and MSN Search will also search for the correct "genealogy" spelling plus other common misspellings of the word.
Similarly, MSN Search works to interpret URL as words. MSN Search estimates that 10 to 15 percent of searches contain URLs, such as "www.hotmail.com." MSN Search will equate this to a search for Hotmail itself, and thus you see a link for Hotmail in the Featured Sites area.
After the Web Directory Sites comes the "AltaVista Sites" section. These are matching web pages retrieved from the AltaVista search engine, which now takes over from Inktomi as MSN Search's crawler-based partner. One thing that MSN Search loses in the transition is the page clustering that Inktomi provided. For instance, search for "collecting beanie babies," and you'll see how pages from one site can crowd out others. With page clustering, something AltaVista sorely needs, only a single page from the site would have been presented. That would allow greater representation in the results.
In some cases, only AltaVista results will be presented. This happens if MSN Search feels AltaVista is providing better information that can be retrieved from the directory.
Direct Hit also has a presence, but it is a minor one. After you do a search, look to the left-hand side of the page. You'll see an option to view Direct Hit's "Top 10 Most Popular Sites" for your search.
So there you have it -- a completely revamped service. It's one that Microsoft has so much faith in that as of Oct. 1, it stopped promoting other search services on the main MSN.com home page. Instead, the search box now queries MSN Search and only MSN Search.
"Where putting our money where our mouth is and betting the farm on MSN Search," said Bliss. Of course, those looking to access other services can still find them by selecting the More Searches option from the MSN Search home page. Plus, Internet Explorer users still have options to other places. But the move clearly shows MSN deciding that search is not just a commodity. They don't send their users to a variety of free email services -- instead, it's Microsoft-owned Hotmail that's promoted. Similarly, visitors are no longer to be sent as easily to competing search services.
Need some more proof of their conviction? Try search for "Yahoo." MSN Search will direct you there, but it also pops up a separate window telling you "Why MSN Search beats Yahoo."
Among the other changes is a new Add URL system. Submissions are sent to both LookSmart and AltaVista at the same time, and here are some tips to keep in mind if you use the system.
If you have submitted to LookSmart, there's really no reason to use the MSN Search interface. You are only duplicating your efforts, because the information is simply sent to LookSmart. Moreover, you can't pick an exact category the way you can if you submit directly via LookSmart. So, I'd recommend you only submit to LookSmart.
Having said this, if you have submitted to LookSmart in the past and still haven't been listed within six weeks or so, then I'd do a resubmission to LookSmart plus try a new submission via MSN Search. It may be that you'll increase your odds of landing in front of a favorable editors. There certainly is no penalty for submitting in both places.
If you do use the MSN Search interface, be sure to keep your title to 35 characters and your description to 145 characters, which are the LookSmart limits. Additionally, MSN Search doesn't allow you to pick an exact category for your site. That's OK -- choose the closest category, and editors at LookSmart will decide where exactly your site should be classified.
Similar to LookSmart, if you have submitted to AltaVista, then you are covered for MSN Search's AltaVista section. There's no reason to use the MSN Search submit form. You would just be duplicating your work. In fact, there are real disadvantages to submitting to AltaVista via MSN Search.
For one thing, you can only submit one URL per day at MSN Search, while AltaVista will accept several per day. Moreover, if you do submit multiple URLs via MSN Search (doing one per day over the course of several days), you are also sending off duplicate submissions to LookSmart.
So to recap, submit directly to LookSmart and AltaVista, and you will have covered yourself safely at MSN Search.
Microsoft moves to revive MSN
News.com, Sept. 23, 1999
More details about the future of the MSN portal site here and in the story below.
MSN Finds Focus - Finally
Inter@active Week, Sept. 20, 1999
A Yard Sale in Redmond
Industry Standard, Sept. 10, 1999
Could Microsoft be planning to tighten its focus by selling off sites like MSN CarPoint? A look at possible transactions.
How MSN Search Works
Covers what you've read here, but with illustrations.
How LookSmart Works
Tips on submitting to LookSmart.
How AltaVista Works
Tips on submitting to AltaVista.
New AOL Search Unveiled
AOL retired its longstanding AOL NetFind service in favor of the new AOL Search in September, though the official launch of AOL Search actually comes later this week. That's to coincide with the release of AOL 5, where AOL Search will serve a major role in helping AOL users find things within their service and across the web with a single search.
I'm holding off on a longer review until after the AOL 5 launch, so expect it in the next issue. In the meantime, non-AOL members can access the service via the URL below.
Also, as I mentioned in the last issue, site owners who wish to be listed within the new AOL Search should ensure they are listed within the Open Directory and within the Inktomi database. That will cover you for AOL Search.
How The Open Directory Works
How Inktomi Works
Google Gets Out Of Beta
Google finally lost its "beta" moniker on September 21, with the new site featuring a cleaner look and a new "GoogleScout" related pages feature.
When you do a search, you'll see a little GoogleScout link appear at the end of each listing. Clicking on this brings up a list of other sites Google feels are similar to the site you selected.
Google says it also plans an index expansion to 200 million web pages, in the coming weeks. And for those statistic-loving types, the service say that it now averages about 65 searches per second at peak times and 3.5 million searches per day.
Go Guides Goes Live
Go's new Go Guides directory went live in September. Similar to the Open Directory, Go Guides makes use of volunteers to catalog the web.
The system has been in beta testing since July, and Go Guide's product manager Ramsey Ksar says small changes have since been made to make it work better for editors. In particular, a new "Please Edit" button allows those reviewing submissions to send them back to the original guide for minor corrections.
"People were declining submissions, even if there were little typos," Ksar said. "With the Please Edit system, the reviewer sends a message to original person to correct the entry. It's done miracles for the system."
While Go still accepts email submissions to the Go Guides area, Ksar said it's better and faster for site owners to simply sign-up and submit their own sites using the Go Guides system, even if they don't plan to actively participate.
"People are going to submit their own content. It's just a matter of how much self promotion they are going to do. Just don't go overmarketing," he said.
Sign-up from here.
Go Beta Tests User-Assisted Directory
More information about how the Go Guide system works.
Northern Light Launches Ads, Alerts Service
Northern Light launched its first advertising campaign in September, with ads appearing in print and on television and radio. The service also gave its home page a facelift and introduced a new "alert" service for searchers. Search Alerts notifies users via email when there are new finds for their registered search topics.
Northern Light Search Alerts
New Search Lookup Tool Available
GuruNet, still in beta testing, lets you highlight words in a document and then send them to an online dictionary or thesaurus. You can also send the words as a query to one of several search engines. It includes an ability to view and suggest related links for a site, but little of this actually works yet. Additional information is offered, as appropriate, such as stock charts and news headlines.
The main downside to GuruNet is that it requires a 700K download. Nor is the program integrated into your browser, though I didn't mind this as much with other standalone search utilities. Overall, GuruNet puts an amazing amount of quality information at your fingertips.
A mature product that offers similar functionality to GuruNet and far more search functionality. It also directly integrates into your browser. Internet Explorer users can even download it in less than a minute.
Allows you to "jump" directly to major companies by entering their names, stock ticker symbols or trademarks. More importantly, it provides easy access to a wealth of data about these companies: financial data, stock quotes, management teams, SEC filings and more. Use it directly online or download a software companion for Windows 95/98/NT.
Webopedia Personal Search Tool
Easy access to technical definitions from within your browser, and no download time to speak of to install it.
Like GuruNet? Check out other search tools that are available.
AltaVista's Oddities Explained
AltaVista says the odd results that I mentioned in the last newsletter were due to some experimenting it had underway -- visitors were randomly diverted to a testbed area, which was generating a different set of results than if the same query was run against the normal servers.
AltaVista to climb new heights with IPO
News.com, Sept. 27, 1999
AltaVista's IPO is now scheduled for next year, and the service has a relaunch scheduled for this month.
Lycos 50 Features Our Searches
Forget live search displays. Lycos has realized that all our searches can make for great content, if assembled correctly. The Lycos 50 shows top searches each week, plus glimpses at what's up-and-coming from the query logs, along with other features. Extremely well done -- make it a regular visit!
Lycos Top 50
Search Engine Articles
ClickZ, Sept. 28, 1999
Excite has instituted a new auction system for banner ad buys. Good tips on how to cope.
The Perfect Search
Newsweek, Sept. 27, 1999
Nice review of current thinking in search.
Internet icons on parade?
Salon, Sept. 24, 1999
Ask Jeeves will be in the next Macy's Thanksgiving Day parade, and Salon pictures it as the start of a net-related character rush in future years.
A Search Site for Search Sites Is Accused of Trespassing
New York Times, Sept. 24, 1999
A meta search service for online shoppers taps into another meta search service for shoppers, and a lawsuit results.
Bertelsmann plans IPO for merged search engines
Bloomberg, Sept. 23, 1999
Bertelsmann plans to merge Lycos Europe with the popular German Fireball search engine and take them public next year.
Best Search Sites on the Web
PC Magazine, Sept. 20, 1999
Yahoo, Northern Light and HotBot take top honors out of the 15 services reviewed.
PC Magazine, Sept. 6, 1999
This review looked specifically at portal features. Yahoo came in tops again, with Excite as runner-up.
10 Questions About Meta Data
Builder.com, Sept. 7, 1999
Comprehensive introduction to using meta tags designed for search engines and for other uses.
How do I unsubscribe?
+ Follow the instructions at the very end of this email.
How do I subscribe?
+ The Search Engine Update is only available to paid subscribers of the Search Engine Watch web site. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to email@example.com
I need human help with my subscription!
+ Send a message to firstname.lastname@example.org. DO NOT send messages regarding list management or site subscription issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at
This newsletter is Copyright (c) internet.com corp., 1999