THE SEARCH ENGINE REPORT
March 3, 1998 - Number 16
By Danny Sullivan
Editor, Search Engine Watch
About The Report
The Search Engine Report is the email companion to Search Engine Watch, http://searchenginewatch.com/. It keeps you informed of changes to the site and general search engine news.
The report has 22,000 subscribers. High-speed email list hosting services are provided by SparkNET, http://www.SparkLIST.com/. You may pass this newsletter on to others, as long as it is sent in its entirety.
Please note that long URLs may break into two lines in some mail readers. Please cut and paste, should this occur.
In This Issue
+ Site Updates
+ Bridge Page Follow-Up
+ GoTo Sells Positions
+ AltaVista Changes Direction
+ HotBot Adds Directory
+ WebCrawler To Be Relaunched
+ Search Engine For Web Developers
+ Search Engine Discussion Areas Aplenty
+ Infoseek Add URL Off On Weekends
+ Search Engine Notes
+ Search Engine Articles
These pages are new:
Search Engine Results Survey
This is a project I'd planned to do for some time. It shows you how well each search engine is ranked on the various other search engines. Guess what? Not very well, in some cases. This is more than just trivia. It helps illustrate the particular problem of searching for companies on the web. Top honors go to Infoseek, for listing the appropriate home page at the top of the results more than the others.
Search Engine Spam Survey
One good survey deserves another. This page looks at the quality of results returned for a search on "monica lewinsky," in particular to spot what opportunistic spam made it into the top ten. Infoseek wins top marks again for keeping out spam.
These pages have been updated:
Search Engine EKGs
The Search Engine EKGs show how well the search engines are crawling the web, based on activity from sample sites. They have been significantly redone. A third survey site has been added. Each search engine now also has its own page, topped by an estimate of how complete and fresh the search engine is based on observed activities. The EKGs provide only a rough glimpse into how well things are working under the hood, but even these few sample sites can be revealing.
Search Engine Alliances Chart
Reflects the latest search engine positions on the Netscape and Microsoft search pages, as well as other important venues that funnel traffic toward search engines.
Media Metrix Search Engine Ratings
Media Metrix now shows Infoseek in the number two spot, above Excite.
RelevantKnowledge Search Engine Ratings
RelevantKnowledge shows Infoseek just barely behind Excite and AltaVista tied with Lycos for fourth place.
Search Engine Profits
Now shows the latest numbers from Lycos, which posted a $300,000 profit, its second in a row. Only Yahoo has had two consecutive profitable quarters in a row, to date.
Search Engine News
There was quite a bit of reaction to last month's article about bridge pages. In particular, I heard from the company that manages State Farm's bridge pages, Green Flash Systems. CEO D.R. Peck has written up a guest response that's well worth reading. A link is below.
Peck discusses whether the methodology deployed by Green Flash (bridge pages coupled with IP delivery) is grounds for penalizing the high rankings they achieved, or whether the positions should be judged as to their relevancy for a given search phrase.
Also, more clarification on last month's article. If you recall, only Infoseek said they had a problem with the pages Green Flash had submitted for State Farm. At the time, they said the content of the pages appeared to violate their spamming guidelines. Since then, they have been more specific, saying it was the redirection itself that they disliked.
Bridging The Gap
Search engine GoTo debuted its new pay-for-placement service on Feb. 21. It allows web site owners to bid for placement. Those willing to pay more can appear higher in the search results.
Pay-for-placement is not new. Open Text experimented with it in mid-1996, and numerous small search directories currently offer the ability to pay for a better ranking.
Despite this, the major search engines have avoided such a service. This is mainly because they see their results as akin to editorial copy. They aren't something that advertisers are supposed to influence. By allowing pay-for-placement, search engine users may begin to mistrust the results they receive. Such complaints helped cause Open Text to give up its experiment.
Does that mean GoTo's results are bad? Not necessarily. Founder Bill Gross makes an interesting argument on how money produces better results.
"This is an opening shot of changing the search engines from a white pages to a yellow pages," Gross says.
What he means is that with a white pages phone book, numbers are organized alphabetically. Money does not get you higher in the results, though some people use company names such as "A Typographical Service" to try and jump ahead of others.
With a yellow pages phone book, money is central. The more you pay, the bigger ad you can have. That doesn't make the yellow pages any less useful. It's a very convenient way to find businesses, and often those with the bigger ads are the better ones.
Of course, there are plenty of good businesses without big budgets, and there are plenty of bad businesses that can afford large ads. So the ability to spend money does not guarantee quality. But it can be a useful ranking tool, and one that Gross says makes more sense than leaving it to search algorithms.
"All you're really getting back is how good people are at tricking the search engines, not how good people are, Gross said."
It's overstated to say only people who know how to trick the search engines get top listings. However, it's not far off the mark when it comes to some popular phrases or one word search terms. The more general and more popular the search, the more likely you'll see people who are aggressively fighting for a top placement.
A search for "monica lewinsky" illustrates this rather easily. A recent survey I did revealed that on average, one-third of the top sites listed in the major search engines were "clickthrough" spam sites.
These are sites that earn referral fees from people clicking through from their sites to other sites. The site owners make pages that the search engine algorithms think are relevant but which most people would probably consider junk.
Beyond this opportunistic activity, there's continuing competition to do well for all sorts of product related terms. While it's heartening to still see small sites rise to the top "naturally," the reality is that people are spending more and more time clawing their way to the top. The days of being listed for popular terms without expending effort aren't gone, but they are receding.
Since time is money, it's easy to argue people are already buying their listings. For those without time or skill, they can turn to search engine placement services. These services will oversee the submission of bridge pages and monitor them for you constantly. They get paid by the click, so if you've got the budget, they can seem a saving grace to the otherwise confusing world of improving your placement. That's assuming you get the listing you are looking for, of course.
Given all this activity, it might be advantageous to search engine users if the search engines took back control of the process. By selling placement for the most popular terms, they might help defuse the arms race of spamming and aggressive placement that exists. Results would certainly benefit, if not just for the fact that opportunistic spam would lose some of its incentive.
It would also come as a great relief to web marketers, who scratch their heads trying to keep up with what's the best way to get listed. Rules about hidden text, repetition in meta tags and the need for HTML text in a graphic-intensive site are confusing and complex. In contrast, pay-for-placement is simple and makes sense.
What about those with little or no budget? Would material at universities be locked out, for example? Again, this material is probably already poorly ranked for many popular and general searches. Pay-for-placement wouldn't hurt it, and the material would still remain accessible for longer and more specific searches.
For example, pay-for-placement might mean a realtor in Newport Beach, California would no longer be listed "accidentally" for "real estate." However, they may still do very well for "newport beach homes for sale."
So there are many reasons why pay-for-placement makes sense. There's also a big reason against it. It just doesn’t feel right.
If you go to a search engine and enter something like "microsoft," many people would expect the service should probably list the Microsoft web site. That's a relevant result, and you want it regardless of whether Microsoft has made an ad buy to ensure it happens.
Likewise, if you do a search for "real estate," you might appreciate having a human editor point you toward a few sites about real estate, not just toward major realtors. You might also appreciate some mechanism that helps you be more specific about what you are looking for.
In that light, what's probably needed is more human intervention in results, not commercialization of them. Search engines might create pages of links for these topics, carefully chosen for relevance. There would probably be a place for advertising on these pages, yet done right, it shouldn't offend anyone concerned over the results being "influenced."
This move could please everyone. Those with money could pay to be placed. Those with quality sites could request being listed on the merit of their content. Users would be given more guidance than they get sometime from raw search results. Spamming should decrease naturally, as incentive disappears.
Raw search results would remain, of course. You can't create pages for everything, and even then, people may still want more choices. However, these popular topic pages would likely satisfy many users.
You can see these types of pages growing already. Excite's Entertainment channel or the Lycos Web Design guide, both which can be found from the home page of either service, provide more organized and relevant information that a blind search for "entertainment" might yield. At some point, it is likely they will make the leap to supplant raw results, in some cases.
Ironically, GoTo had these precise type pages. A search for "travel," for example, brought up "The Travel Page." It was nicely organized, with sites listed in side-by-side columns under topics such as "car rental" and "air travel." At the bottom were raw search results. Now, only raw results are presented.
Enough about the pros and cons of paid listings. How about GoTo itself? For one, GoTo is not a new search engine. It was launched in 1997 and incorporates the former University of Colorado-based World Wide Web Worm. However, its results have been algorithm ranked until now.
To make a purchase, advertisers open an account and note how much they wish to "bid" per click. For example, Toyota currently comes up tops on a search for "cars," and the amount of $0.02 is displayed next to the listing. That means Toyota has agreed to pay 2 cents for anyone clicking on its link.
Honda could open an account and choose to pay 3 cents per click. If so, it would rise above Toyota. In the case of a tie, the site which users have voted best gets the priority.
The way users get to vote is probably the most annoying thing about GoTo. Selecting a listing launches a frame, with the actual destination page embedded in the frame. The frame allows users to vote on the site, similar to how Lycos sets up a frame when people browse links in its community guide. It can also be easily turned off. But it seems too intrusive, and certainly anyone who actually paid to be listed might prefer to have the screen all to themselves.
What impact will GoTo have on the other search engines? Probably little. A quick call to representatives at Excite and Lycos found minimal interest.
"It will be interesting to see how this plays out. My feeling that the consumer wants something more cleaner than commercialism," said Brett Bullington, Executive Vice President of Strategic and Business Development at Excite
Likewise, Lycos search manager Rajive Mathur said, "I'm not sure it's really providing value to the user, in the long term. I think they want some independent sorting"
The closest Lycos is likely to come to pay-for-placement for the time being are the Bullseye ads it has been experimenting with for the past month or so. They put advertisers just above the raw search results section, though they remain physically separated from it. They should be available to all advertisers within the next week or so, at $75/CPM.
How about the impact GoTo will have on marketers. Will they suddenly flock to the service? Perhaps if it grows more, but it has a way to go.
GoTo has well below the traffic of the major search engines. It's at 5 million page views per month, about one-fifth of what Excite generates. But Gross is a well-known Internet entrepreneur. His company, idealab, is backing GoTo and about to launch a new advertising campaign. Awareness may grow, which in turn may help make pay-for-placement more acceptable than it has been in the past.
Gross stresses that he can experiment with the model more easily. "Since we have no brand to worry about, we can get away with it easier than most," he said. In fact, Gross hopes that his service fills a niche that the other services won't be able to immediately, either because of consumer opposition they would face, or due to lack of interest.
GoTo Advertising Placement Page
Engines Battle Irrelevance of Results
Internet World, Feb. 23, 1998
GoTo Searches With Capitalistic Engine
Wired, Feb. 24, 1998
Pay-for-placement gets another shot
News.com, Feb. 19, 1998
New Search Engine Goes Commercial
News.com, Feb. 18, 1998
Lycos Adds New Features,
Reorganizes Suggested Links
The Search Engine Report, Jan. 9, 1998
Talks about the "Bullseye" links advertisers can purchase to appear above the raw search engine results, along with other changes.
Over the past year, search services such as Excite, Yahoo and Lycos have added free email, chat, discussion areas and other features designed to help capture their audiences. The moves were meant to turn the services into destinations, rather than web transit points.
Now AltaVista is heading in a similar direction. In February, AltaVista added free email to its service. It has also added business and name searching, as well as a browsable directory, in recent months.
In the coming months, expect to see AltaVista unveil its own version of topical or channel content called "zones." They will feature information ranging from entertainment to health.
The changes put AltaVista firmly on a new path that its competitors have been walking for some time.
AltaVista launched back in Dec. 1995 as a demonstration project, and it has mainly been used as a technology showcase for Digital since. However, it's been a showcase with incredible brand recognition.
AltaVista rivals Lycos in terms of traffic, and yet it spends nowhere near the amount that service or others spend on marketing. That sort of traffic helped AltaVista earn nearly $20 million in revenue last year, according to marketing director Kathy Greenler.
Furthermore, AltaVista broke even with those earnings, officials say. That would put it just behind Lycos and well ahead of Excite and Infoseek, in terms of overall profitability.
It seemed that AltaVista would continue as a brand-building exercise for the AltaVista Internet Software division, which originally supposed to be spun-off into its own company last year. Digital then nixed those plans. In the wake of changes, it was decided to take AltaVista away from its search-centric roots.
"We decided not to stay a technology showcase but to instead compete as a media site," Greenler said.
AltaVista didn't necessarily have to go this way. It licenses out its results to numerous services, and it could have continued with simple banner sales. But its competitors see big money in expanding their content and incorporating commerce partnerships into it. AltaVista plans to follow suit, in order to better leverage the traffic it receives.
Meanwhile, for those still lacking a free email account, AltaVista joins Yahoo, Excite and Lycos in offering one. It has a partnership with iName, which also powers the free email offered by Lycos. Names are of the format firstname.lastname@example.org.
AltaVista to 'Zone' out
PC Week, Feb. 24, 1998
Some details on revenues and future plans.
AltaVista Picks A Strategy
Red Herring, Feb. 19, 1998
Excellent article on where AltaVista has come from and where it is heading strategically against its competitors.
AltaVista unveils free email
News.com, Feb. 10, 1998
Free email: the next Catch-22?
PC Week, Feb. 10, 1998
Quotes on why adding new free services like email might bring support woes.
HotBot has followed on AltaVista's lead last month and partnered with LookSmart to offer a browsable directory of web sites. LookSmart lists about 300,000 web sites in over 17,000 categories.
The directory maintains HotBot's branding, while still providing the same results reachable via the main LookSmart service. Users can take advantage of the service by clicking on any of the links below "Browse By Subject" on the HotBot home page.
"Enhancing HotBot with the Web's most comprehensive directory from LookSmart provides HotBot users another way to locate the information they seek. This new improvement reinforces HotBot's position as the Internet's pre-eminent search site," said Hunter Madsen, senior vice president for marketing at Wired Digital, which runs HotBot.
Keeping Tabs On AltaVista
The Search Engine Report, Feb. 3, 1998
Talks about the addition of the branded directory from LookSmart and how it works, which is similar to how it works with HotBot.
Excite announced Feb. 23 that it would be unveiling a new WebCrawler service this month designed for home users. The service is meant to relaunched as a fast, simple guide to the web.
WebCrawler will continue to have search listings and topical channels, but the intent will be to make the information less overwhelming than it may seem to some who use the company's flagship Excite service.
Excite also plans daily programming aimed to provide helpful hints and "news you can use" type information.
devSearch is a new search engine that crawls only selected sites with information useful to web developers. Over 20 sites, including WebMonkey, Builder.com and WebReference.com are on the crawl list. More are planned to be added. The search engine is backed by the creators of Project Cool.
The advantage to crawling a small list of select sites is that results tend to be much more relevant. It is something that Search.com has offered since the middle of last year. By default, it offers web wide searches (drawn from the Infoseek database). However, it also has Infoseek crawl a select list of sites in various categories, such as Computing or Entertainment. Searching within these categories may reveal better results for some people than a web wide search.
A Search Engine by Developers, for Developers
Wired, Feb. 6, 1998
There have been many attempts to create search engine discussion areas for marketers, but these usually become overwhelmed by noise. Now three moderated venues have launched, providing some useful forums.
The best so far is I-Search. This is a moderated mailing list run by AudetteMedia (MMG), which runs other well-regarded lists such as I-Sales.
Issues appear weekly. Topics include those of interest to marketers and to general search engine users. I also post answers to some questions in "Danny's Corner," which appears as time allows.
To subscribe, send email to: I-Search@gs4.revnet.com, with the word subscribe in the body or subject.
The Search Engine Secrets Discussion List got off to a rough start in January because it was unmoderated. It has just gone back up in a moderated version, which should help cut down on the noise. The list is devoted to promotion and marketing, not for general search engine users.
To subscribe, send a blank email to email@example.com.
VirtualPromote's newly launched Search Engine Forums have been interesting. They are devoted to each of the major search engines, and moderation helps keep the questions and answers tightly focused. A nice web-based interface makes it easy to post and move around. They can be found at:
With all these venues, keep in mind that many things are posted as fact, when in reality they are only speculation.
There are also places to talk about search engines without the marketing emphasis these venues offer. You'll find them at the link below:
Search Engine Mailing Lists
Infoseek said it is now keeping its instant Add URL feature off on weekends to help keep the index clean of spam. Without someone to monitor and review submissions, spam slips through, causing customer complaints. E-mail submission remains always open, though it takes a few days for URLs to be processed this way.
Infoseek Add URL
Search Engine Notes
DejaNews Offers Web-Based Newsreader
DejaNews is beta-testing a web-based newsgroup reader service called My Deja News. It tracks what you've subscribed to and what you've read.
My Deja News
Deja News offers personal Usenet
News.com, Jan. 27, 1998
Excite Launches Netscape UK Guide
Excite launched its "Netscape Guide by Excite UK" on Feb. 11. The service provides Netscape users with content and links organized into nine channels, such as Entertainment, Sport and Technology. The guide loads by pushing the Destinations or Guide buttons in the UK versions of Netscape 3 and 4. It can also be reached by the link below. This is Excite's third international guide that it has produced for Netscape. It maintains similar ones for Japan and Germany.
Netscape Guide by Excite in the UK
searchUK Comes Out Of Beta
searchUK was officially launched in early February. It has been in beta test since the summer of 1997.
The service indexes only UK-oriented sites. It automatically crawls UK domains, and it adds UK-sites using non-UK domains (such as .com) upon request and review of acceptability.
The site has about 2 million pages listed, and new additions are added each month. For developers, the site offers meta tag support.
MetaCrawler Adds Retailers
It's not just the main search engines that are getting in on retailing partnerships. Metasearch service MetaCrawler has added more online retailers to its MetaCrawler Marketplace. On Feb. 10, it announced that CDnow, Z Auction and eToys would join other recent online retailing partners such as the Audio Book Club and CarSmart. These partners appear prominently along the right-hand side of the MetaCrawler home page.
Yahoo Launches Everything
They've been busy over at Yahoo adding new content. Among the new offerings are a site designed for the older net surfer, another for television addicts in the US, and a third for American tax payers facing the upcoming April 15 deadline. And if you owe taxes, why not apply for the new Yahoo Visa card? It's free and has a web-based management feature, not to mention possible future souvenir value.
Yahoo Seniors' Guide
Yahoo TV Coverage
Yahoo Tax Center
Yahoo Visa Card
justQuotes.com Offers Financial Searching
justQuotes.com is a financial data search engine. Enter a company name or stock symbol, and you'll be linked to a page with quotes, financial data, related web sites and much, much more. It launched in February.
Search Engines Conference Planned
There's still time to book for "Search Engines and Beyond: a Landmark Conference." The meeting will be held in Boston, Massachusetts. Dates are April 1-2 1998. Attendees include speakers from PLS, Infoseek, Northern Light, Inference, Verity, Excalibur and Automony, among others.
Conference Program And Information
Search Engine Articles
Does Yahoo Still Yahoo?
Wired, Feb. 11, 1998
Nothing new here to Search Engine Watch visitors, but it is one of the first mainstream articles about problems getting listed in Yahoo. Imagine calling Wired "mainstream!" But it is, in terms of Internet coverage. For more on the topic, see also the Yahoo Special Report, http://searchenginewatch.com/webmasters/yahoo/.
Search Engines See Huge Value In Community Sites
Internet World, Feb. 9, 1998
Lycos buys Tripod; Yahoo invests in GeoCities. A look at why search engines are interested in these community sites.
Searching for "clean" content
News.com, Feb. 13, 1998
Inktomi is being tapped by N2H2 to power a search engine that hopes to be free of objectionable content. Launch is planned for Spring 1998.
The Internet Search-Off
Searcher, Feb. 1998
The Internet Search-Off asked professional researchers to pit traditional services such as DIALOG against web-based services such as AltaVista. In general, it was found that it takes longer to find information using web-based services, and that searches using web-based services are more likely to bring up irrelevant documents.
However, the Search-Off found that web-based services were useful for certain searches, such as gathering product information. It also lists when to use traditional services (for those with access to them) and when to use either.
The Search-Off also revealed interesting statistics regarding the most popular search engines, as rated by professional researchers. AltaVista was by far the favorite, used for 45% of the searches received in the Search-Off. HotBot came in at 20%, and Excite and Infoseek both rated 14%.
Searchers on the Beachhead
Searcher, Feb. 1998
Internet Librarians gathered in Monterey, California last November for a conference on searching. One of the highlights was a panel of search engine representatives. An account, for those that missed out on the fun.
Keeping 66 Sites Up, Holding Costs Down
Internet World, Feb. 2, 1998
Tech details on what it takes to keep Excite's many servers up and running.
Casting an Information Net
Upside, Feb. 2, 1998
A look at the challenges of information retrieval, ranging from web-based services to information technology. Lots of sidebars, with a listing of main players in different fields.
How Yahoo Won the Search Wars
Fortune, March 2, 1998
Some nice history here on Yahoo's origins and moves it has made in comparison to other search services. I'd disagree with the statement that Yahoo had the "worst technology," since I tend to think humans are the best technology. Interesting statement about Yahoo wishing to deliver the "best of the web." Past articles I've written have documented how easy it is for the "best" to go missing from searches. Also interesting is the statement that Yahoo "spends money on people, not computers." Actually, it spends much more money on its marketing than on its editors.
Getting There, or Not: Why Search Is So Ineffective
Internet World, Feb. 23, 1998
A short but interesting sidebar article that summarizes the many things that can cause searching to miss the mark.
Search Veteran Looks to Video
Internet World, Feb. 23, 1998
Excalibur, a company known for its intranet search products, is aiming toward video search technology.
Fresh Cash Infusion Invigorates Infoseek in Search Race
Internet World, Feb. 23, 1998
Infoseek has raised $40 million from selling stock and plans to use it to increase its market share. Analysts think it’s a good thing.
Netscape Re-Evaluates Search Relationships
Interactive Week, Feb. 27. 1998
That's right. It's coming up time for Netscape to renew its placement contracts for the Net Search page. The article says that Netscape is considering perhaps having its own branded search service. But with the search engines less dependent on Netscape for their traffic, perhaps it is them that will be reevaluating whether it is still worth millions to be listed.
The 7 habits of highly effective surfers
Cnet, Feb. 1998
Habit number one is a short summary of how to search better. It links to a nice chart (second link listed above) on picking the right search engine for the job, and another about advance searching (third link listed).
This newsletter is only sent to those who have requested it.
To unsubscribe, use the form at:
To subscribe, use the form at:
The contents of this report and excerpts of past reports can be found online at:
If you enjoyed this newsletter, consider showing your support by becoming a subscriber of the Search Engine Watch web site. It doesn't cost much and provides you with some extra benefits. Details can be found at: http://searchenginewatch.com/about/subscribe.html
This newsletter is Copyright (c) Mecklermedia, 1998