THE SEARCH ENGINE UPDATE
March 3, 1998 - Number 24
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch, http://searchenginewatch.com/.
Please note that long URLs may break into two lines in some mail readers. Please cut and paste, should this occur.
In This Issue
+ Site Updates
+ Bridge Page Follow-Up
+ GoTo Sells Positions
+ AltaVista Changes Direction
+ WebCrawler To Be Relaunched
+ Infoseek Add URL Off On Weekends
+ Search Engine Notes
+ Search Engine Articles
The Subscriber-Only Area will be moving to a new location shortly. This should not cause a problem for anyone, as I'll simply redirect anyone going to the old location to the new one, if it occurs before the next newsletter. For all of you who have forgotten, the area is currently located at http://calafia.com/subscribers/.
Within the Subscriber-Only Area, this page is new:
Search Engines And Dynamic Pages
Do you deliver web pages using a database, CGI-scripts or through another dynamic means? This page aims to help you avoid problems that sometimes occur.
Within the main site, these pages are new:
Search Engine Results Survey
This is a project I'd planned to do for some time. It shows you how well each search engine is ranked on the various other search engines. Guess what? Not very well, in some cases. This is more than just trivia. It helps illustrate the particular problem of searching for companies on the web. Top honors go to Infoseek, for listing the appropriate home page at the top of the results more than the others.
Search Engine Spam Survey
One good survey deserves another. This page looks at the quality of results returned for a search on "monica lewinsky," in particular to spot what opportunistic spam made it into the top ten. Infoseek wins top marks again for keeping out spam.
These pages have been updated:
Search Engine EKGs
The Search Engine EKGs show how well the search engines are crawling the web, based on activity from sample sites. They have been significantly redone. A third survey site has been added. Each search engine now also has its own page, topped by an estimate of how complete and fresh the search engine is based on observed activities. The EKGs provide only a rough glimpse into how well things are working under the hood, but even these few sample sites can be revealing.
Media Metrix Search Engine Ratings
Media Metrix now shows Infoseek in the number two spot, above Excite.
Search Engine Profits
Now shows the latest numbers from Lycos, which posted a $300,000 profit, its second in a row. Only Yahoo has had two consecutive profitable quarters in a row, to date.
Search Engine News
There was quite a bit of reaction to last month's article about bridge pages. In particular, I heard from the company that manages State Farm's bridge pages, Green Flash Systems. CEO D.R. Peck has written up a guest response that's well worth reading. A link is below.
Peck discusses whether the methodology deployed by Green Flash (bridge pages coupled with IP delivery) is grounds for penalizing the high rankings they achieved, or whether the positions should be judged as to their relevancy for a given search phrase.
Also, more clarification on last month's article. If you recall, only Infoseek said they had a problem with the pages Green Flash had submitted for State Farm. At the time, they said the content of the pages appeared to violate their spamming guidelines. Since then, they have been more specific, saying it was the redirection itself that they disliked.
Bridging The Gap
Search engine GoTo debuted its new pay-for-placement service on Feb. 21. It allows web site owners to bid for placement. Those willing to pay more can appear higher in the search results.
Pay-for-placement is not new. Open Text experimented with it in mid-1996, and numerous small search directories currently offer the ability to pay for a better ranking.
Despite this, the major search engines have avoided such a service. This is mainly because they see their results as akin to editorial copy. They aren't something that advertisers are supposed to influence. By allowing pay-for-placement, search engine users may begin to mistrust the results they receive. Such complaints helped cause Open Text to give up its experiment.
Does that mean GoTo's results are bad? Not necessarily. Founder Bill Gross makes an interesting argument on how money produces better results.
"This is an opening shot of changing the search engines from a white pages to a yellow pages," Gross says.
What he means is that with a white pages phone book, numbers are organized alphabetically. Money does not get you higher in the results, though some people use company names such as "A Typographical Service" to try and jump ahead of others.
With a yellow pages phone book, money is central. The more you pay, the bigger ad you can have. That doesn't make the yellow pages any less useful. It's a very convenient way to find businesses, and often those with the bigger ads are the better ones.
Of course, there are plenty of good businesses without big budgets, and there are plenty of bad businesses that can afford large ads. So the ability to spend money does not guarantee quality. But it can be a useful ranking tool, and one that Gross says makes more sense than leaving it to search algorithms.
"All you're really getting back is how good people are at tricking the search engines, not how good people are, Gross said."
It's overstated to say only people who know how to trick the search engines get top listings. However, it's not far off the mark when it comes to some popular phrases or one word search terms. The more general and more popular the search, the more likely you'll see people who are aggressively fighting for a top placement.
A search for "monica lewinsky" illustrates this rather easily. A recent survey I did revealed that on average, one-third of the top sites listed in the major search engines were "clickthrough" spam sites.
These are sites that earn referral fees from people clicking through from their sites to other sites. The site owners make pages that the search engine algorithms think are relevant but which most people would probably consider junk.
Beyond this opportunistic activity, there's continuing competition to do well for all sorts of product related terms. While it's heartening to still see small sites rise to the top "naturally," the reality is that people are spending more and more time clawing their way to the top. The days of being listed for popular terms without expending effort aren't gone, but they are receding.
Since time is money, it's easy to argue people are already buying their listings. For those without time or skill, they can turn to search engine placement services. These services will oversee the submission of bridge pages and monitor them for you constantly. They get paid by the click, so if you've got the budget, they can seem a saving grace to the otherwise confusing world of improving your placement. That's assuming you get the listing you are looking for, of course.
Given all this activity, it might be advantageous to search engine users if the search engines took back control of the process. By selling placement for the most popular terms, they might help defuse the arms race of spamming and aggressive placement that exists. Results would certainly benefit, if not just for the fact that opportunistic spam would lose some of its incentive.
It would also come as a great relief to web marketers, who scratch their heads trying to keep up with what's the best way to get listed. Rules about hidden text, repetition in meta tags and the need for HTML text in a graphic-intensive site are confusing and complex. In contrast, pay-for-placement is simple and makes sense.
What about those with little or no budget? Would material at universities be locked out, for example? Again, this material is probably already poorly ranked for many popular and general searches. Pay-for-placement wouldn't hurt it, and the material would still remain accessible for longer and more specific searches.
For example, pay-for-placement might mean a realtor in Newport Beach, California would no longer be listed "accidentally" for "real estate." However, they may still do very well for "newport beach homes for sale."
So there are many reasons why pay-for-placement makes sense. There's also a big reason against it. It just doesn’t feel right.
If you go to a search engine and enter something like "microsoft," many people would expect the service should probably list the Microsoft web site. That's a relevant result, and you want it regardless of whether Microsoft has made an ad buy to ensure it happens.
Likewise, if you do a search for "real estate," you might appreciate having a human editor point you toward a few sites about real estate, not just toward major realtors. You might also appreciate some mechanism that helps you be more specific about what you are looking for.
In that light, what's probably needed is more human intervention in results, not commercialization of them. Search engines might create pages of links for these topics, carefully chosen for relevance. There would probably be a place for advertising on these pages, yet done right, it shouldn't offend anyone concerned over the results being "influenced."
This move could please everyone. Those with money could pay to be placed. Those with quality sites could request being listed on the merit of their content. Users would be given more guidance than they get sometime from raw search results. Spamming should decrease naturally, as incentive disappears.
Raw search results would remain, of course. You can't create pages for everything, and even then, people may still want more choices. However, these popular topic pages would likely satisfy many users.
You can see these types of pages growing already. Excite's Entertainment channel or the Lycos Web Design guide, both which can be found from the home page of either service, provide more organized and relevant information that a blind search for "entertainment" might yield. At some point, it is likely they will make the leap to supplant raw results, in some cases.
Ironically, GoTo had these precise type pages. A search for "travel," for example, brought up "The Travel Page." It was nicely organized, with sites listed in side-by-side columns under topics such as "car rental" and "air travel." At the bottom were raw search results. Now, only raw results are presented.
Enough about the pros and cons of paid listings. How about GoTo itself? For one, GoTo is not a new search engine. It was launched in 1997 and incorporates the former University of Colorado-based World Wide Web Worm. However, its results have been algorithm ranked until now.
To make a purchase, advertisers open an account and note how much they wish to "bid" per click. For example, Toyota currently comes up tops on a search for "cars," and the amount of $0.02 is displayed next to the listing. That means Toyota has agreed to pay 2 cents for anyone clicking on its link.
Honda could open an account and choose to pay 3 cents per click. If so, it would rise above Toyota. In the case of a tie, the site which users have voted best gets the priority.
The way users get to vote is probably the most annoying thing about GoTo. Selecting a listing launches a frame, with the actual destination page embedded in the frame. The frame allows users to vote on the site, similar to how Lycos sets up a frame when people browse links in its community guide. It can also be easily turned off. But it seems too intrusive, and certainly anyone who actually paid to be listed might prefer to have the screen all to themselves.
What impact will GoTo have on the other search engines? Probably little. A quick call to representatives at Excite and Lycos found minimal interest.
"It will be interesting to see how this plays out. My feeling that the consumer wants something more cleaner than commercialism," said Brett Bullington, Executive Vice President of Strategic and Business Development at Excite
Likewise, Lycos search manager Rajive Mathur said, "I'm not sure it's really providing value to the user, in the long term. I think they want some independent sorting"
The closest Lycos is likely to come to pay-for-placement for the time being are the Bullseye ads it has been experimenting with for the past month or so. They put advertisers just above the raw search results section, though they remain physically separated from it. They should be available to all advertisers within the next week or so, at $75/CPM.
How about the impact GoTo will have on marketers. Will they suddenly flock to the service? Perhaps if it grows more, but it has a way to go.
GoTo has well below the traffic of the major search engines. It's at 5 million page views per month, about one-fifth of what Excite generates. But Gross is a well-known Internet entrepreneur. His company, idealab, is backing GoTo and about to launch a new advertising campaign. Awareness may grow, which in turn may help make pay-for-placement more acceptable than it has been in the past.
Gross stresses that he can experiment with the model more easily. "Since we have no brand to worry about, we can get away with it easier than most," he said. In fact, Gross hopes that his service fills a niche that the other services won't be able to immediately, either because of consumer opposition they would face, or due to lack of interest.
GoTo Advertising Placement Page
Engines Battle Irrelevance of Results
Internet World, Feb. 23, 1998
GoTo Searches With Capitalistic Engine
Wired, Feb. 24, 1998
Pay-for-placement gets another shot
News.com, Feb. 19, 1998
New Search Engine Goes Commercial
News.com, Feb. 18, 1998
Lycos Adds New Features,
Reorganizes Suggested Links
The Search Engine Report, Jan. 9, 1998
Talks about the "Bullseye" links advertisers can purchase to appear above the raw search engine results, along with other changes.
Over the past year, search services such as Excite, Yahoo and Lycos have added free email, chat, discussion areas and other features designed to help capture their audiences. The moves were meant to turn the services into destinations, rather than web transit points.
Now AltaVista is heading in a similar direction. In February, AltaVista added free email to its service. It has also added business and name searching, as well as a browsable directory, in recent months.
In the coming months, expect to see AltaVista unveil its own version of topical or channel content called "zones." They will feature information ranging from entertainment to health.
The changes put AltaVista firmly on a new path that its competitors have been walking for some time.
AltaVista launched back in Dec. 1995 as a demonstration project, and it has mainly been used as a technology showcase for Digital since. However, it's been a showcase with incredible brand recognition.
AltaVista rivals Lycos in terms of traffic, and yet it spends nowhere near the amount that service or others spend on marketing. That sort of traffic helped AltaVista earn nearly $20 million in revenue last year, according to marketing director Kathy Greenler.
Furthermore, AltaVista broke even with those earnings, officials say. That would put it just behind Lycos and well ahead of Excite and Infoseek, in terms of overall profitability.
It seemed that AltaVista would continue as a brand-building exercise for the AltaVista Internet Software division, which originally supposed to be spun-off into its own company last year. Digital then nixed those plans. In the wake of changes, it was decided to take AltaVista away from its search-centric roots.
"We decided not to stay a technology showcase but to instead compete as a media site," Greenler said.
AltaVista didn't necessarily have to go this way. It licenses out its results to numerous services, and it could have continued with simple banner sales. But its competitors see big money in expanding their content and incorporating commerce partnerships into it. AltaVista plans to follow suit, in order to better leverage the traffic it receives.
Meanwhile, for those still lacking a free email account, AltaVista joins Yahoo, Excite and Lycos in offering one. It has a partnership with iName, which also powers the free email offered by Lycos. Names are of the format [email protected]
AltaVista to 'Zone' out
PC Week, Feb. 24, 1998
Some details on revenues and future plans.
AltaVista Picks A Strategy
Red Herring, Feb. 19, 1998
Excellent article on where AltaVista has come from and where it is heading strategically against its competitors.
AltaVista unveils free email
News.com, Feb. 10, 1998
Free email: the next Catch-22?
PC Week, Feb. 10, 1998
Quotes on why adding new free services like email might bring support woes.
Excite announced Feb. 23 that it would be unveiling a new WebCrawler service this month designed for home users. The service is meant to relaunched as a fast, simple guide to the web.
WebCrawler will continue to have search listings and topical channels, but the intent will be to make the information less overwhelming than it may seem to some who use the company's flagship Excite service.
Excite also plans daily programming aimed to provide helpful hints and "news you can use" type information.
Infoseek said it is now keeping its instant Add URL feature off on weekends to help keep the index clean of spam. Without someone to monitor and review submissions, spam slips through, causing customer complaints. E-mail submission remains always open, though it takes a few days for URLs to be processed this way.
Some people have also had problems with automated submission programs. Infoseek reiterated that it does not accept automated submissions via is Add URL page, nor does it offer any support for those trying to use them despite its ban.
If you are using such a program to submit a large number of legitimate web pages, consider using email submission instead. It is very effective and will efficiently add hundreds of pages to the index quickly. I've also found Infoseek has greatly improved its crawling over the past few months, so it's no longer as crucial to submit each page. You may wish to see what it adds on its own, then follow up adding any important pages it may have missed.
Finally, it's been reported elsewhere that Infoseek may not index web pages which contain "index" in the file name. This is definitely not the case, Infoseek says.
What generally happens is some people use automated template programs that create slight variation of the same pages, often with names like index1.htm, index2.htm, index3.htm, etc. The goal is to hit it "right" with one of these pages. However, Infoseek along with the other major search engines have greatly improved their ability to detect near duplicate pages. They often won't allow them in, or they may allow only one of them to be listed.
This isn't a worry for most people. In short, don't fear that you have to change your file names from index.html or index.htm, or something similar.
Infoseek Add URL
Search Engine Notes
Yahoo Launches Everything
They've been busy over at Yahoo adding new content. Among the new offerings are a site designed for the older net surfer, another for television addicts in the US, and a third for American tax payers facing the upcoming April 15 deadline. And if you owe taxes, why not apply for the new Yahoo Visa card? It's free and has a web-based management feature, not to mention possible future souvenir value.
Yahoo Seniors' Guide
Yahoo TV Coverage
Yahoo Tax Center
Yahoo Visa Card
justQuotes.com Offers Financial Searching
justQuotes.com is a financial data search engine. Enter a company name or stock symbol, and you'll be linked to a page with quotes, financial data, related web sites and much, much more. It launched in February.
Search Engines Conference Planned
There's still time to book for "Search Engines and Beyond: a Landmark Conference." The meeting will be held in Boston, Massachusetts. Dates are April 1-2 1998. Attendees include speakers from PLS, Infoseek, Northern Light, Inference, Verity, Excalibur and Automony, among others.
Conference Program And Information
Search Engine Articles
Getting There, or Not: Why Search Is So Ineffective
Internet World, Feb. 23, 1998
A short but interesting sidebar article that summarizes the many things that can cause searching to miss the mark.
Search Veteran Looks to Video
Internet World, Feb. 23, 1998
Excalibur, a company known for its intranet search products, is aiming toward video search technology.
Fresh Cash Infusion Invigorates Infoseek in Search Race
Internet World, Feb. 23, 1998
Infoseek has raised $40 million from selling stock and plans to use it to increase its market share. Analysts think it’s a good thing.
Netscape Re-Evaluates Search Relationships
Interactive Week, Feb. 27. 1998
That's right. It's coming up time for Netscape to renew its placement contracts for the Net Search page. The article says that Netscape is considering perhaps having its own branded search service. But with the search engines less dependent on Netscape for their traffic, perhaps it is them that will be reevaluating whether it is still worth millions to be listed.
The 7 habits of highly effective surfers
Cnet, Feb. 1998
Habit number one is a short summary of how to search better. It links to a nice chart (second link listed above) on picking the right search engine for the job, and another about advance searching (third link listed).
To unsubscribe from this list, send a message to [email protected], with the following in the subject field:
If you have problems, just send a message to [email protected]
This newsletter is Copyright (c) Mecklermedia. It should not be distributed. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at:
Meet Your Favorite Search Engine Watch Contributors
Many of SEW's leading expert contributors will be at ClickZ Live, the new online and digital marketing event kicking off in New York (March 31-April 3). Hear from the likes of: Thom Craver, Josh Braaten, Lisa Barone, Simon Heseltine, Josh McCoy, Lisa Raehsler, Greg Jarboe, Dan Cristo, Joseph Kerschbaum, John Gagnon, Eric Enge and more!