The End For Search Engines?

The impending closure of Go only underscores the dramatic changes that have been taking place among the major search engines over the past few months. Money is tight; new revenue is being sought anywhere, and no one seems guaranteed a future. Will your favorite search engine be around tomorrow? For searchers, such losses could mean less diversity in search results. For web marketers, a consolidation could mean less likelihood of being found. It's scary sounding stuff, and no one knows the answers. However, a look back can provide us some perspective on how the future may unfold.

We've lost search engines before. The demise of Go is dramatic simply because old search engines are supposed to quietly fade away, not come to a screeching halt. Open Text is a good example of this. In 1995, Open Text was one of the big search engines people depended on. It was arguably bigger than Yahoo. For example, I remember being at Internet World in 1995 and stopping by the fairly large Open Text exhibit. It was big enough to graciously allow Yahoo a table within its confines, where none other than Jerry Yang was personally answering questions.

Open Text faced the classic search engine question of whether it should develop its technology as a product for businesses, concentrate on its consumer web site or try to do both. The company ultimately decided to concentrate on software, and the web search site was allowed to slowly die. By the time the web search engine closed in mid-1997, no one really noticed.

The same is true for Magellan, which was a rival that Excite gobbled up in mid-1996. Magellan has never closed, but Excite has allowed it to wither away and be essentially unsupported. WebCrawler is another example, which Excite also acquired in 1996. WebCrawler is far more popular than Magellan, but it can hardly be considered one of the web's major search services any longer. There haven't been service enhancements for ages, nor does Excite position the service toward users. However, at least it survives, unlike Point (Top 5%), another significant service that which was acquired by Lycos in 1996, then slowly allowed to disappear into oblivion.

So, losing a major service like Go isn't new. It's just the sudden nature that's shocking, especially when it comes at a time when so many major services all struggling at the same time. AltaVista has just gone through another round of layoffs. So have Excite, LookSmart and NBCi. Surely the end is near, and we're going to be left with only a few search engines.

Perhaps, but our memories can also fade. Search engines have been in trouble like this before. Back in late 1996 and early 1997, the situation seemed as dire as it does today. Importantly, the portal route search engines were advised to follow for survival is what caused the current crisis they face today.

"The market for consumer-oriented search and directory services is overcrowded and a shake out could be in the offing in 1997," warned Jupiter Communications, in November 1996. "There are simply too many players offering similar functionality and features, competing for a limited number of advertising dollars and users," the company advised. To survive, the search engines needed content and services to "keep users" at their sites, which would boost ad revenue.

A few months later, in April 1997, MSNBC ran an article called "Searching for success - or survival," which was typical of several other pessimistic business pieces about search engines at the time. It discussed some of the early changes search engines were making to evolve into portals, before that word was even being used (instead, search engines were positioned as the new "online services" to rival big players like AOL, CompuServe and Prodigy).

"1997 is 'fish or cut bait' for some of them," one analyst said in the article. Funding was running low, and the analyst, like practically every other analyst quoted in that article, assumed that there would be a massive search engine consolidation that year.

In May 1997, Excite laid off half its editorial staff, prompting a Red Herring article to ask, "Can Excite make it without a deep-pockets old-media partner like Bertelsmann [which had just invested in Lycos” or Yahoo investor Softbank?"

(Interestingly, Go, Excite and Lycos all went on to get media partners, which hasn't seemed to help them gain on Yahoo. Nevertheless, some analysts continue to hound Yahoo to find its own partner, despite its success in going it alone).

The article went on to say, "While most observers think there's room for at least two navigation sites, it's up in the air which pair will be left standing." A Jupiter analyst was also cited, saying, "A shakeout is definitely imminent."

Doom and gloom, doom and gloom. When it comes to search engines, you can only have Coke and Pepsi and maybe RC Cola, we were told (literally -- Excite CEO George Bell was just one of several to say this, at the time).

The turnabout came in July 1997, when Amazon announced a landmark deal to be a preferred book merchant carried by Excite and Yahoo. Suddenly, the big online retailing deals began rolling in, just the pill the money folks said was needed to save the search industry.

"Analysts have been hounding search companies to break away from the advertising-only revenue model and into a more diversified business," reported in August 1997, in a story that discussed yet another new online retailer deal, this time between Lycos and Barnes & Noble.

To maximize the value of those deals, search engines still needed to keep users at their sites. How to make the sites "sticky?" Excite had made a significant move in July 1997, when it began providing free email. Lycos and Yahoo soon followed, in October 1997. That was the same month that the first mentions of the P word -- portal -- began to appear.

"[Microsoft” is expected to use the search technology to create a web search engine that would be at the center of a revamped web site, positioned as an Internet 'portal,' in which a major portion of the content now available on the proprietary MSN service would be on the web for anyone to see," MSNBC wrote this in October 1997, when Microsoft announced plans to run its own search engine. The idea of "portals" was so new that MSNBC article twice surrounded the word with quotes within the article. Today, it sounds almost like Dr. Evil from Austin Powers carefully explaining a basic concept that everyone knows about.

By 1998, instead of the expected shakeout, the future of search engines seemed bright due to their transformation into portals. Investors loved them. They had tons of traffic that seemed certain to produce huge revenues. As for the portals themselves, accidentally call them search engines, and you'd be informed they'd grown well beyond that. Search was only a small part of what they offered, you'd be told. It was almost as if the portals were embarrassed of their search engine roots.

Well, why not? Being search engines nearly cost the portals their lives. People came, searched and went, ignoring banner ads that were a vulnerable single-source of revenue. Portal features, though inconvenient to those looking just to search, were what seemed to have saved the search engines. Search was really a loss leader to them, except mainly for the saving grace of selling overpriced keyword-linked banner ads. Why did these sell? They were easy to implement, and if an advertiser bought enough, they'd still get tons of traffic, no matter how poor the clickthrough rate was. Advertisers, flush with investment, were happy to shell out for huge orders. Indeed, there were so many banner advertisers, along with would-be retail partners, that some complained they couldn't get search engine sales reps to call them back.

Something else happened at the beginning of 1998, the year when we were supposed to have only two or three search engines remaining. A new search engine was launched in February, GoTo. It dared to do what no search engine wanted to touch since Open Text's failed experiment in 1996: sell paid listings. GoTo's business model was absolutely counter to everything the portals had evolved into. Instead of trying to keep people at GoTo, the service wanted them to leave, because outbound departures were how GoTo earned its money.

It was easy for GoTo to gamble with paid listings. It had no reputation to lose. The major search engines -- oops, portals -- were more conservative. They didn't want a backlash at attempting paid links, nor did they really need to. After all, we're big, we're strong, we're the portals. We've got advertisers coming out of our ears and no need to scrape for pennies per click.

Then, a funny thing happened in April 1999. Jupiter warned that the big retailing deals with portals weren't so hot. "Despite the huge volume of traffic portal sites offer commerce players, revenue expectations for portal tenancy deals are often overly ambitious and are destined to be unrealized," Jupiter advised commerce sites. It was a first shot that would begin to haunt the portals in early 2000, when articles began appearing about the demise of portal deals.

"In addition to these bet-the-farm deals, countless more startups have signed smaller contracts-worth $2 million to more than $10 million-for placement on portal pages, the most expensive real estate on the Net. Now that many e-commerce companies are struggling to stay alive, the future of these deals is in doubt. If they fall apart, or are renegotiated on more lenient terms, it would force portals to replace the lost income with new sources of revenue," wrote the Industry Standard, in April 2000.

In addition to the loss of big retailing deals, portals also suffered from the decline in more ordinary banner advertising, as a result in the crash of dotcom stocks in 2000. That's two strikes against them, and the third was really that it is expensive to run a portal. If you don't have the traffic, you need to get it -- which some portals went after by spending millions on television ads. Or, if you do get the traffic, you still have problems providing all these free features for free. As a result, players started dropping out of the portal game. Both Go and AltaVista pulled back from wanting to be all-purpose portals in September 2000, for example. The consolidation long predicted was finally here.

Perhaps. The consolidation predictions started out talking about search engines, and I've repeatedly said each time they've been made that I don't believe that we must have only two or three major search engines. Not portals -- search engines. Yes, maybe we are going to be left with AOL, MSN, Yahoo as the major portal players. However, that doesn't mean other players can't survive and thrive as focused search services. People have very different views about relevancy, and I think it is entirely possible for us to have several major search engines serving the incredibly huge web audience. The key is monetization. The services have to find new ways of making money that don't require users to stay at their sites.

I've already written before in some depth of the new methods being explored, such as paid listings, paid submission and paid inclusion programs. One or more of these revenue streams has been implemented by every major search engine you care to name -- yes, even at Google. To date, there's been little outcry against the programs, and in fact, the searching public needs to continue to support such moves. Failure to do so means that the search engines will indeed face losing one of the last lifelines they have for staying afloat. This isn't 1996, the portal model doesn't work and wasn't pleasing to searchers anyway, so the various paid options I've named are now the leading way for search engines to continue providing search for free.

In return for public support, the search engine industry needs to ensure it doesn't become a free-for-all. Players such as Netscape Search, Lycos and HotBot need to cease calling paid links "Featured Listings" or "Partner Search Results" and begin using what has become the more common and more honest terminology used by services such as AOL Search, Google, and AltaVista -- "Sponsored Links." Meta search engines that are overloading their results with too many paid links need to reevaluate what they are doing or face users deciding they are going to go elsewhere.

LookSmart, which has a choke-hold on the paid submission and paid inclusion model, needs to loosen up a bit and figure out a guaranteed way for quality non-profit sites to get listed, perhaps by charging a low, token fee to eliminate spam but not make non-profit submission unaffordable. Visit the LookSmart site today, and you'll only find the non-profit submission form after a struggle -- it ought to be listed prominently alongside LookSmart's other submission options. In contrast, Yahoo's system of limiting mandatory fees to commercial categories makes sense. Perhaps we'll see a hybrid approach -- some categories may cost more than others, allowing non-profits to pay a smaller fee while letting search engines better capitalize on what they charge to appear in popular commercial categories.

Similarly, Inktomi has suggested that it will figure out some mechanism to ensure that non-profit sites are better included in its index, as a response in kind to the deeper indexing it does of some sites that pay. Let's see some action on this, rather than reassurances that it will happen sometime "soon."

Webmasters, who are being asked to foot the bill these days, also need some greater benefits in return. Too many complaints are coming in about decent descriptions being rewritten when submitted through the paid submission systems at LookSmart and Yahoo. Perhaps the editors need to take more of the site owners' concerns into account, while still trying to maintain editorial quality. The two can meet.

As for GoTo, which is the big winner these days, distributing its links far and wide, it needs to now consider a way for webmasters to control exactly which search engines may carry their links. For instance, I might find that AOL Search converts well for my site but the same link running at Lycos isn't a performer. Why should I be forced to pay to appear in both places? If I ran banners through a network, I could pick and choose the places I'd want them to appear. Similarly, I could run different banners on different web properties. GoTo really is no longer a search engine and rather a search engine ad network. As such, I should be entitled to the same control an ad network offers, allowing my paid listings to appear where I choose and even to have different descriptions, depending on the service.

The industry also needs to ensure that it keeps improving the quality of search. Google has been the shining example of a service offering good search that has attracted users without needing to spend much in advertising dollars. Yes, many users have tolerated the poor search some portals have provided. But, the long term strategy for survival should be to provide a great search product along with complementary ways to earn money off that product.

The Search Engine Revenues page available to site subscribers provides links to many articles written since 1996 from various publications about the financial aspects of search engines
Click here to learn more about becoming a site subscriber

Monetizing The Search
The Search Engine Report, Sept. 4, 2000

Covers the trend for search engines to seek ways of earning money by turning to site owners and web marketers. Touches on "paid submission" systems at LookSmart, Inktomi and Yahoo, "pay for display" at RealNames and "paid inclusion" at LookSmart.

Portals: Who gets the bigger slice?
Upside, Jan. 31, 2001

Long look at the leaders and wanna-bes remaining in the portal race.

Second-Tier Portals: Going the Way of
Business Week, Jan. 31, 2001

The writing is on the wall -- second-tier portals are doomed to die. Certainly they need to find a business model that works, but perhaps they've all gone down the portal route too far to come back. What I especially love is that Go followed the latest advice of the analysts to go vertical, but it was too late to save the company. Of course, Go followed the exact advice of earlier analysts in its incarnation as Infoseek, to find a big media partner -- Disney -- and turn itself into a portal to survive.

Yahoo Meets Street Expectations, Cuts 2001 Forecasts
Internet Advertising Report, Jan. 10, 2001,,12_557361,00.html

Expect the next year to be rough, Yahoo tells investors -- but it also says that efforts to diversify revenue away from ads will help it thrive going forward. Ads make up 90 percent of Yahoo's income.