Time For The Search Dividend?

When the Cold War ended, there was much talk in the US about the public receiving a "peace dividend," since defense funding could be reduced. Similarly, we may be seeing the beginning of a "search dividend" coming to searchers, now that several of the major search engines are maturing their new business models. That means instead of the news from the search front being dominated by the latest ways to buy your way into search engines, you may instead be hearing more about actual search improvements.

For instance, the news from Inktomi over the past few months has been dominated by its paid inclusion programs. The self-serve "Search/Submit" program went live in November, then new partners and program enhancements were added. Last month, an entirely new paid inclusion program aimed at the corporate market called "Index Connect" was announced. Despite the spin of Index Connect "enhancing" search results (and there is some truth to this), news of yet another program is enough to make some wonder if Inktomi is doing anything search-wise other than figuring out new ways to make money.

Fortunately, there is indeed more going on beyond revenue enhancement. Last month, Inktomi also rolled out new under-the-hood improvements to its core search technology. Known internally as Search 6, the company announced changes that it says should improve the quality of its results.

Also last month, Excite woke up out of its recent search slumber with a new search refinement tool, Zoom In. The company also has some ongoing changes to its search index underway, a sign that search isn't as dead as it may have seemed over there.

These follow changes in February at AltaVista, where the company finally introduced some actual changes designed to improve the quality of its results, rather than just site redesigns.

Meanwhile, Google continues to go from strength to strength. Like some type of search engine Switzerland, Google never allied with the portal model but instead concentrated on offering good search right from the beginning. The search dividends have always flowed from Google and continue to come, such as its recent addition of telephone results. The company says that revenues, which come from paid placement, site search services and providing web-wide search to others, will make it profitable later this year.

It remains to be seen how much the recent changes, in particular at AltaVista and Inktomi, have helped. I'm looking forward to doing some close relevancy comparisons in the coming weeks. Nevertheless, at least some welcome action in this area is being taken.

The various programs that I'm now lumping under the phrase "paid participation" shouldn't be dismissed as not helping to improve search results. First and foremost, they're keeping search engines in business, as aptly illustrated by the recent closure of Go.com.

Last month, money-losing Go.com was finally shut down by its owner, Disney. That is to say, Go's own editorially-driven search results were completely replaced with GoTo's yellow-pages style listings. GoTo's paid links can be a fine supplement to editorial listings, but they are not a substitute for the editorial information itself.

This change means that Disney, known for its high-quality products, is now foisting low-quality results on anyone who still decides to visit Go.com. I joked recently that it is as if Disney decided it was too expensive to license Winnie The Pooh any longer so instead created a different character called Vinnie The Pooh. The names may sound the same, but fans of Winnie don't want Vinnie.

Beyond survival, paid participation programs introduced over the past months allow an important conversation to take place between site owners and search engines. In the past, search engines never wanted to talk with site owners. The attitude was that site owners just wanted to spam the search engines or manipulate results in their favor. Moreover, any attempts at having a conversation took time, especially if a back-and-forth dialog (or argument) erupted.

However, site owners have very real concerns that deserve attention. I often speak about two types of "canaries" in the search engine coal mine, librarians and site owners, either of whom can tell you if something is wrong with search. Librarians have the training and experience to recognize top quality information and can quickly determine if a search engine seems to fail with its results. Similarly, site owners have expertise in the areas they are targeting. They often do have top quality sites that should be appearing in response to certain queries, and they certainly know which other sites ought to be there, also.

In addition, site owners know when things are wrong behind the scenes. Long before the famous Science study in 1998 about how crawler-based search engines don't cover everything on the web, many site owners already knew of this gap. They understood exactly how much -- or how little -- of their web sites were indexed. This remains the case, and site owners also know how out of date a search engine may be or whether it has failed to pick up important pages in preference to randomly collecting documents of little content value.

New programs such as paid inclusion, paid submission and paid placement now let site owners formally participate in the search listing progress, hence my grouping them under the paid "participation" heading. Conversations between site owners and search engines can now occur, because the time is being paid for through these programs. In addition, they appear to be reducing the amount of spam and low-quality submissions that were swamping search engines by introducing a cost barrier. For instance, Inktomi estimates that it gets seven times more quality content via its paid Search/Submit program than through the free Add URL pages its partners offer -- and paid content is twice as good as content it finds by crawling the web without site owner assistance, the company says.

It's also important to understand that search engines are following different paid participation paths that match the services they offer. Google, for instance, doesn't run a paid inclusion program while Inktomi does. Google has expressed concern that all of its content should be gathered freely from the web, and it's an admirable stance. However, Google also has an alternative revenue stream to depend upon, paid placements that appear on its own site. In contrast, Inktomi is not able to offer paid placements, since it does not operate its own portal. Hence, paid inclusion is a model Inktomi is having to pursue, especially to remain attractive to the hard-pressed portals it powers. They want a slice of paid inclusion revenues, in order to maintain their own services.

How about subscription-based searching? Infoseek tried charging people to search when it launched in 1995, and the experiment failed miserably. No one wanted to pay. However, now that paid listings are everywhere and image-based ads may be coming right within search results, as now happing at Ask Jeeves with its Branded Response program, perhaps there is a market for those who want just pure search.

The problem with this idea is that web search is not a service that deters users from seeking other sources, if you charge for it. For example, if Yahoo started charging to use its currently free email service, some people would still pay because they like the service and would be loathe to get a new email address, tell everyone how to reach them at the new address, establish a new online address book and other complications. However, if Yahoo charged for searching, users could easily turn to using one of several other search engines without problems.

Given this, I doubt we'll see general web search offered on a subscription basis. However, we could see premium search services offered. For example, perhaps having a US $49 annual search subscription with Yahoo would include access to an MP3 search engine allowing several legal downloads per month, a clip-art search engine offering copyright-free images that could be used, the ability to save ordinary searches and have them rerun through an alerting service, along with access to download several articles per month from a "premium content" database similar to the Special Collections documents that Northern Light offers.

I think such value-add subscriptions will be possible, but I don't know if ad-free subscriptions similar to what Salon is currently trying with its news and commentary site will work with search engines. One other complication involves what information gets removed. Do you drop paid links and banners? Sure, that's a fairly easy choice to make. But what about paid inclusion content? Losing some of this information would actually reduce the quality of search results.

I suspect that if ad-free search is going to happen, it will occur through companies offering meta search software such as Intelliseek and Copernic. They already have free products that download search results from different services. However, some search engines like Google and Northern Light actively block meta search tools, because they use their resources without providing any support. Perhaps those seeking ad-free search could turn to the search utility makers. They might be able to develop paid versions of their utilities that bring back results from all the key search engines, since these search engines would receive licensing fees from the software companies.

Hopefully, the moves we've seen to monetize search will pay off with search dividends, as we are now seeing emerge. What sort of improvements would you like to see? Let me know, and I may share a summary of comments with everyone in a future newsletter.

Pay For Placement?
Past articles from Search Engine Watch about paid participation programs and the issues, pro and con, on what these programs mean for the quality of web search.

Search Engine Watch Feedback
Comments on the search dividends you want to see? Let me know using the feedback form -- and put dividends somewhere in the subject line.

Search Utilities
You'll find meta search programs from Intelliseek (BullsEye) and Copernic listed here.

Inktomi Launches New Paid Inclusion Program, Search Improvements
The Search Engine Update, April 2, 2001

Excite Gets Search Refinement Feature, Paid Listings Coming
The Search Engine Update, April 2, 2001

Google Adds Phone Book, Languages And More
The Search Engine Update, April 2, 2001

Blending Vertical Results & Other AltaVista Improvements
The Search Engine Report, March 5, 2001

Portals Getting Creative With New Ad Units to Lure Traffic
DMNews.com, March 26, 2001

This article also touches on the Salon ad-free experiment and new Ask Jeeves advertising options, such as Branded Response.

Ask Jeeves Mixes Ads with Content
Internet World, March 23, 2001

More about the new "Branded Response" ads from Ask Jeeves.

All Eyes on Yahoo
Business 2.0, March 22, 2001

To stay afloat, Yahoo is considering subscription fees. Oddly, nothing is mentioned of the subscription-like fees Yahoo has charged for some time for those submitting to the directory. LookSmart has made its multimillion dollar "listing services" products the bedrock of its future. Yet, when it comes to Yahoo, analysts seem to ignore the listing services it offers.