After months of speculation, Yahoo announced today that it has renewed its relationship to use Google's results as part of its search listings. In addition, Yahoo made a substantial change to end its historic barrier between human-powered and crawler-based search results.
Since its birth, Yahoo has used its own human editors or "surfers" as it calls them to organize web sites into categories. However, recognizing that humans can't index everything, Yahoo also has for years partnered with a third-party "crawler-based" search engine to provide answers for when there are no matches within its own human-powered listings.
Historically, the third-party search providers have been paid by Yahoo for the queries they handled. Since Yahoo has consistently been one of the most popular search sites on the web, the amount of queries it generates translates into serious money. In 2001, Google was paid $7.1 million by Yahoo for the search queries it handled, according to a Yahoo proxy statement filed in March.
Fighting For Yahoo
Competition for the latest Yahoo contract was intense. In addition to Google, both Inktomi and FAST were also seeking the contract. However, Google has made history by being the first provider to win the Yahoo deal twice in a row.
Yahoo has been a fickle about its partners. Open Text was the company's first partner, then AltaVista won the contract in mid-1996. It was then dumped for Inktomi in mid-1998, in particular because AltaVista was seen by Yahoo as competitor in the portal space while Inktomi ran a "behind-the-scenes" business model of powering but never competing with portals.
When Inktomi lost out to Google in 2000, this seemed both due to Google's growing reputation of having high quality search results and also what was widely assumed to be a better business proposition by Google to Yahoo. In fact, it turned out later that as part of the deal, Yahoo gained a small investment stake in privately-held Google.
Two Strikes But One Great Hit
Going into the latest competition, Google had two strikes against it. First, some at Yahoo believe that Google may be capturing their visitors. Google certainly does handle a search volume more than double that of Yahoo's as measured by "search hours," and Google has nearly equal Yahoo's 30 percent reach of the US search audience, according to figures from Nielsen//NetRatings.
In addition, a deal earlier this year between Yahoo and Overture precluded Google from giving Yahoo both its editorial and paid listings. As a result, Yahoo -- which is looking to maximize revenues wherever possible -- was potentially looking at a situation where it either had to pay Google to keep it as its partner or go with another provider and earn money.
This is something that both Inktomi and FAST could offer, because some of their crawler-based results are sold on a "paid inclusion" basis. Unlike with Overture, those purchasing paid inclusion are not guaranteed to receive a particular ranking. This means Yahoo could have gone with one of these partners without violating the exclusive deal Overture has to provide the company's US-based web site with guaranteed placement listings.
Despite these two drawbacks, Google had one major factor strongly in its favor. The company is widely acknowledged as being a leader for search relevancy. Indeed, Google has become a synonym for web search to some people.
Yahoo initially partnered with Google so that Yahoo's own users would feel they were getting both the quality of Google and the unique view that Yahoo's human-powered results bring to the web. Given this, dropping Google could have backlashed against Yahoo and made it seem as if the company was selling out search relevancy to gain cash, regardless of the fact that both Inktomi and FAST has very good relevancy themselves.
Using Google More Than Ever
In a unique twist, Yahoo didn't simply renew the deal for Google to be its "backup" partner, used only when Yahoo itself doesn't have an answer. Instead, the company has embraced Google's results even more tightly. Unveiled to the general public today is a new Yahoo search results page, where there is no longer a separation between Yahoo's own human-powered listings and Google's crawler-based results. Instead, the two are blended together.
To explain more, in the "old" Yahoo system, a search for a popular query such as "cars" would generally bring back matches from Yahoo's own human-powered listings on the first page of results, under the "Web Site Matches" heading. Only after going through all of Yahoo's own listings would users come to Google's crawler-based results, which appeared in the "Web Page Matches" area.
In the new system, both Yahoo and Google results are mixed together under an entirely new "Web Matches" section. Any listing drawn from Yahoo's own human-compiled categories is easily identified by having a "More sites about" link that appears below its description, along with a new red arrow pointing to the link, to call the attention of searchers to it. Click on this link, and you'll see where the web page "lives" within the Yahoo directory -- as well as find web pages similar to it. In contrast, listings that come solely from Google lack the "More sites about link."
Yahoo says it is using its own search algorithm to sort through its own listings and those from Google to determine which ones to blend into the top results that are presented. This may be so, but that Yahoo algorithm clearly takes its lead from Google. After testing 30 different queries, only 3 of them brought up radically different results between Yahoo and Google.
If the results are so similar, how can it be said that Yahoo is still using its own human-powered listings? The answer is that pages ranking well for queries in Google tend to be pages that are also listed in Yahoo. In fact, because a page is listed in Yahoo, Google will likely give it a ranking boost. So, Google has always in a way had "Yahoo" listed sites within its results. Now Yahoo is doing the opposite, blending in Google-only listed sites among its results -- and blending and ranking the data in a manner that seems so close to Google that the results sets often have little difference between them.
Better Than Google?
One distinction Yahoo has had over Google in the past is that it often presented a completely different view of the web. I'm not saying that Yahoo's results were better or worse because of this -- they were simply unique, and uniqueness is to be valued. Now Yahoo seems to be speaking with Google's voice -- so why not simply listed to Google directly.
One major reason may be that Yahoo's results clean-up Google's listings. As mentioned, both Google and Yahoo may list the same pages. However, Yahoo will provide the sites it lists with a human-reviewed title and description. Google, in contrast, will automatically generate one based on the content of the web page. That's not always perfect, so Yahoo's listings may be more readable.
For example, here's how one page within Google was described in a search for "nfl":
At Yahoo, the same page was listed in the same position but in this way:
How Blending Improves Yahoo
The new change will be most noticeable -- and an improvement to searchers -- for those queries where Yahoo has only a few matches in its own database. In these cases, the artificial divide has caused a poor search experience.
For example, a search in "old" Yahoo for "gluten free pancake recipes" brought up exactly one single, solitary listing from Yahoo's own human-powered database. That listing is easily lost among all the navigation and other clutter on the Yahoo results page:
Consequently, a searcher -- especially novice searchers -- might feel Yahoo is not providing much selection, not realizing that a further click on the "Web Page Matches" link would bring back nearly 3,000 matches to the query from Google's crawler-based results.
In new Yahoo, such sparseness won't happen. A user won't be forced to make a further click for those times when Yahoo's own human-listings have only a few matches. Instead, Google's results will automatically rise to fill the otherwise empty space.
More About The Deal
Yahoo released no specifics about the financials of the deal nor the time period, saying only that it was "long term." The deal between Yahoo and Google also applies to the Yahoo.com site alone. Other Yahoo properties are free to choose different third-party providers, if they so choose, according to Jeff Weiner, Yahoo's senior vice president of corporate development.
In fact, the deal at Yahoo.com is itself non-exclusive. If Yahoo wants, it can bring in an additional third party provider to provide results alongside Google.
"We are very open to working with multiple vendors," Weiner said. "We don't want to be dependent on any one vendor."
Is that just talk or a real direction Yahoo is heading? It could very well be real. Interestingly, Yahoo's new help pages about its new search results make no mention about some listings being powered by Google but instead refer to some results coming from "third-party search engine providers."
Why add another provider in addition to Google? This could give Yahoo the opportunity to still gain some paid inclusion income from a crawler partner while also maintaining the relationship with Google.
To that end, Inktomi says not to count it out of a Yahoo relationship -- it could yet happen.
"We still continue to talk with them, and weve made a lot of progress," said Vishal Makhijani, vice president and general manager of web search at Inktomi.
What's changed with Yahoo Search?
Yahoo Partners With Google
The Search Engine Report, July 5, 2000
Coverage from when the first Yahoo-Google deal was announced, in particular about the logistics.
Good For Google Does Not Equal Bad For Inktomi
The Search Engine Report July 5, 2000
Coverage from the first Yahoo-Google deal, looking at reasons why Google won.
Nielsen//NetRatings Search Engine Ratings
More about "search hours" measurements and audience ratings can be found here.
Overture Wins Yahoo, What Will Happen With Google?
The Search Engine Report, May 6, 2002
Background on the Yahoo-Overture deal and how it impacted Google's ability to pitch Yahoo on its own ads.
Yahoo is GONE!
WebmasterWorld, Oct. 9, 2002
Comments on the changes at Yahoo, starting from those who saw early tests to when the changes were rolled out for everyone. Leads to:
Google Wins Yahoo
WebmasterWorld, Oct. 9, 2002
Comments on Google continuing its relationship with Yahoo.