Yahoo Renews With Google, Changes Results
After months of speculation, Yahoo announced today that it has renewed its relationship to use Google's results as part of its search listings. In addition, Yahoo made a substantial change to end its historic barrier between human-powered and crawler-based search results.
Since its birth, Yahoo has used its own human editors or "surfers" as it calls them to organize web sites into categories. However, recognizing that humans can't index everything, Yahoo also has for years partnered with a third-party "crawler-based" search engine to provide answers for when there are no matches within its own human-powered listings.
Historically, the third-party search providers have been paid by Yahoo for the queries they handled. Since Yahoo has consistently been one of the most popular search sites on the web, the amount of queries it generates translates into serious money. In 2001, Google was paid $7.1 million by Yahoo for the search queries it handled, according to a Yahoo proxy statement filed in March.
Fighting For Yahoo
Competition for the latest Yahoo contract was intense. In addition to Google, both Inktomi and FAST were also seeking the contract. However, Google has made history by being the first provider to win the Yahoo deal twice in a row.
Yahoo has been a fickle about its partners. Open Text was the company's first partner, then AltaVista won the contract in mid-1996. It was then dumped for Inktomi in mid-1998, in particular because AltaVista was seen by Yahoo as competitor in the portal space while Inktomi ran a "behind-the-scenes" business model of powering but never competing with portals.
When Inktomi lost out to Google in 2000, this seemed both due to Google's growing reputation of having high quality search results and also what was widely assumed to be a better business proposition by Google to Yahoo. In fact, it turned out later that as part of the deal, Yahoo gained a small investment stake in privately-held Google.
Two Strikes But One Great Hit
Going into the latest competition, Google had two strikes against it. First, some at Yahoo believe that Google may be capturing their visitors. Google certainly does handle a search volume more than double that of Yahoo's as measured by "search hours," and Google has nearly equal Yahoo's 30 percent reach of the US search audience, according to figures from Nielsen//NetRatings.
In addition, a deal earlier this year between Yahoo and Overture precluded Google from giving Yahoo both its editorial and paid listings. As a result, Yahoo -- which is looking to maximize revenues wherever possible -- was potentially looking at a situation where it either had to pay Google to keep it as its partner or go with another provider and earn money.
This is something that both Inktomi and FAST could offer, because some of their crawler-based results are sold on a "paid inclusion" basis. Unlike with Overture, those purchasing paid inclusion are not guaranteed to receive a particular ranking. This means Yahoo could have gone with one of these partners without violating the exclusive deal Overture has to provide the company's US-based web site with guaranteed placement listings.
Despite these two drawbacks, Google had one major factor strongly in its favor. The company is widely acknowledged as being a leader for search relevancy. Indeed, Google has become a synonym for web search to some people.
Yahoo initially partnered with Google so that Yahoo's own users would feel they were getting both the quality of Google and the unique view that Yahoo's human-powered results bring to the web. Given this, dropping Google could have backlashed against Yahoo and made it seem as if the company was selling out search relevancy to gain cash, regardless of the fact that both Inktomi and FAST has very good relevancy themselves.
Using Google More Than Ever
In a unique twist, Yahoo didn't simply renew the deal for Google to be its "backup" partner, used only when Yahoo itself doesn't have an answer. Instead, the company has embraced Google's results even more tightly. Unveiled to the general public today is a new Yahoo search results page, where there is no longer a separation between Yahoo's own human-powered listings and Google's crawler-based results. Instead, the two are blended together.
To explain more, in the "old" Yahoo system, a search for a popular query such as "cars" would generally bring back matches from Yahoo's own human-powered listings on the first page of results, under the "Web Site Matches" heading. Only after going through all of Yahoo's own listings would users come to Google's crawler-based results, which appeared in the "Web Page Matches" area.
In the new system, both Yahoo and Google results are mixed together under an entirely new "Web Matches" section. Any listing drawn from Yahoo's own human-compiled categories is easily identified by having a "More sites about" link that appears below its description, along with a new red arrow pointing to the link, to call the attention of searchers to it. Click on this link, and you'll see where the web page "lives" within the Yahoo directory -- as well as find web pages similar to it. In contrast, listings that come solely from Google lack the "More sites about link."
Yahoo says it is using its own search algorithm to sort through its own listings and those from Google to determine which ones to blend into the top results that are presented. This may be so, but that Yahoo algorithm clearly takes its lead from Google. After testing 30 different queries, only 3 of them brought up radically different results between Yahoo and Google.
For the curious, two of these were suggested to me by Yahoo: "pizza" and "mexico." You can run them yourself to see how in these cases, the listings at Yahoo and Google do look substantially different. The other I discovered was "las vegas." But for queries such as "dragonball," "lord of the rings," "san jose realtors" and many others, differences are slight if any.
If the results are so similar, how can it be said that Yahoo is still using its own human-powered listings? The answer is that pages ranking well for queries in Google tend to be pages that are also listed in Yahoo. In fact, because a page is listed in Yahoo, Google will likely give it a ranking boost. So, Google has always in a way had "Yahoo" listed sites within its results. Now Yahoo is doing the opposite, blending in Google-only listed sites among its results -- and blending and ranking the data in a manner that seems so close to Google that the results sets often have little difference between them.
Better Than Google?
One distinction Yahoo has had over Google in the past is that it often presented a completely different view of the web. I'm not saying that Yahoo's results were better or worse because of this -- they were simply unique, and uniqueness is to be valued. Now Yahoo seems to be speaking with Google's voice -- so why not simply listed to Google directly.
One major reason may be that Yahoo's results clean-up Google's listings. As mentioned, both Google and Yahoo may list the same pages. However, Yahoo will provide the sites it lists with a human-reviewed title and description. Google, in contrast, will automatically generate one based on the content of the web page. That's not always perfect, so Yahoo's listings may be more readable.
For example, here's how one page within Google was described in a search for "nfl":
At Yahoo, the same page was listed in the same position but in this way:
Here's another example, this time for a search for "grand theft auto" and the listing from Google:
Flash Detection In Progress...
The official GTA2 Web site - check it out for all the latest info. and downloads for GTA2
Description: The official GTA2 page. Contains a demo and movie for downloads.
Category: Games > Video Games > ... > Grand Theft Auto 2
www.gta2.com/ - 3k - Cached - Similar pages
Here's the Yahoo listing for the same page:
In this comparison, Google gets credit for coming up with a nicer description than that written for the page by Yahoo editors. However, it's the title that leaps out at you -- and here, the title Google automatically generated is clearly inferior to the one written by Yahoo's editors.
How Blending Improves Yahoo
The Yahoo change will probably give the most improvement to its searchers for those queries where Yahoo has only a few matches in its own database. In these cases, the artificial divide has caused a poor search experience.
For example, a search in "old" Yahoo for "gluten free pancake recipes" brought up exactly one single, solitary listing from Yahoo's own human-powered database. That listing is easily lost among all the navigation and other clutter on the Yahoo results page:
Consequently, a searcher -- especially novice searchers -- might feel Yahoo is not providing much selection, not realizing that a further click on the "Web Page Matches" link would bring back nearly 3,000 matches to the query from Google's crawler-based results.
In new Yahoo, such sparseness won't happen. A user won't be forced to make a further click for those times when Yahoo's own human-listings have only a few matches. Instead, Google's results will automatically rise to fill the otherwise empty space.
The Directory Is Not Dead
Yahoo has been testing its new format randomly with some users over at least the past week, and many more started seeing the changes early today. Some initial reactions were that the Yahoo directory was dead -- that there was no value in being listed in it any longer. It can certainly feel that way given that the Yahoo results seem so identical to Google's. I've seen many post asking "Why pay to be listed with Yahoo" if you can just get listed with Google for free.
Here are some reasons to make those of you who've paid Yahoo's listing fee in the past feel better. First, Yahoo is continuing to display directory category links even above its highly-placed, Overture-powered "Sponsored Matches." This means that some people will continue to "detour" into categories and may perhaps find you this way. Nothing about this has changed.
Second, as mentioned, being listed in Yahoo is probably one of the factors that helped you get a good ranking on Google. Now that Yahoo is presenting results similar to those of Google, you may be getting a double payoff on that listing.
Having said this, those people who are currently pondering a Yahoo commercial listing -- and the $299 minimum per year fee that it requires -- may now want to wait a bit and see what happens with their rankings on Google. It could be that you'd naturally do well there without a Yahoo listing -- and if so, then you'll probably do well in the new results at Yahoo that so closely mimic Google's.
That's mainly advice I'd offer to those really watching their budgets. For those who are already spending ample amounts on search engine marketing, the Yahoo fee is a pretty cheap insurance policy for helping yourself on both Google and Yahoo. And should you fail to feel it paid for itself, then don't renew.
Was Google Really A Threat?
Earlier, I'd mentioned that some people have believed that "Yahoo built Google" and so felt that Yahoo would see Google as a threat.
Yes, the Yahoo-Google deal in 2000 clearly gave Google the credentials to be seen by potential search partners as a serious player, given that Yahoo trusted them. But if the deal hadn't happened, would Google be as big as it is now? I believe so. I don't believe people discovered Google through Yahoo by seeing the Google logo on Yahoo's results pages. Word-of-mouth about Google's great search results has always been to me the key factor in the company's success.
My gut feeling is fine, but can I provide more proof than that? To help me do so, Nielsen//NetRatings provided some custom analysis of the traffic to and from both Google and Yahoo, for 2001 and 2002. Unfortunately, they couldn't provide me with 2000 data, so I can't go back to the very beginning of the Yahoo-Google relationship. Still, the two years I do have provide some interesting findings.
Today, the web site sending the most traffic to Google comes from the search.yahoo.com domain, according to data for August 2002. Ah, ha! So the Yahoo deal has been helping Google. Not exactly.
The search.yahoo.com domain is what people saw (in August) when they did a search on Yahoo and got Yahoo's own human-powered listings. The Google logo was not displayed on these pages. So, how did people get driven from Yahoo to Google. The vast majority of them probably searched on Yahoo for "google" and found a listing for Google in Yahoo's own human-powered results. These were people who already knew they wanted Google when they came to Yahoo, not those who "discovered" Google by seeing the logo.
Also, while search.yahoo.com was the single non-Google site that sent Google the most traffic, it was responsible for only 2.3 percent of Google's unique visitors that month. Google is hardly dependent on it.
The domain sending Google the second highest amount of traffic to Google is google.yahoo.com, responsible for 1.7 percent of Google's traffic. Now this is the domain used by Yahoo during that month to show Google results -- which until recently, also had the Google logo on it. The same time a year ago, domain sent Google 3 percent of its traffic. Why the drop? One reason might be that during the last week of August 2002, Yahoo dropped the Google logo and the link to Google. The move may have indeed stemmed some outbound flow to Google.
That drop certainly could be used to argue that Yahoo has at least in part built Google. However, let's look at some last figures to show why Yahoo may be far wiser to embrace Google than push it away.
In August 2001, aolsearch.aol.com was Google's fifth largest referral, sending it 1 percent of its unique visitors. That domain is for those people who did a search at the time on AOL's own search engine and came to Google. Since AOL and Google didn't have a partnership at the time, these are people who almost certain got to Google by searching for it by name on AOL Search.
A year later, the figure for that domain is down to 0.3 percent of Google's traffic. AOL Search's traffic usage hasn't dropped enough to be responsible for this fall. Instead, the major change is that AOL now uses Google's results for its own search engine. Now, rather than its users leaving AOL to use Google, some of them certainly seem to have gotten the message they can get what they want right on Google-powered AOL Search.
Given this, rather than the Yahoo-Google deal further "hurting" Yahoo, it may well be that renewing it could keep some people with Yahoo.
And by the way, in Aug. 2002, Google was Yahoo's third largest external source of traffic, with www.google.com sending Yahoo 0.3 percent of its unique visitors.
More About The Deal
Yahoo released no specifics about the financials of the deal nor the time period, saying only that it was "long term." The deal between Yahoo and Google also applies to the Yahoo.com site alone. Other Yahoo properties are free to choose different third-party providers, if they so choose, according to Jeff Weiner, Yahoo's senior vice president of corporate development.
In fact, the deal at Yahoo.com is itself non-exclusive. If Yahoo wants, it can bring in an additional third party provider to provide results alongside Google.
"We are very open to working with multiple vendors," Weiner said. "We dont want to be dependent on any one vendor."
Is that just talk or a real direction Yahoo is heading? It could very well be real. Interestingly, Yahoo's new help pages about its new search results make no mention about some listings being powered by Google but instead refer to some results coming from "third-party search engine providers."
Why add another provider in addition to Google? This could give Yahoo the opportunity to still gain some paid inclusion income from a crawler partner while also maintaining the relationship with Google.
To that end, Inktomi says not to count it out of a Yahoo relationship -- it could yet happen.
"We still continue to talk with them, and weve made a lot of progress," said Vishal Makhijani, vice president and general manager of web search at Inktomi.
Given that Inktomi has 10 million paid inclusion URLs in its database -- far more than any other crawler -- it's not surprising that Yahoo may want to pursue a two-or-more vendor strategy. In addition, there's an interesting twist it could do to minimize the impact some might feel paid inclusion has on its results.
Meta search engines take results from multiple search engines and blend them together into one place. IxQuick.com is a great example, where the more search engines that report listing a particular page for a query, the more likely the page will rank well.
Now imagine that Yahoo does this. It takes the listings that Google provides, which many consider the best out there. It continues to enhance them with its own titles and descriptions. Finally, it compares them to the paid inclusion databases from Inktomi, FAST and any other major crawlers running these programs. If a page is in one of these programs -- especially on a cost-per-click basis -- then it might "technically" use the listing from one of those providers and earn a fee for doing so. Since the rankings would be the same as with Google, it would be difficult to accuse Yahoo of selling out -- yet it would be earning money all the same.
One way or another, paid inclusion in crawler-based results seems likely at Yahoo. Another new help page about Yahoo's search results clearly seems to be prepping the way: "Third-party search engine providers may include listings for which the third party has received payment to review and/or include the site within its database." Since Google doesn't do paid inclusion, this statement seem ready-made to allow the addition of another partner that does.
Here is some additional information received from Google after the above article was written:
+ The Yahoo.com deal involves only Google's editorial listings
+ No news on whether AdWords will be picked up on any non-US Yahoo sites. "Nothing to announce at this time, but we'd like to provide search monetization services to all of our partners," said Google spokesperson David Krane.
+ Yahoo is paying Google on a per query basis. Yahoo also says that Google is guaranteed a certain level of search volume.
+ Yahoo did not increase its stake in Google as part of the deal.
+ Yahoo's directory will not be integrated into Google or replace Google's use of the Open Directory as part of the deal.
+ Google will not be working on the new Yahoo shopping search service that Yahoo has said will be coming.
Also, I've seen a number of articles and comments that Google is no longer credited on Yahoo's search results page. As a matter of fact, Google is now actually far more credited than ever before.
Now at the bottom of ALL web searches at Yahoo is the text "Search Technology provided by Google." Prior to the deal, Google was only mentioned on the pages it provided if Yahoo found no matches of its own -- what I'd estimate to be one-third of Yahoo's web search queries. Having said this, Google is clearly less visible in that it has no logo presence nor hyperlink leading to its site.
What's changed with Yahoo Search?
Page from Yahoo explaining changes to its results.
What are the different sections on the search results page?
This page opens the possibility of paid inclusion within the Web Matches results.
Yahoo Partners With Google
The Search Engine Report, July 5, 2000
Coverage from when the first Yahoo-Google deal was announced, in particular about the logistics.
Good For Google Does Not Equal Bad For Inktomi
The Search Engine Report July 5, 2000
Coverage from the first Yahoo-Google deal, looking at reasons why Google won.
Nielsen//NetRatings Search Engine Ratings
More about "search hours" measurements and audience ratings can be found here.
Overture Wins Yahoo, What Will Happen With Google?
The Search Engine Report, May 6, 2002
Background on the Yahoo-Overture deal and how it impacted Google's ability to pitch Yahoo on its own ads.
Yahoo is GONE!
WebmasterWorld, Oct. 9, 2002
Comments on the changes at Yahoo, starting from those who saw early tests to when the changes were rolled out for everyone. Leads to:
Google Wins Yahoo
WebmasterWorld, Oct. 9, 2002
Comments on Google continuing its relationship with Yahoo.