Yahoo Announces Content Acquisition Program

Just two weeks after launching a brand new search engine, Yahoo has announced a content acquisition program that consolidates all of its paid inclusion programs and marks the beginning of an aggressive new campaign to significantly expand both the scope and quality of content available via Yahoo search.

A longer version of this article that goes into more details about the Yahoo CAP program, including pricing information, potential ranking problems, and an analysis and response to rumors about the new program is available to Search Engine Watch members.
Click here to learn more about becoming a member

The new program, called CAP (Content Acquisition Program), focuses on both commercial and non-commercial web sites. On the commercial side, the new program is called Overture Site Match. The non-commercial effort is geared toward quality sites that traditionally have been overlooked by search engines -- e.g. parts of the invisible web. "It reflects our desire to invest heavily to get more content into our search engine experience," said Tim Cadogan, vice president of search for Yahoo.

A key aspect of the new program is outreach, both to existing and new customers. "Inclusion in its own right is not the most exciting component going forward," said Cadogan. "What we are excited about is having a better relationship with the people who are providing content. There's currently a lot of uncertainty and guesswork. We think there's a lot of room for improvement. There's a lot of demand for knowledge in how to participate in search."

As with previous paid inclusion programs, ranking is still determined by the quality of the site. Participating in the paid inclusion program doesn't give a site any boost in Yahoo's relevance algorithms -- at least explicitly.

"Site Match helps us understand more about the site," said Cadogan. "We have more meta information about your site, and that will allow us to position you to users more appropriately. If you're a good site that's going to be great for you, and if your site is bad, it's probably going to hurt you."

Rolling Up Six Programs Into One

With its acquisitions of Inktomi and Overture, Yahoo inherited six different pay for inclusion programs (from Inktomi, AltaVista and AlltheWeb). Over the past year, the company has continued to run all six programs. For search marketers, this posed a problem, because all of the programs had separate terms, pricing and distribution. From a practical standpoint, it was necessary to participate in several of the programs to assure the broadest reach.

The new program consolidates all six programs into a single, unified program. By participating in the new program, search marketers continue to get the broad distribution offered by the previous programs, with one new, very important addition -- inclusion in the new Yahoo index itself.

This is a major benefit -- last month, Yahoo handled about 30% of all U.S. search traffic, according to Nielsen//NetRatings. However, until the switchover to the new engine, the actual search results were powered by Google, which has no pay for inclusion program of its own. In effect, Yahoo is now offering search marketers a major new source of search traffic, through one unified program.

"It radically simplifies the situation from having six programs to one that gets you a ton of distribution and gets you a lot of benefits from interacting with us," said Cadogan.

Creating Transparency and Structure

Search marketing has traditionally been a virtual cat and mouse game between the search engines and content providers. "Search engines do what they will, and the web site has to guess what they're doing," said Cadogan. While this has often frustrated legitimate content providers, search engines have justified this less than forthcoming approach as necessary, to prevent site owners from gaming the system in their favor.

"We're going to continue to do that," said Cadogan. "But on top of that we're going to reach out to providers and engage them. Part of the way we want to do that is to create a much more transparent, structured relationship between us and content providers," said Cadogan.

As part of this structured relationship, Yahoo is establishing guidelines for what it considers to be quality content. This should help content owners have a better understanding of the ranking criteria Yahoo uses, without revealing too much detail that could be helpful to spammers.

Bucking a trend, Yahoo is encouraging content providers to provide detailed metadata for each web page. For the past several years, most of the major search engines have spurned metadata as a reliable source of information about content.

"We want to know more about the content, we want more metadata about the content," said Cadogan.

Participants in the CAP program will also be part of a regular review process, that has both automated components and input from Yahoo's editorial team. Cadogan says that a key benefit of participating in the CAP program is feedback from these quality reviews, though he won't say exactly what type of feedback will be offered, or whether sites receiving favorable reviews will get a ranking boost.

Pages that don't measure up to Yahoo's standards will be given the opportunity to improve -- or risk a potential rank demotion.

Embracing the Invisible Web

In addition to consolidating the company's paid inclusion programs, Yahoo is also embarking on an aggressive program to work with non-commercial sites that offer valuable content, but have technical issues getting included in the index.

"We want to get much much more content. That means reaching out to providers that are not currently on the web -- invisible web databases," said Cadogan. "We are reaching out to major content providers, in academia, government, libraries and so on, and creating a structured interaction with them."

Yahoo is working to resolve technical issues on a case by case basis, surfacing some terrific content that previously was hidden to search engines.

For example, Yahoo has worked with National Public Radio to make more than 17,000 hours of audio content accessible. Also included is Northwestern University's online OYEZ project, with more than 2,000 hours of U.S. Supreme Court audio, including all audio recorded since 1995. Another new source is UCLA's Cuneiform Digital Library Initiative, content documenting Babylonian history back to 3500 B.C.

These content providers are responding very favorably to Yahoo's efforts to make their "hidden" content more accessible. "What is most exciting about this is the level of engagement," said Cadogan. "They're just delighted to participate. It opens up a whole new set of worlds."

In general, Yahoo does not have exclusive arrangements with these content providers. Cadogan says that's not important. What's important is broadening Yahoo's reach into many more kinds of content than before. And it also helps Yahoo's engineers better understand how to solve some of the technical challenges associated with the Invisible web. "Having spent the time to see how someone's database works is quite valuable," he said.

What About AltaVista and AlltheWeb?

AltaVista and AlltheWeb will continue to operate as stand-alone search properties, though they both now use the same underlying index that powers Cadogan notes that even though they're using the same index, they each have different features that should continue to appeal to fans of each service. AltaVista's advanced search capabilities remain strong, for example.

Over time, Cadogan says that Yahoo may attempt to differentiate the two engines. "We're going to optimize the properties for each audience," he said. "We may experiment a little bit more on those sites."

Potential Disclosure Issues

Yahoo's new paid inclusion program could potentially open up new controversy surrounding the issue of disclosure. While the company clearly labels paid placement "sponsor results" and places them in separate areas of result pages, the new paid inclusion listings are not explicitly differentiated -- even though they generate cost-per-click revenues for Yahoo.

This may raise issues with the Federal Trade Commission, which issued a letter of recommendation to the search engines about disclosure practices in the summer of 2002. While all of the major search engines have moved to clearly label sponsored listings since then, few have fully confronted the issue of clear disclosure for paid inclusion results -- one of the central concerns of the FTC.

Yahoo does do a limited form of disclosure, through the "what's this" link found next to the "web results" label. Click the link, and you'll see a page describing the structure of the search results page. On this page, the following appears: "Web Results may include sites that participate in the Overture Site Match or Site Match Xchange programs. Sites from both of these programs pay to be included in our search engine index, and for Site Match Xchange, sites also pay to be reviewed for relevance."

There's no way for a searcher to easily distinguish between paid inclusion results and content that Yahoo's crawler found on its own. And there's no indication that Yahoo gets paid when a searcher clicks on a paid inclusion link.

Cadogan insists that this isn't a problem, because there is no difference in treatment between pages participating in the inclusion program and those found naturally. "Payment is linked with the interaction. The payment does not connect with the ranking," he said. "We're going to do everything we can to reward good sites and punish bad sites, period."

That may be the case, but it's virtually certain that vocal consumer watchdog groups will have issues with Yahoo not being fully forthright about the commercial nature of these listings.

We're here at the Search Engine Strategies conference in New York this week, and have the opportunity to closely follow reactions from the various key constituencies to Yahoo's new program. Once a clearer picture emerges we'll come back with more complete details.

A longer version of this article that goes into more details about the Yahoo CAP program, including pricing information, potential ranking problems, and an analysis and response to rumors about the new program is available to Search Engine Watch members.
Click here to learn more about becoming a member

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Yahoo to search audio and video content...
Computer Shopper Mar 2 2004 12:34PM GMT
If ICANN can't, who can?...
CNET Mar 2 2004 12:13PM GMT
DoubleClick: Web Is Major Awareness Builder for Consumers... Mar 2 2004 6:25AM GMT
Search Engine Strategies: Something For Everyone... Mar 2 2004 1:20AM GMT
Domain names are big again... Mar 1 2004 2:24PM GMT
New Web tools aim to customize searches...
San Jose Mercury News Mar 1 2004 12:48PM GMT
Google reportedly plans to offer free email service...
Seattle Times Mar 1 2004 10:26AM GMT
Icann 'disappointed' in VeriSign... Mar 1 2004 9:20AM GMT
Search Startup Scores $5M in Funding... Mar 1 2004 6:22AM GMT
Google too confusing?...
About Web Search Mar 1 2004 3:06AM GMT
Google Founders Keep 'Top 100' List of New Ideas...
Boston Globe Mar 1 2004 0:07AM GMT
Internet May Not Have It All, Says Web Search Expert...
Financial Express Feb 29 2004 9:55PM GMT
Google Prospects Party Like It's 1999...
Boston Globe Feb 29 2004 3:10PM GMT
powered by