Just two weeks after launching a brand new search engine, Yahoo has announced a content acquisition program that consolidates all of its paid inclusion programs and marks the beginning of an aggressive new campaign to significantly expand both the scope and quality of content available via Yahoo search.
The new program, called CAP (Content Acquisition Program), focuses on both commercial and non-commercial web sites. On the commercial side, the new program is called Overture Site Match. The non-commercial effort is geared toward quality sites that traditionally have been overlooked by search engines -- e.g. parts of the invisible web. "It reflects our desire to invest heavily to get more content into our search engine experience," said Tim Cadogan, vice president of search for Yahoo.
A key aspect of the new program is outreach, both to existing and new customers. "Inclusion in its own right is not the most exciting component going forward," said Cadogan. "What we are excited about is having a better relationship with the people who are providing content. There's currently a lot of uncertainty and guesswork. We think there's a lot of room for improvement. There's a lot of demand for knowledge in how to participate in search."
As with previous paid inclusion programs, ranking is still determined by the quality of the site. Participating in the paid inclusion program doesn't give a site any boost in Yahoo's relevance algorithms -- at least explicitly.
"Site Match helps us understand more about the site," said Cadogan. "We have more meta information about your site, and that will allow us to position you to users more appropriately. If you're a good site that's going to be great for you, and if your site is bad, it's probably going to hurt you."
Rolling Up Six Programs Into One
With its acquisitions of Inktomi and Overture, Yahoo inherited six different pay for inclusion programs (from Inktomi, AltaVista and AlltheWeb). Over the past year, the company has continued to run all six programs. For search marketers, this posed a problem, because all of the programs had separate terms, pricing and distribution. From a practical standpoint, it was necessary to participate in several of the programs to assure the broadest reach.
The new program consolidates all six programs into a single, unified program. By participating in the new program, search marketers continue to get the broad distribution offered by the previous programs, with one new, very important addition -- inclusion in the new Yahoo index itself.
This is a major benefit -- last month, Yahoo handled about 30% of all U.S. search traffic, according to Nielsen//NetRatings. However, until the switchover to the new engine, the actual search results were powered by Google, which has no pay for inclusion program of its own. In effect, Yahoo is now offering search marketers a major new source of search traffic, through one unified program.
"It radically simplifies the situation from having six programs to one that gets you a ton of distribution and gets you a lot of benefits from interacting with us," said Cadogan.
Creating Transparency and Structure
Search marketing has traditionally been a virtual cat and mouse game between the search engines and content providers. "Search engines do what they will, and the web site has to guess what they're doing," said Cadogan. While this has often frustrated legitimate content providers, search engines have justified this less than forthcoming approach as necessary, to prevent site owners from gaming the system in their favor.
"We're going to continue to do that," said Cadogan. "But on top of that we're going to reach out to providers and engage them. Part of the way we want to do that is to create a much more transparent, structured relationship between us and content providers," said Cadogan.
As part of this structured relationship, Yahoo is establishing guidelines for what it considers to be quality content. This should help content owners have a better understanding of the ranking criteria Yahoo uses, without revealing too much detail that could be helpful to spammers.
Bucking a trend, Yahoo is encouraging content providers to provide detailed metadata for each web page. For the past several years, most of the major search engines have spurned metadata as a reliable source of information about content.
"We want to know more about the content, we want more metadata about the content," said Cadogan.
Participants in the CAP program will also be part of a regular review process, that has both automated components and input from Yahoo's editorial team. Cadogan says that a key benefit of participating in the CAP program is feedback from these quality reviews, though he won't say exactly what type of feedback will be offered, or whether sites receiving favorable reviews will get a ranking boost.
Pages that don't measure up to Yahoo's standards will be given the opportunity to improve -- or risk a potential rank demotion.
Embracing the Invisible Web
In addition to consolidating the company's paid inclusion programs, Yahoo is also embarking on an aggressive program to work with non-commercial sites that offer valuable content, but have technical issues getting included in the index.
"We want to get much much more content. That means reaching out to providers that are not currently on the web -- invisible web databases," said Cadogan. "We are reaching out to major content providers, in academia, government, libraries and so on, and creating a structured interaction with them."
Yahoo is working to resolve technical issues on a case by case basis, surfacing some terrific content that previously was hidden to search engines.
For example, Yahoo has worked with National Public Radio to make more than 17,000 hours of audio content accessible. Also included is Northwestern University's online OYEZ project, with more than 2,000 hours of U.S. Supreme Court audio, including all audio recorded since 1995. Another new source is UCLA's Cuneiform Digital Library Initiative, content documenting Babylonian history back to 3500 B.C.
These content providers are responding very favorably to Yahoo's efforts to make their "hidden" content more accessible. "What is most exciting about this is the level of engagement," said Cadogan. "They're just delighted to participate. It opens up a whole new set of worlds."
In general, Yahoo does not have exclusive arrangements with these content providers. Cadogan says that's not important. What's important is broadening Yahoo's reach into many more kinds of content than before. And it also helps Yahoo's engineers better understand how to solve some of the technical challenges associated with the Invisible web. "Having spent the time to see how someone's database works is quite valuable," he said.
Overture Site Match Pricing
As with previous pay for inclusion programs, there are two versions of the program depending on the size of the web site. The base version is called Site Match, for sites with fewer than 1000 URLs. The self-serve format of the program works in much the same way as the former Inktomi Site Submit program. You can sign up for the program directly through Overture, or through an authorized reseller.
Site Match pricing is based on an annual submission fee, starting at $49 for the first URL. The price drops for URLs 2-10, at $29 each, and further still for URLs 11-99, at $10 each.
A new twist on the Site Match program is that in addition to the up-front annual review fee, there's also a new cost per click (CPC) fee. For most URLs, this will be a 15 cents per click. For content in certain competitive categories, such as travel, the CPC will be 30 cents.
For larger sites with 1,000 URLs or more, Yahoo offers the Overture Site Match Xchange program at no annual fee and a fixed cost-per-click of up to $1.00, depending on content category.
"It was very important to add the CPC component because it brings the content provider into alignment with us," said Cadogan. The rationale is that rather than simply providing a fixed annual cost, content providers will consider which pages they want included in the program more carefully, as the new variable pricing model will almost certainly increase annual costs for well-ranking pages.
"We're also using this as a quality discipline. That pricing helps them help us make the results more relevant," said Cadogan.
For customers who signed up for the Inktomi program between February 18 and today, Yahoo will waive the annual fee for the new program for the remainder of the year.
Confusion and Fear Over a Trail PeriodTwo weeks ago, the following announcement appeared on several of Inktomi's paid inclusion parter web sites:
"Yahoo Search has transitioned to its own search technology powered by Inktomi's search platform and is will soon launch a new inclusion program. As a bonus to Inktomi Search Submit customers, Yahoo Search is providing a free trial of Yahoo traffic that will run until April 15, 2004. When the new paid inclusion program launches, new Search Submit subscriptions will no longer be accepted.
"Existing Inktomi Search Submit customers will have the option of maintaining their subscriptions (receiving continued traffic from partners previously in the Inktomi network) or joining the new program for ongoing distribution on Yahoo Search and a full network of portals including MSN, Excite, HotBot, About.com and others."
Some interpreted this announcement as Yahoo adding yet another, additional paid inclusion program. Word of the transition sparked fears and more questions on a WebmasterWorld thread. A knowledgeable Yahoo employee responded to the post, confirming that search marketers should not buy into the Inktomi program thinking that it was a way to get into Yahoo.In fact, Yahoo has essentially killed the Inktomi brand. All of its former paid inclusion programs, including Inktomi's, have now been rolled up into the new Overture Site Match program.
No Need to Panic Over Changes
Two persistent rumors on search marketing forums are that Yahoo will drop pages from participants that don't participate in the new program, or that non-participation may trigger a performance penalty, pushing pages lower in result lists.
Both of these rumors are false. Cadogan was emphatic: "It would be crazy for us to do that." He explained that the free trial period was a bonus to expose customers to the new traffic available via the Yahoo index. Customers opting not to continue with the new program will continue to see the full benefits of their existing program, until their current subscriptions expire. They simply won't get the benefit of exposure to Yahoo users.
How is this possible if there's just one, unified index that's used for both Yahoo.com and all of its distribution partners? The answer is simple, really. While it's true that there's just one primary index, "we are able to distribute various parts of the index to our partners," said Cadogan.
Bottom line: If you are currently participating in one of Yahoo's older programs, and decide not to participate in the new Site Match program, nothing will change. Your pages will remain in the Yahoo index, and will be distributed to the same partners they always were.
Furthermore, even if you completely opt out of the program altogether, your content will remain in the Yahoo index -- no pages will be removed, and there is no "penalty" for not participating in the program -- with one possible exception.
As part of the ongoing quality review process, Yahoo selects random sites from both participating sites in the CAP program, as well as from the unpaid crawled index. If your site happens to be randomly selected, and fares poorly in the evaluation process, it's likely to have a negative impact on ranking.
If this is the case, what's the incentive for paying Yahoo to be included in the program?
"What you'll forego is the value of the structured interaction with us -- feedback, and a quality review," said Cadogan. He did note, however, that the quality review could be a two-edged sword, amounting to a benefit for good sites or a disadvantage for poor sites, which could see a negative ranking impact.
"For people who are already doing very well, they may not need to use the program," said Cadogan. "In many ways it's an insurance policy."
What About the Yahoo Directory?
Just two years ago, inclusion in the Yahoo directory was considered to be an absolute requirement for successful search engine marketing. But with Yahoo's promotion of algorithmic search results over directory listings, the importance of that listing has diminished.
"Increasingly users were comfortable searching free text, rather than needing the structured format," said Cadogan. But he says that people continue to use the directory, so a Yahoo directory listing is still a valuable potential source of traffic.
Should you continue to fork over the annual $299 review fee for your site?
For most sites, the answer is yes. While Yahoo is not currently using sites included in the directory to influence search results, they may at some point. "We're spending a fair amount of time thinking through how to use all of that structured information in the directory to make search better," said Cadogan.
He also noted that while Yahoo itself is not using inclusion in the directory as a relevance variable, other search engines might, in the same way Google uses inclusion in the Open Directory Project as one input to their relevance calculations.
What About AltaVista and AlltheWeb?
AltaVista and AlltheWeb will continue to operate as stand-alone search properties, though they both now use the same underlying index that powers Yahoo.com. Cadogan notes that even though they're using the same index, they each have different features that should continue to appeal to fans of each service. AltaVista's advanced search capabilities remain strong, for example.
Over time, Cadogan says that Yahoo may attempt to differentiate the two engines. "We're going to optimize the properties for each audience," he said. "We may experiment a little bit more on those sites."
Potential Disclosure Issues
Yahoo's new paid inclusion program could potentially open up new controversy surrounding the issue of disclosure. While the company clearly labels paid placement "sponsor results" and places them in separate areas of result pages, the new paid inclusion listings are not explicitly differentiated -- even though they generate cost-per-click revenues for Yahoo.
This may raise issues with the Federal Trade Commission, which issued a letter of recommendation to the search engines about disclosure practices in the summer of 2002. While all of the major search engines have moved to clearly label sponsored listings since then, few have fully confronted the issue of clear disclosure for paid inclusion results -- one of the central concerns of the FTC.
Yahoo does do a limited form of disclosure, through the "what's this" link found next to the "web results" label. Click the link, and you'll see a page describing the structure of the search results page. On this page, the following appears: "Web Results may include sites that participate in the Overture Site Match or Site Match Xchange programs. Sites from both of these programs pay to be included in our search engine index, and for Site Match Xchange, sites also pay to be reviewed for relevance."
There's no way for a searcher to easily distinguish between paid inclusion results and content that Yahoo's crawler found on its own. And there's no indication that Yahoo gets paid when a searcher clicks on a paid inclusion link.
Cadogan insists that this isn't a problem, because there is no difference in treatment between pages participating in the inclusion program and those found naturally. "Payment is linked with the interaction. The payment does not connect with the ranking," he said. "We're going to do everything we can to reward good sites and punish bad sites, period."
That may be the case, but it's virtually certain that vocal consumer watchdog groups will have issues with Yahoo not being fully forthright about the commercial nature of these listings.
We're here at the Search Engine Strategies conference in New York this week, and have the opportunity to closely follow reactions from the various key constituencies to Yahoo's new program. Once a clearer picture emerges we'll come back with more complete details.