Last month, I wrote about how an AltaVista sales representative had pitched the company's "Trusted Feed" paid inclusion program as a way to get guaranteed top rankings. AltaVista quickly denied that this was the case, repeatedly emphasizing that content in the program is not given any ranking boost over content that AltaVista's crawler-technology finds and lists for free. Despite this, it turns out that Trusted Feed content may indeed get a bump into the top results, in the right circumstances.
Try a search for "oliver stone" at AltaVista, and you'll see an example of what looks to be this ranking bump of paid inclusion content. The first five listings come from Buy.com, PriceGrabber.com and NexTag.com, pitching opportunities to buy the director's movies.
If the search was for "oliver stone movies," then getting such product-oriented listings might feel more relevant. However, getting these listings as top matches for a generic search about Oliver Stone feels very odd. Take this listing, which AltaVista's algorithm supposedly has decided is the most relevant out of nearly 340,000 possible matches for an "oliver stone" search:
oliver stone - Buy.com
Easy oliver stone Shopping at Buy.com. Everyday
Great Deals on Hot New Products and Leading Brands.
Similarly, a search for "pentium iii" brings up results dominated by what look to be paid inclusion listings from the companies named above, as well as DealTime.com, and several of these companies miraculously get top rankings for "microsoft trackball," as well. These are just some of the examples where Search Engine Watch found a high correlation between suspected paid inclusion pages and high rankings for the terms those pages were targeting.
AltaVista itself admits that paid inclusion content from its Trusted Feed program may indeed be doing better than it should. The company puts the blame on problems with index blending, which is when it tries to mix matches from its "free" editorially-compiled listings with those from its paid inclusion database.
"They may come up when there are more meritorious URLs," said Jan Pedersen, AltaVista's chief scientist, when asked about the "oliver stone" and "microsoft trackball" examples above and the correlation found in general. "That's a demonstration of a problem with our blending rather than a matter of policy."
Specifically, Pedersen said the problem lies in the fact that Trusted Feed "pages" aren't typically web pages at all. Instead, the program allows companies to supply AltaVista with a list or "feed" of many URLs and some key information describing those URLs, such as a page title and a description to shown to the reader.
Sending AltaVista information this way is radically different from when AltaVista understands what a page is about by reading its HTML content. Cues useful for ranking "ordinary" web pages such as the frequency of terms, the location of terms and link analysis may not be effective when trying to ranking pages based on the extremely limited information described about them via the Trusted Feed program.
"They [trusted feed listings” are not web pages and don't benefit from the usual cues," said Pedersen. "We make an estimate on how to blend them in."
Pedersen said that AltaVista is constantly working to ensure that paid inclusion listings don't appear more often than what their "entitlement" should be, or that they aren't crowding out ordinary editorial results inappropriately. However, he qualified this by saying that for commercially-oriented queries, it may not be bad in AltaVista's view for users to have more paid inclusion content ranking well.
"Most of the paid inclusion content is indeed basically catalogs from various commercial sources, and the queries that they do tend to come up with are queries of commercial intent," Pedersen said. "The notion of relative relevancy there needs to be carefully assessed."
In other words, many of those seaching for "microsoft trackball" may actually want to purchase the product. Paid inclusion is a mechanism that allows some online merchants to show up in editorial results when they might ordinarily stand no chance, because there web sites are not search engine friendly.
"The base motivation here is to provide a better experience," Pedersen said. "The reason its called 'Trusted Feed' is that we want to get good content from merchants of reputable intent."
It's a good argument, and one that I've written about many times before when explaining paid inclusion. However, Trusted Feed content seems to be coming up well not just for more unique, unusual or expressly commercial queries. From various tests conducted last Thursday, Trusted Feed URLs seem to have a magical ability to rank well for nearly anything.
Apparent Trusted Feed content from PriceGrabber and Walmart ranked in the top ten for "die fledermaus," "jazz guitar" and "biorhythms." Buy.com and PriceGrabber ranked well for "saturday night live." PriceGrabber came up in top results for "waldorf astoria" and "x-files." DealTime, eBay, Hammacher, AllPosters.com, Amazon all ranked tops for "potholders."
It's important to note that no wrong-doing should be attached to any of the companies named that are ranking well. I'm not suggesting that they are spamming AltaVista or doing something incorrect. They are simply companies that appear to be feeding AltaVista with Trusted Feed content and who, with a good deal of consistency, seem to have their content rank well for the terms they are targeting. And unlike is usually the case in search engine optimization, their good fortune seems much more to do with AltaVista giving them a ranking boost than with the content of their pages being somehow exceptional.
Other companies beyond those named may be in the Trusted Feed program and are seeing unusually high ranks, as well. The difficulty is that it is hard to tell when looking at an AltaVista listing whether it is in the Trusted Feed program. AltaVista itself makes no attempt to disclose this (nor does any other search engine with paid inclusion programs, to ordinary users). However, one can make some good guesses. Pages that have long URLs containing ? symbols and embedded tracking codes are very likely to be paid inclusion candidates.
All of the companies named here had URLs that met this criteria in some way. AltaVista was also sent a list asking if the companies named were in the Trusted Feed program. AltaVista didn't confirm this, but the email did prompt a call from the company to talk further about the Trusted Feed program, so presumably many of the companies are indeed in it.
Certainly AltaVista is being singled out here, but that's because it has had the most damning evidence pointed at it about potentially boosting paid inclusion content.
We've had reports that other crawlers with paid inclusion programs might drop editorial content for "spam" reasons, only for the web site owner to be called later about getting the content back in through paid inclusion (see the "Search Engine Standards, Please!" article, below). And there are whispers and rumors that paid inclusion will give a boost with others, despite their denials.
However, none of these other search engines have ever had a sales person put this out in writing, in the way that happened with AltaVista, recently. The denial of that sales pitch by AltaVista is fine, but the boosting for whatever reason certainly seems to be happening.
AltaVista does having a fairly large amount of paid inclusion content that potentially can rise in its main results for general queries, 10 million URLs, according to the company. Inktomi has about the same amount, but if these are ranking well on Inktomi for things like "oliver stone," it's less noticeable. That's because Inktomi's major partner MSN Search only really makes use of Inktomi's data for very specific queries. As for FAST and Teoma, they have relatively little paid inclusion content, so it's harder for this content to dominate queries.
To be fair, testing at some of the other search engines should be done, which I may conduct in the future, and it may reveal similar preference for paid inclusion content happening. Indeed, a quick peak at "pure" Inktomi results for "oliver stone" reveals possible paid inclusion pages from Amazon and eBay, while the cast of characters coming up for "microsoft trackball" includes both those companies along with possible paid inclusion URLs from DealTime, Nextag and perhaps some others.
Is this wrong? A search for "microsoft trackball" Google brings up a page from Amazon's UK site, a "free" or "natural" listing, since Google doesn't offer paid inclusion. So if some "product" pages come up naturally, what's wrong with getting site owners to foot the bill for the traffic they receive? In addition, wouldn't it be better to get product pages from large, trusted brand owners rather than those from perhaps affiliate sites, one of which appears in that same Google search mentioned?
Sure, there's nothing necessarily wrong with such a thing happening. In fact, AltaVista could have a new business model by perhaps taking in only paid inclusion content and touting itself as the place to go for product-oriented searching. The problem is that is doesn't do this now. There is the presumption that editorial and paid content will be balanced. It's a messy situation, one that leaves the search engines offering paid inclusion open to claims of abuse.
What's the solution? I'm uncertain, but it may be that paid inclusion content has now grown enough that it will need to be separated from editorial content, so that users who want both product listings and editorial can get a blend of the two.
Perhaps part of the acceptance process might be to begin classifying this content. You have great news articles and want them included? OK, that might be considered for the editorial database. Have a product database of 100,000 items? Clearly that's commercial content, and we'll dump it into our commercial index. But don't worry -- we always provide both commercial listings as part of our results and in fact list them above our editorial results, so it's a benefit to be there.
The downside remains that as soon as you have 11 people in any paid inclusion program offering a bump, there's every incentive to play the "relevancy game" of trying to make little changes to be one of the 10 getting the top listings. The answer here may be that ultimately for commercial content, paid inclusion sites may be ranked just like paid listing sites, by who is willing to pay the most.
AltaVista Sales Pitch Suggested Paid Inclusion Boost
The Search Engine Report, Oct. 1, 2002
Past article about that AltaVista sales pitch
Hurry into AltaVista's Discount Warehouse Blowout Clearance Event!
Traffick.com, Sept. 27, 2002
Traffick.com's Andrew Goodman received the original pitch, and you can read about what happened here.
Search Engine Standards, Please!
SearchDay, October 8, 2002
Search engines should set standards and adhere to them, say webmasters, advertisers and a senior attorney for the Federal Trade Commission. Uniform standards would serve both web site owners and searchers.
Pay For Placement?
Collection of past articles by me and others on the subject of paid inclusion.