Paid Inclusion Listings May Get Boosted At AltaVista
Last month, I wrote about how an AltaVista sales representative had pitched the company's "Trusted Feed" paid inclusion program as a way to get guaranteed top rankings. AltaVista quickly denied that this was the case, repeatedly emphasizing that content in the program is not given any ranking boost over content that AltaVista's crawler-technology finds and lists for free. Despite this, it turns out that Trusted Feed content may indeed get a bump into the top results, in the right circumstances.
Try a search for "oliver stone" at AltaVista, and you'll see an example of what looks to be this ranking bump of paid inclusion content. The first five listings come from Buy.com, PriceGrabber.com and NexTag.com, pitching opportunities to buy the director's movies.
If the search was for "oliver stone movies," then getting such product-oriented listings might feel more relevant. However, getting these listings as top matches for a generic search about Oliver Stone feels very odd. Take this listing, which AltaVista's algorithm supposedly has decided is the most relevant out of nearly 340,000 possible matches for an "oliver stone" search:
oliver stone - Buy.com
Easy oliver stone Shopping at Buy.com. Everyday
Great Deals on Hot New Products and Leading Brands.
Similarly, a search for "pentium iii" brings up results dominated by what look to be paid inclusion listings from the companies named above, as well as DealTime.com, and several of these companies miraculously get top rankings for "microsoft trackball," as well. These are just some of the examples where Search Engine Watch found a high correlation between suspected paid inclusion pages and high rankings for the terms those pages were targeting.
AltaVista itself admits that paid inclusion content from its Trusted Feed program may indeed be doing better than it should. The company puts the blame on problems with index blending, which is when it tries to mix matches from its "free" editorially-compiled listings with those from its paid inclusion database.
"They may come up when there are more meritorious URLs," said Jan Pedersen, AltaVista's chief scientist, when asked about the "oliver stone" and "microsoft trackball" examples above and the correlation found in general. "That's a demonstration of a problem with our blending rather than a matter of policy."
Specifically, Pedersen said the problem lies in the fact that Trusted Feed "pages" aren't typically web pages at all. Instead, the program allows companies to supply AltaVista with a list or "feed" of many URLs and some key information describing those URLs, such as a page title and a description to shown to the reader.
For instance, Buy.com has a high-ranking listing at AltaVista for a search on "saturday night live," yet the company actually has no static page on this topic. Instead, clicking on that listing at AltaVista leads you to a dynamically generated page at Buy.com.
Specifically, that page is a listing of any products at Buy.com that appear in response to a search for "saturday night live." No crawler would typically find such a page, since it is only generated when someone enters a search for those specific words.
So if AltaVista couldn't crawl this page, how could it possibly rank well? Enter Trusted Feed. Through the program, participants can tell AltaVista what a particular URL is supposed to be about.
Scratching your head? Stay with me. The doorway page had these major page elements:
- Title: saturday night live
- Keywords meta tag: saturday night live
- Description meta tag: saturday night live - click here for the best selection on the web for saturday night live
- Body copy: You can find the information about saturday night live here
With Trusted Feed, rather than have AltaVista extract the information from the page above by crawling it, the page's owner can submit the information in a delineated way. Think of how information is contained in the rows and columns of a spreadsheet, and you'll see how this works. The example below shows how the page above and two other real life examples of listings from Buy.com at AltaVista might be represented in a delineated feed to AltaVista:
|URL|| Display |
|Title||Description||Body|| Key |
| http://www.buy.com/clickfrom/ |
|saturday night live||saturday night live - click here for the best selection on the web for saturday night live||You can find the information about saturday night live here||saturday night live|
| http://www.buy.com/clickfrom/ |
|tape media||tape media - click here for the best selection on the web for tape media||You can find the information about tape media here||tape media|
| http://www.buy.com/clickfrom/ |
|marine life||marine life - click here for the best selection on the web for marine life||You can find the information about marine life here||marine life|
In this example, each row represents a different "page" that will be sent to AltaVista in a feed format. AltaVista will understand what's on each page not be reading the actual page but instead by storing the information sent to it via the feed.
The URL column shows the URL that users will be sent to when they click on a listing, should it appear in response to a search. The Display URL column shows the URL that will actually be shown in AltaVista's listings. That's handy if you want a shorter, more user friendly URL to be shown to potential visitors.
The Title column shows both the title of the listing shown to potential visitors, as well as the text AltaVista will record as being in the title tag for the page. The description tag is the description shown to visitors, as well as what AltaVista records as the page's meta description tag content.
AltaVista's not supposed to be still recording meta keywords information, but should it decide to do this again in the future, the keyword column would show the content for each page. Finally, text that is in the body copy of a page is shown in the body column.
Sending AltaVista information this way is radically different from when AltaVista understands what a page is about by reading its HTML content. Cues useful for ranking "ordinary" web pages such as the frequency of terms, the location of terms and link analysis may not be effective when trying to ranking pages based on the extremely limited information described about them via the Trusted Feed program.
"They [trusted feed listings” are not web pages and dont benefit from the usual cues," said Pedersen. "We make an estimate on how to blend them in."
For example, since the pages may not have any link analysis value, AltaVista tries to come up with one based on how important the page's parent web site is deemed to be.
"We synthesize an authority score by looking at the authority score of the site they are associated with," Pedersen said. "That's degraded to some degree because [the Trusted Feed content” is deeper on the web."
Pedersen said that AltaVista is constantly working to ensure that paid inclusion listings don't appear more often than what their "entitlement" should be, or that they aren't crowding out ordinary editorial results inappropriately. However, he qualified this by saying that for commercially-oriented queries, it may not be bad in AltaVista's view for users to have more paid inclusion content ranking well.
"Most of the paid inclusion content is indeed basically catalogs from various commercial sources, and the queries that they do tend to come up with are queries of commercial intent," Pedersen said. "The notion of relative relevancy there needs to be carefully assessed."
In other words, many of those seaching for "microsoft trackball" may actually want to purchase the product. Paid inclusion is a mechanism that allows some online merchants to show up in editorial results when they might ordinarily stand no chance, because there web sites are not search engine friendly.
"The base motivation here is to provide a better experience," Pedersen said. "The reason its called 'Trusted Feed' is that we want to get good content from merchants of reputable intent."
It's a good argument, and one that I've written about many times before when explaining paid inclusion. However, Trusted Feed content seems to be coming up well not just for more unique, unusual or expressly commercial queries. From various tests conducted last Thursday, Trusted Feed URLs seem to have a magical ability to rank well for nearly anything.
Apparent Trusted Feed content from PriceGrabber and Walmart ranked in the top ten for "die fledermaus," "jazz guitar" and "biorhythms." Buy.com and PriceGrabber ranked well for "saturday night live." PriceGrabber came up in top results for "waldorf astoria" and "x-files." DealTime, eBay, Hammacher, AllPosters.com, Amazon all ranked tops for "potholders."
It's important to note that no wrong-doing should be attached to any of the companies named that are ranking well. I'm not suggesting that they are spamming AltaVista or doing something incorrect. They are simply companies that appear to be feeding AltaVista with Trusted Feed content and who, with a good deal of consistency, seem to have their content rank well for the terms they are targeting. And unlike is usually the case in search engine optimization, their good fortune seems much more to do with AltaVista giving them a ranking boost than with the content of their pages being somehow exceptional.
Other companies beyond those named may be in the Trusted Feed program and are seeing unusually high ranks, as well. The difficulty is that it is hard to tell when looking at an AltaVista listing whether it is in the Trusted Feed program. AltaVista itself makes no attempt to disclose this (nor does any other search engine with paid inclusion programs, to ordinary users). However, one can make some good guesses. Pages that have long URLs containing ? symbols and embedded tracking codes are very likely to be paid inclusion candidates.
All of the companies named here had URLs that met this criteria in some way. AltaVista was also sent a list asking if the companies named were in the Trusted Feed program. AltaVista didn't confirm this, but the email did prompt a call from the company to talk further about the Trusted Feed program, so presumably many of the companies are indeed in it.
It's a real eye opener to see just how well some of these companies do, once you have a list of their pages. For example, after seeing a URL for Walmart rank well for a particular search, I suspected it might be in the Trusted Feed program. I then ran query to find all the pages from Walmart in AltaVista, like this:
Many of the pages seemed likely to be in the paid inclusion program, since they contained ? symbols in their URLs and might not generally be picked up for free. I further could determine that they were probably paid inclusion URLs because the titles and descriptions in AltaVista did not match the content of the pages.
So, how can you then determine if the pages are getting a ranking boost? Look at the titles for each page, for an indication of what the content is aimed to do well for. One of the Walmart pages I found was this:
Discusses the lives of penguins, particularly Emperor penguins, describing
where they live, what they eat, their mating habits and parental behavior
Clearly Walmart would love to see this page do well for "penguins," and it does, ranked number 9 at AltaVista. It really stands out -- why is a page selling a book about penguins at Walmart deemed to be one of the best matches from among 162,000 possible matches that AltaVista finds? Neither Google nor FAST deem it worthy of such an honor, and rightfully so, I'd say.
Another page from Walmart with the title "mammals" gets ranked third, as does one for "jane goodall." A search on "ultimate x-men" brings up Walmart content at number one, and the same is true for "animal crafts." Walmart comes up fifth for "rushmore" and for "zach," number two.
In some cases, pages I checked didn't make it into first ten results but they could be found lurking on the second, such as for "doris day," "calculus" and "dune." The worst ranked page of those I checked was for "boats" at 50 and "traffic," which didn't rank well at all.
Again, no blame or wrong-doing should be attached to Walmart. They are simply one of the several sites that seem to be benefiting from AltaVista paid inclusion program. Rather than fault Walmart, it makes more sense for marketers looking for traffic to consider the program AltaVista's offering, if it will continue to work this way.
Certainly AltaVista is being singled out here, but that's because it has had the most damning evidence pointed at it about potentially boosting paid inclusion content.
We've had reports that other crawlers with paid inclusion programs might drop editorial content for "spam" reasons, only for the web site owner to be called later about getting the content back in through paid inclusion (see the "Search Engine Standards, Please!" article, below). And there are whispers and rumors that paid inclusion will give a boost with others, despite their denials.
However, none of these other search engines have ever had a sales person put this out in writing, in the way that happened with AltaVista, recently. The denial of that sales pitch by AltaVista is fine, but the boosting for whatever reason certainly seems to be happening.
AltaVista does having a fairly large amount of paid inclusion content that potentially can rise in its main results for general queries, 10 million URLs, according to the company. Inktomi has about the same amount, but if these are ranking well on Inktomi for things like "oliver stone," it's less noticeable. That's because Inktomi's major partner MSN Search only really makes use of Inktomi's data for very specific queries. As for FAST and Teoma, they have relatively little paid inclusion content, so it's harder for this content to dominate queries.
To be fair, testing at some of the other search engines should be done, which I may conduct in the future, and it may reveal similar preference for paid inclusion content happening. Indeed, a quick peak at "pure" Inktomi results for "oliver stone" reveals possible paid inclusion pages from Amazon and eBay, while the cast of characters coming up for "microsoft trackball" includes both those companies along with possible paid inclusion URLs from DealTime, Nextag and perhaps some others.
Is this wrong? A search for "microsoft trackball" Google brings up a page from Amazon's UK site, a "free" or "natural" listing, since Google doesn't offer paid inclusion. So if some "product" pages come up naturally, what's wrong with getting site owners to foot the bill for the traffic they receive? In addition, wouldn't it be better to get product pages from large, trusted brand owners rather than those from perhaps affiliate sites, one of which appears in that same Google search mentioned?
Sure, there's nothing necessarily wrong with such a thing happening. In fact, AltaVista could have a new business model by perhaps taking in only paid inclusion content and touting itself as the place to go for product-oriented searching. The problem is that is doesn't do this now. There is the presumption that editorial and paid content will be balanced. It's a messy situation, one that leaves the search engines offering paid inclusion open to claims of abuse.
What's the solution? I'm uncertain, but it may be that paid inclusion content has now grown enough that it will need to be separated from editorial content, so that users who want both product listings and editorial can get a blend of the two.
Perhaps part of the acceptance process might be to begin classifying this content. You have great news articles and want them included? OK, that might be considered for the editorial database. Have a product database of 100,000 items? Clearly that's commercial content, and we'll dump it into our commercial index. But don't worry -- we always provide both commercial listings as part of our results and in fact list them above our editorial results, so it's a benefit to be there.
The downside remains that as soon as you have 11 people in any paid inclusion program offering a bump, there's every incentive to play the "relevancy game" of trying to make little changes to be one of the 10 getting the top listings. The answer here may be that ultimately for commercial content, paid inclusion sites may be ranked just like paid listing sites, by who is willing to pay the most.
AltaVista Sales Pitch Suggested Paid Inclusion Boost
The Search Engine Report, Oct. 1, 2002
Past article about that AltaVista sales pitch.
Hurry into AltaVista's Discount Warehouse Blowout Clearance Event!
Traffick.com, Sept. 27, 2002
Traffick.com's Andrew Goodman received the original pitch, and you can read about what happened here.
Search Engine Standards, Please!
SearchDay, October 8, 2002
Search engines should set standards and adhere to them, say webmasters, advertisers and a senior attorney for the Federal Trade Commission. Uniform standards would serve both web site owners and searchers.
Pay For Placement?
Collection of past articles by me and others on the subject of paid inclusion.