The Problems With Rating Services

The Problems With Rating Services

From The Search Engine Report
April 4, 2000

I'm often asked which are the web's most popular search engines. To answer, I mostly rely on figures from the major ratings services of Media Metrix and Nielsen//NetRatings. Their figures are far from perfect, however. In fact, it's rather alarming how often their figures are cited as proof of any site's popularity without a lot of qualifying by journalists or analysts. Web ratings can be and are manipulated much more by web property owners than could ever happen when compared to television ratings.

I don't mean to knock the numbers entirely. They are the best estimates we have and can be extremely useful. But as with any statistics, they can't be accepted at face value. Reporters and others who quote these figures to the public ought to be critically analyzing them, rather than just parroting back whatever top ten list appears in the latest press release they've been given. Since many of them are not, I thought it would be useful to go through the type of questions I have to consider when pondering the ratings for search engines. By doing so, I hope you'll be able to do your own questioning.

Let's start with Media Metrix, which is the web's oldest rating service. Media Metrix has a sample of at home and at work web surfers that it monitors, in order to estimate what we all are doing. How exactly sampling is done itself can impact whether the data received is trustworthy. But let's set that aside. Just assume that the sampling is perfect, because there will still be enough problems remaining to take issue with the numbers.

Each month, Media Metrix releases data about top web properties and top web sites ("digital" properties not necessarily on the web are also measured). There's a world of difference between a "property" and a "site," yet it is not uncommon to see the terms used synonymously when the figures are reported. For instance, the San Francisco Chronicle article below talks about AltaVista changing its web site as part of an effort to "catch up" with more popular portal sites. The discussion is mostly about web sites, yet web property numbers are what's offered as proof that AltaVista is behind. As we'll see, if web site numbers had been used -- as they should have been -- AltaVista looks much better off.

"Web property" is defined as a collection of web sites owned by one company. For instance, Lycos owns several web sites, such as its Lycos flagship portal site, the HotBot search site and the Tripod home page building service. The combined traffic of all these and other Lycos-owned web sites make up the Lycos "web property" figures.

Let's compare that to television. A television network like NBC has many TV shows, such as ER, Third Watch and The Tonight Show. Ratings companies can estimate how many people tuned into the entire network during a given week versus estimating how many people watched a particular show. They are different numbers, and they both have their uses.

Web property numbers are like network numbers. They are especially helpful if you want to know how many people a web media owner can reach. Want to get your ad out in front of a lot of eyeballs? According to Media Metrix, Yahoo's network had 45 million of them in February. Lycos had 32 million, and AltaVista had 12 million.

Web site numbers are like television show numbers. They are useful if you want to know if a particular web site is popular among surfers, just as you might use television ratings to determine if a particular TV show is a hit with viewers. For example, I use web site numbers because I'm specifically interested in which search engines are popular. The Lycos web property number is useless, in this regard. It mixes in people who went to Lycos, HotBot, Tripod and other properties. Only the web site numbers will let you know if any of these sites is a hit.

Now comes the manipulation part. Media Metrix leaves it to the web property owners to decide what makes up their web site numbers. That's why in January 1999, the web site jumped from having a 21 percent to a 34 percent reach of among users surfing the web from both at home and at work. All Go sites, such as Disney and ESPN were rolled up into the web site figure. In essence, the web site figure was simply allowed to be corrupted into a web property number.

More recently, the same thing happened with the Lycos web site figure. Since March 1999, the Lycos site reach among at home and at work surfers had been dropping from its high of 26 percent down to a low of around 22 percent. Then last December, it suddenly spiked to about 28 percent. It shot up further to about 39 percent in February.

What happened? I don't know exactly, but one major driver was the fact that Lycos decided to combine HotBot site figures with the Lycos figures, according to Media Metrix. These are two completely separate web sites, and it makes no sense to combine them. Nevertheless, this is allowed.

Can you imagine if the Fox television network was able to combine the ratings for shows like The Simpson and Ally McBeal into the ratings for Titus, in order to show how it was more popular than NBC's Friends? No one would stand for it, yet this exact situation is being tolerated in the web ratings world.

It's crucial to know the hit shows on the web. As with television, hit shows can build or break a network. The combination of distinct sites into one site number can allow web media owners to potentially hide their bad shows. The Forbes article below talks about the success of Lycos, while the Chronicle article I mentioned generally looks down upon AltaVista. But the Lycos web site, as the network's flagship show, was on a downward trend until this latest change to how the Lycos web site numbers are calculated. AltaVista's flagship site was on a generally upward trend that threatened to surpass the Lycos site. The web property numbers don't reflect this. It's only when you look at the web site numbers that such patterns emerge.

Of course, even when you know the hit shows, you still don't necessarily know which are the hits for search. For instance, many people go to the Yahoo web site in order to access their email, and there's no way to break them out of the search-specific traffic. That's one reason you are beginning to hear some search engines quote the number of queries per day that they process. Those figures are a better measure of search-specific traffic. Numbers from StatMarket also provide another look at search engine popularity, based on the traffic they send to web sites.

For its faults, at least Media Metrix does regularly cite both web property and web site numbers. In contrast, the monthly releases from NetRatings focus on only web property numbers. You can get web site numbers, but you have to make a special request. That why the NetRatings page that I maintain within Search Engine Watch is so out of date. I'm still waiting for figures to be sent back to me from NetRatings.

Ideally, I'd like to see NetRatings release both web property and web site numbers regularly, as Media Metrix does. In fact, the new format Media Metrix introduced last month is ideal. It shows the web property figure, then a web site breakdown for that property (though not a complete breakdown). But both companies also need to ensure that they, not their clients, define what makes up a web site, and that those definitions align with what the public generally perceives those sites to be.

Ratings, Reviews and Tests
You'll find numbers for major search-related sites from Media Metrix, NetRatings and StatMarket here, plus links to each of those companies from these pages.

Alexa 1000

A new resource from Alexa that I've only just begun to explore. It offers the ability to look up traffic for any web site and even to see traffic "rolled up" within those sites. For instance, Lycos is listed as the 10th most popular site for February. By drilling down, you can see that Alexa is combining any Lycos-owned sites into this number, making it really a web property number. You can also discover that Alexa estimates that HotBot draws more page views that the Lycos main site.

AltaVista Switches Web Portal Into High Gear
San Francisco Chronicle, March 27, 2000

Yet another article with the usual analyst quotes of how AltaVista is trying to catch up in the portal game, struggling against being "late" to the market. As a site, AltaVista is closing in on Excite, which the same analysts would acknowledge as being one of the portal leaders. AltaVista would also probably be doing the same to Lycos, if HotBot numbers weren't merged into the Lycos total.

Bob Davis, Lycos' Savior
Forbes, March 30, 2000

Much focus here on the recent European spin-off by Lycos, plus mention of how Lycos has grown.