Getting Away From Words-On-The-Page Relevancy

Getting Away From Words-On-The-Page Relevancy

From The Search Engine Report
March 3, 1999

HotBot's move to using Direct Hit's popularity results makes it the second major search engine to radically break from the words-on-the-page ranking system that has dominated the industry.

Typically, the major crawler-based search engines have ranked pages based on where and how often search terms have appeared on them. I'm simplifying the process, nor have these been the only factors -- but they've remained the predominate ones.

Infoseek made a significant move away from this last year, when it debuted its "ESP" ranking system. That system gives more weight to pages which come from sites that are also listed in Infoseek's human-compiled directory, or which have some degree of link popularity, or both. In general, I've felt the change has been an improvement for the service.

Expect to see more services significantly abandon the words-on-the-page approach for their top results. It will always remain important, especially for more obscure searches. But it's rapidly becoming an outdated method for finding the most relevant pages for popular queries.

The system was fine for a time when the web had only a few million text-heavy pages created by webmasters who were largely ignorant of search engine ranking algorithms. Words-on-the-page ranking worked then, because the playing field was relatively level.

As we go into the new millennium, we have a web measured in hundreds of millions of documents, many of which are so design-heavy that they get downgraded in relevancy by existing systems. Add to this the fact that many webmasters now actively work to influence rankings, and it make sense that search engines need something new to determine relevancy.

"The good answer to the query 'car' has nothing to do with the text. I'd rather use a medium with a crystal ball," said AltaVista's chief technical officer Louis Monier, when I spoke with him on the issue earlier this year. AltaVista hasn't made dramatic move like Infoseek or HotBot yet, but don't be surprised when it does.

Adding to the pressure on the other services is the growing popularity of Google, which relies primarily on link popularity for ranking pages. Google isn't breaking away from words-on-the-page because it never started there.

Even GoTo, with its pay-for-placement model, is thriving. Not only are users not shunning the service, but the last NPD survey of the major search engines, GoTo ranked number one for users reporting they found what they were looking for "every time."

HotBot Integrates Popularity Into Top Results
The Search Engine Report, March 3, 1999

More about the HotBot change to Direct Hit.

Counting Clicks and Looking at Links
The Search Engine Report, Aug. 4, 1998

More details about getting away from words-on-the-page ranking, and systems like Google and Direct Hit, which do this.