How Search Engines Make Sense of the Web

Search engines are essentially massive full-text indexes of web pages. The quality of the indexes, and how the engines use the information they contain, is what makes — or breaks — the quality of search results.

We’re all familiar with back-of-the-book indexes. They’re simply alphabetized lists of the important words in the book, and the pages on which they appear.

Search engine indexes are similar, but vastly more complex that back-of-the-book indexes. Although most of us will never want to become experts on web indexing, knowing even a little bit about how they’re built and used can vastly improve your searching skills.

A good way to learn about web indexing is to spend some time with a page compiled by The School of Library, Archival and Information Studies at the University of British Columbia. Indexing Resources on the WWW is a lengthy list of links to research, articles and web sites concerned with the process of indexing of all kinds.

SearchDay Readers will want to investigate two sections in particular. The first, Information Retrieval, looks at all types of searching.

Although most of the links point to technical information, there are a number of excellent articles for non-professionals, such as Vannevar Bush’s classic As We May Think, the recent U.C. Berkeley research on How Much Information, and others.

The second section, Indexing Resources on the WWW is focused on indexing specifically for the world wide web. Three parts of this page will be of most interest:

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication’s search facility, which most have, and search for the headline.

University plans bot museum
USA Today May 5 2003 12:41PM GMT
Google: An engine of change May 5 2003 12:34PM GMT
Google listens to your questions
The Register May 5 2003 9:55AM GMT
Signs of a Revival for Online Ads
New York Times May 5 2003 5:52AM GMT
Microsoft Takes XML Mainstream
EContent May 5 2003 4:02AM GMT
UK internet surfers are weird
Web-User May 5 2003 0:16AM GMT
Hints on tracing stock splits on Yahoo
San Francisco Chronicle May 4 2003 12:32PM GMT
eBizSearch Search Engine for E-Business Info
Research Buzz May 3 2003 12:43PM GMT
U.S. may add to copyright confusion
ZDNet May 2 2003 12:58PM GMT
Are spam blacklists killing legitimate emails? May 2 2003 11:44AM GMT
Report: Web Traffic Rises With Consumer Confidence May 2 2003 5:24AM GMT
Search Engine Optimization for Pure Content Sites
High Rankings May 2 2003 4:02AM GMT
Is Yahoo making eyes at Overture?
CNET May 1 2003 9:15PM GMT
powered by

Related reading

Simple Share Buttons