Search Engine or Directory?

The web offers two fundamentally different search tools to help you locate information -- search engines and directories. The differences between the two, in design, scope, and even bias, are huge, and choosing the proper tool can often make the difference between finding what you want and hopelessly floundering in a sea of irrelevant information.

I can hear some of you saying, "Pah, only newbies don't understand the difference between an AltaVista or a Yahoo." But that's not always the case. Even experienced searchers can benefit from a quick refresher on the distinctions between these two primary web search tools. So let's take a quick, closer look at the differences between search engines and directories, and when to use them for best results.

The simplest way to think about the difference between a search engine and a directory is to consider the finding aids available in a book. In the front of the book is the table of contents, which provides a high-level overview of the book, with descriptions of chapters and sections. In the back of the book is the index, with lists of important words in the book and the exact pages on which they appear.

If you're just browsing a book, using the table of contents is a great way to quickly locate an interesting section. It'll get you near what you're looking for, but you'll have to do some additional reading to find what you're looking for.

By contrast, an index lets you quickly pinpoint specific pages with only minimal need for additional reading.

Most non-fiction books include both a table of contents and an index precisely because they function as totally different finding aids for readers. As experienced readers, we know how -- and when -- to use either for maximum effectiveness.

A directory is like a table of contents for the web. Directory entries are often annotated with brief descriptions of the page or site the link points to. But like a table of contents entry, a directory listing is arbitrary -- it's a short abstract written by a person to help another person find information.

The key point to remember is that a good, well-written directory listing will mean roughly the same thing to everyone who reads it. A poorly written listing, or a listing that points to a page containing complex material, may mean entirely different things to different readers. You simply have no way of knowing without clicking through and reading the page for yourself.

A search engine is more like a book's index, because search engines literally index the full text of web pages and store them in a huge database that is keyword searchable. You enter a word, and the search engine will point you to specific web pages where that word appears. Unlike a directory listing, there is no conceptual ambiguity in a search engine result.

The analogy tends to break down here, because most directories have a search form or even supplemental results provided by a search engine partner, and many search engines also supplement their results with information from directories. But as a general rule, think of directories as tables of contents and search engines as indexes.

Why is this distinction important? For several reasons.

A table of contents has a structure (it reflects the organization of the book), while an index is just a list of words. Similarly, directories offer structure that's completely missing from most search engines. Directories are usually organized in a hierarchal fashion, with major topics or categories and subcategories beneath them, often several levels deep. This structure makes it easy to browse for information simply by drilling down through subcategories that interest you.

Browsing is an excellent way to begin a search on a topic you're unfamiliar with, since you can easily move from a general category to a specific resource just by clicking links -- you don't need to know the "right" keywords to find what you're looking for. But browsing is inefficient and time-consuming if you know exactly what you're searching for. A search engine, with its index-like structure, is a far better choice for finding a specific needle in the haystack of the web.

It's important to keep in mind that search engines are created by automated software, whereas directories are painstakingly assembled by human beings. Search engines benefit from the full power of the computer resources used to create their indexes of the Web, and therefore are generally orders of magnitude larger in scope than directories created by comparatively pokey human beings.

This means that a search engine will be a better choice if you're looking for comprehensive, exhaustive results. A directory, on the other hand, will serve you better if you want a selected guide, in theory limiting your results to only the best or most useful web resources for your information need.

And that brings us to the potentially sensitive issue of bias. Directories are inherently more biased than search engines, simply because they are compiled by people. Bias isn't necessarily bad -- a directory like Yahoo, for example, is biased in favor of high-quality sites and against sites that don't offer much for users. Other subject-specific directories will be biased in favor of sites relevant to the specific subject and against all other sites.

Search engines, on the other hand, tend to be less biased, simply because they don't put as much effort into evaluating resources they include in their indexes, and they have to be all things to all searchers. Search engines are not completely bias-free, of course, because most exclude some web content from their indexes. And all approaches to determining relevance have certain built-in biases toward what constitutes a "good" match for a query.

We'll take a closer look at these and other differences between search engines and directories in future issues. But the next time you're pounding you head on the table in frustration with your search results, step back for a moment and consider whether you're using a search engine or a directory, and whether it's the appropriate tool for your current information need.

The Major Search Engines and Directories

How do you know whether you're using a search engine or a directory? This page has an overview of the major search services on the web, indicating whether they are primarily a search engine or directory.

Search Engine Alliances Chart

Most web search services offer both search engine and directory information, though they will predominately feature one type of results over the other. This chart provides an at-a-glance view of who's playing with who in the world of web search.

Elsewhere on

Faster Wireless Won't Save WAP

A new study says that, despite the claims of some, faster wireless access won't overcome the shortcomings of Wireless Application Protocol (WAP).

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

That's it for this issue. Thanks again for subscribing, and watch for tomorrow's issue - Browser companions promise welcome relief from information overload - that is, until the companies offering them change business models or go out of business entirely.