Navigating the Invisible Web

You've likely heard the phrase "Invisible Web" over the past year. In a nutshell, the Invisible Web consists of material that general-purpose search engines either can not, or perhaps more importantly, will not include in their collections of Web pages. It's not that the search engines and Web directories are "stupid" or even badly engineered. Rather, they simply can't "see" millions of high-quality resources that are available exclusively on the Invisible Web.

The Invisible Web contains vast amounts of authoritative and current information that's accessible to you, using your Web browser or add-on utility software -- but you have to know where to find it ahead of time, since you simply cannot locate it using a search engine like Google or Lycos.

Why? There are several reasons. One is technical -- search engine technology is actually quite limited in its capabilities, despite its tremendous usefulness in helping searchers locate text documents on the Web. Another reason relates to the costs involved in operating a comprehensive search engine. It's expensive for search engines to locate Web resources and maintain up-to-date indices. File formats other than simple text web pages also pose challenges for search engines.

By far the largest part of the Invisible web consists of databases. There are tens of thousands of web-accessible databases that are simply impenetrable to current generation search engines.

To paraphrase the razor salesman, I was so impressed with the Invisible Web that I wrote a book about it. "The Invisible Web: Uncovering Information Sources Search Engines Can't See," written with co-author Gary Price, is a 439 page volume covering everything we know about the web's hidden treasures.

The book takes a detailed look at the nature and extent of the Invisible Web, and offers pathfinders for accessing the valuable information it contains. We include a history of Web search engines, a detailed examination of what the Invisible web is (and is not), show you how to find your own way around the Invisible web, and include links and descriptions of more than 1,000 Invisible Web resources that we consider to be among the very best on the Net.

The book is available at most major bookstores, and directly from the publisher at a 20% discount from the cover price. We've also created a companion web site with sample chapters and an ever-growing directory of Invisible web sites that's freely available to everyone.

The Invisible Web: Uncovering Information Sources Search Engines Can't See

Order the book directly from the publisher and get a 20% discount from the cover price. ISBN 0-910965-51-X.

The companion site for the book, including the directory of Invisible web resources. An overview, the book's introduction and table of contents, and all of chapter 12 are also available.

Book Reviews,9868,166169,00.html

Reviews of The Invisible Web from USA Today, Time, and Free Pint.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Web titans welcome content ratings...
ZDNet Oct 23 2001 12:39PM GMT
Post-Napster, Peer-to-Peer Computing Gets Ready for Prime Time...
Business 2.0 Oct 23 2001 7:38AM GMT
New Yahoo service targets MSN search...
Business 2.0 Oct 23 2001 7:38AM GMT
AltaVista to sell corporate solutions via its search engine in Russia... Oct 23 2001 7:07AM GMT
The Keyword at Google: Growth...
Business Week Oct 23 2001 3:09AM GMT
AOL, Microsoft: Youve got a conflict...
ZDNet Oct 22 2001 1:52PM GMT
Yahoo signs Garbage for Instant Messenger...
ZDNet Oct 22 2001 11:52AM GMT
Farewell web freedom?...
Media Guardian Oct 22 2001 8:18AM GMT
The Rant: Whats in a Name? If You Give It Away, a Box of Spam and a Lot of Aggravation...
Network Computing Oct 22 2001 7:22AM GMT
VeriSign expands its behind-the-scenes domain... Oct 22 2001 5:42AM GMT
Newsweek: Rumor-busting on the Web...
MSNBC Oct 22 2001 3:39AM GMT
Personal technology: Whats that saying? Google can tell you...
Seattle Times Oct 21 2001 12:13PM GMT
powered by

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was's Web Search Guide.