By Chris Sherman Executive Editor, October 23, 2001
You've likely heard the phrase "Invisible Web" over the past year. In a nutshell, the Invisible Web consists of material that general-purpose search engines either can not, or perhaps more importantly, will not include in their collections of Web pages. It's not that the search engines and Web directories are "stupid" or even badly engineered. Rather, they simply can't "see" millions of high-quality resources that are available exclusively on the Invisible Web.
The Invisible Web contains vast amounts of authoritative and current information that's accessible to you, using your Web browser or add-on utility software -- but you have to know where to find it ahead of time, since you simply cannot locate it using a search engine like Google or Lycos.
Why? There are several reasons. One is technical -- search engine technology is actually quite limited in its capabilities, despite its tremendous usefulness in helping searchers locate text documents on the Web. Another reason relates to the costs involved in operating a comprehensive search engine. It's expensive for search engines to locate Web resources and maintain up-to-date indices. File formats other than simple text web pages also pose challenges for search engines.
By far the largest part of the Invisible web consists of databases. There are tens of thousands of web-accessible databases that are simply impenetrable to current generation search engines.
To paraphrase the razor salesman, I was so impressed with the Invisible Web that I wrote a book about it. "The Invisible Web: Uncovering Information Sources Search Engines Can't See," written with co-author Gary Price, is a 439 page volume covering everything we know about the web's hidden treasures.
The book takes a detailed look at the nature and extent of the Invisible Web, and offers pathfinders for accessing the valuable information it contains. We include a history of Web search engines, a detailed examination of what the Invisible web is (and is not), show you how to find your own way around the Invisible web, and include links and descriptions of more than 1,000 Invisible Web resources that we consider to be among the very best on the Net.
The book is available at most major bookstores, and directly from the publisher at a 20% discount from the cover price. We've also created a companion web site with sample chapters and an ever-growing directory of Invisible web sites that's freely available to everyone.
The Invisible Web: Uncovering Information Sources Search Engines Can't See
http://store.yahoo.com/infotoday//invisibleweb.html
Order the book directly from the publisher and get a 20% discount from the cover price. ISBN 0-910965-51-X.
Invisible-web.net
http://www.invisible-web.net/
The companion site for the book, including the directory of Invisible web resources. An overview, the book's introduction and table of contents, and all of chapter 12 are also available.
Book Reviews
http://www.usatoday.com/usatonline/20011015/3539178s.htm
http://www.time.com/time/europe/biz/magazine/0%2C9868%2C166169%2C00.html
http://www.freepint.com/issues/041001.htm#bookshelf
Reviews of The Invisible Web from USA Today, Time, and Free Pint.
NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.
