A Search Engine Gateway to the Invisible Web

The Invisible Web is an enchanted realm for searchers, but only if you know how to access its abundant treasures. The Resource Discovery Network (RDN) is an outstanding gateway to thousands of Invisible Web sites that's as close to a search engine for the hidden web as you're likely to find.

The RDN is a web directory compiled by subject and information experts in colleges, universities and related organizations throughout the United Kingdom. These individuals identify, catalogue and describe high quality Internet resources relevant to teaching, learning and research.

Like the U.S. based Librarian's Index to the Internet, the RDN is not a "pure" invisible web directory, but a considerable portion of its high quality content consists of material indexed poorly (if at all) by conventional search engines.

The RDN is structured as a cooperative network consisting of a central organization and a number of independent service providers called hubs. Experienced searchers will recognize many of these hubs, which include:

BIOME - Health and Life Sciences

EEVL - Engineering, Mathematics and Computing

Humbul - Humanities

PSIgate - Physical Sciences

SOSIG - Social Sciences, Business and Law

While these hubs can be accessed independently, browsing the RDN lets you easily access all of them under a unified interface. Even better, the site's search function provides cross-disciplinary querying of all RDN resources with a single search.

The service currently links to more than 35,000 human selected resources organized into eleven topical categories.

The RDN also offers a news service called "Behind the Headlines" that offers links to in-depth resources and information for a wide range of current events. It's an excellent way to get information not always provided by the mainstream media. For example, related to the current instability in Zimbabwe, there are links to both government controlled web sites and independent groups advocating democratic reform in the country.

The RDN's "Virtual Training Suite" is another useful resource. This is a set of online tutorials designed to help students, lecturers and researchers improve their Internet information skills. The tutorials take around an hour each to complete, and include quizzes and interactive exercises. The tutorials provide both an excellent way to sharpen research skills and to learn what's available online for specific subject areas.

The RDN is also pushing the envelope when it comes to resource discovery, according to Simon Jennings, Manager
of the Resource Discovery Network Centre.

"In the medium term we will be developing an advanced search and a search engine based on harvesting one hop away from all the links in our 35,000 hand selected and described records," says Jennings. "The software will store (and we hope, in future, utilise) the linking relationships between all items in the database."

In other words, the RDN is applying Google-like techniques to find additional web resources based on the "recommendations" made by links in its existing database of selected sites. This "focused crawler" approach to resource discovery is providing excellent results, when a bit of filtering is applied, says Jennings.

The RDN is a first-rate gateway to some of the best resources available on the Web. And, given that it points the way into numerous regions of the Invisible Web, it's a tremendously valuable pathfinder for all of us.

The Resource Discovery Network

RDN "Behind the Headlines"

RDN "Virtual Training Suite"

The Librarians' Index to the Internet
The Librarians' Index to the Internet (LII) is a searchable, annotated subject directory of more than 8,500 Internet resources selected and evaluated by librarians for their usefulness to users of public libraries.

Navigating the Invisible Web
A brief overview of the Invisible Web.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Stanford celebrates 10th anniversary of first U.S. Web site...
SiliconValley.com Dec 4 2001 1:31PM GMT
AT&T drops bid for Excite@Home...
CNET Dec 4 2001 1:29PM GMT
Value of Online Information Market To Grow...
Content-Wire Dec 4 2001 1:07PM GMT
Google vs. AllTheWeb...
Linux Journal Dec 4 2001 4:58AM GMT
Big guys usually win best addresses...
Chicago Tribune Dec 3 2001 4:59PM GMT
Macromedia opens Net to the disabled...
ZDNet Dec 3 2001 3:58PM GMT
ExciteAtHome Users Scrambling...
Wired News Dec 3 2001 3:58PM GMT
What fate for Excite@Home subscribers?...
Business 2.0 Dec 3 2001 9:00AM GMT
At M.I.T., a Fake Web Site Pokes Fun at the Media Lab...
New York Times Dec 3 2001 8:12AM GMT
Microsoft Reorganizes Web Divisions in Battle Against AOL...
Quicken.com Dec 1 2001 10:23AM GMT
DomainPeople Launches Customized Domain Registration System...
Web Host Industry Review Dec 1 2001 10:12AM GMT
Playboy Claims Domain Registered By The Anti-Porn Flynt...
BizReport.com Dec 1 2001 7:28AM GMT
powered by moreover.com

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was About.com's Web Search Guide.