Mapping the 'Dark Net'

Researchers at Arbor Networks have discovered that up to 5% of the Internet is completely unreachable. This isn't a case of Invisible Web sites not being found by search engines. Rather, the study draws a map of true black holes on the Net that are simply impossible to access by web browser or search engine alike.

The three year study focused on the topology, or connectivity, of servers connected to the Internet. Internet access is provided by a vast number of service providers, ranging from your local ISP to titans like AOL and MSN. In turn, each of these providers rely on a small number of network providers who operate the long-distance "pipelines" that make up the "backbone" of the Net.

The dark space discovered by the researchers consists of parts of the Net accessible from one provider but unreachable via one or more competitor.

Citing "failures, filter errors and disputes," the study paints a portrait of behind-the-scenes activity that can directly influence what you're able to find on the Net. Failures and filter errors often result from misconfigured routers, the virtual traffic cops that direct Net activity.

Occasionally, however, Net traffic is blocked due to contractual disputes between providers. Most providers have "peering" agreements that allow them to handle traffic from one another. End users may never realize that sites are unreachable due to a spat between two or more essentially faceless Net "plumbing" providers.

On a more disturbing note, the study also found malicious use of dark address space. This includes spam, attacks taking the form of IRC (Internet Relay Chat) battles, and squatting or address stealing. For example, some spammers use a pernicious technique whereby routes are announced and then immediately withdrawn around the time of a mass email transmission. IRC battles and address stealing are essentially hacker tricks.

The good news from the study is that foul play and shenanigans makes up a relatively small percentage of the total amount of dark space. The majority of unreachable Net connected computers are owned by broadband customers and the United States military. Broadband customers presumably don't want outsiders accessing their computers, and may be blocking access with protective measures like personal firewalls. And the US military has its own reasons for restricting access to its networks.

If you're able to handle a bit of dense technospeak, the report makes interesting reading, illustrating yet another reason that it's not always easy to find what you're searching for on the Net.

Shining Light on Dark Internet Address Space (pdf)

Power Point Abstract:

Full Paper:

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Save the Internet...
Boston Globe Jan 24 2002 2:22PM GMT
Netscape seeks more than money...
ZDNet Jan 24 2002 9:28AM GMT
Labels ask time-out in Napster battle...
The Register Jan 24 2002 8:37AM GMT
The worst is over - Autonomy boss... Jan 24 2002 8:22AM GMT
.biz proves popular in Europe... Jan 24 2002 1:24AM GMT
Exactly What Is AOL Up To?...
Wired News Jan 23 2002 7:47PM GMT
Terry Semel: Can he Yahoo?...
CNET Jan 23 2002 1:30PM GMT
AOL Time Warner suit restarts browser battle with Microsoft...
San Francisco Chronicle Jan 23 2002 1:21PM GMT
powered by

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was's Web Search Guide.