Carrier Pigeons for the Web

Panic gripped Britain in June of 1815, as word reached London that the empire's forces were being routed by Napoleon. The financial markets crashed, as securities dealers frantically unloaded government bonds at fire-sale prices. Amid the chaos, few took note of a banker named Nathan Meyer Rothschild who was going against the trend, quietly but aggressively buying everything he could get his hands on.

A few days later news of Napoleon's catastrophic defeat at Waterloo arrived, and securities prices soared. Rothschild's courage in buying what everyone else shunned made him fabulously wealthy -- but was it really courage? Not really, according to biographies of the banker.

For Rothschild had a "secret weapon" -- a network of agents using carrier pigeons that fed him with a constant stream of news of the battle hours or even days before it arrived by other means. Rothschild's early warning system gave him an advantage that he didn't hesitate to exploit.

Fast-forward to today, with news being posted to the web almost instantly, and its easy to think that there are no more early warning systems like Rothschild's. But a new project underway at MIT's Media Lab may actually reinvent the concept of "carrier pigeons for the web."

Blogdex is the creation of 24-year-old PhD student Cameron Marlow. It's an index of more than 10,000 weblogs, the often idiosyncratic online journals that typically combine personal musings with commentary on web sites the weblog's author, or "blogger," considers interesting.

Good bloggers take pride in finding content that's both fresh and cutting-edge. If you find a blog written by someone with similar interests to your own, it can be a gold-mine for finding content that you often wouldn't find on a traditional media web site, let alone using a search engine.

The problem is that many bloggers tend to follow their instincts rather than a consistent editorial mindset. Fortunately, there's a strong sense of community among bloggers. Bloggers often link to other blogs they like (a practice called "blogrolling"), so it's relatively easy to browse a number of different "recommended" blogs based loosely on your own interests.

In addition to linking to other blogs, bloggers often link to the same sites or pages on the web. By examining the link structure of the thousands of blogs in its index, Blogdex "can get an instantaneous look at internet fashion from democratic means," according to the site's information page.

Examining the link structure of web pages is essentially how Google works. Blogdex uses similar techniques to examine thousands of blogs and generate a list of the most popular pages or sites the blogger community has linked to. The list is updated daily.

Blogdex also offers a cumulative look with a list of the top all-time links -- the sites the blogger community has "voted" as the most noteworthy on the web.

Blogdex's creator Marlow believes this will both "democratize" news by allowing readers to bypass major media outlets in favor of sites that have been voted as best by the blogging community. He also believes that it might change the nature of news gathering, if Blogdex catches on among journalists as a source for emerging trends.

I asked Steve Outing, an expert in online journalism (and the maintainer of a noted blog himself) what he thought of Blogdex and its capabilities.

"I do agree with Blogdex's creator that this could be a useful tool for journalists, to help them spot new trends," Outing wrote in an email. "Weblogs are well known for reporting on things that the mainstream press picks up much later.

"If I understand it correctly, Blogdex is about ranking articles linked to by the thousands of weblogs. My first reaction is that while this will be interesting data (ranking the popularity or interest in news stories), what are we supposed to do with it? I mean, weblogs are still primarily personal projects; while commercial weblogs are growing, they're still far outnumbered by the one-person-in-their-spare-time blogs. And most of them are quirky, taking on oddball topics. So I'm not sure that the statistics Blogdex generates will be terribly meaningful.

"It's interesting, no doubt. I think it would be cool to have it also offer a search function (e.g., find all the weblog items that mentioned Jenna Bush yesterday), which would provide a nice news-clipping tool that covered the "alternative" media world of weblogs. Also a subscribe service (e.g., send me a daily mailing of all the weblogs that mention the Denver Broncos). I don't see the search functionality on the site yet. I'd find the search feature much more compelling and useful than the links ranking."

Blogdex creator Marlow is hard at work expanding the capabilities of Blogdex, and full-text search is one of the likely functions to be added in the future. Several of the major search engines have expressed interest in weblogs recently, noting their freshness and ability to target obscure but often useful or interesting content.

In the short run, however, Blogdex is the 21st century kin to Rothschild's carrier pigeons. It's unlikely Blogdex will make you wealthy, but it does serve as a terrific advanced warning system for emerging trends and web content that's often difficult to find by other means.


Pass Me the Blog, Please
"How in the world did you find THAT?" The answer, quite often, is by searching through blogs, the web's equivalent of a sophisticated early warning system.

Anatomy Of A Weblog
Longtime blogger Cameron Barrett describes the weblog phenomenon, using a variety of different blogs to illustrate widely different approaches.

Weblogs: A History And Perspective
A look at the evolution of the weblog phenomenon, by Rebecca Blood.

E-Media Tidbits: A Group Weblog
Maintained by Steve Outing, this weblog bills itself as "news, analysis, & interesting stuff from the sharpest minds in online journalism/content/publishing."

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.

Search start-ups seek Googles throne...
ZDNet Aug 28 2001 11:48AM GMT
Gator plays dirty tricks with personal data, ads, Web sites...
Chicago Tribune Aug 28 2001 11:43AM GMT
Yahoo to play the web host... Aug 28 2001 11:08AM GMT
Internet Homesteaders Scramble for the Right Domain...
International Herald Tribune Aug 28 2001 2:35AM GMT
Better Browsing with Google Groups... Aug 28 2001 1:23AM GMT
Do search engines tell the truth?...
CNN Aug 27 2001 5:00PM GMT
Half a Dozen Things Google (Or Any Search Engine) Could Do...
Research Buzz Aug 27 2001 3:58PM GMT
Branding Your Web Site and Search Engine Optimization...
Rank Write Aug 27 2001 8:35AM GMT
Parody defense falls flat in cybersquatting case...
Freedom Forum Online Aug 26 2001 7:58PM GMT
Personal portals get you started on research...
Chicago Tribune Aug 26 2001 6:39PM GMT
Exploration of World Wide Web Tilts From Eclectic to Mundane...
Yahoo Aug 26 2001 4:37PM GMT
Google Wins by Not Hiding the Banana...
Traffick Aug 26 2001 9:47AM GMT
Image Search Faces Renewed Legal Challenge...
Search Engine Report Aug 24 2001 8:15AM GMT
powered by

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was's Web Search Guide.