He considers the Open Directory (or any web directory) no longer necessary, given how the web has evolved and grown, saying:
It achieved these goals and has fulfilled its mission of becoming the largest human-edited directory of the web. But the web moved on, and while directories were very
interesting in the mid ’90’s, keyword search has eclipsed them as the main ways consumers find information on the Internet.
Skrenta also provides some background on how to automatically classify the news and finding copies of stories where registration is not required, when possible.
Some time ago, I wanted to do my own long piece on the decline of directories but never got to it. So I’ll dive in a bit here.
Back in 1999, it seemed directories had won in search, making up the majority of services when compared to crawler-based search engines. News.com has a nice piece on this
from back then, Web search results still have human touch.
So what happened? I use a library metaphor to explain the decline. In the early days of crawlers, it was like walking into a library, asking for information about cars and being given
thousands of matching pages from within the various books to sort through.
Sure, some of the good pages might be on top. But it was overwhelming to get so much junk and other
information as well. In contrast, a directory was like using a card catalog. It helped you locate a few books on the topic of cars, a much more manageable list to deal with.
Crawler-technology has improved since 1999, of course. Google led the way and gave us the ability to search on every page of every book in the library AND largely get some
very good matches right up front. The need for directories as a filtering device has diminished.
Again from News.com, The changing face of search engines from 2003 looks at some of this flip-flop, including the
idea of Yahoo losing its directory "religion" when it put crawler-results first and the challenges the ODP has faced.
Even if directories are in decline, humans still have a role. In our recent coverage of AOL’s search
engine changes, it was heartening to hear that the company has about 60 people working to active shape, refine and customize some of the results its service. That’s 60
more people than Google has actively intervening in keyword-specific results.
There are times when it is helpful for human beings to review results and do hand manipulation of them, to "program" the most important queries to help ensure quality. As
I’ve written before, sadly most of the major search engines have abandoned such review.
You can’t intervene in every result, and doing so raises other issues, such as when AOL removed
the George W. Bush home page from coming up tops for a search on miserable failure. But it can also
ensure that your users are not being served solely by an automated process that can and will make mistakes.
Sure — airplanes can take off, fly and land themselves. But it’s nice to have a pilot there as a double-check and vice versa.