In the search battle of human versus machine, 1999 is the year that the tide turned in favor of humans. One-by-one, major services that served up predominately crawler-based results have switched over to using human-power for their main listings.
Let's recap. In April, Lycos changed to the Open Directory. In June, Netscape Search made the same move. September saw AOL Search begin using Open Directory information, while MSN Search changed to using LookSmart-powered results. Later this month, AltaVista becomes the latest Open Directory convert, listing its categories within its main results.
Here's some perspective. At the beginning of 1996, only one of the six major search services provided human-dominated results -- Yahoo. The others -- Excite, Infoseek, Lycos, WebCrawler and AltaVista -- served up crawler-dominated results.
Now at the end of 1999, only three of the top six major search services remain predominately crawler-based. These are Excite, Go (Infoseek) and AltaVista. Expand the list to the top ten major services, and you only pick up one additional crawler-based service -- HotBot. (For those keeping score, the human-powered ones are Yahoo, MSN Search, Netscape Search, Lycos, AltaVista, LookSmart and Snap).
Crawlers aren't obsolete, of course. Humans are used for the main results at the services above, but crawlers provide the necessary and welcome backup for when there aren't any matches from a human-compiled database.
Let's look at the remaining "holdouts" and make some predictions. I'd say the next to switch is Go, which already has a substantial, quality directory that it could leverage at any time. HotBot already floats Open Directory categories at the top of its listings, but I suspect its deal with Direct Hit means that we won't see Open Directory sites supplanting Direct Hit information in the main results in the near future. AltaVista has just recommitted to using crawler-based results, so it seems unlikely that they will change over soon. That leaves Excite. Given that most of its competitors have changed over to a human results first, crawled results second model, I think it will be inevitable that Excite will also make the change. Moreover, Excite recently cut a deal with LookSmart to license that company's directory. That gives Excite the ability to more easily promote human-powered results.
Of course, the big question is, why has this change happened? For one thing, the Open Directory has sparked a revolution by making human results available for free. That's given Lycos and AltaVista the easy ability to change. And since AOL owns the Open Directory, it makes sense for Netscape Search and AOL Search to use it.
A bigger reason is that directories tend to provide better results because of their nature. A user searches for "travel," and they get a list of categories that are related to that incredibly broad topic. They select a specific category of interest, and in doing so, they painlessly narrow their search without even being aware of doing so. In contrast, a user searching for "travel" on most of the crawler-based services will get millions of matches. They can feel overwhelmed by this large number, and it's worse if the top results aren't relevant to what they are looking for.
So what's next? Better ranking. It's not enough to categorize web sites. They need to be ranked in order of quality. Currently, users do a search, see a category of interest and click through. There, they discover a list of sites where the alphabet is the main ranking criteria involved. Maybe a few "editor choices" top the list. But pretty much, sites that begin with A come first, then the Bs, then the Cs, and so on.
Enough of this. It's like telling someone to get something to drink by pointing them at a soda fountain with 100 different taps, one for each brand. The variety is nice, but some guidance would be appreciated.
Go does guidance right. Check out its directory, via the home page. Sites are ranked in order of quality, three stars being best. I find myself continually drawn back because of this feature. It helps me sort through the mess and decide which sites are worth my time.
What does this change to the directory format mean for the webmaster? I've always said that the most important promotional activity that you can do is concentrate on building a quality web site. Doorway pages are not going to help you impress an editor, when you submit your site for review. Instead, editors want to see a polished web site that loads quickly, is easy to navigate and which offers substantial content to visitors. Build your content. Make your site a place to visit that's related to your underlying products or services. For instance, if you sell car tires online, then ensure that your site has lots and lots of good general information about buying car tires -- in addition to information about your actual products.
The popularity of the Open Directory also means you need to adopt a specific strategy there: multiple listings with different descriptions. At any directory, you have to submit to a particular category. The best way to find the right category is to search for your most important search terms, see which category comes up first and try to get listed there. The problem is that the Open Directory now powers four different services, each of which uses slightly different ranking systems. So, if you pick the "right" category via AOL Search, you may miss out on being in the right category over at Netscape Search.
Here's what to do. Go to each of the Open Directory powered services. Search for your top terms and review the categories that appear at each of them. Submit to all the appropriate ones for your site. There will be some overlap -- for instance, let's say all of them list a category for online car sales. That's fine -- submit to that category once, and you'll be covered on all of them for it. But you may notice that one of them also lists a category about tire buying tips. Perfect, here's another appropriate place for you to submit a relevant subsection of your web site. Now you've expanded your presence and the chances that you will rank well at various Open Directory-backed services.
Furthermore, don't use the same description for all of your submissions. Use a slightly different or even a dramatically different description for each of your different listings, but always ensuring to make use of your key search terms. By doing so, you increase the odds of doing well at the different services, which as I mentioned, have their own ranking criteria.