THE SEARCH ENGINE UPDATE
November 23, 1998 - Number 41
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch, http://searchenginewatch.com/.
Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
In This Issue
I'm in the midst of site revisions and working through some new material. I should have many new and revised pages -- and an update of the offline edition -- available around Dec. 1. In the meantime, you'll find some interesting resources emerging out of my work in the Webmaster and Marketer Tools section, below.
Additionally, I've had so many requests over the past month or so for information about agent and IP delivery that I created a new page explaining important issues involved and providing more assistance for those wishing to enable such delivery options. It's just in time, too -- because the issue of "cloaking" web pages has just erupted on some prominent marketing mailing lists. I hope you'll find the page informative about an issue that can cause concern and confusion.
I've also created a new page that compares key data between various directories. You can see the number of editors, categories and listings for major directories, in as much as this information is known.
Additionally, the Measuring Link Popularity page has been updated.
Agent Name and IP Delivery
Measuring Link Popularity
Search Engine News
Rumor is that AOL may purchase Netscape this week, with the online company to take over Netscape's portal site and browser operations, while Sun would take over Netscape's business software operations. I won't list actual story links, as new articles will be online by the time you receive this. Instead, check in to one of the news sites below for updated headlines.
Netscape has acquired the volunteer-compiled NewHoo web directory, which has been rebranded as the Netscape Open Directory. It is currently being made available through Mozilla.org and will be offered later within the Netscape portal site. The announcement came on Nov. 18, and terms of the deal were not disclosed.
NewHoo launched in June with an interesting premise: volunteers would be enlisted to compile the directory, which aimed to be more comprehensive and fresher than the Yahoo directory.
Since that time, NewHoo has compiled over 100,000 web site listings. That's well below those available from competitors such as Yahoo, LookSmart and Snap, but NewHoo also has a huge staff of over 4,000 editors. None of the major directories comes close to having this much surfing power.
On paper, NewHoo's staffing structure meant it could be a powerful force -- but few were using it. "Traffic was nowhere near major portal level," said Rich Skrenta, NewHoo's former CEO who now leads engineering for the Netscape directory.
Now Netscape will now be routing significant numbers of visitors to the directory. The site is already available under the auspices of Mozilla.org, where Netscape has made its browser source code available to developers for reuse and new development. Netscape intends for its directory listings to be open for reuse in the same way.
"Hopefully it will make people more comfortable about participating in the open directory," said Skrenta. "The goal is really to make the biggest and best directory of the web. We want there to be one of them, one big giant authoritative directory of the web. In order to do that, you have to give it away."
It will be interesting to see exactly how people make use of the listings, especially if Netscape's rivals decide to repackage them within their own sites. But the move with Mozilla.org primarily seems to be a way for Netscape to put a positive spin on making money off a directory compiled by thousands of unpaid workers.
That money will come from the other place where Netscape will publish its directory: within its own portal site. This will occur over the next six months, as Netscape decides exactly how to integrate the listings into its channels.
While Netscape has been busy recasting itself as a search-and-navigation portal this year, the NewHoo acquisition is the first real-step the company has made toward backing up its words with original content.
Netscape has always repackaged its portal content, first from Yahoo, and now even more dramatically from Excite. Repackaging may be fine for the captive audience Netscape controls via its browser and its default web pages, but compelling content is what's needed to let Netscape be a player independent of its browser.
Yahoo is the classic example. Its reputation as the best place to locate information on the web is what drives millions to its site, and that reputation is built on its human-compiled directory.
Netscape has made a number of content deals recently, but none of these has substantially improved its search and navigation offerings. NewHoo offers a solid core that Netscape can truly build upon -- and a solid reason for users to consider the service.
NewHoo's volunteer editors are its greatest strength. One analyst quoted in the Wired story below sounds almost dismissive of these volunteers: "How valuable an editor is Joe in Podunk who happens to like Pearl Jam," he asks. The answer is a lot more valuable than an overworked generalist editor with no real passion about the band.
After all, volunteers usually take up subjects in which they have an interest or a degree of expertise. That means they may know the topic far better than a generalist editor may, and so are likely to produce higher quality listings.
One need only look at the web to see this. Many of the best entertainment web sites are volunteer efforts created by fans, which often make "official" web sites look poor in comparison.
At the moment, this potential is best seen in what Skrenta calls NewHoo's "bright spots," scattered areas in the directory where editors with special interests have expanded listings well beyond Yahoo. Compare Yahoo's Thyroid Disease category to NewHoo's Thyroid Disorders category, for example.
In another example, I found a NewHoo category called "Urban Speleology," which is about exploring manmade tunnels and caverns, such as missile silos. Yahoo has no matching category -- this is clearly a case where someone has an interest in a subject that's expressed by organizing web sites for the web as a whole.
(Correction: I learned after this was written that Yahoo does have a similar category: Urban Exploration)
Of course, NewHoo's top level subject of Spelelogy is woefully under populated when compared to Yahoo. The directory has a long way to go until its bright spots become the norm. But by their sheer numbers, NewHoo's editors have the potential to divide and conquer the web in much greater depth than can rival directories.
NewHoo's editors can also be its greatest weakness. Some editors sign up with their own agendas. Shortly after the directory launched, there were several public complaints about editors that listed their own sites as cool and then did little more to maintain their categories.
Skrenta says this is mostly corrected now, with editors tending to police themselves and report any undesirable actions.
Behind the scenes, NewHoo operates dramatically differently than directory leader Yahoo.
Yahoo believes in centralizing its surfing staff as much as possible: most of them work in the same building in Santa Clara, California -- even some of those responsible for non-US listings. Srinija Srinivasan, who runs Yahoo's listings, has told me in the past that she feels this central gathering is vital to assembling an effective guide.
A central gathering doesn't mean central control. Classification rules at Yahoo don't come down from on high. Instead, editors may respond to the rapidly changing web on a day-to-day basis. While they have a high degree of individual control, they also work with each other to decide when to create new subcategories or to reorganize existing ones. That's where face-to-face contact is seen as essential.
In contrast, NewHoo editors are spread across the globe. Skrenta says they live in 229 countries and speak at least 20 languages among them. Face-to-face contact is impossible, but they still work together via email.
"Usually, editors within a particular category will communicate quite a bit," Skrenta said. "People tend to converse with editors around them, and it does tend to flow down in a tree," he added, referring to the fact that editors that oversee categories higher in the directory tend to watch over the progress of subcategories that are produced by other editors.
NewHoo even lacks someone watching over the entire directory structure. It has no counterpart to Yahoo's Srinivasan. NewHoo's five founding employees, now employed by Netscape, have left things mostly to the volunteers. Nor is there a desire to change this.
"We don't have any plans to build up a big editorial staff inside of Netscape. The model is to put this in the hands of the people on the web and let them at it," Skrenta said.
I'd be surprised if this hands-off approach lasts long. I suspect it will be essential that the guide eventually have at least some top-level editors on staff, as it continues to grow. But the crucial question is whether the mass of volunteer editors will keep working for free.
Netscape's not the first to take advantage of free labor. Community sites like GeoCities and Tripod have made use of their members' efforts to sell ad space for some time. But in return, members have been given free web space.
In contrast, it seems NewHoo editors have been working under an esprit de corps, inspired by the opportunity to build a new and valuable resource for the web community. They still have that opportunity, but they are also building something valuable for a large company that stands to benefit.
Of course, without Netscape's support, the NewHoo directory likely would have continued to be a good idea, but one that few people would know about. The greater exposure in partnership with Netscape is a huge benefit that editors are receiving, and a reason for them to stay on, Skrenta says.
"For them, as a NewHoo editor, they have a lot more exposure now. More people are going to see and use their work. They all seem pretty jazzed about it," he said.
For webmasters, the acquisition means that NewHoo goes on your must-do list of places to submit. If you aren't in the directory, go submit now. The article below covers basic tips that will help.
Netscape Open Directory Project
NewHoo: Yahoo Built By The Masses
The Search Engine Update, June 22, 1998
More details about the service, with information on the end that covers submission.
Netscape Acquires NewHoo
Wired, Nov. 18, 1998
As part of my updating and working through new material, I've come across a number of small sites and tools that are mainly of interest to webmasters and marketers.
GoTo Database Info
Search Spy is a database of search terms available for desktop use. Using the program is straightforward. You enter a term, and the program scans to find matches. You can sort results by count or by keyword.
The data is gathered principally by sampling the live search displays at MetaCrawler and WebCrawler, though data from some other live displays is also included. The sampling program runs every 30 seconds, every day.
The program comes in three versions. The Lite edition contains 50,000 terms. The Classic edition contains 250,000 terms, and the Pro edition contains a month's worth of data, or 3 million terms.
I compared results from Search Spy to the publicly-available GoTo database (see second link, above, for more information on this) and felt that you really need to use the Pro edition to get what feels like a comfortable sample web searches.
Be aware that searches can take several minutes to complete, when using the Pro dataset. It’s the sort of program you want to run on a separate computer, if one is available.
One handy feature is the meta tag generator. You can review the list of top terms, select those you think are important, and the program will generate a meta keywords tag for you.
The Lite version is free; Classic is $30, and Pro is $95. There is also a subscription version of Pro that is updated every two months, for a year, priced at $295. The program is available for Windows 95/98/NT.
The company is also offering Search Engine Watch subscribers discounted rates: Classic $20, Pro $75 and Pro w/subscription, $255. Use the second URL above for this. Click on the "Cart" graphic, then chose the product you want from the drop-down options menu.
Overall, most people will probably find the online GoTo database to do everything Search Spy does, but without a price tag. Some people may find the product worthwhile as a backup to GoTo's results, especially for more obscure terms where GoTo's own sample is small. And should GoTo's results be taken offline, then Search Spy would be a good substitute.
This freeware package taps into various live search displays, allowing you to create your own database of search terms. Options include saving the most popular searches and saving the current day's new searches.
This is a free Unix log analysis program that provides search terms support for nearly 60 search engines, including some non-US services such as Virgilio and Fireball.
The log analysis program is notable for its graphical displays of crawler activity from the major search engines. It does not do search term analysis, however.
Various search engines offer ways to integrate their search boxes into your web site. Now LookSmart offers a way for webmasters to tap into some or all of its directory listings, with an offer of free advertising on the service as an additional incentive to participate. The step-by-step instructions take about 10 minutes to complete, and then HTML code for the SmartLinks panel is sent via email for insertion on your web pages.
Enter a URL, and this free web-based service will report back on the number of times and percentages various keywords appear within it. Remember, keyword density and frequency as a ranking criteria is just one part of the relevancy equation, but some webmasters may still find this a useful service.
This is a very cool service that allows you to enter a search term, then see banner ads from various major search engines that appear in response. It's a fast way to see who is buying keyword-linked ads across the various services. To use the service, look for the "Bannerstake" section on the lower left-hand side of the home page. Thomson & Thomson, a trademark and copyright services firm offers the service.
U.S. Trademark Boolean Search Page
Enter a few words, and you can quickly discover if a US trademark has been registered containing them. It's free, easy and comes courtesy of the US Patent and Trademark Office. The link above is to the "Combined Marks" search page, which is very easy to use.
News Index News Ticker
News Index is a search engine that crawls selected news web sites. It is now providing a small, Java news ticker that site owners can place on their web sites. Simply enter the search terms relating to a news topic, and appropriate HTML code appears on screen for your use.
Search Engine Notes
Excite Add URL Bug
Excite has had problems over the past two weeks or so with its Add URL page. If you submitted recently and never got a confirmation screen, your submission may have failed to be registered. Excite says the bug is corrected, and the page appears to be working now.
Search Engine Articles
A Flap over Snap
Wired, Nov. 20, 1998
Snap Online is being sued by Snap Technologies, claiming trademark infringement. Details about the complaint.
Search Engine Optimization: Half-Art, Half-Science, Or Half-Baked?
ClickZ, Nov. 10, 1998
Joel Gehman, a marketing analyst with Creative Labs, argues that spending too much time on search engine optimization can be a losing proposition, in terms of the traffic gained (and he's right). The article sparked an interesting follow up conversation in the Online Ads marketing list (see below).
Search Engine Optimization Discussion
Online Ads, Nov. 1998
Moderator Richard Hoy posted his agreement to Joel Gehman's ClickZ article that search engines can consume time better spent on other efforts. That prompted a stream of interesting and intelligent responses. You can browse them from the link above.
Search Engine Optimization: The Science
And The Art
ClickZ, Nov. 19, 1998
A follow up to Joel Gehman's ClickZ article, where I agree that aggressive optimization can be a time waster, but urge that passive changes built into a site from the start can make a dramatic difference for web site traffic. Lots of tips -- and probably nothing new to regular readers of this newsletter. However, if you feel lost in all the details of search engine optimization, this is a good place to get a bigger picture of the very basic things you should be doing systematically throughout your site.
Netscape renegotiating with Infoseek
News.com, Nov. 18, 1998
It appears Netscape wants to drop Infoseek from its Net Search page or lower the existing 15 percent rotation the company has under a contract running through April 1999. Perhaps concern over the new Go site from Disney and Infoseek is causing Infoseek to be targeted?
Portals on new search
News.com, Nov. 16, 1998
Those upstart portal companies better watch out -- once digital television and other hybrid electronic devices become widespread, more established media companies will take the lead. Well, so some analysts believe. Of course, portals like Yahoo and Excite already attract much traffic without being a default home page within Internet browsers. I suspect the real upstarts will be players who think their next-generation access boxes that no one has will translate into instant web market share.
Uneasy Allies: When Portals Go Retail
Industry Standard, Nov. 16, 1998
An interesting look at the relationship between retail merchants and portal sites such as Excite.
Keywords Threaten Domain Name System
TechWeb, Nov. 9, 1998
The owner of whitehouse.com is upset with SmartBrowsing and keyword addressing systems that may stop him from cashing in on those actually trying to reach whitehouse.org. So, he's considering a lawsuit.
Why Yahoo is Good (But May Get Worse)
Jakob Nielsen's Alertbox, Nov. 1, 1998
A short but interesting look at some key strengths and weaknesses with Yahoo, ranging from the technical to the financial.
Trademark Spies Get Better Tools For Keeping Up With Competition
Wall St. Journal, Oct. 20, 1998
More details about the free trademark search service offered by the US Patent Office (see above), which went live in August 1998, and how it can be used by individuals to monitor important business moves.
How do I unsubscribe?
+ Send a message to [email protected] with the following as the first line of the body:
How do I subscribe?
+ The Search Engine Update is only available to paid subscribers of the Search Engine Watch web site. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to [email protected]
I need human help!
+ Send a message to [email protected]. DO NOT send messages regarding list management or site subscription issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at http://searchenginewatch.com/about/contact.html.
This newsletter is Copyright (c) Mecklermedia, 1998
This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) will bring together the industry's leading online marketing practitioners to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, the comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!