THE SEARCH ENGINE UPDATE
July 6, 1999 - Number 56
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch, http://searchenginewatch.com/.
Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
In This Issue
Search Engine Briefs
+ Direct Hit Deals Soon
+ Fee-Free Searching At Gov.Search For Schools, Libraries
+ Does Yahoo Own You?
+ New Position Checker Extends Coverage
+ RealNames Coming To MSN
+ New Search Engine Performance Study Out
I'm happy to say that my new system for managing links is nearly ready. I've been transitioning everything from static HTML pages into -- dare I say it -- a Yahoo-like directory system. It will make it much easier for me to add new search sites and resources related to searching. The new system uses software called Hyperseek, and I'd encourage anyone with many link resources to check it out at http://www.hyperseek.com.
Everything should go live by next Monday, if not earlier. When it does, it will contain all the various search engines links that currently reside in the Search Engines Facts section, plus a large number of new sites. Any bookmarks you have to the old pages will automatically go to the right category in the new system, when it goes live.
My next step will be to begin transitioning pages within the Search Engine Resources section into the new system. This will happen slowly over the next two months.
Watch the What's New page to know when the new system goes live -- the location will be posted there. I've also updated a variety of pages within Search Engine Watch, which are listed on What's New. These are:
+ The Search Engine Alliances Chart
+ The Search Engine Features For Webmasters
+ Search Engine Reviews
+ Search Engine Reviews Chart
+ Media Metrix Search Engine Ratings
+ NetRatings Search Engine Ratings
Both the ratings pages are current through April 1999, and May 1999 data should be added in a week or so. The Search Engine Sizes page will also be updated shortly.
I'd planned to do an update of the Search Engine EKGs, which are meant to track freshness and coverage, but I've decided to retire them instead. When I began them, crawling was closely related to freshness and web coverage. In recent months, the relationship to what's spidered and what gets listed is less direct. There are better ways to reflect freshness and coverage, and I'll be looking at posting new types of reports on these topics in the future.
I will try to give the SpiderSpotting Chart a much needed update sometime later this month. I'm also looking forward to doing a new series of relevancy tests on a regular basis. More news as this develops.
In the Subscribers-Only area, you'll find new pages that cover Netscape Search and Google. I've also given the Open Directory page a slight update to reflect Netscape Search changes. Links can be found via the What's New page below. If you've forgotten your password, please try the Password Finder to gain access.
Subscribers-Only Area: What's New
Search Engine News
Netscape released a new version of its Netscape Search service in June. Previously powered by Excite, the new service now draws on the Open Directory and Google for its listings. It's a powerful combination that makes Netscape Search worth considering among your top search choices.
Netscape Search also integrates information from Netscape's Smart Browsing index. Smart Browsing connects Netscape browser users to web sites when they enter ordinary words into the Location box, where you normally enter a web address.
"We've been building out this database of official web sites and other related content. We've now taken that dataset and made it a core set of search," said Dariusz Paczuski, senior program manager for Netscape Search.
I've always been very impressed with the quality of Smart Browsing results, and integrating the information into Netscape Search makes the service even better. I wish Netscape would finally set up a formal mechanism for site owners to suggest sites for Smart Browsing, however. Netscape says it is working on this, but that's exactly what it said a year ago, when Smart Browsing debuted. How much more time is needed?
You can begin using Netscape Search in three main ways. First, you can go directly to the Netscape site and chose "Netscape Search" as your search option at the top of the home page. Second, those using Netscape's browser can just push the Search button. When the Net Search page appears, select the "Netscape" tab, if it is not already selected. Third, you can go straight to the Netscape Search home page, http://search.netscape.com.
When you do a search, you'll be presented with information from one of several different data sources. First will be "Official Sites," if any relevant matches from the Smart Browsing database are found. For instance, search for "hotmail," and a link to the HotMail web site appears right at the top of the page. Search for "united," and you are shown several "official" web sites that use that word, such as United Airlines and the United Nations.
Next, "Netcenter Pages" may be listed. This is content within the Netscape Netcenter portal site that seems related to your search. For example, a search for "horoscopes" brings up a link to the daily horoscopes page within Netcenter.
"Web Site Categories" shows you any matching categories from the Open Directory that match your query. Search for "star wars," and you'll see links to topics such as Star Wars trading cards and Star Wars action figures. Clicking on a category link displays a list of sites relevant to that category.
For those unfamiliar with the Open Directory, it is essentially a Yahoo run by volunteer editors. These editors, over 12,000 of them now, organize web sites into categories. Netscape owns the Open Directory, but it runs it as an ad-free independent site that also offers its data for anyone to use. That's what Lycos did in April -- it began powering its primary results using the Open Directory. Now Netscape Search is also Open Directory-powered.
After categories are displayed, a "Reviewed Web Sites" section of the results page shows selected web sites from within the Open Directory that seem to match your query. So while the example above listed Star Wars-related categories, the Reviewed Web Sites section displays individual Star Wars sites, such as fan sites.
At the very bottom of the results page, you'll see an option to get "Additional Results" from the Google search engine. Just click on the link, and your query will be sent to Google. Netscape will also do this automatically if it fails to find any matches from within its Smart Browsing or Open Directory information.
Google, unlike the other major crawler-based search engines, makes extremely heavy use of link popularity to rank web sites. Many users rave about Google's relevancy, and I've also continued to be impressed with how on target it can be.
An odd thing is that earlier in June, Netscape-owner AOL announced an agreement for Inktomi to power its various search services. So why wasn't Inktomi chosen for AOL-owned Netscape Search? There was no clear answer on this from Netscape, other than Inktomi would be involved in AOL-related search elsewhere.
"I think you'll see a number of announcements coming up where Inktomi will be involved," said David Beckwith, senior director of search for Netscape.
As for why Google was selected, Netscape was foremost impressed with its results. "We are interested in their technology. We are interested in seeing how they play out as a partners and how that could impact other things we want to do, Beckwith said. "Netscape seemed a place where we could start with them and see how it goes."
Users can also browse listings at Netscape Search by starting out at one of the top level categories listed on the home page The home page also lists "Hot Topics" along the right-hand side of the page, which take you directly to popular categories such as "MP3" and "Britney Spears." Topics are changed from week-to-week, Netscape says.
The Netcenter portal itself has also changed. Previously, its channels were mostly filled with content provided by Excite. Now Netscape says it has replaced this with content from other partners and from AOL. Within the channels, you'll also see a "Search Categories" section that will take you into relevant portions of Netscape Search.
FYI, if you compare a category in Netscape Search to the identical one at the Open Directory, you may notice some slight differences. This is because the Open Directory will always be slightly more up-to-date than Netscape Search.
In other changes, the line-up on the Netscape Net Search page has changed. Gone are AltaVista and Infoseek, though the latter remains a search option from the home page. Yahoo is also gone, and a "Where's Yahoo" link has appeared in its place. That takes you to a long page that explains how Netscape Search now has categorized results like Yahoo.
The page does eventually link to Yahoo, but it would seem more user friendly just to keep Yahoo on the Net Search page in a secondary position, whether they pay for it or not, if that's what users are expecting. It's similar to what happens when you search at Lycos for Yahoo -- a page promoting Lycos appears, instead. These things feel a bit too much like a marketing intrusion into the editorial for my taste, though to Netscape's credit, if you actually search for Yahoo using Netscape Search, the Yahoo site appears at the top of the list.
Lycos Transforms Into Directory
The Search Engine Report, May 1999
Background about Lycos using the Open Directory and the Open Directory itself.
Counting Clicks and Looking at Links
The Search Engine Report, Aug. 1998
Background about Google.
Netscape's Smart Browsing Matures
The Search Engine Report, April 5, 1999
How Netscape's Smart Browsing works.
Netscape Net Search Page
Where's Yahoo Page
While I've considered Google a "major player" in the search space for some time, the deal with Netscape gives it a mass audience for the first time, along with its first business deal.
So what's next? Is Google planning to follow an Inktomi-style model and power other services, or was Netscape a one-off, with future efforts focused on finally taking the Google.com site itself out of "beta" mode and into prime time? Google cofounder and CEO Larry Page doesn't see the two as necessarily exclusive.
"We'd like to provide services to people where there's an opportunity to do that," Page said. "I don't see that there's a conflict."
In particular, he feels that way because the Google site itself will stay focused mainly around search. There's no intention to build a full-fledged portal property meant to take on the likes of Yahoo and gang. "That reduces the competition we have with people we might work with," Page said.
That doesn't mean Google.com will feature search and only search, however.
"I won't say we won't add services, but we wouldn't put free email on our site unless we thought we could do a much better job," Page said. "Google is about high quality products. If we add random services, we don't think that adds value."
As for what and when things will happen at Google.com, Page is staying quiet. "You're likely to see some changes soon, but I can't say when or what they are going to be," he said.
Whatever happens, search will remain the top priority. "Our goal is to produce no holds barred, the best search available," Page said.
Google's well on its way toward that goal. I've spent more time at the service recently, trying a variety of queries from the simple to complex. It performs extremely well on all of them. There's a lot of hype on the Internet, but Google lives up to its buzz.
Of course, much is made about Google's use of link popularity to rank results, but another thing to like about it is the fact that it may return results for pages it can't visit due to robots.txt bans or other crawling problems.
How can Google do this? It comes back to links. One of the things Google does is look at the text in and around hyperlinks, which it can then use to help define the pages the links point at. If it sees many pages point at the same site and using particular words, then it can tell that site may be relevant for those words -- even if it has never visited the actual site.
This naturally improves Google's comprehensiveness. Page says Google has indexed about 70 to 100 million web pages, but through link analysis, he estimates searches cover up to 300 million pages on the web. No other major search service comes close to that number. True, it's not the same as full-text indexing of 300 million pages, but the point is that Google has a pretty broad reach.
As for freshness, Page said Google tries to update the bulk of its index each month, though some pages may be refreshed on a daily basis.
Counting Clicks and Looking at Links
The Search Engine Report, Aug. 1998
Background about Google.
Do You Google?
About Web Search Guide, June 29, 1999
Nice, recent profile on Google, with some search tips.
Can Google's search engine find profits?
MSNBC, June 14, 1999
Lots of speculation on how Google plans to make money, in lieu of real details.
It was a busy month for AltaVista -- a new owner, a new direction, new paid links auction and a freshness guarantee were among the major developments for the service. Here's a wrap up:
CMGI, an Internet investment company, has agreed to acquire 83 percent of AltaVista in cash and stock swap valued at US $2.3 billon. AltaVista-parent Compaq will retain the remaining shares. The announcement came on June 30.
This is good news for the service. AltaVista's fate has always been second to the fate of its parents, first Digital, then Compaq. Every time it has seemed ready to move forward as a major online property, some problem with its parent has put things on hold. The CMGI sale cuts the apron strings and lets AltaVista finally have a chance of succeeding properly on its own.
A big question remains as to how the sale will affect Lycos. CMGI is a 20 percent shareholder there, and so now it has links to three major portals: Lycos, Lycos-owned HotBot and AltaVista. Will it push for AltaVista over the Lycos properties? The article below has CMGI's chief executive saying he thinks all the sites can coexist, but analysts seem to think AltaVista will become the favored child. That's especially because CMGI helped scupper the Lycos-USA Networks deal that was proposed earlier this year.
Just two days before the CMGI deal, AltaVista had a press conference to announce new services and a direction for the service. In particular, AltaVista President Rod Schrock said the service would be targeting web savvy users as its audience of choice. "We will focus on the most experienced, most prolific users on the net," Schrock said.
We've not had any major search service say that it was going after a particular type of audience since Infoseek tried for the business crowd in 1997. It quickly pulled back from that attempt and since then, the various portals have essentially competed for the same audience.
One of AltaVista's initiatives is a new "Search Freshness Guarantee" that I feel has gone largely overlooked. That's too bad, because it's important. AltaVista is pledging that its index will be refreshed at least once a month, if not sooner.
I'd like to see all the major services offer a similar pledge and live up to it. We went through Lycos last year getting away with a stale index for months, a situation that's been corrected now. HotBot is now saying it may take up to two months to add new pages. These are both examples of freshness issues which aren't immediately apparent to users. With freshness guarantees, the search engines would be publicly accountable for an important criteria of indexing for the first time.
A monthly update may not sound very fresh, but this is the worst case scenario for pages at AltaVista. It updates some portions of its index on a daily basis, plus new pages are also added daily. AltaVista also expects to begin a new crawling system later this year where each page would be revisited as often as seems necessary. It does some of this today, but the new system would be much more comprehensive.
AltaVista also plans to continue expanding its index. "Our long term goal is to be able to crawl the entire web," Schrock said. How big is that? Barry Rubinson, AltaVista's VP of Engineering, estimated the service would be at 400 to 500 million pages indexed within a year. It's currently at 150 million pages, just behind size-leader Northern Light, which is in the 160 million page range.
In other news, AltaVista opened its paid placement program to the general public last Friday. An automated ordering system is now online that allows anyone to bid on terms. I may make a page that describes the process in more depth, in the future. In the meantime, there's plenty of help available at the AltaVista site, including a helpful tutorial in PDF format.
By the way, Lycos has introduced new "Start Here" links that appear to be paid placements, in many cases. To see them in action, just search for "cars" and look for the first link in the category section. Unlike AltaVista's links, these placements are not identified as advertisements. I've seen one advertiser report that they are very effective, so contact Lycos for more details.
Now some miscellaneous housecleaning:
Some readers have reported problems with pages going missing or strange results at AltaVista over the past month. Rubinson said this is related to engineering works that have gone on, and the problems should be over now. It probably is worthwhile to do a check on any pages you've submitted recently, just to see that they are still listed.
A few readers have commented on the disappearance of AltaVista's Refine feature. Yes, it is gone, and it's not coming back soon. "We are looking at a replacement for it, and stay tuned," Rubinson said.
AltaVista has also debuted a new form for those that find offensive material slipping past its Family Filter (to enable this, choose the link under the search box, on the home page). AltaVista says that the form can also be used to report spam. Pages are not immediately removed. Instead, they are placed on a list for review.
AltaVista is to begin powering MSN Search in the third quarter of this year, AltaVista President Rod Schrock said. He also said its likely that Microsoft-powered email and instant messaging options will appear on AltaVista at the same time.
What can CMGI do with two portals?
News.com, June 29, 1999
An excellent article looking at how CMGI may deal with its ownership in both Lycos and AltaVista.
AltaVista Paid Relevant Links
AltaVista Launches Paid Listings
The Search Engine Report, May 1999
Background on the program.
Offensive Page Block Form
I haven't tried it, but another recent announcement was the availability of AltaVista MicroPortal. This is apparently a desktop toolbar that lets you search via AltaVista, receive stock and news updates, and other features. For Internet Explorer 4/5 users only, and not available until the end of July.
Can't wait for MicroPortal? Get something similar from Excite, which is ready now. You can check on your Excite email and even tune in an Internet radio station.
Snap is now using GlobalBrain technology to refine its results and make the most popular sites appear first. As with Direct Hit, the technology monitors what sites people are clicking on and moves the top choices higher.
"A clear example that the technology is working is that the number one answer when you search for WWF is Sable," said Sam Parker, Snap's Vice President of Product Development. Sable is an extremely popular WWF player. "The users have spoken, and that's what they are looking for, so that site rises to the top," he said.
Similarly, while Snap thought the official Pokemon site would be a top choice among users, the popularity ranking shows that personal fan sites are the draw. "Pokemon.com keeps getting pushed down. That was quite a surprise," said Snap producer Paul Wood.
Within categories, sites are still arranged alphabetically. But Snap expects that this may change in the future. Sites would be listed by popularity, or an option to order them this way might be offered.
GlobalBrain also has the ability to produce personalized results, so that results are formatted based on a person's age, sex or geographical location, among other factors. There are no immediate plans to introduce this type of refinement.
Snap has an exclusive, multiyear license to GlobalBrain's technology and has gained a equity stake in the company, as part of the deal.
GlobalBrain To Offer Profile Searching
The Search Engine Report, Nov. 1998
More information about GlobalBrain.
Direct Hit Debuts Personalized Search
The Search Engine Report, Feb. 1999
Direct Hit has been doing popularity results for over a year. The company also has a personalized results system waiting in the wings.
Lycos and IntelliSeek, maker of the BullsEye desktop search utility, have teamed up to produce an index of search databases to help users find information that is invisible to search engines. The "Invisible Web Catalog" provides links to more than 7,000 specialty search resources. Users can browse listings, or Lycos will suggest appropriate databases within its own search results.
This is a great new tool because there's lots of helpful information locked away in databases that can never be indexed by search engines. No, Lycos isn't automatically searching these databases when you perform a search, which some people have mistakenly assumed. That's no simple task, though IntelliSeek says it is something that could possible happen in the future. Instead, the goal is to make these search resources more visible as alternative sources of information.
For instance, say you searched for "cancer." You'll notice in the search results that there's a link to "Reference > Searchable Databases > Health > Diseases > Cancer." If you click through, you'll discover some important cancer-related sites listed, with links that lead straight to their search pages. You can then select a resource and try a specific search there.
So to get the most out of the Invisible Web catalog, change your search strategy at Lycos. If you see a searchable database link in the results, consider clicking through to explore the resources there.
You can also browse the Invisible Web Catalog's listings by going to its home page. Follow the link below or just search for "invisible web" to find it. Once there, you can also choose to search just within the catalog for databases of interest.
Lycos Invisible Web Catalog
This takes you straight to the catalog. Those using non-US versions of Lycos should also go here to access the catalog, as listings are not being integrated into your search results.
My co-panelist Nora Paul pointed this resource out at a recent search presentation we gave, and a great find it is. Like the Invisible Web Catalog, Direct Search takes you to all types of helpful specialty databases that may have the information you are looking for.
Another comprehensive guide to searchable databases. Browse or search through listings.
The Invisible Web
About Web Search Guide, June 11, 1999
A longer introduction to searching where search engines can't go, featuring some of the resources above and some additional ones.
Search Engine Briefs
Direct Hit Deals Soon
Expect to see Direct Hit appear on two new search services later this week. It will become the default data source at one service, as it is at HotBot, and will appear in the more common "Top 10" button model at the other. An appearance on a European search service is also imminent. I'll post more details on the Search Engine News page when they become public. Meanwhile, Direct Hit is also completely powering a new search service for Zap.com. That means that Direct Hit is both spidering the web and refining results, rather than working in conjunction with other people's databases, which has been its usual model.
Search Engine News
Fee-Free Searching At Gov.Search For Schools, Libraries
Northern Light's new Gov.Search government search engine shifted to a pay basis on June 14. In changes since my last report, Northern Light has lowered the daily search pass to US $5. Northern Light is also offering free search accounts to US public libraries and schools. To apply, just send a short message to Northern Light customer service at [email protected], saying you'd like to be part of the free pass program. Everyone that emails will get more details in the near-future on how to apply when a formal system is in place. Finally, the NTIS is still involved with the site as a joint venture, despite any reports you may have heard about a pullout, Northern Light says.
US Government Search Engine Launched
The Search Engine Report, June 1999
More about the service and other government search options, such as via Google.
Does Yahoo Own You?
GeoCities members got a rude surprise when parent-company Yahoo revised the terms of using the free web page service. Under the revisions, Yahoo was granted the right to use a member's content without having to get permission or pay royalties. Yahoo has since revised its terms, but not all critics are happy. The stories below will bring you up to speed on the dispute:
Warner Bros. targets Yahoo's backlash
News.com, July 1, 1999
Yahoo relents on GeoCities terms
News.com, June 30, 1999
Yahoo's Lawyers Take Over the Asylum
WebDeveloper.com, June 29, 1999
Yahoo Angers Homesteaders
InternetNews.com, June 29, 1999
A protest site that emerged over the dispute.
New Position Checker Extends Coverage
TopDog is a relatively new position checker that's just come out with a new release. I haven't tested it yet, but a big plus to the software is that it apparently covers about 150 search services, including many international ones. It also provides submission support to about 80 of them. The software is for Windows platforms and costs US $149.
RealNames Coming To MSN
MSN will be adding RealNames information to its search results, and they'll also be incorporated into the AutoSearch feature of Internet Explorer 5. Expect the changes to happen in the fall of this year.
New Search Engine Performance Study Out
A new study of how search engines cover the web will be released tomorrow in Nature magazine. It was conducted by researchers at the NEC Research Institute, who did the landmark study of coverage that appeared in Science magazine last year. I can't provide specific findings until the public release tomorrow, but there's lots of interesting data. Look for details at the URL below, or watch the Search Engine News page.
Accessibility and Distribution of Information on the Web
Search Engine News
Search Engine Articles
Q&A: Ask Jeeves CEO Robert Wrubel
ZD Interactive Investor, July 2, 1999
Ask Jeeves went public, then its stock went on a nearly-record setting climb. Ask Jeeves CEO comments on the rise.
Ask Jeeves quadruples in debut
Redherring, July 2, 1999
More about the business side of Ask Jeeves.
AOL beats Netizen in domain spat
News.com, June 25, 1999
With all the effort AOL went through to gain control of aolsearch.com, it seems likely that's the address the new search service will use.
Below are sponsor messages that ran in this month's issue of the Search Engine Report, which may be of interest to Search Engine Update readers.
Submitting alone is not enough to bring new traffic to your site. People must be able to find your site in the top 10 to 30 matches. Check out the award-winning new product WebPosition GOLD which optimizes pages to rank in the top 10, submits, reports your positions, tracks your traffic, and much more. ZD Net 5 Star rating!
Get your FREE download at:
EASIEST SOLUTION FOR INTRANET DOCUMENT PUBLISHING
Build a maintenance free intranet today! Your intranet documents
will look just like your desktop documents. Try the product
rated 9.85 by NetworkWorld & InfoWorlds HOTTEST (98 & 99).
Try FREE today & enter to WIN a Digital Camera!
HOW-TO: ADD COMMUNITY TO YOUR SITE (FREE TRIAL)
Gain the life and fire only *people* can bring.
Add chat, bulletin boards, free homepages, instant "telegrams,"
and "who's online" to your site in minutes!
No downloads, plug-ins, cost, or obligation!
+++++++ TRY IT NOW!
INTRODUCING INTERNET STOCK REPORT'S HOTWATCH
HotWatch is an email newsletter that features the top 10 noteworthy Internet stocks as chosen by Internet Stock Report's Steve Harmon. For $99 per year, subscribers will receive 12 timely issues bringing you in-depth analysis each month on the top ten Internet stocks.
Subscribe Today at
WebReference.com - The Webmaster's Reference Library
FREE NEWSLETTER and more info at <http://www.webreference.com/>.
Your one-stop Web dev resource, 8-time Winner PC Mag's Top 100.
How do I unsubscribe?
+ Follow the instructions at the very end of this email.
How do I subscribe?
+ The Search Engine Update is only available to paid subscribers of the Search Engine Watch web site. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to [email protected]
I need human help with my subscription!
+ Send a message to [email protected]. DO NOT send messages regarding list management or site subscription issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at
This newsletter is Copyright (c) Internet.com LLC, 1999
Meet Your Favorite Search Engine Watch Contributors
Many of SEW's leading expert contributors will be at ClickZ Live, the new online and digital marketing event kicking off in New York (March 31-April 3). Hear from the likes of: Thom Craver, Josh Braaten, Lisa Barone, Simon Heseltine, Josh McCoy, Lisa Raehsler, Greg Jarboe, Dan Cristo, Joseph Kerschbaum, John Gagnon, Eric Enge and more!