THE SEARCH ENGINE REPORT
4/1/97 - Number 5
Note: This was originally in two parts. Both part are on this page.
================ About The Report ================
The Search Engine Report is the email companion to "A Webmaster's Guide to Search Engines," http://calafia.com/webmasters/. It keeps you informed of changes to the site and general search engine news useful to web developers. Feel free to pass on this newsletter to others.
================= Introductory Note =================
Because there was a lot of search engine news in March, this month's report has been divided into two parts.
At long last, the Search Engine Report is now delivered via an automated mailing program. Thanks to Michael Tratz of Esosoft for his help in getting Majordomo 1.94.1 running to serve this list. Esosoft does mailing lists, low-cost web sites and Virtual Servers. You can visit them at http://www.esosoft.com/ or write to [email protected].
Remember, you don't have to wait until the end of the month to get your search engine news. Subscribers to the site receive "The Search Engine Update" twice per month, along with an offline version of "A Webmaster's Guide to Search Engines" and access to "in progress" additions to the guide. Each month, one new subscriber will also win a free ad in the guide. Subscription details can be found at:
================= Sponsor Message =================
Are Your Search Engine And Directory Listings Paying Off?
Chances are, the majority of your Website traffic comes from the popular directories and search engines. Try NetGambit's PositionAgent now and find out where your selected URL's (using selected keywords)are ranked in the leading search engines and directories. Use our FREE TRIAL version at http://www.positionagent.com.
Subscribe to the full version and PositionAgent will automatically monitor all results for up to 5 selected URL's and keywords report to you weekly. Then use our search engine listing resources, and you'll be getting more hits in no time!
============ Site Changes ============
The Search Engine Features chart has been completely updated. There are new categories and updated information about how search engines work. I highly recommend revisiting this page to see what's new with the search engines.
Beyond the features page, the "How It Works" pages for each search engine have been updated. These can be found from links at:
Other minor changes have been made throughout the site.
================== Search Engine News ==================
New Netscape Net Search Line-Up
Netscape has renewed position contracts with several major search engines for its immensely popular Net Search page. "Premier Providers" will be Excite, Infoseek, Lycos and Yahoo They'll each have a tab at the top of the page, as under the current system.
The fifth tab, now occupied by WebCrawler, will be devoted to other engines called "Marquee Providers." WebCrawler now joins this group.
The change will take effect May 1st. Last time, the engines paid $5 million each for a year's placement. This time, the engines are paying based on the amount of traffic they receive from the page. Netscape is saying the cost should be about the same.
Currently, the top engines are rotated as default choices. Under the new system, users will be able to pick their favorite engine and have it always be selected. They can also pick which of the marquee providers they wish to appear in the fifth spot.
The ability to set a default search engine is significant. If people begin using a preferred engine, traffic to the other engines will drop -- possibly by a noticeable amount.
Netscape Press Release http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/oracle/dbtmp/77729&EDATE=
Netscape, Search Engines Team Up, CMP TechWeb http://www.techweb.com/investor/newsroom/tinews/mar/0318search.html
Netscape Net Search page
Instant Listings On Infoseek
Infoseek has made its page listing service even faster. An online form is now available, and pages submitted via the form are added to the index within minutes. You can even find the exact time the page was added, up to three days after submission, using the Check URL form.
Many pages can be added at once via the old email option, though it may take up to a week to process them this way.
This new system makes Infoseek the fastest of the major search engines to index submitted pages. Previously, it was tied with Alta Vista in adding pages within a day or two of submission.
Infoseek's Add URL page notes that a page may not be added to the index if it is submitted more than once in a 24 hour period. No doubt this is an anti-spamming measure, designed to hinder spammers who would otherwise change a page, resubmit, then keep changing until the page made the top ten.
Infoseek does not seem to penalize for submitting many pages within a short period of time. Alta Vista does this as an anti-spamming measure.
ADD URL page http://www.infoseek.com/AddUrl?pg=DCaddurl.html
Check URL Status page http://www.infoseek.com/AddUrl?pg=CheckURLStatus.html
HotBot Launches SmartCrawl
HotBot quietly launched its SmartCrawl system on 3/15, which will eventually index up to 10 million web pages per day. Its major competitors are in the 3 million web pages per day range.
The new system should greatly improve HotBot's freshness. Its crawling had been interrupted while SmartCrawl was brought online.
Submitted pages are to be added within three weeks, along with other pages found when visiting a web site.
NOTE: Above item was corrected from the original mailing, where I swapped "week" for "day."
AOL Launches AOL NetFind
On March 13, AOL unveiled its AOL NetFind service at Internet World in Los Angeles. In the past, AOL has directed its members to a search page, where they could choose between using Excite or WebCrawler. Now they are directed to the AOL NetFind page.
Under the hood, AOL NetFind is Excite with an AOL look. Excite is crawling the web as normal, and AOL NetFind taps into the Excite listings.
For example, a search for "bill clinton" will bring up the same number of matches and the same ranking order whether Excite or AOL NetFind is used. If there are any differences, it is because AOL's copy of the Excite listings may occasionally be a day or two behind the "main" Excite database.
So, don't worry about trying to submit to AOL NetFind. Your submission to Excite will cover you in both places.
Want to have some fun? Pop over to NetFind and do a search. Now in the address box, look for this at the end:
Cut that part off and reload the page. Surprise! Welcome to Excite, or at least the Excite look & feel. Don't stop there. Pop over to Excite and do a search. Now add those characters to the end of whatever is in the address box. Welcome back to AOL NetFind!
AOL NetFind http://www.aol.com/netfind/
Excite Switches To Channels
Excite announced in mid-March that it would regroup its content into "channels," which are subjects such as "news" and "sports." Search engine results will remain general in nature.
For example, a query sent to the Excite database from someone visiting the sports channel will still bring back results from across the web. There is no technology that will be used to identify or classify "sports" sites.
Of course, most people at the sports channel will probably send sports-related queries, such as "dodgers" or "basketball," so sports-related results will likely be found.
Expect to see the changes slowly occur over the next few months.
Excite changes channels, C|Net News.Com http://www.news.com/News/Item/0,4,8933,00.html
Lycos Launches Redesign
Yesterday, Lycos unveiled a new look, along with a new slogan, "Get Lycos or get lost," which is part of a rebranding campaign now underway.
Content has been reorganized into subject areas, such as "news" and "sports." Sound familiar? Well, it's not channels here. Lycos calls its groupings "WebGuides."
Search engine results are not filtered to match these areas. There are "power searches," but these are merely links that send keywords to the search engine. For example, the "car rental" power search simply sends those words to the Lycos search engine.
Under the hood, there's no sign that the crawling mechanism has been upgraded. That's supposed to occur sometime in April and will mean that submitted web pages will be indexed in near real-time. At the moment, Lycos still promises to add pages in 2 to 4 weeks -- though many webmasters have found it can take twice this time.
Lycos Press Release http://www.lycos.com/press/pressroom.html
Yahoo Takes Over Netscape Destinations Page
Somewhat related to Netscape's Net Search page is the Destinations page. This is reached via a button in Netscape Navigator, just like the Net Search page. A site listing here is a guarantee of increased traffic.
Yahoo is to take over production of this page and related pages such as What's New and What's Cool. It is paying Netscape $5 million to do so, plus Netscape will get a share of ad revenue generated by the site.
For web developers and users, the takeover probably means better information on those pages. For example, the Netscape What's Cool page was last updated February 20. The takeover is to occur by July 1.
Netscape Press Release http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/oracle/dbtmp/78735&EDATE=
Yahoo, Netscape strike deal, C|Net News.Com http://www.news.com/News/Item/0,4,8920,00.html
Netscape Destinations page http://www.netscape.com/escapes/index.html
UPS Partners With Search Engines Sending a package via UPS? The delivery company now has buttons on the home pages of Infoseek and Lycos that take you directly to a tracking query form. A similar set-up is planned for Yahoo, though it wasn't installed when I checked on 3/13. UPS also tried for placement with Excite, but the two companies didn't reach and agreement. This is apparently a move to raise UPS's presence on the web, where Federal Express is more widely known, according to Web Week.
UPS wraps deal with search firms, C|Net News.Com http://www.news.com/News/Item/0,4,8732,00.html
PR Newswire http://www.prnewswire.com/cgi- bin/stories.pl?ACCT=104&STORY=/oracle/dbtmp/75201&EDATE=
Rating the Raters
Who's Marketing Online put together an interesting look at review sites and how they rate each other. For example, NetGuide Live gives Magellan 5 stars (best), while Magellan rates NetGuide Live only 2 stars.
More important, there's a good look at the quality of reviews, exploring issues such freshness or inconsistent praise and criticism.
================= Sponsor Message =================
Need a home for your website? Do you have clients who want to get online?
Virtualscape's full-featured web hosting starts at $14.95 and includes a special reseller service for those people needing multiple websites. At Virtualscape, we go beyond static web pages by offering services such as Real Audio, Real Video, ListServs, On-Line forums, chat rooms, and FrontPage support. As a premier Cold Fusion web host, Virtualscape can even help non-programmers take advantage of database-driven, dynamic websites.
Do you have clients with their own website ideas? Join Virtualscape's reseller club and become an Internet Presence Provider for only $99.00 per month Virtualscape's reseller services make it easy to get multiple websites online without the hassle of maintaining your own server or paying for dedicated lines.
Alta Vista Mirror Semi-Perfect Reflection
A number of people have noticed that results from Alta Vista's European mirror don't exactly match those from the main US-based search engine. This is because the European mirror is slightly behind the US site. If the indexes are not exactly the same, then results will not be exactly the same.
Alta Vista says the European site should never be more than a week behind the US. My testing supports this. A page submitted to Alta Vista appears in the US edition in a day and in the European edition in a week.
There's no way to submit directly to the European site to bypass this delay. Submissions done via pages at the European site simply prompt the US-based spider to visit your site. Next time you submit a page, notice the URL that appears in the confirmation page. You'll see that it's from the altavista.digital.com domain. It'll look something like:
Meanwhile, Alta Vista has made some changes that make using the European edition easier. By entering a country code, you can do directly to language-specific version for your country. For example, those wishing the English version from Great Britain would enter:
A full list of country codes is linked to below.
Alta Vista US http://www.altavista.digital.com/
Alta Vista Europe http://www.altavista.telia.com/
Alta Vista Europe Country Codes http://altavista.telia.com/telia/tmpl/gb/news/countrycodes.html
Submissions Remain Off At Open Text
The Open Text URL submission page has been closed since late November while the index has been upgrading its crawling capabilities. It was promised to be reopened in mid-March, but then the date slipped back until late March. Watch the Submit URL page to see if the index will eventually reopen submissions.
Open Text Index http://index.opentext.net/
Open Text Submit URL page http://index.opentext.net/main/submitURL.html
Yahoo Categories To Get Own Sites?
Popular Yahoo categories could be broken off into their own sites in the future, Yahoo's President and CEO told C|Net in a story on earnings. The directory is also planning a launch later this year of a search engine geared toward product/merchant searches.
Chief Yahoo looks into future http://www.news.com/News/Item/0,4,8268,00.html
Alta Vista Vs. Alta Vista
Ever try to reach Alta Vista by accidentally typing http://www.altavista.com? If so, you ended up at the home page of Alta Vista Technology, a completely unrelated company.
Lots of people apparently make that mistake, so much so that Alta Vista Technology changed its home page to look very similar to the Alta Vista search engine page, complete with search box. The company sold ads to earn money off the confusion.
As you might expect, the Alta Vista search engine people weren't happy. They sued and now have won an injunction that requires Alta Vista Technologies to place a disclaimer on every web page, as well as a link to the Alta Vista search site.
AltaVista wins injunction, C|Net News.Com: http://www.news.com/News/Item/0,4,8900,00.html?latest
PC WEEK: Digital wins injunction in AltaVista legal snit http://www.pcweek.com/news/0317/18malta.html
Alta Vista Search Engine http://www.altavista.digital.com
Alta Vista Technologies http://www.altavista.com
G.O.D gets E-mail
G.O.D. (Global Online Directory) has gained email and telephone number search abilities through a partnership with Four11. G.O.D. is one of Europe's leading search directories.
More Infoseek News
Some more news from Infoseek, which had a busy month in March:
Infoseek will now be featured throughout CNN's various web sites as the preferred method of searching the Internet. In addition, news stories will have an option allowing users to search the Internet using Infoseek for sites related to the story.
Infoseek is also shipping its software for those who wish to index web sites or Intranets. The product is called Ultraseek Server.
Italian and French editions of Infoseek have been launched.
Infoseek What's New http://info.infoseek.com
Ultraseek Information http://software.infoseek.com
Excite Joins With Quote.com
Excite is now partnered with Quote.com, allowing visitors to the Excite web site to access stock quotes, stock headlines and stock articles.
Send in the search party http://go2.guardian.co.uk/theweb/859389571-alta.html
A journey behind the scenes at Alta Vista, from the United Kingdom's Guardian newspaper.
The Internet: Bringing Order from Chaos, Scientific American http://www.sciam.com/0397issue/0397intro.html
A series of articles on search engines, heavy on technical issues of improving searching and categorizing. There's a nice sidebar in advances in image searching. Find out how software could identify 43% of the nude photos in a sample collection.
April Fools Headlines
In honor of April Fool's Day, some fictional headlines I hope brighten your day. I expect a commission check if any of these stories come true :)
Alta Vista LoudTopics First it was LiveTopics, and now Alta Vista has rolled out LoudTopics. The new feature makes Alta Vista the first fully-audio search engine. Users pick a mode ranging from "Newbie-Helpful" to "Rude." Those selecting the latter are treated to the engine shouting "Get a dictionary!" when they misspell a keyword. Less-adventurous souls get prompts like "Don't worry. Lots of people get over 2,000,000 matches to a search. Shall we try again?"
Star Trek: Excite Excite announced a Star Trek-branded version of its search engine today. Star Trek: Excite has the look and feel of a Star Trek computer PADD. Fans have the choice of hearing "Working" or "Make It So" when clicking on the search button, and a result without any matches causes the engine to prompt, "Illogical."
Microsoft Search Microsoft entered the search engine market with Microsoft Search. The engine employs a unique ranking mechanism that gives pages a boost if they use Active X. The more Active X on a page, the better it performs. Unfortunately, a security flaw marred the debut. A 10-year old boy discovered a way to cause Microsoft Search to unlock the doors of a user's home anytime the Search button is pushed. Microsoft is rushing a patch.
Netscape JavaSearch Not to be outdone by Microsoft, Netscape is launching its own search engine. The search engine employs a unique "Netscape Now" crawling mechanism. If it detects a Netscape Now button on a web page, the page is included in the index. The engien will be reachable via the Net Search button, where it will now be listed as "Supreme Provider," over the Premier and Marquee Providers already on the page. Netscape has apparently paid itself $5 million for the placement.
=========================== New Search Engine Resources ===========================
Search Engine Mailing Lists There are two excellent mailing lists relating to search engines. The "search" mailing list is for those who want to discuss searching the web. The "robot" mailing list deals with technical issues relating to search engines and robots.
Search Engine Message Board The authors of "Search Engine Tutorial for Web Designers" have added a web-based message area for discussions about search engines and web promotional tactics. Aside from the board, the site covers search engine design issues. If you lke the Webmaster's Guide to Search Engines, then the Search Engine Tutorial is another book to add to your virtual bookshelf.
Search Engine Tutorial http://www.digital-cafe.com/˜webmaster/set01.html
Message Board http://www.digital-cafe.com/˜webmaster/set08.html
1,001 Best Internet Tips: Search Engines, PC Computing Pick up some tips on searching the web.
========= End Notes =========
To subscribe to this mailing list, send email to [email protected] with the following as your message:
Alternatively, you may use the form at http://calafia.com/webmasters/list.htm
To unsubscribe, send email to [email protected] with the following as your message:
The contents of this page can be found at http://calafia.com/webmasters/whatsnew.htm
This newsletter is Copyright (c) Danny Sullivan, Calafia Consulting.
This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) will bring together the industry's leading online marketing practitioners to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, the comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!