THE SEARCH ENGINE UPDATE
July 2, 1997 - Number 8
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to "Search Engine Watch."
Please note that long URLs may break into two lines in some mail readers. Please cut and paste, should this occur.
Site Changes - Search Engine Watch
The usual number of changes has been made to pages throughout the site, such as to add new providers of search engine software and to update the Search Engine Features chart. In addition, several new pages have been added since last month.
Tops In PC Meter - But What Does That Mean?
PC Meter used to release some basic results publicly. It no longer does that, which leave the numbers open to confusion and easier for search engine companies (and no doubt others) to provide their own spin. This page takes a look at some of the problems.
What is MCF?
Netscape is supporting a new type of meta tag system. Don't panic. It is almost certainly not going to be adopted by the search engines. This page tells you more.
How Big Are The Search Engines?
A look at why pages may not make it into the different search engines.
The Alta Vista Size Controversy
Related to the How Big page, it details discussion about how Alta Vista, like some other search engines, only samples pages from web sites.
LiveTopics: Love It or Hate It?
A look at some comments gathered from different mailing lists about Alta Vista's LiveTopics, which seems to draw people into one of two camps.
Search Engine EKG
Now also features two sites, rather than one. It will be updated with June numbers shortly, and a third site may be added this month.
Don't forget to keep an eye on the site's What's New page, where I will post headlines of important stories.
Site Changes - Subscriber-Only Area
The More About Meta Tags page has been expanded, and a new Meta Robots page is in the In Progress section. You'll also find the Keyword Analysis page there, too.
I've updated the Lycos and WebCrawler pages and will be working in July to bring the remaining individual search engine pages up to the new site format.
The downloadable version of the guide will be posted within a few days, if not sooner
Also, ordinarily I remove all the sponsor messages from the Search Engine Report, when creating the beginning of the month Search Engine Update. However, I realized that some of you might be interested in the messages. So, I'm now going to include them at the end of the Update.
Search Engine News
Lycos Launches Lycos Pro
Lycos launched a significant upgrade to its service called Lycos Pro on June 23. Lycos Pro features enhanced power searching commands and an interesting Java-based "Power Panel." But the changes don't stop there. Under the hood, the entire search engine has been rebuilt.
"Lycos Pro represents a total revamping of our search engine structure," said David Burns, Director of Product Marketing. A completely new search algorithm is being used to find and rank web pages, and the spidering system has been improved, Burns explained.
While relevancy depends mostly on personal opinion, I found the new results provided by Lycos Pro to be far superior to the regular Lycos service and think others will agree.
Previously, Lycos refreshed its catalog monthly. Now, new and updated information is being added on a weekly basis, which covers about half the index. So no web page should ever be more than one to two weeks out of date, according to Burns.
Lycos has also enhanced its crawling abilities. The service is capable of retrieving 10 million web pages per day, which sets it up as a rival to leader HotBot, which is at that level. However, Lycos will more likely to run in the 6 million web pages per day range, Burns said. It will be ramped up to a higher crawl rate if needs demand.
Those submitting pages to Lycos should expect to begin to see them appearing within a week, and a real-time submission service may eventually develop. Submission covers both Lycos and Lycos Pro. Also, Lycos is now doing full-text indexing, rather than forming abstracts of web pages, as was the case in the past.
Catalog size remains around 30 million web pages, or up to 50 million URLs when audio, image and other media files are included. The 100 million page figure cited on the site is in reference to the pages the crawler knows about, not the actual number in the catalog.
Lycos, like several other search engines, remains selective. It does a more thorough crawl of popular sites as determined in part by link popularity. This is part of a trend toward recording "the best" of the web, rather than trying to index everything.
"The old Lycos was when bigger was better," Burns said. "Now everyone is saying, "Give me better.'" He added that the index size may be increased should Lycos feel it's necessary to meet user demand.
The Lycos Power Panel is unique in providing an unprecedented degree of control to users who want to regulate how the search engine ranks web pages. Some other search engines allow the weight of particular words to be increased. Lycos goes far beyond that. By sliding controls on the Power Panel, users can set the emphasis of how pages are ranked by Lycos in six major areas:
Match every word
Frequency of words
Appear in title
Appear early in text
Appear close together
Appear in exact order
Webmasters looking for a web page might set the title control to 100% and search for their page titles. Meanwhile, search engine users might decide to try the opposite and search for pages without keywords in the title. That might reveal perfectly relevant documents but ones that may be overlooked because of the emphasis Lycos (like other search engines) places on page titles by default.
Lycos isn't certain exactly how visitors may use the panel. It expects a lot of experimenting and is already receiving interesting examples, which it may eventually publish in its help pages. An HTML version of the panel is in the works, and an Active X version is being considered.
An expanded list of Boolean and other operators is also available. Up to 50 results can be displayed at a time, and these can be titles only, standard descriptions or a detailed description that include information such as when a page was added to the index.
At the moment, Lycos and Lycos Pro are using different catalogs, so there are some differences between them. Lycos Pro started out with a brand new catalog, while Lycos is using the old one. New finds are being added to both catalogs.
Over the next three months, and more likely much sooner, Lycos will begin using the Lycos Pro catalog and the Lycos Pro searching algorithms. While Lycos will tap into the new Lycos Pro infrastructure underneath, on the surface it will be maintained as a simple interface to the search engine, so as not to overwhelm existing users with power features.
"We'll keep Lycos Pro for those who want to keep their hands on the controls," Burns said.
If you haven't been by Lycos recently, its worth a visit and a reevaluation. If you are a regular Lycos user, I would recommend going straight to Lycos Pro. Don't be frightened by the name. There's no need to use the optional power panel. Using the main interface has a few more options that regular Lycos, but it remains very much an enter-your-keywords-and-push submit system.
Lycos Pro, with Java Power Panel
WebCrawler Gets Face Lift
(Repeat from last issue, but with new information)
WebCrawler got a major facelift on June 16, along with some enhanced searching functionality. The service now features WebCrawler Shortcuts, which suggest alternative links to classified ads, travel guides, weather and other material related to the search topic.
WebCrawler also continues to offer some very nice features, such as the SearchTicker and the WebCrawler 100. Just click on "Fun" from the home page, and you'll find these. The WebCrawler 100, which lists the most linked to sites in the WebCrawler database, has finally been updated. It was languishing with February 97 results for months, but now it has been updated twice in June.
The upgrade is part of Excite's commitment to maintain WebCrawler as a distinct search service. Excite acquired WebCrawler from America Online in November 1996.
"This is a first step for WebCrawler, its first major release since its been under Excite,' said Executive Vice President Brett Bullington, who oversees strategic business development at Excite. "You're going to see the WebCrawler team migrate and build the product in a different way than Excite."
Keeping WebCrawler a distinct service makes sense in part because of a built-in base of dedicated users, many of whom first explored the net via AOL and thus were directed to WebCrawler as a guide to the web.
"We think there's definitely an attachment to that brand," Bullington said. We would be foolish to mess around with it"
In contrast, the Magellan service, which was also acquired, is run from under the Excite team. While being maintained as its own brand, it is not planned to have the enhancements that Excite and WebCrawler have seen.
Bullington said the service draws a different user than the Excite service, which offers many options beyond searching, in particular since the debut of its channel format earlier this year. In contrast, WebCrawler centers around its search box.
"Excite gives them the noise up front, where WebCrawler tries to filter the noise," Bullington said. "People like the simpleness of it."
Webmasters will find that WebCrawler has revamped its submission information. Take the time to review its Adding URLs help page. It details problems relating to frames and dynamically-generated URLs. It also clarifies that simply submitting a particular page does not mean that an entire site will be crawled and added to the WebCrawler index.
WebCrawler adds pages to its index in two ways. It crawls URLs that it knows about or discovers, paying particular attention to do deep crawls of sites that are popular in terms of links to them. Links are one measure of quality, so this is a way for WebCrawler to concentrate on what seems to be the best of the web.
WebCrawler will also add any PAGE directly submitted to it, assuming the page does not contain spam. It will immediately spider the page, to ensure it exists. Then it will come back to the site within a few days to retrieve the page, and the page should appear in the index within a week or two after that.
Note the emphasis on page. WebCrawler will not crawl an entire site, as Excite or HotBot might, just by submitting one URL. Therefore, in order to ensure a site is properly represented in the index, do exactly what WebCrawler recommends and submit KEY pages using its Add URL form.
Again, note the emphasis on KEY. WebCrawler does not want hundreds of pages submitted from the same site. WebCrawler is the smallest of the search engines and happy that way, though it may increase its index from 2 million to 4 million pages in the near future.
"We feel that we're able to present our users with a broader and less repetitive range of results by keeping our index smaller and more selective," WebCrawler spokesperson Melissa Walia said.
Should WebCrawler receive mass submissions, it may not add the pages and will probably review the site. How much is too much? There are no specifics, but use common sense, pick a good representation of pages from your site, and you will avoid trouble.
It's a bit harder to find the Add URL form than in the past. Use the "Getting Listed" link at the bottom of the home page, and it will take you through the Add URL help page before you arrive at the submission form. Finally, WebCrawler is now also supporting both meta keyword and meta description tags.
WebCrawler Add URL Help Page
WebCrawler Add URL Page
WebCrawler Fun Page
You'll find the WebCrawler 100 and the SearchTicker from this page.
Search engine to share revenue with content firms
Ad Age, June 16, 1997
Provides details about WebCrawler's revenue sharing model with partners that provide information for its Shortcuts service.
Alta Vista Goes Multilingual
Alta Vista now offers the ability to search for web pages written in one of 25 different languages. Simply enter your query, select the language of your choice, and pages in that language containing your search term will appear.
This is much different than using a country-specific version of a search engine, where results are usually taken from the parent search engine but filtered by domain. For example, a UK-regional search engine might find matches from UK domains, such as co.uk.
A language-specific search on Alta Vista searches through pages the search engine had determined match a particular language. Domain filtering is not involved.
"For example, more than half of all Web pages written in French are located outside of the .fr (France) domain," said Louis Monier, chief technical officer for AltaVista. "Conversely, over 30 percent of the pages in the .fr domain are not written in the French language. With the new multilingual search capability of AltaVista Search, users have a powerful advantage to be able to quickly and accurately find pages in their chosen language anywhere on the Web."
Languages included are Chinese, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hebrew, Hungarian, Icelandic, Italian, Japanese, Korean, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Spanish and Swedish. More will be added in the near future.
Alta Vista has also launched additional mirror sites, one in Southern Europe and another in Asia.
Alta Vista Southern Europe
Alta Vista Asia
LookSmart Looks Smart
(Repeat from last issue, but with new information)
LookSmart has undergone a major redesign this month, dropping the last vestiges of its original Java-based browsing system. It now has a very clean, bright look. LookSmart moved away from the Java interface in order to speed up load times and improve stability for users.
"Our HTML version mimics the core functionality of the Java version, but is quicker to load, more stable, and easier to maintain," said Marketing Manager Elizabeth Connaghan.
Users can continue to browse or search the guide. Browsing, by default, taps into LookSmart reviews. Searching, by default, taps into the Alta Vista database of sites.
Despite the default search using Alta Vista, LookSmart remains committed to reviewing the web. It has added two new categories (People & Chat, Hobbies & Interest) and currently has 200,000 reviews, Connaghan said.
"We hold the view that our review style is the best written with the best focus of any directory and that consumers find them valuable," Connaghan said.
Sites can continue to submit themselves for review using the site submission form, found via the Feedback link on the home page. Sites already listed in Alta Vista will be available for those performing Alta Vista searches via LookSmart.
LookSmart Site Submission Form
Search Engine Challenge Details Online
A round-by-round account of the 2nd PC Computing Search Engine challenge is now online. In case you missed the item in last month's Search Engine Report, PC Computing put several search engines to the test at NetWorld+Interop 97 in May. HotBot won with 13 points. Excite came in a close second with 12 points, followed by Alta Vista with 6 points and Infoseek with 4 points. A variety of questions were asked over two days.
The 1st PC Computing Search Engine Challenge planned for October 1996 never occurred, due to Internet problems. Instead, it devolved into a laser tag war between Excite and Infoseek. A quite amusing round-by-round account of that battle, which Excite won, can be found below.
2nd PC Computing Search Engine Challenge
May 9. 1997, Las Vegas, Nevada
1st PC Computing Search Engine Challenge
October 11, 1996, Mountain View, California
WebPosition Monitors Rankings From Desktop
The first software product to check page rankings has been released. WebPosition runs from the desktop. You can enter different URL/Keyword combinations, then send it on "missions" to report back on page position from a variety of major search engines.
A 45-day trial version is available for evaluation and will check results from three search engines. The commercial versions support 10 search engines.
Yahoo-Visa Shopping Site Nixed
News.Com reports that a plan for Yahoo and Visa to build an online shopping site called Marketplace is now dead.
Yahoo-Visa deal on shaky ground
News.Com, June 20, 1997
Excite and Ticketmaster Partner
Excite and Ticketmaster have teamed up, allowing users to find event listings and buy tickets from Excite's services: Excite, WebCrawler, Magellan and Citi.Net. Listings began on June 30.
Alta Vista To Stay Part Of Digital
Digital has decided not to spin off its Alta Vista Internet Software division as a separate company. Digital said the move was to help it use the Alta Vista brand to promote its other Internet products.
Digital likes AltaVista so much it can't let go
InfoWorld Electric, Jun 24, 1997
Digital Nixes AltaVista IPO
TechWeb, June 25, 1997
Digital Press Release
Digital, June 24, 1997
FlashStats Analyzes Keywords
FlashStats 1.2 has been released. It is a server-based log analysis program meant to provide a quick look at key web site statistics. Its "Search Phrases" report breaks out the keywords used to reach a site from 15 search engines, as well as provides general linking and site activity reports. It's available for a 30-day trial period via the link below.
FlashStats Download Page
Infoseek Republicizes Instant Add URL feature
Infoseek has reannounced its instant Add URL feature. Any page submitted to the index appears within minutes, in most cases. The instant service has been running since February, though the submission form now also sports a Remove URL button. You can enter a dead URL into the box, push the button, and it will be removed from the index.
The Add URL feature is both a joy and a pain. It's wonderful to know that any page can be added within seconds. There's no worries about whether the spider will come. You get instant gratification. You can also discover if there is anything wrong with your page that won't allow it to be accepted, such as an unacceptable URL.
However, the feature it also makes it easy for people to submit page-after-page in an attempt get higher in the rankings. No single page can be submitted more than once in a 24-hour period, but there's no limit on how many different pages from a site can be submitted in one day.
As a result, spammer try to flood the index with alternative versions. The weekend after the reannouncement, I watched results from one high-profile search term change literally every hour. Some sites were clearly submitting pages, then changing their code in an attempt to "hide" the reason behind their success. The next day, others would come along and resubmit those sites -- causing them to drop because the code no longer existed on the page.
If the feature is abused in this way, it is quite possible that it will become restricted. Alta Vista had the same instant indexing feature for a while. But since the beginning of the year, it has instituted a day delay before pages appear and a limit on how many pages per site can be submitted. The number varies, but generally try not to exceed five in a day. The measures were specifically created to block spamming attempts.
Crawling For Playboy Pirates
Playboy is going to begin digitally watermarking its images through technology from Digimarc. Part of Digimarc's service is it's MarcSpider, which searches the publicly-accessible portions of the web to find any violators using the coded images.
Playboy to tag and track images
News.Com, June 30, 1997
WiseWire Upgrades Agent Search Service
WiseWire.com has enhanced its free, agent-based search service and dropped the user registration process to make it easier for people to try it. Headline news, newsgroup access and moderate chat are some of the new additions.
WiseWire subscribers select favorite topics, then read and rate documents for relevancy. Based on what the user likes, WiseWire builds an individual profile that is supposed to grow more accurate with each use. WiseWire also compares the ratings of all members, allowing it to recommend documents rated highly by one member to other members with similar interests.
Which Way For Infoseek?
(Repeat from last issue, but with new information)
Media Central has an interesting article that sheds light on the departure of top executives from Infoseek, in recent weeks. According to sources, there has been a split between whether the company should pursue the corporate market or promote the licensing of its technology.
The tug-of-war described is something all the search engines face. Do they continue to invest in consumer-oriented search services, or do they instead focus on selling their search engine technology?
While Excite still provides its search engine software, the focus of the company is firmly aimed at the consumer. In contrast, Open Text continues to run its search service, but the focus there is clearly on the search engine software that it sells to corporations.
Others remain in a balancing act. Alta Vista has been moving forward strongly in developing its technology for personal and corporate use. However, it has made notable enhancements to its World Wide Web search service. Likewise, Lycos is partnering with another company to spin off its technology, yet it has also recently upgraded its search service.
A few months ago, Infoseek had said it was going to focus on the corporate consumer and formed a division devoted to last April. Media Central reports that this vision of former CEO Robin Johnson clashed with that of founder Steve Kirsch, who sees software sales as crucial to the company's future.
Johnson resigned in May. On June 9, it announced that chief operating and financial officer Leonard J. LeBlanc would leave as of July 7. Also leaving on that date is Peter D. Rip, vice president and general manager of the Infoseek Network. He oversaw Infoseek's advertising operations and advertising marketing.
Meanwhile, new promotions were announced April 24: Dr. William I. Chang to chief technology officer and Arthur H. Clark as vice president of Business Development.
Infoseek Exodus Doesn't Slow Corporate Info Division
Media Central, June 24, 1997
Summarizes the struggle between promoting consumer products vs. software development, leading to the resignation of top executives.
Infoseek Exec Exodus Continues
TechWeb, June 9, 1997
Infoseek CEO Abruptly Resigns Position
Web Week, May 19, 1997
Background on the departure of CEO Robin Johnson, with observations on the mixed directions Infoseek seemed to be heading.
StudyWEB Categorizes Academic Sites
Looking for an academic or research site? StudyWEB has categorized and reviewed about 15,000 sites to date.
Search Engine Articles
Lost in cyberspace
New Scientist, June 28, 1997
An excellent look at why some search engines are moving away from an "index everything" attitude and instead adopting an "index the best" or "sample the web" method. Does it make a difference to searchers if some pages aren't included? Search engine execs explain why they believe a sample is good enough.
The Great Search Engine Turf War
Go2Net, June 30, 1997
A nice look at search engine battle for users since the early days, with the emphasis on some recent developments. It's well worth a read, though it makes an odd mention of Inktomi having died a "slow death." Inktomi gave birth to HotBot, so it makes perfect sense that the original experiment has not continued.
How To Get To The Top Of The List
Internet Magazine, July 97
If you're looking for a nice, concise summary of things I cover in Search Engine Watch, this is the article for you. I wrote it for Internet Magazine, here in the UK. It's not available online, but newsagents everywhere carry it here in the UK, and places like Barnes and Nobles carry it in the US.
Search Engines - The Future
.net, July 97
A nice look at the way some search engine are trying to be more useful to users. It's got a UK emphasis, but it's interesting to anyone, all the same. As above, it's not available online. In the UK, see newsagents. In the US, high-end bookstores.
Search Engine Notes
Still Trouble Adding To Lycos From Outside The US
Lycos is still having problems related to adding URLs from outside the US. Below is the situation from the UK, and it may be applicable to other countries where domain shifting is occurring.
The Add URL form allows you to check to see if a URL is in the index before adding it. If you do this -- and if the page is not already in Lycos -- a new page appears that lets you automatically add the URL.
However, trying to use this form will return a "malformed url: scheme missing" error message. The page will not be added.
To get around this problem, go back to the main Add URL form and submit your page from there -- not from the one that appears after you check on a URL. You should have no problems submitting.
Waiting For Excite
The new Excite Add URL feature has yet to launch. It was supposed to be up by mid-to-late June, but it's obviously taking longer than expected. Excite is continuing to crawl the web, however, and it does appear that new sites are still being added.
When completed, the new feature will add each page submitted to the index within seven days, as opposed to the usual three-week wait. In addition, the new feature will notify webmasters when the page is actually added to the index. Should the spider fail to get the page after three attempts, the webmaster will also be notified and asked to resubmit.
Below are sponsor messages that ran in this month's issue of the Search Engine Report, which may be of interest to Search Engine Update readers.
Eric Ward's URLwire(tm)
Web site news & events in over 100 subject areas, matched by topic and delivered via Email to thousands of Internet journalists worldwide.
NEW!!--POWER SEARCHING WITH ALTAVISTA--FREE ISSUE!
If you're a serious AltaVista Search user, I'm inviting you to accept a FREE ISSUE of POWER SEARCHING WITH ALTAVISTA today.
Every month, POWER SEARCHING WITH ALTAVISTA brings you 16-pages of information about one of the Internet's most powerful search engines.
POWER SEARCHING WITH ALTAVISTA supplies tips and techniques to help you pinpoint the data you need with each search. Each monthly issue shows you how to save and reuse complex searches; use the latest features of the AltaVista Search; use Live Topics; use Private eXtensions to access data; find information without paying online database fees.
Plus, the newsletter features a regular column by Richard Seltzer, co-author of "The AltaVista Search Revolution."
POWER SEARCHING WITH ALTAVISTA is available in electronic format (Web site and E-mail alert) only, or you can get both the printed newsletter and electronic edition together.
For your FREE Issue, fill out the form at:
I-Advertising Digest is a free email discussion digest focusing on Internet Advertising and Marketing. The list is moderated, which means that you won't be inundated with messages. In fact, you'll get one digest on Mondays, Wednesdays and Fridays and that's it (3 messages).
Come join our online community and learn how to increase traffic to your site or promote your business on the Internet!
In the BODY of your message, put: JOIN
Archives and an online sign-up form can be found at:
To unsubscribe from this list, send a message to [email protected], with the following in the subject field:
If you have problems, just send a message to [email protected]
This newsletter is Copyright (c) Danny Sullivan, Calafia Consulting. It should not be distributed. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at:
Last Week to Save on SES London Tickets!
Learn to engage customers and increase ROI by distributing your online marketing efforts across paid, owned & earned media. Join the leaders of today's digital marketing & advertising industry at SES London. Find out more ››
*Saver Rates expire this Friday, Dec 13.