THE SEARCH ENGINE UPDATE
Sept. 3, 1997 - Number 12
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to "Search Engine Watch."
Please note that long URLs may break into two lines in some mail readers. Please cut and paste, should this occur.
Site Changes - Search Engine Watch
Some of the pages for the Yahoo Special Report are online, for your review. Keep in mind these are rough -- the final pages should be up on the regular site on Monday, Sept. 8, as mentioned below. Until then, you can preview them in the Projects in Progress section of the Subscriber-Only Area. If you preview them, please be sure to return and read the final pages.
The offline edition will be updated mid-month, around Sept. 15 to Sept. 20. Ordinarily, this is done at the beginning of the month. However, I will doing a series of major updates to pages within the site. It makes sense to revise the offline edition after these changes are in place.
I'll be doing a variety of updates throughout September, especially working to integrate material from some of the past reports into the site. As pages are posted or updated, they'll be noted on the What's New page.
Search Engine News
Yahoo Submission Survey Results Online
The results of the Yahoo submission survey are in, and the most important theme that has emerged is that many people seem frustrated or confused with the Yahoo submission process. "Merry-Go-Round" and "black hole" are among the terms used in comments.
There were 162 usable responses gathered from the survey, which ran during August on the site. This small sample means that the numerical statistics gathered are hardly conclusive, but many people will find them of some interest.
Just a quarter of those responding said that they had gained a listing in Yahoo. Also, it seems generally that Yahoo is accepting a smaller percentage of submissions than in the past, but processing those it does accept much faster than it used to.
Other findings were that having a root URL, such as http://www.mysite.com, seemed to slightly increase the chance of being listed over those using extensions of a hosting company's domain. Also, odds seem much more against those who submit sites using free web space.
Full numbers, comments and more can be found in a Yahoo Special Report area at the web site. I'm still finishing some of the pages, but expect them up by Monday, Sept. 8. Watch the URL below, and you'll know when the pages are ready. Specifics are below:
Yahoo Special Report
Encompasses a variety of documents, including:
Yahoo Submission Survey
Stats about what percentage gets in, how often people resubmit, and more, from the Aug. 97 survey.
Not Listed In Yahoo Comments
What those who didn't get listed in Yahoo have to say, from the Aug. 97 survey.
Listed In Yahoo Comments
What those who did get listed in Yahoo have to say, from the Aug. 97 survey.
Yahoo Submission Problems
A tour of the submission process, with a look at all the ways people might be confused, and clarification along the way.
Yahoo: Delays Expected
An essay on the problems site owners face when the web's most important guide can't keep up with submissions. But is it a concern for users? Not really, Yahoo says.
Yahoo Front Lines: A Surfer's Perspective
Quotes and some tips from one of Yahoo's editors.
Lycos and Barnes & Noble Partner
Last issue, I talked about the Lycos - Barnes & Noble Partnership. Below are some additional stories about the deal, since the last update:
Lycos, Barnes & Noble take on Amazon.com
Advertising Age, Aug. 25, 1997
Story has analysts' estimates that the deal is worth $15 million to Lycos over three years, plus commissions, and other details.
Barnes & Noble and Lycos Ink Deal
WebWeek, Aug. 25, 1997
HotBot Launches Instant Indexer, New Search Option
(Repeat from last issue, but with new information)
HotBot quietly became the third major search engine to offer instant indexing of web pages. The feature launched in mid-August. Any page submitted to HotBot via its Add URL page is now added within 48 hours. Non-submitted pages at the same site will be retrieved within 2 weeks.
"One of the key features of HotBot is that it is one of the freshest search engines on the web," said David Pritchard, HotBot's marketing director. HotBot refreshes its entire database of known web pages once every two weeks. "It was a natural step once we had the fast crawling in place," he said of the change to quickly add new pages.
The 48 hour delay is to deter spammers who might try to fine-tune their pages to dominate the top ten. AltaVista has a similar delay for its instant indexing service, and Infoseek requires that some using free web space submit URL via email, both as spam deterrents.
There is also a 50 page per day submission limit, which is more liberal than AltaVista and on par with Infoseek.
HotBot also added a new "Page Type" search option to its home page. The option can be used to help narrow down a search to particular pages.
There are three narrowing options: "Front Page," "Index Page", " Page Depth."
Front Page will find any web page at the top level of a web site, such as http://www.website.com. It will not find any sub-pages. This is an interesting way to narrow a search, because often times people don't properly optimize or tag inside pages. Also, design issues may keep search engines from getting inside a site. Limiting a search to just the front pages of a web site may help increase the chance of finding relevant sites that might otherwise be missed.
Index Page does what Front Page does, but it will also find the front page/index page of any subdirectories within a site. For example, it would find these type of URLs:
Page Depth controls how many levels into a site search should go. Set to 1, it would find URLs such as:
Set to 2, it would go further down:
Set to 3, that list would also include URLs such as:
Users can now also search only in page titles, using a drop-down box next to the search box.
Finally, HotBot is planning a major upgrade for launch in late September. Expect to see a new, customizable interface and an associated directory, among other changes.
Search Engine Survey Provides Wealth of User Data
NPD has released the results of a survey that provides a range of interesting statistics about search engine users. The survey was conducted during the summer on AltaVista, Excite, Infoseek, Lycos, WebCrawler and Yahoo. A random number of visitors to each site received an invitation to fill out a survey. There were 22,874 respondents. Some key findings:
Searching by keyword is far more popular than browsing listings. Only 6 to 20 percent said they used predefined topics, which all the search engines but AltaVista provide.
Multiple keywords are used more than single keywords, users say. Search engines generally report that those searching use less than two words, on average. But 50 to 77 percent of those surveyed said they use multiple keywords.
Users stick with the same search engine, if they don't find what they are looking for the first time. Between 66% and 80% said they'll try again using different words or methods, while only 16% to 25% said they would shift to a new search engine.
Surprisingly, there was a significant difference between those in the NPD survey saying they used a particular search engine "most often" versus the last reported audience shares for the search engine category from Media Metrix (formally PC Meter). Media Metrix is part of the same company that conducted the survey.
The May 97 Media Metrix results put Yahoo 1st (37.2%), Excite 2nd (19.9%), Infoseek 3rd (16.5%), Lycos 4th (14.0%) and WebCrawler 5th (12.9%).
In contrast, the NPD survey found Yahoo staying 1st (29%), while AltaVista shot up to 2nd (17%), WebCrawler rose to 3rd (14%), Infoseek dropped to 4th (11%), Excite dropped to 5th (9%) and Lycos dropped to 6th (7%).
The survey also asked which features users found important. For 95% or more, sites need to be easy to use, provide search results quickly, provide relevant results and load pages quickly. Just below this, 93% or more said information needed to be well organized and have up-to-date information.
In-depth information was rated slightly lower. Between 86% to 90% wanted comprehensive information, while 83% to 88% wanted detailed information.
Clearly, the emphasis is on speed and results. In contrast, many features that have been given greater importance recently by some search engines were ranked much lower:
Site is interactive: 56%-69%
Site is fun to use: 43%-68%
Site looks good: 44%-60%
Site provides personal news/information page: 26%-38%
Site has contests and surveys: 24%-34%
The survey also provides a demographic breakdown of respondents, listing information about income, age and browsers used, among other categories. As this is from a random sample taken from the search engines, this breakdown provides a good look at what a search engine user may be like.
NPD Search Engine Survey
NPD, Aug. 26, 1997
Yahoo Press Release
Yahoo, Aug. 26, 1997
New Search Engine Launched
Northern Light is a new search engine that opened to general use on Aug. 12. This is the first major search engine to begin crawling the web since the launch of HotBot back in May 1996.
While there have been many new search services appearing since that time, these have been either directories or branded-versions of existing crawlers, such as how AOL NetFind uses Excite's listings. None of these services have created new, comprehensive listings by spidering the web.
In contrast, Northern Light has already crawled nearly 30 million web pages, easily putting its index on par or above all the existing major search engines except Excite and HotBot, which are at the 50 million+ level. That may change, as Northern Light says it plans to continue expanding the guide.
"We plan to be exhaustive on the Internet," said Director of Engineering Marc Krellenstein.
Northern Light has a few twists to help differentiate it from the existing major search engines. One of these is its ability to classify documents by topic.
For example, search for "bill clinton" on Northern Light, and standard search engine results appear on the right side of the screen. But on the left side, "Custom Search Folders" group all the returned documents into browsable topics such as "Non-profit sites," "White House" or by site, such as "www.rnc.org" for the Republican National Committee.
You can select a folder to see sub-topics. Each selection narrows the number of returned documents, but all the documents listed remain relevant to the initial search terms.
The Custom Search Folders are likely to be compared to similar features on other services, so it's important to understand the differences.
AltaVista's Refine function, formerly LiveTopics, will automatically group documents by relevant keywords. However, these aren't browsable. You can use the system as a means of selecting new words to assist in conducting a new search. It's more a dynamic thesaurus than a document categorization feature.
Infoseek automatically categorizes documents in its collection by topics, which are browsable. It even goes further and will suggest likely topics relevant to a particular search. However, selecting a topic immediately takes you away from the original results. In contrast, Northern Light's topics are directly related to the original results and are meant to help narrow the results.
The Inference Find meta-search service comes closest to matching Northern Light's folder system. It groups documents in a similar way. However, as a meta-crawler, it only pulls back the top results from a variety of search services. If there are good documents listed "further" down in the results, these won't be returned. Northern Light's results, in contast, are based on the original document set from its own index.
Northern Light plans to eventually recrawl its index every two weeks, plus have the spider revisit sites based on frequency of changes. Webmasters can register their sites with the search engine for a visit. Meta tags are indexed but not implemented. The text will be noted, but the description will not be controlled by the tag, nor is any special weight attached to the keywords.
Northern Light also has a set of "special collection" documents that are not readily accessible to search engine spiders. There are documents from about 1,800 sources, including newswires, magazines and databases. Searching these documents is free until Sept. 11. After that, the documents can continue to be searched, but there will be a charge of up to $4 to view the full-text, depending on the actual article. This is an approach originally tried by Infoseek when it launched, but it was later abandoned.
Searching web-based documents will remain a free service. Fees will only apply to the special collection documents.
Technologically, Northern Light aspires to take on the majors. The big question will be if it can win traffic away from the existing players, some of which spend millions on promotion. "Clearly we mean to be the dominant search service," Krellanstein said. That's a big challenge, but search services such as AltaVista and HotBot established themselves basically through word of mouth. While they draw less traffic than the other majors, they do have a significant presence.
Northern Light Add URL Form
AltaVista Refine Search Tips
Yahoo + CDnow
News.Com reports that CDnow will become Yahoo's primary music partner. Links to CDnow will be integrated throughout Yahoo's editorial content.
Yahoo, CDnow strike a deal
News.com, Aug. 26, 1997
Lycos Loses Less Money; All Stocks Rise
Lycos posted its fourth quarter earnings, with a loss less than analysts had expected. The news, along with the Lycos-Barnes & Noble announcement, boosted the value of Lycos shares significantly. Other search engine stocks also saw significant rises. The latest quarterly losses for all the major search engines are below. Keep in mind that Yahoo would have posted a slight profit, had it not been for the Visa Marketplace buyoff.
Excite: $7.9 million
Infoseek: $11.9 million
Yahoo: $20.5 million
Lycos lights up search sector
News.com, Aug. 27, 1997
Lycos Trims Losses, Eyes Profitability
TechWeb, Aug. 29, 1997
Under the hoods of search engines
News.com, Aug. 28, 1997
Quotes from a variety of analysts about the valuation of search engines, in light of current earnings and future expectations.
Excite stock rocketing
News.com, Aug. 28, 1997
Directory Stocks Go Nutty
Wired News, Aug. 28, 1997
Infoseek Gets New Look, New CFO And New Strategies
(Repeat from last issue, but with new information)
Infoseek unveiled a new design for the search service on Aug. 20. The home page has been redesigned to better emphasize particular areas of the service.
A box at the top of the page emphasizes the pure search service, while below it, the directory area is highlighted. Along the left side of the screen are special search features that Infoseek offers, such as news, reference, yellow pages and image searching.
"Before, we had the stuff grouped that way, but it wasn't real obvious," said product manager Sue LaChance Porter. "We found with usability testing, that there is much more a comfort level with this new user interface."
On the inside search result pages, a suggestions column runs alongside the left side of the screen, while standard search results appear on the right side. In the suggestions column are related services that may be of interest.
"Best Bets," at the top of the column, highlights specialty business information, when relevant to the topic. "Related Topics" highlights categories of interest within Infoseek's associated directory. "Related News" brings up news stories that may be of interest, from the news sources that Infoseek crawls.
In the past, a search box appeared at the top of the results pages, and the default was to narrow a search with words that were entered. It was rather awkward to scroll back up to perform a new search, and the default setting made it easy to narrow a search accidentally, when the intent was to start a new search.
Now, a search box and new buttons appear at the top and bottom of each search results page suggesting "New Search" or "Search These Results."
To perform a completely new search, enter your terms and select New Search. Otherwise, narrowing a search is easy. Add a few terms within the box, select "Search These Results," and Infoseek will now search only within the original list of documents it found.
Small changes have also been made throughout the service. Search tips randomly change and appear near the search box. The help files have been cleaned up and slightly reorganized.
Infoseek's look is not all that is changing. Infoseek named former Fractal Design-exec Les Wright as its new chief financial officer on Aug. 12. Wright will oversee all of Infoseek's financial and administrative operations. Previously, Wright was chief financial officer and chief operating officer for Fractal Design Corporation.
Wright completes the replacement of major management position left empty after departures earlier this year. Infoseek is also planning to announce major strategy changes sometime this month.
Infoseek Press Release
Infoseek, Aug. 12, 1997
Infoseek names Les Wright CFO
News.Com, Aug. 12, 1997
NewBot Becomes NewsBot, Now Available On The Web
HotWired has changed the name of its news searching service from NewBot to NewsBot, to better reflect the function of the service. The change occured in mid-August.
NewsBot is now also available as a web-based search service. Previously, it was only available to those who had downloaded an ActiveX applet. It will remain a beta service for the next few weeks.
The service, such as Excite's NewsTracker, scans key news websites on a daily basis to find top news stories. If you are still looking for news using "normal" search engines, stop doing it! You'll find either of these services to be a much better way to search for news related stories from hundreds of sources.
Yahoo Australia & NZ Launched
Yahoo launched a regional version of its guide for those in Australia and New Zealand on Sept. 1.
Yahoo Australia & NZ
Lawsuit Over Meta Tag Keywords
A lawsuit has been filed over the use of someone's trademark in the meta tags of another company's web page. The law firm of Oppedahl & Larson has gone after several companies that used its name within its tags.
The suit already has people wondering if they are violating trademark law by using competitor's or trademarked names in meta tags. But the issues raised are not clear-cut. Moreover, this lawsuit and future ones like it are quite likely unnecessary, if the situation is simply treated from a search engine spamming perspective.
Many people seem to think that merely adding a term to a meta tag is enough to make a page appear well for that term. This is almost never the case.
For example, imagine I have a page about sporting equipment and I add a single instance of the term "Rollerblades" to the meta keywords tag. The word itself never appears in my HTML body text. It is extremely unlikely the page will appear when a search is done for that term. There are simply too many other pages rich in body copy content for that term. They are more likely to come up first.
Chances are, the other pages that do well will be using the term "Rollerblades" in an accepted manner. Someone who sells Rollerblades may have an online catalog of offerings. Using the term in body copy will be a necessity, and adding it to the meta tags is an acceptable way to classify the page. The trademark owner is unlikely to have a claim for misuse, nor a reason to complain for search engine spamming if the page should do well.
In contrast, the dispute in this case is with companies that used the trademarked terms for no legitimate reason. The pages cited in the suit are for web design and hosting firms that seemed to think that using the firm's name might bring them some traffic. The law firm deals with Internet domain disputes, along with other intellectual property issues. Assumedly, the companies believed adding the name to their meta tags might find people looking for domain dispute information, who in turn might have been interested in web hosting services.
Clearly, this wasn't a legitimate use. Had the law firm reported the page to any of the search engines, they would likely have been banned from the listings because of an attempt to be misleading.
Most search engines do not require that the terms used in meta tags be also used in body copy -- one of the reasons for creating meta tags was to help web designers have a place to classify their pages if the body copy was not sufficient. However, all the search engines will require that the tags be related to page content. An attempt to be misleading isn't tolerated.
Moreover, the particular pages cited were loaded with spam. The law firm name was used over and over, along with many other terms. Any of the search engines would have thrown these out and quite possibly banned the site.
The law firm says its filing the suit to help protect others. Perhaps so, and it will certainly be helpful as a means for others to think twice before using trademarked terms. But had the situation been dealt with from a spamming point of view, a lawsuit would likely be unnecessary. The search engines themselves would have tossed out these pages.
Meanwhile, from a legal view, the main issue cited is unlikely to hold up in court:
"Such use by defendants of web pages bearing a substantially identical mark to plaintiff's mark 'OPPEDAHL & LARSON' is misleading and is likely to cause confusion and mistake, and to deceive the public into believing falsely that defendants' web pages are connected with and/or sponsored or authorized by Plaintiff."
The tags are used to possibly help the site come up for those terms, but the page title and meta description clearly do not represent these pages in the search engine listings as being part of the law firm's site. There is no way that anyone viewing the listings would have thought these pages were part of the law firm's site.
Should you stop using trademarked terms in your meta tags? Probably not, if you have a legitimate reason to use them. But remember, they aren't going to help you unless you have relevant copy on your pages. Otherwise, you'll probably only do well by attempting to spam the search engines. Your success might bring a trademark lawyer coming after you, but even more feared should be the chance you find your site banned for search engine spamming.
Oppedahl & Larson Complaint
Civil Action No. 97-Z-1592, July 23, 1997
The actual complaint, along with examples of the pages and their listings in AltaVista.
Trademark Battles Simmer Behind Sites
Web Week, Aug. 25, 1997
Keywords said to violate trademark
News.com, Aug. 27, 1997
InterNetGain Checks Positions, Analyzes Content
The InterNetGain position checking service provides web site owners with reports of how their pages are ranked for particular search terms. It also provides an analysis of how the pages compare content-wise in relation to top ranked pages. The service launched Aug. 13.
The free service runs position reports on either AltaVista, Excite, Infoseek or Lycos. The paid service, $49.95 for six months, reports on the position and count of keywords in the title, meta tags and body text of top ranked web pages. It also provides a checklist of deficiencies and corrective actions after analyzing page content. Results are delivered online.
Lycos Spider Now Patented
Lycos is to be granted a US patent on its spidering technology, it was announced on Sept. 2.
Lycos Press Release
Lycos, Sept. 2, 1997
Move To Improve Robots.txt Standard
An effort is underway to revise and improve the robots.txt standard, which provides guidelines on how web site owners can restrict the access of spiders to their sites. The current guidelines are part of an informal standard, but one that has gained widespread acceptance by robot operators, such as search engines.
Those behind the revision effort would like to see the standard expanded and improved. The move got a somewhat ambivalent reception recently in the robots mailing list, which is populated by those involved in the robot industry.
Standard for Robot Guidance
Robots Mailing List
WiseWire Added to Netscape Net Search Page
WiseWire has been added the Netscape Net Search page. It now appears listed along with other services, below the Premier Provider search boxes. WiseWire is a free service where subscribers select favorite topics, then read and rate documents for relevancy. Based on what the user likes, WiseWire builds an individual profile that is supposed to grow more accurate with each use.
Yahoo European Demographics Out
Yahoo released some results of a survey it conducted among UK, German and French Yahoo users. Those looking for some demographics about search engines users in these countries will find the results a good starting place. The company also announced that surveys conducted by marketing and research firms in France and Germany found Yahoo to be the top-ranked guide in those countries.
Yahoo Press Release
Yahoo, July 31, 1997
Philp Devoted To Channels
Excite's not the only one with a directory to channels. Phlip allows you to search for listings in its directory, and those maintaining channels can list themselves.
Excite Guarantees Shopping, Adds Polls
Excite launched a new Shopping Channel on Sept. 2, along with an unprecedented guarantee to protect those making online purchases via the channel from credit card fraud.
If a user's credit card is used fraudulently as a result of a transaction from one of Excite's Certified Merchants, listed in the channel, Excite promises to reimburse the user for any fraud liability that the credit card company does not cover.
Certified Merchants are another part of Excite's new online shopping strategy. Those certified must offer secure transaction capability and meet specific customer service guidelines drawn up by Excite.
Credit card fraud on the Internet remains more a worry than a reality, but the move to offer a guarantee might draw people in who continue to feel nervous about online shopping.
Excite also added online polls in August. Visitors can now vote, see instant results and even propose questions.
Excite Shopping Channel
Excite Shopping Guarantee
Below are sponsor messages that ran in this month's issue of the Search Engine Report, which may be of interest to Search Engine Update readers.
------------- Web Tip ---------------
Visit WebPromote Monthly at:
Keep apprised of the latest web site promotion developments, opportunities, and tips affecting your web site. Register now for your FREE email subscription to WebPromote Monthly. Subscribe today.
Informative back issues are also available at
------------- Web Tip ---------------
Did you know that you can easily submit every unique URL within your web site to each of the major search engines? What a valuable time saver!
------------- Web Tip ---------------
Need to increase your search engine ranking? The 'Top Ranking Guide' is now available!
When a prospect does a search for a keyword related to your products or services, do YOU appear in the TOP 10 or does your competition? Submitting alone does nothing to insure good VISIBILITY. WebPosition is the FIRST and the only software product to monitor and analyze your search positions in the TOP 10 engines. WebPosition has been compared to similar "services" on the web and has been overwhelmingly voted the best and most accurate tool for search position management.
* You'll know your exact positions for an unlimited number of keywords.
* You'll know if you drop in rank!
* You'll know when a search engine FINALLY indexes you!
* You'll know when you've been dropped from an engine!
* You'll even know how you rank in relation to your competition!
Try WebPosition yourself for FREE at:
To unsubscribe from this list, send a message to [email protected], with the following in the subject field:
If you have problems, just send a message to [email protected]
This newsletter is Copyright (c) Danny Sullivan, Calafia Consulting. It should not be distributed. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at:
Twitter Canada MD Kirstine Stewart to Keynote Toronto
ClickZ Live Toronto (May 14-16) is a new event addressing the rapidly changing landscape that digital marketers face. The agenda focuses on customer engagement and attaining maximum ROI through online marketing efforts across paid, owned & earned media. Register now and save!*
*Early Bird Rates expire April 17.