THE SEARCH ENGINE UPDATE
September 3, 1999 - Number 60
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch, http://searchenginewatch.com/.
Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
NOTE: There is only one part to this newsletter, this issue.
In This Issue
+ General Notes
+ Excite Enlarging Index, Partnered With LookSmart
+ New MSN Search, AOL Search Available in Beta
+ [More” Search Boxes That Pay
+ Bye To Go's Instant Add Feature
+ New Search Engine Promotion Forum Launched
+ Search Engine Articles
+ Subscribing/Unsubscribing Info
First, in a bit of non-site news, I'm very pleased to announce the Search Engine Strategies '99 conference. I'm organizing this first-of-its-kind conference in partnership with Search Engine Watch's parent, Internet.com.
Held on November 18 in San Francisco, it will be a one-day conference with sessions entirely about search engine marketing issues. If you know nothing about search engines, I'll be leading off with a Back to Basics session. Then there will be sessions devoted to meta tags, submission issues, dealing with directories and other topics. We've assembled a wide-range of experts to talk about these topics, and it will be an exciting and informative day. The final session is a "Meet The Search Engines" panel, where site owners can put those perplexing questions of how they are listed directly to the search engines themselves.
The conference agenda has just gone up via the URL below, and we'll be tidying up the information over the information, adding speakers bios, etc. over the next few days.
Search Engine Strategies '99
In search engine news, there are lots of interesting things floating on my radar screen to write about, but I'm keeping this edition of the newsletter relatively light so I can finish some updating throughout the web site itself.
Within the public area, I've just posted the latest results to the Company Name Test. In the search engine category, HotBot and Google got perfect scores for delivering users to official company sites in the test. As for directories, Netscape Search and MSN Search take top honors. Honorable mentions also go out to Go, Ask Jeeves, Yahoo and Snap -- they all did very well. A link can be found from the What's New page.
This marks the resumption of regular relevancy tests that I have planned. I hope to do a different one every month -- but depending on how much news there is, I may slip this to every two months. Once I've done enough continuing tests, I'll also begin a cumulative total for the different services.
I also expect to post the latest Media Metrix and NetRatings figures shortly, along with a number of other changes this month. Keep an eye on What's New, and you'll know when these go up.
Additionally, expect a new home page for Search Engine Watch to go up within a few days. There won't be any dramatic changes, but I am doing a slight organization of the site's content and using new section names that I think will be more descriptive to new visitors.
Within the Subscribers-Only Area, I'm busy doing some updates to the individual pages of search engines that have had major changes. A quick note here: my priority is always to get the latest information out via the newsletter, then go back and update the page for each search engine as time allows. That's why there's a link at the top of each search engine's page that takes you to the newsletter archive area. If anything new has happened, you'll find it in the archives.
I hope to have a number of revisions posted, along with a new version of the Offline Edition, by the middle of next week. Watch the Subscribers-Only Area What's New page to know when this happens.
Subscribers-Only Area What's New Page
Search Engine News
Excite Enlarging Index, Partnered With LookSmart
In August, Excite began the first phase of an ambitious plan to enlarge its search index to 250 million web pages and improve the relevancy of its search results. The search engine also debuted new LookSmart-powered directory listings.
Under its new indexing system, which has been in the works for the past year and a half, Excite plans to visit 500 million or more pages across the web on a regular basis. It will then retain only those pages that it determines are most popular, or which offer the best quality information, or which seem to satisfy the queries its users make.
This "visit many, keep some" approach is how Excite hopes to expand its index coverage without simultaneously overwhelming users with irrelevant or off-topic documents.
"We don't think just adding more content will do the job for us," said Kris Carpenter, Excite's Director of Search Products. "We view that as our number one challenge, understanding what's out there and producing that top quality content in the first two pages of results."
Excite is using a number of "off-the-page" criteria to determine both which pages to retain in its index and how to rank those pages in response to queries. By off-the-page, I mean factors that are not tied to what's on the page itself.
For instance, search engines have traditionally ranked pages by criteria such as where and how often search terms appear in them. Since these factors happen "on-the-page," webmasters could make changes to their pages to try and increase rankings.
In contrast, off-the-page criteria are those not directly in a webmaster's control. A good example is link popularity. It is very difficult for a webmaster to try an outwit a good system that uses link popularity as a ranking criteria. That's because such a system leverages information from across the web, which a single webmaster cannot control.
Excite has long made use of link popularity, and that criteria is now being given heavier weight in its new system. Some have also noticed that Excite has been measuring clickthrough from its results. Carpenter said the Excite has experimented with using this data to influence rankings, but that it is not currently being used as part of its relevancy system.
Excite is also using another set of off-the-page information that I can't disclose publicly. I can say that it is unique among the major search engines in using this type of information, and that it would seemingly offer yet another way of getting the best information to the top of search results lists. Of course, the proof will be if relevancy actually does improve in the long term.
Each of these off-the-page criteria are weighted differently, but term frequency and location still come into play. In general, the mixture should work to reward sites with good content or that at least somehow distinguish themselves online.
This has been the overall trend with all the major search engines, and smart webmasters should be doing everything they can to build up the "reputation" of their sites in order to tap into this trend. Reputation? Yes -- just like people, sites can have reputations. Here are some key ways you can build up yours, in terms of what search engines want:
+ Loving Links: Search engines are making more use of link popularity, so getting people to link to your site is important. However, it's not just a numbers game. You want quality links from sites that are contextually related to you. In other words, getting links from 100 different sites may not be as important as getting links from 10 sites that are similar to you in content. So, get out there and find non-competitive sites that are related to you. Link to them, and ask them to link back.
+ Content Is King: People visit and link to sites that offer unique and substantial information. So, start developing more content if your site is lacking it. Build up FAQ pages and articles about topics related to the search terms that you want to be found for. This is especially important for those that have been devoting most of their energy into "doorway" pages. These are pages designed to rank well for particular search terms, but which typically offer no real content to visitors. Yes, they can be effective, and certainly don't abandon anything that works for you. But these pages do little to build your site reputation, so depending on them too much leaves you unprepared to do well in the future.
+ Get Your Own Domain: Search engines are far more likely to favor your site if you have your own domain, rather than if you reside within free web space such as that offered by GeoCities or Tripod. I know these type of places host many quality web sites. However, if you are concerned about search engines, you should move to your own space.
One big plus to the expanded Excite index will be that good pages should no longer suddenly disappear from the service for no apparent reason. This problem has plagued Excite over the past year. It would constantly drop pages out of its index to make room for new finds. As a result, webmasters with good representation in Excite might suddenly find all their pages gone. Similarly, this had an adverse impact on searchers, because pages that were satisfying their queries one week might no longer be present the next.
With the new system, pages that are deemed popular or high quality in some way should be retained. Excite is also planning to upgrade its submission system to help ensure that new pages or those that its crawler may have missed will also have a presence in the index.
"We want to give every site a shot at being in there long enough to demonstrate that they should stay," said Carpenter.
In particular, pages submitted via the Excite Add URL form that are not already in the database and that are not identified as spam will be far more likely to appear in the index than is currently the case. These pages would then remain within the index for a period of time, the length of which is still being determined. After this period, they might be dropped unless Excite's new crawling and ranking system has somehow tagged them as important.
Excite is also introducing new spam detection systems that are especially aimed at removing duplicate content. This has become a real problem for the service. Over the past year, about the only page a site owner could expect to get listed and keep listed was the home page of a web site. Thus, many have set up multiple web sites, in hopes of increasing their representation at Excite. These "mirror" or "satellite" sites often have only one or two pages that in turn link back to the "real" web site.
As a result, it is not uncommon to do a search for a popular topic and find multiple sites listed that seem independent but which in reality link back to the same place. Excite says it intends to crack down on this practice, as well as the intentional creation of duplicate or near-duplicate pages. So far, I haven't seen a real impact, but the rollout is still continuing.
So when does all this happen? Excite says it is currently at about 113 million web pages indexed, and that they will increase their volume of pages indexed by, on average, a rate of over a million pages per day. It is also introducing a new system meant to revisit pages based on how often they change, in order to keep the entire index as fresh as possible.
As for the Add URL system improvements, expect these to come around mid-September, though I suspect it may take longer than this.
In addition to crawling the web, Excite has also maintained a human-compiled directory of web sites. As at Yahoo, this is where sites have been reviewed by editors and organized into categories. A new deal struck in August means that this web directory will now be produced by LookSmart. In fact, LookSmart's information has already be integrated into Excite.
Just like at Yahoo, you can access the directory by selecting a main category from the Excite home page. You'll find them just under the search box. These links take you into one of Excite's "channels," which are filled with information beyond just web site listings.
On the left-hand side of each channel page, you'll see a box called "Directory" filled with topics related to that channel. For instance, in Excite's Lifestyle channel, the first topic in the directory box is "Beauty & Fashion." By selecting this topic, you'll then be shown a list of Beauty & Fashion web sites.
Only a few top sites will automatically be displayed for any topic. To see more, click on the "More Web Sites" link. You'll also see that as you drill down, even more topics will be revealed.
A faster way to get to relevant directory listings is just to do a search at Excite. If Excite finds any categories that match, it will display them in the search results under the heading of "Directory."
Many webmasters have been frustrated in the past about the inability to submit to the Excite directory. With the transition to LookSmart, those worries are lessened. Now if you submit to LookSmart and get accepted, you'll be included in the Excite directory -- along with the AltaVista directory and at the new version of MSN Search.
Unfortunately, LookSmart's submission system can be rather sluggish. On the plus side, you can submit to multiple categories, as long as you're relevant for them. Also, plans to have an expanded "self-publication" index that I've reported on in the past have been dropped, LookSmart says.
A couple of other Excite notes. A new Adult Content filter was introduced earlier this year. You'll find it on the advanced search page. It has to be enabled each time you do a search, unlike filtering options offered by AltaVista, Go and Lycos. A more permanent solution may appear later this year. Filtering is done by a combination of looking for the presence of certain words at the time a page is spidered and through the use of a site block list.
Excite is also offering the ability to search by language. As with other services doing this, language determination is made by looking for the presence of certain words unique to a particular language. You'll find this option on the Advanced Search page.
I also wanted to take a moment and briefly provide an update on Excite's two other search properties, WebCrawler and Magellan.
Magellan is now essentially a stripped-down version of Excite's directory listings and search index. Magellan's home page features the directory -- click on a topic, and you'll get web sites and only web sites -- no channel bells and whistles as you might get at Excite. Do a search, and your query goes against about two million pages from the Excite index, which are predominately site home pages. Magellan also uses Excite's ranking algorithms, so for popular queries, you may get the same results as at Excite.
Magellan also used to feature the ability to view "green light" web sites; however, this kid-friendly feature is temporarily gone. A replacement should appear by end of the year, Excite says.
WebCrawler is similar to Magellan in being a lighter-version of Excite. It also presents directory information, and web searching also goes against only two million page from the entire Excite index. However, the service has much more personality than Magellan, plus it does have expanded channel content that Magellan lacks. Additionally, WebCrawler uses a much different ranking system than Excite, so expect to see differences if comparing the two.
In the future, both services may have their web search ability expanded to tap into about 3 and 5 million pages from the Excite index. And webmasters -- if you have submitted your web pages to Excite, there is absolutely no reason to also submit them also to Magellan and WebCrawler. They use Excite's spiders and index.
Excite Advanced Search
Click on the words "Advanced Search" on this page to get complete options, including the adult content filter.
How LookSmart Works
Covers tips on submitting to LookSmart.
Kids Search Engines
Listing of services offering kid-friendly searches
New MSN Search, AOL Search Available in Beta
Two major portals are readying new versions of their search services. Both MSN and AOL have made unveiled their next-generation search offerings at beta sites that have just gone live.
The new MSN Search nicely integrates information from RealNames, the LookSmart directory and AltaVista. You will also find an option to search at Direct Hit to the left of the search results screen. It should take over from the current Inktomi-powered service in mid-September.
The new AOL Search service will be the successor to the current AOL NetFind. A version that works internally for AOL members will blend matching content from within AOL and from across the web into one results screen. AOL's internal keywording system will also be integrated into the system. The external version that will be accessible via the web won't list content only available to AOL members. Both versions will offer Inktomi-powered web-wide searching, but they will also be presenting Netscape's Open Directory Project's categorized listings, as well.
As both beta sites have just gone live, I'm holding off on longer reviews for the next newsletter. Meanwhile, the curious can explore them via the URLs below. Remember -- both of these services are still in beta, so they are going to be some rough edges.
MSN Search Beta
AOL Search Preview
[More” Search Boxes That Pay
Last issue, I mentioned a new program at Lycos that lets you get paid for adding a search box. Now Direct Hit has launched its own affiliate that pays 3 cents per search.
Direct Hit Affiliate Program
Bye To Go's Instant Add Feature
Go (Infoseek) has dropped its instant Add URL feature. Previously, any page directly submitted would appear within a day or two. Now that time frame has been dropped back to a week, and a faster add won't be coming back, says Jan Pederson, Go's Director of Search. Moreover, because of on-going engineering works, new submits will take longer than a week to appear, in the short term. On the other hand, the spidering changes that Go has underway will allow it to more in-depth crawling of sites than in the past, Pederson said. Expect a follow up in a coming newsletter.
New Search Engine Promotion Forum Launched
First Place Software, which makes the WebPosition ranking and submission tool, has launched a new online forum at MarketPositionTalk.com. It's devoted to search engine marketing issues. Also be sure to check out the Search Engines Forums, if you've never visited them before. Run by VirtualPromote, they offer on-going, moderated discussions of submission issues.
Search Engines Forums
Search Engine Articles
AltaVista Reaches Record Users With Free Internet Access
AtlaVista Press Release, Aug. 25, 1999
AltaVista's new free Internet access service has already drawn 225,000 subscribers in its first two weeks.
The Quality of Researchers Searches of the ERIC Database
Education Policy Analysis Archives, Aug. 25, 1999
Typical users of an in-house database were found rarely go beyond the first page of hits and to examine only about 3 to 4 matches, in contrast to advanced searchers who would look longer and harder.
The Art of Backward Searching
About Web Search Guide, Aug. 24, 1999
How to find your site's popularity on search engines or find related pages to one you like.
Browser multitasks as "Internet Desktop"
News.com, Aug. 23, 1999
Lycos is partnering with NeoPlanet to produce a browser that integrates portal services like instant message into the software.
Q & A: Jeff Bennett, Lycos V.P. of E-Commerce
InternetNews.com, Aug. 20, 1999
Lycos VP discussed where the company is going in e-tailing and e-commerce.
Search engines can be tricked into serving up porn sites
MSNBC, Aug. 18, 1999
A look at a metajacking case, where someone stole someone else's meta tags. Comments from search engines and the FBI. The usual caveat applies -- just using someone else's meta tags doesn't mean that you will rank well for their terms. In this case, the real issue is what happens when someone is actively trying to mislead consumers into thinking they are reaching another company and who should be responsible for taking action -- the search engines, the site owners, the police, the courts?
AltaVista's International Mirrors
EContent, August. 1999
Covers differences between AltaVista's mirror sites.
How do I unsubscribe?
+ Follow the instructions at the very end of this email.
How do I subscribe?
+ The Search Engine Update is only available to paid subscribers of the Search Engine Watch web site. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to email@example.com
I need human help with my subscription!
+ Send a message to firstname.lastname@example.org. DO NOT send messages regarding list management or site subscription issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at
This newsletter is Copyright (c) internet.com corp., 1999