THE SEARCH ENGINE UPDATE
October 20, 1998 - Number 39
About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to those people who have subscribed to Search Engine Watch, http://searchenginewatch.com/.
Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
In This Issue
Things are heating up once again. AltaVista has added several new features, while Lycos has unveiled a redesign and is to acquire HotBot. Below you'll find a quick update on these developments, with more details coming in the next newsletter.
Within the public section of Search Engine Watch, all the Search Engine EKGs now have been updated with data through September 1998. The SpiderSpotting Chart has also been updated, as has the More SpiderSpotting page.
New results from the latest feedback test can be found on the Search Engine Response Time page. Northern Light was the big winner, with a feedback response of one hour. Several others showed fast responses, also.
Search voyeurs will be pleased to know that the MetaCrawler Top Search Terms page has been updated with September 1998 terms.
All the pages above can be found from the What's New page except the More SpiderSpotting page. You can find that via the Subscribers Only home page, in the More About section.
Subscribers Only Home Page
Search Engine News
AltaVista has introduced a number of significant new features this month. Most important is a partnership with answer service Ask Jeeves. Matching results from Ask Jeeves now appear first on the results page, in response to queries. Next comes the RealNames link, then the actual search results.
The search results have also been impacted in two major ways. First, AltaVista has introduced an algorithm change that it says produces better results for simple queries. There are no details on new important factors that influence why a page may be top ranked, though home pages seem to be getting even greater weight than in the past. AltaVista is now also doing automatic phrase searching, in response to many queries.
There is also a new and impressive image search capability. Additionally, the service has introduced a filtering option. You'll find both in the specialty searches area, below the main search box.
Expect a more in-depth look next issue, plus a revisit to the Ask Jeeves service itself.
In Redesign, AltaVista Is Returning to Its Roots
Internet World, Oct. 18, 1998
Lycos has introduced a new look, as well as a relevancy tweak that seems to be improving results that have been relatively disappointing over the past few months.
Of course, the bigger news was that earlier this month, Lycos announced that it would be acquiring Wired Digital, which owns the HotBot search engine.
My understanding is that plans are to continue running HotBot as a separate service, and that HotBot will continue being powered by Inktomi, not by the Lycos search engine.
More news on the facelift and acquisition next issue.
Wired Digital Is Bought by Lycos
Internet World, Oct. 12, 1998
Wired Finds New Network Home
Red Herring, Oct. 9, 1998
In a few months, the backers of GlobalBrain.net hope to bring profile-based searching to the web, through a deal with one of the major search services.
GlobalBrain is technology that uses clickthrough data combined with user data to deliver highly targeted search results. It has been developed by Dr. Grant Ryan and a team of Ph.D. and programmers in New Zealand, and is being marketed to search firms by a Bay Area business development consulting company called Double Impact.
"Based on the incredible interest and feedback we've received from test users, Portal CTOs and CEOs and industry analysts, we expect to solidify a major exclusive relationship and release the technology to the public within a few short months," said Double Impact principal and GlobalBrain board member Steven Marder.
At the core of GlobalBrain is a system of measuring what users select from search results. That sounds like Direct Hit, which currently has a partnership with HotBot. However, GlobalBrain is not a copycat system attempting to cash in on Direct Hit's success. I heard about both technologies at nearly the same time, about six months ago, before either launched publicly. It seems very much the case of a good idea -- counting clicks -- being developed independently.
Both systems watch to see which pages users actually select from among the search results. These pages are considered better than others, especially if a person spends much time viewing them. In this way, searchers invisibly vote for the pages they like.
Direct Hit uses this clickthrough data to rerank HotBot's "normal" listings, and it often produces more relevant results. HotBot users tap into Direct Hit by performing a search, then selecting the "Top 10 Most Visited Sites" link that appears about the HotBot results.
GlobalBrain steps beyond Direct Hit with its concept of profiles. GlobalBrain asks searchers to voluntarily register themselves by country, age, occupation, favorite sport and gender. This user data is then linked to clickthrough data. This allows GlobalBrain to deliver results targeted to particular profiles, an extremely powerful feature.
For example, a person living in the United States will have a completely different concept of what is relevant for "football" than someone who lives in the United Kingdom. With profiling, GlobalBrain can intelligently deliver results about American football to the US surfer, while the UK surfer gets information about what an American would call soccer.
Likewise, profiling can be done by interest area. At Excite, a search for "giants" brings up special information about the New York Giants football team. That's a great enhancement, unless you are a fan of the San Francisco Giants baseball team. With GlobalBrain, someone who indicates that their favorite sport is baseball, or that they live in San Francisco, would be more likely to get information about the Giants baseball team, rather than the football team.
GlobalBrain also offers an innovative keyword suggester that leverages what people click on to build a thesaurus of related terms.
For example, imagine someone searches for "bicycling" and then selects a page from the results that are displayed. They may then search for "cycling." Because the two keywords were entered one after the other, the system assumes they may be related. The next time someone searches for bicycling, it will then suggest "cycling" as an alternative.
The system also measures which of the suggested keywords users click on. This helps it learn which keywords are best related to each other, in the same way it learns which web pages people find useful.
Currently, Excite is most advanced in this area of suggesting alternative words. It watches for links between words that appear on pages, rather than within searches. For example, if it sees many pages where Apple appears with computer, it may then suggest computer as a word that should be included in the search.
The GlobalBrain solution is not necessarily better than what Excite does, but rather another approach toward achieving the same goal.
GlobalBrain also provides automatic bookmarking, which is probably more understandable as saved searches. The system will show you past searches, along with the link you visited and found most relevant for each keyword. This bookmarking is also used to help refine search results to an individual level.
The data gathered through bookmarking means that two people with identical profiles (country, sport interest, occupation, etc.) might still get two different sets of results in response to a search, because the system is tailoring matches to what they've selected in the past.
One possible concern with this automatic mechanism is the fear that searches could be tied to an individual. I've long been expecting for some journalist to track down what a politician is searching for while at work, in the way that some have sought video rental records.
It's theoretically possible to do this now, but only with a great deal of difficulty. For example, personalization services at Yahoo and Excite, among others, allow for search topics to be saved. This makes it easy for a user to run the search in the future -- you just click on the saved link.
Someone who could access this personal information might be able to discover if some embarrassing searches are being stored. However, the first problem is that the companies aren't likely to release it willingly. In fact, several of them have just pledged ad space to promote privacy issues.
"We would never communicate data on an individual level," said Excite search product manager Kris Carpenter.
Second, it would be difficult to conclusively prove that any individual actually created an account, Carpenter pointed out. After all, no positive identification is required to open a personalized account.
For example, someone could go to a coworker's computer, quickly create a fake account, then store some embarrassing searches. The computer could be linked to the account via a cookie, but there's no way to prove that the computer's owner actually created the account.
Beyond these problems, the main difficulty in gathering this data would be that most people probably have not bookmarked their searches. Someone who actually managed to force their way into an account by some means would likely discover nothing incriminating there.
This is why GlobalBrain's system might cause a concern with some people. It would store searches automatically. All the other hurdles in getting this information would remain, of course, but people can still be hypersensitive about privacy. To address this, GlobalBrain's solution would be offline storage of searches.
"Privacy has been key concern when designing the GlobalBrain system, hence there will be an option to download software that stores the personal bookmark information on a users computer, said Ryan, "The only person that will ever have access to this data is the person that created it"
In addition to the above features, GlobalBrain also provides an ability to sort pages into different groups, such as "popular" or "high flyers." Popular displays pages just as Direct Hit does, where pages are ranked in order of popularity based on clickthrough and time viewed. High flyers list pages in order of those with increasing popularity. A "New" option shows the newest pages that have been found relating to particular terms. "Past Preferences" shows results based on a user's individual preferences.
By default, GlobalBrain results show a mixture of the four groups. In this way, relevancy is kept high, yet an element of discovery is also mixed in.
GlobalBrain even makes some provision for the use of clickthrough relating to the submission process. The basic idea is pages would be submitted with associated keywords and a targeted profile. In this way, a site owner could submit a page intended only for a UK audience, or for a baseball fan. If the page ranked well, it would keep its ranking only if people were visiting it.
While the system is meant to discourage spam, I think it would actually attract more, at least in terms of submissions. However, the concept of linking pages, or perhaps even sites, to profiles is an interesting one that warrants investigation. One easy twist might also be to require registration for those submitting pages. Such a system could even learn if pages from some submitters are ranked well by users, perhaps thus giving future submissions priority.
GlobalBrain is taking a different strategy with its technology than Direct Hit. That company is offering its technology to anyone, through branded partnerships. In contrast, GlobalBrain has been talking with various search engine companies to find a single partner to purchase its technology.
GlobalBrain says that two companies are currently testing its system, to see how it works in the real world, and a third is about to evaluate it. Ultimately, it hopes the system will launch with one of the companies within a few months.
"The system has worked so well in our extensive trials that we are very excited about launching it and providing users with a more satisfying and relevant search experience," said Ryan.
Double Impact / GlobalBrain
Privacy ad campaign to launch
News.com, Oct. 7, 1998
Search Engine Notes
Yahoo Buys Yoyodyne
Yahoo announced it will purchase target marketing firm Yoyodyne in a $30 million stock swap.
Yahoo Buys Marketer, Hoping To Build Ad Base
Internet World, Oct. 19, 1998
Excite Partners With Netscape Internationally
Excite announced on October 14 that it will be powering Netscape's country-specific search services for Australia, France, Germany, Italy, Japan and the United Kingdom. Excite will also be developing content for several Netcenter channels aimed at these countries.
MSN Out Of Beta
MSN's redesign lost its beta moniker on October 7. Changes went live on the site that day.
Also, last issue I said the name Microsoft Network was now being used for the separate Microsoft Internet access service. In fact, this service is called MSN Internet Access.
Microsoft Unveils MSN Search
The Search Engine Report, Oct. 5, 1998
Details about the improved MSN and Inktomi-powered MSN Search.
Search Engine Articles
Net search engines find TV ads
San Francisco Business Times, Oct. 12, 1998
A nice summary of the current round of search engine television ads that are underway.
Netscape Bets Portal Users Will Pay to Play
Industry Standard, Oct. 8, 1998
Netscape is planning to charge for extra services offered through its portal. Will it work, and might others follow? Certainly, if successful. That remains to be seen. Article provides some details of Netscape's plans.
How do I unsubscribe?
+ Send a message to email@example.com with the following as the first line of the body:
How do I subscribe?
+ The Search Engine Update is only available to paid subscribers of the Search Engine Watch web site. If you are not a subscriber and somehow are receiving a copy of the newsletter, learn how to subscribe at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to firstname.lastname@example.org
I need human help!
+ Send a message to email@example.com. DO NOT send messages regarding list management or site subscription issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at http://searchenginewatch.com/about/contact.html.
This newsletter is Copyright (c) Mecklermedia, 1998