About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to Search Engine Watch members. Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
In This Issue
+ Site News
+ Search Engine Strategies Conference Only Two Weeks Away!
+ Search Engine Marketing Finally Getting Respect
-- (full story online, link provided)
+ New Articles On FTC Complaint
+ LookSmart Overhauls Submission Process, Increases Prices
-- (full story online, link provided)
+ New AllTheWeb.com Goes Live
+ A Webby, More Images, Date Range Search & Search Voyeurism At Google
+ Search Engine Security Concerns
+ Search Engine Resources
+ SearchDay Articles
+ Search Engine Articles
+ List Info (Subscribing/Unsubscribing)
I've been plugging away to update my pages about searching tips, power commands and special features. This will also include updates on how to locate your URL in search engines and do backward link lookups. It's been a massive job, and there are lots of new commands and tips to share. I expect to have all that ready by next week, fingers crossed. Keep an eye on the What's New page, and you'll know when it is ready.
Search Engine Strategies Conference Only Two Weeks Away!
It's two weeks and counting down until the next Search Engine Strategies conference. To be held in San Francisco on August 16th and 17th, the conference features an entire day devoted to marketing your site on search engines. Sessions are designed to bring beginners up to speed on promotion issues, while there are also in-depth sessions designed for more advanced marketers.
The conference is also unique in having several panels that feature representatives from the actual search engines themselves. Get first hand advice, direct from the source! The second day of the conference also allows marketers more opportunity to hear sessions on promotion issues.
Web searchers -- you're not left out! Day 2 features two "searcher-oriented" tracks designed for you. These panels will give you tips on searching better, as well as a glimpse into behind the scenes issues that influence how well search engines work for searchers.
Developers and systems administrators also have a track for them on Day 2, designed to help you understand how to add a search engine to your own web site, intranet or to create a vertical search feature. Learn how improving site and ecommerce search can lead to more sales or how having better intranet search can mean productivity savings.
I'll be speaking at the conference, along with other search engine marketing and research experts. There will also be speakers from the search engines themselves, including About.com, AltaVista, Ask Jeeves, Excite, FAST Search, Google, GoTo, Inktomi, LookSmart, MSN Search, Netscape/The Open Directory and Yahoo.
The conference is sponsored by AltaVista and exhibiting companies include ah-ha.com, AltaVista, Build Your Own Directory, Business.com, Did-it.com, EasyAsk, Fast Search & Transfer, FireSpout, Inceptor, INFERNOsearch, Inktomi, LexiQuest, LingoMotors, Position Technologies, Quigo Technologies, Search Boss, SiteLab, Sprinks, Web Ignite, WebSeed and WebGenius.
Those interested in sponsoring or exhibiting should contact Frank Fazio Jr, email@example.com, for more information. Attendees can find the agenda or sign-up for the conference via the URL below:
Search Engine Strategies: San Francisco 2001
Search Engine Marketing Finally Getting Respect
To some degree, search engine marketing has been like the Rodney Dangerfield of online advertising -- it's gotten no respect. Or, at least it has gotten no respect in relation to the time and effort analyst firms have put into understanding it compared to banner advertising. The good news is, that's all about to change. The positive financial performance of GoTo is making people take notice, though all the signs have been there for ages. The full story can be found below and includes links to several interesting surveys, as well as a recap of some financial reports.
Search Engine Marketing Finally Getting Respect
The Search Engine Report, Aug. 2, 2001
New Articles On FTC Complaint
Here are two new articles relating to the FTC complaint that I mentioned in the last newsletter, as well as a link to my original article, in case you missed it.
Watchdog group awaits FTC action on search engines
ZDNet eWeek, July 20, 2001
The FTC speaks -- but only to say they can't confirm or deny if they got the complaint. It also explains that assuming it does investigate, it would try to reach a settlement with the search engines first, then take legal action should that fail.
Search Engines and Editorial Integrity
Online Journalism Review, July 24, 2001
Examines the complaint and comments on the growth of paid listings, ending with a call for users to let search engines know that they want paid content clearly labeled.
Consumer Group Asks FTC To Investigate Search Ads
SearchEngineWatch.com, July 17, 2001
Is it deceptive advertising to include paid listings in your search results and not clearly label them as ads? A group backed by consumer advocate Ralph Nader believes so, and it's asking the US Federal Trade Commission to take action against seven major search companies.
LookSmart Overhauls Submission Process, Increases Prices
The good news is that over the past month, LookSmart has given site owners greater flexibility than ever before to get listed with the service. The changes are especially beneficial for those with non-commercial content. The bad news is that getting listed is now actually more complicated and will cost many commercial sites 50 percent more. The article below explains how the new integration of Zeal.com and LookSmart's listings should help non-commercial sites, as well as the new "Express Modify" and "LookListings" options available to those with marketing budgets.
LookSmart Overhauls Submission Process, Increases Prices
The Search Engine Update, August 2, 2001
New AllTheWeb.com Goes Live
FAST Search's AllTheWeb.com got both a new look and new functionality in July, improvements that have turned the site into a top-notch resource for searchers.
AllTheWeb has always been an important site, because it has consistently offered one of the largest collections of web pages available. Consequently, it was a great resource when you needed to search for information on unusual or obscure topics.
The site's weakness was that it lacked many features that other search engines offered, such as automatically clustering listings, so that one web site didn't dominate the results. It also had generally poor spam filtering and fairly average relevancy for more popular queries. As a result, it wasn't a resource I'd typically recommend as a first stop for average web searchers.
All that's changed, now. There's every reason to consider AllTheWeb as one of your top choices when searching. Its new features and relevancy improvements have made the site far more appealing.
One of the most notable things about the new AllTheWeb is the what it calls "universal search," where the search engine automatically brings back information from different collections that it maintains. For instance in addition to a web page catalog, AllTheWeb also has database of pictures, video clips, MP3 and FTP files from across the web. When you do a search, results from several of these different sources may be presented, in response.
For example, take a search for "britney spears." By default, you are shown matching pages from across the web, leading off with Britney's own official web site. However, on the right hand side of the screen is a picture of Britney and links to bring up more pictures or video clips that seem to be about her.
Generally, the suggestion of pictures or video clips in what AllTheWeb calls its "side bar results" is most likely what you'll see. However, FTP or MP3 suggestions might also appear. Of course, you can also specify exactly which database you'd like to search against by using the links that appear under the search box, on the AllTheWeb home page.
AllTheWeb has also added new search tips that may appear to the right of search results, in a "Search tips" box. Look back at the "britney spears" search, and you'll see that the tip suggests using quotation marks to perform a phrase search. In addition to educating users, you can also select the link in the box to perform the suggested tip.
Clustering, which AllTheWeb calls "site collapsing," is another welcome new feature. It wasn't uncommon to do a search at AllTheWeb and find that all the top results came from the same web site. This problem has now been greatly reduced, and you shouldn't see more than two pages in the top results from any particular site.
If you do want to see more pages from a particular site listed in the results, simply select the "more hits from" link that appears below a listing. You can also override site collapsing for all your searches by using the options on AllTheWeb's customize page.
That new customization page also offers a variety of other features. You can control the porn filter, disable search tips or side bar results, stop term highlighting and more. In addition, there are a variety of new search commands that let you search within URL text or link text. These are summarized on the "Basic Help" page, and all the help files are an easy read and provide a well-done summary of how to use the service.
AllTheWeb also used to have noticeable problems with spam, but you should now find that this has been greatly reduced, due to new filtering that the service is using.
In particular, AllTheWeb is making use of several new methods to reduce the spam problem. Most important, Fast says, is watching for unusual linkages, sites that appear to be linking together for purposes of making themselves more popular.
"We discover all the rings, and we can effectively exclude them," said Knut Risvik, Fast's director of engineering.
AllTheWeb is also examining the frequency of terms on pages and removing those that seem excessively abnormal.
"We analyze the page and look at distribution of words," Risvik said. "If the distribution is significantly different from any distribution that should be normal for a language, we will remove them."
That can sound scary -- what if you accidentally create an abnormal page? This isn't likely to happen, Fast says. Fast is checking to see if terms appear excessively in different locations of a document and in high frequency, which can be indicative of those creating "doorway" style pages that they hope will be highly targeted toward a particular term.
In other words, let's say that you wanted to be found for "movies," so you place that word in your title tag ten times, within an H1 header, within link text, within ALT text and use it repeatedly throughout your body copy. That would seem abnormal when compared to the more common collection of documents in the same language about movies, where the word might appear once or twice in the title tag and within the body tag, but not in an extremely high proportion.
There are no hard and fast rules about where and how much is too much that Fast will release. Naturally, that would defeat the spam analysis they are doing. The main advice to take away is not to try and overly engineer your pages. Make use of the terms you want to be found for in your body copy and in your title tags, but don't go overboard.
AllTheWeb is also watching for "gibberish" pages, those where the text may make no sense to a human reader, despite having a sentence structure intended to make it appear normal and relevant to crawlers.
In another fairly recent change, AllTheWeb is now generating descriptions in one of three ways. First, it will use your meta description tag if you provide one and if the tag seems to reflect the content of your web page. Next, in lieu of a meta description tag, it will use your page's description from the Open Directory, if it is listed there. Finally, it will default to using the first 215 characters or so that appear in your visible HTML body copy. By the way, meta keyword tags are still not indexed by Fast.
Along with better spam detection, AllTheWeb also now has relevancy improvements that Fast collective refers to as "FirstPage." At the core of this is better link analysis. Also, the level of pages has an impact. Pages in "upper" levels are more likely to be ranked better, Fast says.
The service has also announced that it will refresh its index every nine to 12 days. Most search engines update on a roughly monthly basis. The last search engine to make such an explicit freshness promise was AltaVista, back in June 1999. It almost immediately failed to keep that promise. It will be interesting to see if AllTheWeb does better.
Index size is 625 million pages, where it has been since around March. That's just below the 700 million or so pages that Google has in its full-text index. FAST also has 70 million listings in its multimedia picture and video catalogs, 2 million MP3 listings and 150 million FTP listings.
All these changes have been done to make AllTheWeb a more attractive destination for users. This is completely opposite of what happened when the site launched back in April 1999. It was meant primarily to demonstrate Fast's technology to prospective portal partners, so special features for searchers weren't added.
Does the change now mean that Fast wants to compete against the portals that it also wants to power? It's more an attempt to coexist, in the way that Google operates alongside its partners such as Yahoo and Netscape, as well as turn AllTheWeb's successes into benefits for FAST's customers.
"The focus of our site is not to abandon all the portal customers and become the number one search destination ourselves. Our focus is toward monetizing AllTheWeb and using that to build monetization technology for potential customers," said Stephen Baker, Fast's director of business development and marketing.
The key is continuing to develop AllTheWeb's ability to dynamically and intelligently pull information from different databases. Fast believes this can help e-commerce sites and others better monetize their services.
Of course, Fast will still continue providing web search services to those who want it. It just expanded its agreement with Terra Lycos in July, so that Fast information can be used by all Terra Lycos sites worldwide. New addition at Lycos using Fast data include Lycos properties for Argentina, Brazil, Mexico and Venezuela. Lycos now also has the ability to sell advertising to appear on the AllTheWeb site.
AllTheWeb Customize Page
It's Fresher at FAST
SearchDay, July 25, 2001
Guest writer and search expert Gary Price puts Fast's claim to be the freshest search engine to the test. It easily beat AltaVista but fared less well against Google.
A Webby, More Images, Date Range Search & Search Voyeurism At Google
Google has enlarged its image search service, added a new date range searching option and unveiled a way to discover popular search topics at the service. The service also received its third Webby award in July. More details are below:
Google Image Search
Google released a new, 66 percent larger index for Google Image Search in July. Google Image Search now enables users to search and browse 250 million digital images, 100 million more than the first index, which was released in June.
Google Date Range Search
Google has introduced the capability to restrict your search to pages that have been updated with a certain time frame. Google date range search uses a drop-down menu to restrict searches for web pages that have been updated in the past three months, six months, or year. Once a date range has been specified for a search, another drop-down menu will appear at the top of the results page, enabling users to manipulate the date range without back-clicking to the advanced search page. Google's date range search feature is available at the advanced search form for both Google and Google Groups.
Brand new, this provides a look at what people are searching for at Google. Shown are top gaining queries and declining queries, and these are being archived each week. Other treats currently shown are most misspelled queries, as well as languages and operating systems used when interacting with Google.
Google Wins 2001 Webby Award for Best Practices
Google Press Release, July 19, 2001
Google wins the new "Best Practices Award" at the recent Webby Awards, which means it was judged best among competing sites in content, structure and navigation, visual design, functionality, interactivity, and overall experience. Last year, Google won for Best Technical Achievement Award and the People's Voice Award, also in the Technical Achievement category.
Search Engine Security Concerns
Security issues with Lycos and Google came up in July. They aren't likely to impact many, if any, users, so don't get panicked. Here's a rundown on what happened.
At Lycos, the search engine was found to render some site descriptions as actual HTML code. Specifically, when it saw the characters < and > in a description, rather than render them as text, it was turning them into HTML commands, if they surrounded appropriate tags. For example, if this was in a description:
The security report filed about this makes it sound like it would be fairly easy for someone to get high ranking pages and then take advantage of users by manipulating descriptions in this way. The reality is that such pages are far more likely only to appear for obscure searches, making the value of this hack minimal.
Nevertheless, it should be something that is corrected. Despite having been reported to Lycos in mid-July, the problem was still happening yesterday.
"We are aware of this, and I was assured earlier this morning that our engineers are working on the fix immediately," said Terra Lycos spokesperson Kathy O'Reilly.
Meanwhile, Google's advancements to index dynamic content means that it was possible for crackers to get into DCShop shopping cart systems and perhaps find credit card information. Google has since removed links to dcshop.cgi URLs, once the potential problem was reported.
Search Engines HTML Parsing Vulnerability (Lycos)
SecuriTeam.com, July 27, 2001
Security warning about the Lycos problem. Second URL has additional information.
Lycos Example Query
If the bug is still there, you'll see how an input form appears for this entry (don't worry. There's no security problem with viewing this example).
Google removes links to credit card loophole
Fairfax IT, July 26, 2001
More details about the problem at Google.
Search Engine Resources
Designed to let you search for messages posted to message boards across the web. Only the automotive and computer topics are currently live. Unfortunately, you can't do a keyword search against all boards. Instead, you must choose a topic and subtopic in order to search.
Like BoardReader (above), this lets you search message boards across the web. Unlike BoardReader, you can do a keyword search against all boards or browse categories to narrow in on a particular area.
Want a search engines that searches through your programming source code? This application might be your solution.
Canadian search engine, with results gathered primarily from user submissions. Thumbnails of web sites shown next to each listing.
Another site emerging from the ashes of Go Guides is Info World, which catalogs sites primarily related to travel and tourism. It was built initially by spidering all the web sites that were cataloged as regional or travel-related from Go Guides, and adds to this collection come from user submissions and continued spidering. It doesn't look pretty, but some test queries I ran related to travel and tourism weren't bad at all.
Dancing with Crawlers
Comprehensive article on making your site friendly to crawler-based search engines, with an emphasis on why user agent detection is a better solution to deliver tailored content to search engines than full-blown IP cloaking. Includes a tutorial on making agent detection work through PHP.
Here are some recent articles that may be of interest from Search Engine Watch's daily SearchDay newsletter:
SearchDay, July 24, 2001
While Napster remains shuttered, another search and play service offers a compelling alternative for getting your online music fix. MSN Music is about as opposite from Napster as you can get -- and for both music fans and musicians, that's a good thing.
The Extreme Searcher's Guide to Web Search Engines
SearchDay, July 19, 2001
The Extreme Searcher's Guide to Web Search Engines is one of the most comprehensive, authoritative and just downright useful guides to what goes on under the hood of the major search services.
Search Engine Cloaking: The Controversy Continues
SearchDay, July 18, 2001
SearchDay readers sound off on the ethics, practicality and effectiveness of search engine cloaking. Also links to original article with feedback defending cloaking.
New Search Patents
SearchDay, July 17, 2001
Newly issued patents offer fascinating glimpses of emerging search technologies -- including those that may pose serious threats to your favorite search engine.
You can also sign-up for SearchDay on that page to get more articles like these during the workweek, along with search engine headlines from across the web.
Search Engine Articles
eBay Listings Now On AltaVista
SiliconValley.internet.com, August 1, 2001
eBay listings are now being integrated into AltaVista's shopping results.
AltaVista: In search of a turning point
News.com, July 31, 2001
A look at the rise and decline of AltaVista, with some estimates on how much the company may be earning and losing, along with future directions.
GoTo.com Spins Off Auction Business
InternetNews.com, July 30, 2001
GoTo is staying focused on paid listing by selling off its auction service.
Search technology gains recognition
InfoWorld, July 30, 2001
Short news on new search and categorization technologies for corporations from Verity and SmartLogik.
The author of the new book Search Engine Positioning answers questions and discusses the topic. Part 1 is the first URL, Part 2 the second.
Increase your Click-Throughs with Killer Title Tags
SearchEngineGuide.com, July 24, 2001
Using the terms you want a page to be found for in the HTML title is essential. Unfortunately, some people don't take the extra step of ensuring the title is also written to attract clickthrough. Good tips on making your titles standout.
eRetailer Grows Revenues 24% in Six Months Through Search Engine Optimization
ConsumerMarketingBiz.com, July 20, 2001
A camera retailer goes from getting four percent of his traffic from search engines to 24 percent, after using the Inceptor search engine optimization software (now called Excedia). In addition, conversion rates were higher than with other online advertising efforts.
Optimizing for International Search Engines: Part 2
ClickZ, July 18, 2001
Many suggestions from readers of Paul Bruemmer's column on important search engines for various countries and regions of the world. Also links to his original article on the topic.
Tasty New Search Engines
About.com Web Search Guide, July 16, 2001
Review of two new crawler-based search engines, Teoma, which I wrote about last month, and Wisenut, which sounds similar to Teoma and well-worth a look.
Yes, Virginia, Viral Marketing Really is a Key Search Engine Optimization (SEO) Component
Viral Marketing Monthly, July 2001
By running a viral marketing campaign, you are really running a link building campaign -- and thus getting the benefits that can bring from search engines.
How do I unsubscribe?
+ Follow the instructions at the very end of this email.
How do I subscribe?
+ The Search Engine Update is only available to paid members of the Search Engine Watch web site. If you are not a member and somehow are receiving a copy of the newsletter, learn how to become a member at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to firstname.lastname@example.org
I need human help with my subscription!
+ Send a message to email@example.com. DO NOT send messages regarding list management or membership issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at