About The Update
The Search Engine Update is a twice-monthly update of search engine news. It is available only to Search Engine Watch members. Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.
In This Issue
+ Conference News
+ Time For The Search Dividend?
-- (full story online, link provided)
+ Inktomi Launches New Paid Inclusion Program, Search Improvements
+ Google Adds Languages, Phone Book And More
+ AltaVista Submission Changes
+ I-Search Paid Participation Survey
-- (full story online, link provided)
+ Excite Gets Search Refinement Feature, Paid Listings Coming
+ Interesting Search Engine Articles
+ List Info (Subscribing/Unsubscribing)
By Wednesday, I'll have an entirely revised page about Inktomi posted. It incorporates all the many changes that have happened with the service, over the past few months. Even if you've kept up with all the newsletters, it will probably be helpful to get an all-in-one refresh of how the service operates.
I'm also then going to move immediately to update the page on AltaVista, now that changes with that service have stabilized. Then I plan to do a series of additional updates throughout the Members-Only area. You can keep up with anything new that is posted by watching the Members-Only What's New page:
Members-Only Area What's New
Something else on my project list is a survey of the size of the search engine optimization industry. While it's relatively easy to get statistics on how large people believe the banner advertising industry is, I have never seen any statistics on what SEO is worth. I think this is because there is no major industry organization that monitors SEO.
I expect to post a survey form next month and attempt a self-reported survey. It would help to know what those of you in the SEO industry would feel comfortable answering and what you'd like to know from your colleagues. For example, I think everyone would like to know what the annual revenue of the industry is estimated to be -- which would mean people would need to answer this for themselves.
I'm also sure people would like to know who the industry leaders are, in terms of revenue -- but the problem here is that there's no easy way to audit the figures to watch for those who would inflate the size of their company (nor would that be the best way to judge the quality of a vendor). Another problem is that not everyone might want to reveal this publicly.
I suspect that the survey form will allow people to post information anonymously, though I would require email addresses as some way to qualify the information. I also plan to only survey Search Engine Watch members, which means the survey might be less encompassing, but quality would probably be more trustworthy.
Finally, one SEO company I talked with about the idea said they'd love to do it, as long as they could see the data, so they could get a sense of how they are placed within the industry. They weren't concerned about actual company names. They just wanted a sense of what companies outsourced work, versus those that had in house staff (and how many), revenues, payment models, etc.
That's a good idea, so I will post all the complete data (without company names, assuming these are gathered) in some way for everyone who participates in the survey.
Got ideas of what you want to know or what would make you feel comfortable about participating? Use the feedback form below and put "SEO Survey" in the subject line (please don't respond to this email -- it doesn't send mail back to me).
Search Engine Watch Feedback Form
The first two-day Search Engine Strategies conference was held last month in Boston, and judging from the feedback I received from attendees, it was a rousing success. I've included some articles about the conference in the Articles section at the end of the newsletter. My thanks to all the speakers, roundtable facilitators, sponsors and exhibitors who made the conference a success -- plus to all those who came, especially Search Engine Watch readers. It's always great to interact firsthand with people and hear about their concerns with search engines.
Our next conference comes to San Francisco on August 16th and 17th. We'll again have the first day focusing on search engines and promotion issues, with the second day featuring panels designed to help Internet searchers better understand how to use the search tools available to them. A preliminary agenda should go up toward the end of this month. Those interested in sponsoring or exhibiting should contact Frank Fazio Jr, [email protected], for more information. Attendees can signup to be notified of when more session information is available via the URL below:
Search Engine Strategies: San Francisco 2001
Time For The Search Dividend?
When the Cold War ended, there was much talk in the US about the public receiving a "peace dividend," since defense funding could be reduced. Similarly, we may be seeing the beginning of a "search dividend" coming to searchers, now that several of the major search engines are maturing their new business models. That means instead of the news from the search front being dominated by the latest ways to buy your way into search engines, you may instead be hearing more about actual search improvements. A full article exploring this potential trend can be found below:
Time For The Search Dividend?
The Search Engine Report, April 2, 2001
Inktomi Launches New Paid Inclusion Program, Search Improvements
Inktomi has rolled out a new paid inclusion program aimed at large content providers. In addition, the company has also unveiled new changes to how the service indexes and ranks web pages.
"Index Connect" is a program that offers cost per click pricing to those wishing to list 1,000 pages or more with Inktomi. In contrast, Inktomi's existing "Search/Submit" program, introduced in November, charges a per page fee. The new program is designed to be more economical for big publishers with lots of content.
"It's aimed at much larger sites than what we were doing with Search/Submit," said Troy Toman, Inktomi's general manager of search. "For them, per page pricing isn't good."
Among the initial partners are companies and sites such as eBay, Epinions, IDG and RollingStone.com. They can now ensure that selected content from their web sites is included in the Inktomi index and refreshed according to schedules that they determine. Without Index Connect, they would instead depend on Inktomi doing a generally random selection of documents from their sites and typically checking for changes only once per month.
Inktomi is also extending the program for free to charitable, educational and other not-for-profit organizations, allowing them greater control over their content. Examples of these included in the initial launch of Index Connect were KQED public broadcasting in San Francisco and the Hunger Project. Inktomi says that not-for-profits interested in participating in Index Connect should use the standard request form and indicated that they are non-profit. Arrangements will then be made for indexing.
The new program is another method for Inktomi to earn revenues from search, as well as being able to share those revenues with portal and search sites that make use of its services. However, the system does also let Inktomi better understand what content from various web sites that it should be indexing.
"By having a relationship directly with those sites, we can make sure we are doing a better job," Toman said. "It helps us be more intelligent about what we are crawling."
In particular, despite making use of link analysis to locate what it feels are the best pages on the web, Inktomi says it still needs site owner input about what content they think is important and how often it should be revisited.
"Having our WebMap and understanding the link structure [of the web” has gone a long way, but what we've realized is that we are pulling all we can out of the WebMap," said Andrew Littlefield, Inktomi's chief strategist of search.
For instance, at KQED, there are some web pages that are inaccessible to standard crawlers. By working with the organization, Inktomi now has access to URLs that let its crawler index important content from the site, the company says.
What's missing from the paid inclusion programs Inktomi offers is a way for what I call "hobbyists" to participate. These are people who produce great content out of personal interest or as a hobby. They aren't non-profits, but their sites generally are not run to make much money. While Inktomi's self-serve Search/Submit program is designed for smaller site owners, it may still be too pricey for them. So, how does Inktomi interact with these people?
One idea might be a low cost registration fee: US $10, $25 or $50 per year, for example. That might let a site owner get their home page included on a guaranteed basis, plus allow them to indicate to Inktomi which other pages they think are a priority to include. There would be no guarantee that these additional pages would be included, but the extra information might be used to more intelligently guide the Inktomi crawler, rather than let it randomly wander through the site. In a way, such a program would be a reverse robots.txt file, with the emphasis on what should be included, rather than excluded.
Inktomi's says it is aware of the hobbyist gap, and the company hopes to come up with some type of offering for this group, in the near future.
In addition to the new paid inclusion program, Inktomi also has rolled out changes to its search engine that it hopes will improve the relevancy of its results.
Chief among these is human modeling. Inktomi has been using an internal editorial staff to run massive numbers of searches and then select documents that they consider relevant. The company has then been tweaking its various relevancy controls to try and automatically match the human selections. In this way, the company hopes to model the human qualities of what's relevant into its ranking software.
"We didn't make huge changes in the algorithm," said Paul Karr, Inktomi's director of web search. "Essentially, what we are able to do is take the modeling technology and apply it to the database. We can fine tune it, experiment, and try to look at what's best."
In addition to the human modeling, Inktomi says it has improved its use of link analysis and is now also doing automatic proximity searching. For example, if you were to search for "george bush," it would favor pages with those words appearing on them in that order.
"Most search engines do not do some type of proximity, because it is very expensive computationally, but you do tend to get this sort of organic proximity without specifying it due to words in the title or the anchor text," said Doug Cook, Inktomi's director of web search engineering. "It's sort of a poor man proximity search."
In other words, while it may have seemed like major search engines have always done automatic proximity searches, that's been more accidental than intentional, according to Inktomi. For instance, a search for "george bush" would bring back pages containing that phrase at the top of the results because both words were in the important ranking area of the title tag, rather than because they were in the order you specified. That's no longer the case with Inktomi. Proximity searching is now always being done.
"This is the first time we've explicitly done proximity on all the queries. Essentially, we had to rewrite all of our core algorithms. Initially, there was a performance cost, but the team came up with some really clever new algorithms that were actually faster than our existing ones," Cook said.
Inktomi has also introduced index blending into its search results, which means that you may get information from the LookSmart directory, the paid inclusion index or the non-paid content that comes from crawling the web blended seamlessly in the same set of results. This also means that Inktomi could include news content, shopping content or other specialty search results into the results set.
Behind the scenes, query modeling is used to decide which databases to hit and when. For example, Inktomi has a collection of 500 million documents, but 110 million of these are kept in a "Best of the Web" index. A search for a popular subject, such as "Britney Spears," may be sent only to this smaller collection while a search for a more obscure topic might go against the entire index.
Unfortunately, there's no way to force Inktomi to search against a particular set of documents, nor can you even tell which partners may even prevent searches from digging as deep as possible. Inktomi does say that most of its partners do hit its entire 500 million page index, and its index of paid inclusion content is always queried by its US and European partners.
Inktomi Index Connect
More details about the pay-per-click inclusion program.
Inktomi Search/Submit Partners
More details about Inktomi's self-serve paid inclusion program.
Pay For Placement?
Past articles from Search Engine Watch about Inktomi's paid inclusion programs and other paid inclusion programs can be found here.
How Inktomi Works
A fully-updated guide to getting listed with Inktomi will be posted by Wednesday of this week, at the latest.
Inktomi Gets Relevant
PC World, March 14, 2001
Another look at Inktomi's relevancy and indexing changes.
Google Adds Languages, Phone Book And More
Want your search engine to talk to you like a Swedish chef? The new "Bork Bork Bork" option is just one of new language interfaces you can set for Google. The search engine has also unveiled a new page translation option and a phone book feature.
From the Google preferences page, users can now find options to have Google speak to them in Afrikaans, Catalan, Czech and Russian, which are just some of the many language options that have been added. Much of the work is now being done by volunteers, who provide translations for words such as "search," "cached" and "search within results" in the language they are working on.
In addition to traditional languages, volunteers are currently working on releases for Elmer Fudd, Hacker and Klingon. Bork Bork Bork is already live. It makes Google speak like the Swedish chef from the Muppets (not like the failed US Supreme Court nominee, Robert H. Bork).
The language interface options I've discussed control how Google talks to you. In other words, Google can make its "Next" link at the bottom of the results page appear in whatever language you've selected. However, that language option won't carry through to the pages you view from Google's search results. For that to happen, you need a page translation option -- and there's good news here, because Google's just added one.
Next to the title of each search result, you'll now see a "Translate this page" link for any page originally written in Italian, French, German, Portuguese or Spanish. Selecting the link will bring up a translation of the page in English. Using the preferences page, you can even cause Google to automatically translate the page descriptions it displays into English for any page originally written in one of the languages I've named.
Be aware that the program is still in beta, so you might encounter bugs. Google also plans to add new languages and features, in the near future.
Leaving languages behind, another new feature at Google is the "PhoneBook" offering. Just enter the first and last name of someone you know, along with a US zip code, and Google will bring back any matching telephone and address information available from public records at the top of the search results page.
Up to two matches are displayed, and if there are more than this, the rest are available by selecting the "More listings" option. In addition, the address information links to online maps.
Other ways to trigger phone book information include entering a name with a telephone area code, US city or US state. For example,"george w bush texas" brings back two Dubya's that live in Texas, though I strongly suspect neither of them is the US president. You can also enter just a phone number to see who it is registered to, turning Google into a cool reverse look-up directory.
Speaking of telephones, Google is also working on something it calls "Voice Search." This would allow you to speak into a telephone and see your results appear in a traditional browser or WAP phone. There's no release date yet for this service, but Google already has a partnership with BMW to build it into future versions of their cars.
In some other Google developments:
+ Google Korea has now gone live, and Korean users will eventually be automatically directed to it.
+ Google's AdWords program has now been released for its German site. You'll find a link on the Google Germany home page, with instructions in German on enrolling in the program.
+ Google, which has been providing backup results to Yahoo.com and 17 other Yahoo sites, is now also going to be powering secondary results at Yahoo Japan.
+ Dr. Eric E. Schmidt, currently chairman and CEO of Novell, has joined Google's board of directors as chairman.
Set your preferred language and other Google options here.
Translate Google Into Your Language
Hold on there, all of you who want to make Google speak Picard just so the search button will say "Engage." When you sign up, you have to select a language you are fluent in from a preset list. Once enrolled, that's the language you get to work on. You can't work on languages not already on the list, and you won't be able to work on any language you've not initially said you were fluent in. What if the language you want isn't on the list? Try contacting Google first using the email address on the translation FAQ page, tell them what you want to do, and see if they'll add it. If so, then enroll.
Translate Web Pages Automatically With Google
More information about the page translation feature can be found here.
Google Special Features: PhoneBook
More about the Google phone book.
Google PhoneBook Name Removal
The phone and address information Google provides comes from public records, but you may prefer to make it less accessible via Google. If so, use the form above. This will cause Google to suppress display of your information.
Novell's Schmidt joins Google at critical time
News.com, March 26, 2001
Schmidt is expected to add a firm business hand to Google, which also continues to predict it will be profitable later this year. Good information here on Google's business activities. For the first time that I've seen, Google discloses some actual figures as to how much it earns from its deal with Yahoo: around several million dollars per quarter. Google's income is also split roughly between search licensing, as with Yahoo, and advertisement sales on the Google site itself.
AltaVista Submission Changes & Other Developments
Forget everything you ever knew about submitting to AltaVista. A completely new system went up at the site about two weeks ago, one that promises faster addition of web pages to the index, as long as you are not using automated submit software.
When you access the Add URL page, it will display a submission code that must be entered. The code is a series of letters and numbers, but because they are displayed in a graphic format, automatic submission tools cannot read the information. As a result, AltaVista says the new system has stopped submission robots in their tracks.
This is important because submission tools generally send AltaVista so much spam that "good" documents get lost among the junk. For instance, AltaVista said at the recent Search Engine Strategies conference in Boston that virtually all of the submissions it receives come from robots, and that 95 percent of all submissions are considered spam. Thus, AltaVista explained, stopping the robots means that the good stuff can get through. This caused the audience of mostly webmasters, web marketers and search engine optimization specialists to break out into applause.
Rather than just percentages, here are some real numbers to consider. AltaVista says that:
+ Of the roughly 300,000 submissions it receives per day, half of these are invalid URLs, such as broken or dead links.
+ Of the remaining 150,000 valid URLs submitted per day, 95 percent of these (142,000) are near-duplicate pages, such as when someone submits hundreds of slightly altered doorway pages to target the same term.
+ This means that per day, AltaVista only receives 7,500 valid, unique pages it deems worthy of adding to its index via the Add URL system (naturally, the AltaVista spider finds many more pages to add on its own).
+ On a typical day, only 400 human beings are estimated to use AltaVista's Add URL page (which I found a surprisingly small number, but there you have it).
+ Since the good pages (7,500 per day) come mostly from humans, and there are about 400 human visitors using AltaVista's Add URL per day, this means an average 19 pages are submitted per day, per human visitor via AltaVista's Add URL page.
Coinciding with the new submission code, AltaVista has also removed the five pages per day, per web site limit it used to observe. There is no longer any limit as to how many pages you can submit to AltaVista, and the company says new pages should appear within a week.
After submitting five pages, the system forces you generate a new submission code. Don't worry about this. It is perfectly acceptable to use the new code to do another batch of five URLs, and so on, until you are done.
The "ransom note" look of the submission code is designed to combat against text recognition programs, but that also means the codes can be hard for even humans to read. AltaVista is working to make them more legible, it says. However, should you be unable to read the code, you can refresh the Add URL page to have a new one generated, which may be clearer.
Finally, AltaVista apparently has a "bulk" paid inclusion program in the works, as it was accidentally pitching this on its Add URL page for a short time. This would be ideal for those who want to submit a large number of pages without fiddling with the new submission code system. I expect to bring more details about this, as soon as they are available.
AltaVista has also debuted a new "Featured Site" paid link that appears above its search results. These are distinct from the "Sponsored Listings" links that appear at the bottom of the search results page and which come from GoTo.com. The Featured Site links are sold directly by AltaVista itself, at a $20 CPM rate. Linked to keywords, they are sold to the first advertiser requesting them. To sign up, I'd suggest contacting AltaVista's advertising department, URL below. To see examples, try a search for "hosting" or "diamonds."
In other AltaVista news, a bug has been preventing it from performing phrase searches properly. For example, if you enter several search terms surrounded by quotation marks, AltaVista is supposed to find pages that contain that exact phrase. This bug seems to be fixed now, but if you find it returning, AltaVista suggests prefacing the phrase search with a + symbol, such as:
+"when in the course of human events"
AltaVista has also launched four new regionally-oriented services: Basque, Galician, Catalan and one for New Zealand. In addition, its BabelFish translation service can now translate from Chinese, Japanese and Korean into English, and vice versa.
AltaVista Add URL Page
AltaVista Advertising Contact Page
AltaVista New Zealand
Offers nearly 3 million New Zealand-specific web pages, as well as access to AltaVista's global listings.
AltaVista BabelFish Translations
I-Search Paid Participation Survey
In March 2001, moderator Detlev Johnson of the I-Search mailing list asked his readers about their experiences with search engine paid participation programs, such as paid placement, paid inclusion and paid submission. The results can be found online, below:
I-Search Paid Participation Survey
The Search Engine Report, April 2, 2001
Excite Gets Search Refinement Feature, Paid Listings Coming
Excite's new "Zoom In" search refinement feature debuted on the site last month, and the service will be gaining paid listings later this month.
You'll find a new Zoom In button on the Excite home page, next to the regular search button. It also appears by the search box, on the results page. When you enter a word, pushing Zoom In will open a new window to suggest other terms that might help you refine your search. These are primarily a list of the most popular searches related to the word you originally looked for.
If you see a term you like among the Zoom In suggestions, select it, then push the "Apply and Search" button. Your selection will be sent as a query to Excite.
Zoom In also has a spell checker built in, so that if you are uncertain how a word is spelled, it may offer the correct way. For example, search for "leonardo dicapreo," and Zoom In will suggest "leonard dicaprio (sp)." The (sp) part indicates that this is a spelling correction. Similarly, try "geneology," and you'll instead get the correct spelling, "genealogy." Even "speling" gets "spelling" as a suggestion.
I did a quick check on the status on spell checking with other services, and here's a recap of those offering it in some way:
+ Ask Jeeves: Some common spelling mistakes will be corrected through behind the scenes mapping. For example, a search for "leonardo dicapreo" is changed to bring back results matching "leonard dicaprio." A search for "geneology" is also changed to the correct spelling. However, "speling" is not changed to "spelling."
+ Google: Spelling suggestions appear at the top of the results list, with the words "Did you mean" next to the correct spelling, shown as a link. Select the link to rerun the search with the correct spelling. Google caught "speling" as incorrect, but missed "leonardo dicapreo" and "geneology" (though it did get "genielogy"). This is a very limited test, so Google's spell checking may have been much better if I had done a comprehensive set of queries.
+ MSN Search: As with Ask Jeeves, there is behind the scenes mapping for popular terms.
+ Yahoo: Again, behind the scenes mapping for popular terms occurs, as with Ask Jeeves.
Other places such as Direct Hit and NBCi had correct spellings among their "related searches" suggestions, but unlike Excite's tool, they didn't call these out in a way that you could recognize them.
In other news, Excite was the last remaining major search engine not to offer some type of self-serve paid placement listings. Now they are coming to the service, through a partnership with FindWhat.com.
Similar to GoTo.com, FindWhat allows advertisers to appear higher in its search results if they agree to pay more money through an auction system. The distinction between the two has been that GoTo has distribution agreements with several major search engines to carry its results, giving its advertisers far broader reach. FindWhat's deal with Excite now gives its advertisers reach into one of the web's most popular search engines.
The paid listings are supposed to begin at Excite by mid-April. The top four results for a particular keyword at FindWhat will be carried at the bottom of Excite's search results page. They will also appear on Excite-owned WebCrawler.
Major Search Engines
Links to other major search engines mentioned above.
Search Assistance Features
Has more about Related Searches functionality at several major search engines.
GoTo Going the Distance, Despite Industry-Wide Turmoil
InternetNews.com, March 22, 2001
GoTo's still losing money, but it expects the loss to be less than predicted, with revenues to likely exceed $45 million in the first quarter of this year.
I-Search GoTo Special Edition
I-Search, March 13, 2001
People continue to vent about the increase in minimum bid price at GoTo.
Buying Your Way In
A guide to where paid placement and other paid participation listings appear at major search engines.
Search Engine Articles
Smarter Tools to Scour a Wider Web
Business Week, March 26, 2001
Review of four search companions for your desktop.
Making Search Engines Speak Naturally
Interactive Week, March 26, 2001
Focuses on two natural language search companies, EasyAsk and LingoMotors, who were at the Search Engine Strategies conference in Boston.
Portals Getting Creative With New Ad Units to Lure Traffic
DMNews.com, March 26, 2001
I'll be looking at the new Ask Jeeves advertising options in the near future, but in the meantime, here's a recap of some new programs.
Ask Jeeves Mixes Ads with Content
Internet World, March 23, 2001
More about new "Branded Response" ads from Ask Jeeves.
Do Portals Still Matter to E-Commerce?
E-Commerce Times, March 23, 2001
Online shoppers now know that all journeys needn't start at portals, yet portals still remain one of the top places where they'll turn to for product purchasing.
MusicCity Emerges as Top Napster Alternative
InternetNews.com, March 21, 2001
As Napster struggles in the face of a court-order restricting what files it can offer, free MP3 fans are flocking to MusicCity. The question is, how long until the music companies turn their attention MusicCity's way?
Search Engine Strategies Hits Hub
Boston.internet.com, March 21, 2001
Recap of my presentation on the basics of submitting to search engines.
Searching for Solid Ground at Terra Lycos
BusinessWeek, March 16, 2001
Challenges Terra Lycos faces in taking on both Yahoo and a declining ad market.
A Linking-Campaign Primer
ClickZ, March 15, 2001
Building links can help with search engines and traffic in general. Here are tips on how to approach a link building campaign.
Terra Lycos bets on portal power
News.com, March 14, 2001
Despite pessimism about the Internet in general, Terra Lycos says it will break even or post a profit this year.
Portals In Peril: Excite@Home
Forbes, March 12, 2001
Just how dedicated is Excite@Home to maintaining the portal that makes up the first part of its name? There are suggestions that Excite might be cast loose, but there are also arguments for keeping it.
Yahoo's Head of Sales Resigns
InternetNews.com, March 14, 2001
Yahoo loses more executive staff, this time its chief sales and marketing officer.
iWon: What Value for the Information Professional?
Online, March 2001
Review of the iWon service, from the perspective of the professional researcher.
Interview: Shari Thurow
Interviews.com, March 2001
Shari Thurow is one of the Search Engine Strategies conference's most popular speakers, covering ways to make existing content in a web site more friendly to search engines. A succinct summary of her tips and advice on achieving good rankings.
How do I unsubscribe?
+ Follow the instructions at the very end of this email.
How do I subscribe?
+ The Search Engine Update is only available to paid members of the Search Engine Watch web site. If you are not a member and somehow are receiving a copy of the newsletter, learn how to become a member at: http://searchenginewatch.com/about/subscribe.html
How do I see past issues?
+ Follow the links at:
Is there an HTML version?
+ Yes, but not via email. View it online at:
How do I change my address?
+ Send a message to [email protected]
I need human help with my subscription!
+ Send a message to [email protected]. DO NOT send messages regarding list management or membership issues to Danny Sullivan. He does not deal with these directly.
I have feedback about an article!
+ I'd love to hear it. Use the form at
This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) will bring together the industry's leading online marketing practitioners to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, the comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!