Search Satisfaction And Behavior Results Released
From The Search Engine Report
April 4, 2000
For the past three years, NPD New Media Services has conducted what I'd consider to be the most extensive survey of search engine user satisfaction available. Every three months, some users at the major search engines are selected randomly by NPD and asked about their visits. The results are fascinating, but they've not been available to the general public until now.
The survey is funded and conducted on behalf of the major search engines, and the results have been used for their own internal purposes. Occasionally, bits and pieces have been released, usually when a search engine performs well in a particular area. For instance, after the latest survey was finished in December, Google announced that 95 percent of its users find what they are looking for all or most of the time, making it number one for that category.
As you'd expect, search engines that rank poorly in some areas do not announce those findings. In fact, some search engines have specifically insisted that comprehensive results of the survey not be released to the general public. As a result, I've been unable to report on the standings.
That's a shame. Numerous search engines representatives have spoken to me about the desire to have some type of mechanism to rate them in terms of relevancy. That's an incredibly difficult task, as relevancy can change by person and by search query. Yet the NPD survey provides one such measure, and I think the search engines should all agree to an open release of key data. It may be embarrassing to those that rank poorly, yet it would also serve as an incentive for them to do better.
While I can't provide comprehensive search engine-specific figures at the moment, I am now able to provide a recap of how the industry as a whole is doing. The answer? Not as well as in the past. Overall, the success rate of finding what you are looking for most of the time or all of the time has dropped to 77 percent, the lowest point since the survey began in the Summer 1997.
What's happened? According to Lisa Manuzza, Director of NPD New Media Services, the main culprit seems to be the emergence of more cluttered and complicated results pages.
"By leaps and bounds, what we hear is that there is too much on the pages," Manuzza said. "People are all confused. 'I just want to know what the results are,' they say."
Could other factors be behind the drop? For instance, if more new web users are being surveyed, they might be less sophisticated about searching, which could cause the success rate to plunge. But Manuzza said demographic factors like this don't appear to be to blame. Likewise, the addition of new search engines to the survey such as Google, Ask Jeeves and GoTo could potentially have caused a slump, if these services themselves earned bad scores. But that's not the case. In fact, these newer services have actually helped keep the overall success rate from slipping further, Manuzza said.
There is some good news. Preliminary results from the Winter 2000 survey are nearly complete, and they show a strong increase in the search success rate, bringing it to its highest point since Fall 1997. Expect a closer look at this change in the future, when the final results are available.
The survey also examines what people do when their searches fail. Nearly 80 percent of people try searching again in a different way at the first search engine they use, rather than trying the same search at a different search engine. This comes from a belief that all the search engines provide basically the same information. That's certainly not true, but the impression is strong enough to keep people from moving elsewhere.
"You'll even see that people are less likely to jump ship than in the past. 'I'm not going to get anything better anyway,' people have said," Manuzza explained. Another factor is the time investment users make, Manuzza said. They get comfortable with a service and understand how things are laid out, making them less inclined to leave.
That should be a warning to some search engines that are constantly changing how they operate. The more they tamper, the more likely they'll impact the comfort level of their users.
Another search behavior surveyed is how we pose our questions. While asking questions is the least popular method, it has seen a huge increase recently due to what I'd call the "Ask Jeeves" factor.
All the major search engines can accept natural language, question-style queries. However, Ask Jeeves explicitly encourages its users to do so. Many users were first exposed to this encouragement when Ask Jeeves began providing some results to AltaVista just over a year ago. Even more have been exposed as the Ask Jeeves site itself has rocketed higher in terms of popularity. Now the Ask Jeeves factor of encouraging questions now seems to have rubbed off everywhere.
"That was non-existent, a year ago. I didn't have any numbers for any of the sites," Manuzza said, about the rise in question asking. "AltaVista was the first to grow, and now, as more people are getting more comfortable with the question asking thing, this was first year I was putting in numbers for practically every guide."
Multiple word queries are still the most popular method of seeking information, but they've lost some ground to single word searching. In the past, single word searching has been bad, since it doesn't provide enough information to help the search engine understand what the user is seeking. But now most search engines have "Related Searches" features. This allows you to enter a single word, then click on a list of multiple keyword suggestions in order to refine your search. This is one reason why single word method seems to be growing in popularity. It allows users to be more comfortable knowing that they'll narrow in on what they are looking for, Manuzza said.
NPD Search and Portal Site Study
Charts and more details from the latest survey.