Google's Matt Cutts is skeptical about the latest Hitwise data that showed Bing provided users with more accurate results than Google.
As we reported earlier this week, Bing had its highest search success rate in January (81.54 percent), while Google had a 65.58 percent success rate. Hitwise defines a successful search as one that results in a visit to a website from a search engine's result page.
On Google Buzz, Cutts last night voiced his objection to how Hitwise reports data by reposting an old 2009 comment:
It sounds like Hitwise's definition is "A successful search is defined as one where the consumer leaves the search engine after performing a search." In another words, the user does a query and then goes somewhere else. That doesn't sound the same as success to me; it just sounds like leaving the site.
Are you able to determine whether the user clicked on a search result vs. just left the search engine to go to another site? There's a difference between an abandoned search and clicking on a search result, but both result in the user searching and then going to a different site. By Hitwise's definition, wouldn't doing a query on Bing and then going to Google or Yahoo count as a "successful search" on Bing? I'm also assuming that you can't measure if the user got the information that they needed from the search results without needing to click to another site.
I think the phrase "successful search" is considerably less accurate than "left the site after searching," because someone can leave a site for lots of different reasons.
Are the Hitwise numbers flawed? Probably. All sample data has flaws, even Google's data can be quite flawed.
Hitwise bases their numbers on data from 10 million U.S. Internet users from ISP data:
Hitwise collects aggregate usage data from a geographically diverse range of ISP networks and opt-in panels, representing all types of Internet usage, including home, work, educational and public access. To ensure this data is accurate and representative, it is weighted to universe estimates in each market. Because of the extensive sample size (25 million people worldwide, including 10 million in the US), Hitwise can provide detailed insights into the search terms used to find thousands of sites as well as a range of clickstream reports, analyzing the movement of visitors between sites.
True, Hitwise doesn't take into account if a Google search provides an answer for a user directly in the search results. For example, if you search for [weather], you probably won't have to click-through to the Weather Channel, Weather Underground, or AccuWeather sites because Google gives you that info in the SERP:
Some have also argued that Bing and Yahoo users are at a "novice" level when compared to more experienced Google users.
However, the job of a search engine traditionally has been to send users to where they want to go by giving them accurate results. For whatever it's worth, Hitwise's numbers say Bing is more accurate than Google.
Cutts' argument is also odd considering Google AdWords advertisers have to pay every time a user clicks on an ad. So, is every ad click on Google "successful"? Or should Google also re-evaluate how it determines a successful ad click?
Speaking of Bing, toolbar startup Conduit, which dropped Google for Bing in December, may have been another factor in Bing's recent search market share gains, TechCrunch suggests. Looking at Alexa and comScore data, Bing gained 138 million page views while Conduit lost 165 million after Conduit began sending searches to bing.com.
And comScore also has reported that Bing's market share grew to 13.1 percent in January, up from 12 percent in December. For the same period, Google's dropped to 65.6 percent, down from 66.6 percent.
What do you think? How should we define a "success search"?