Conductor this week released a whitepaper, "Why You Shouldn't Trust comScore's Numbers for Search Engine Market Share [Data]". It raises the issue that comScore's search share data they release monthly is incorrect, based on Conductor's own research of 100 million organic search visits.
I spoke with both comScore and Conductor to get to the bottom of the discrepancies and why that the data between the two is so different, especially with Conductor's data seemingly more closely in line with what many webmasters also report seeing.
Conductor's study covered 100 million organic search visits across 63 different sites. While the search visits number is high, the number of sites used is a small sample size. Nathan Safran, director of research, Conductor, feels that the sample size is large enough with the large number of visits covered.
"I'm not aware of any comparable study of similar sample sizes," Safran said.
Conductor didn't disclose the types of sites or market areas for the 63 sites, raising questions about how competitive – or not – the same sites were.
"The sites analyzed were a mix of B2B, B2C, and publisher sites and were both highly and moderately competitive sites were represented," Safran said.
Were even a few of the sample sites impacted by Google Panda or Penguin, possibly skewing the results?
"We did not see a significant impact from Panda/Penguin in our sample set of 100 million+," Safran said. "I think given a sample set of this size I'd expect to see the impact of Panda/Penguin to be normalized."
Now looking at the figures – comScore's latest data found Google with a 67.6 percent search market share while Conductor saw an 85 percent search share with their sample. What could cause such a difference?
People have commented for quite some time about the differences people see for Google's share compared to traffic coming to their own site. ComScore said there are good reasons why the figures are so far apart.
ComScore's figures reflect the share of actual searches performed, while Conductor's figures reflect actual click-throughs from the search results. So search behavior, and how those searchers interact – or not – with the results they see, impacts comScore's search share.
"For example, often times I may run a search for a phone number, a stock price, the weather, or something along those veins, and the SERP answers my question without the requirement of a click," said comScore's Eli Goodman, who leads the Agency Business Development team and serves as comScore’s Corporate Evangelist for the Marketing and Communications group. "So in this example we could have just had three or four searches, but no clicks."
Multiple searches with zero clicks is only one scenario. There is also the opposite, where a single search and result in multiple clicks on that search result page.
"Another example is running a single search for a term, and then clicking on multiple links from the SERP to comparison shop, read different articles on the subject, etc.," Goodman said. "Think of times you run a single search, click on a link, read something on that destination page, hit the back button and return to the Google results, and then click on another link....to further this example, you could have clicked on two separate landing pages within the same destination, such as two different product pages on Amazon that are selling the same product, and both came back in you search results on that single search. In this example you have performed one Google search, but Google delivered two clicks to Amazon."
The role of mobile search also skews the results when comparing Conductor's results with comScore's. ComScore's data only accounts for desktop searches, as they don't track mobile searches.
The majority of mobile devices have Google as the default search engine, including Android and Apple devices, with the only notable exception being Windows phones. Conductor didn't differentiate between mobile versus desktop, and didn't have the breakdown when I asked if it was available.
"It is also important to note that tour search share numbers are only for PC/desktop searching, as we do not report on mobile search share at this time," Goodman said. "So if you also factor in the fact that mobile search clicks are overwhelmingly coming from Google via Android smartphones and iPhones with default to using the Safari browser with Google as the search engine, you can also see why the figures are so disparate."
The direct results and percentages of what each study found is valid, however comparing the two very different methods of calculating search share mean that while both are valid, neither can be used to discount the other's findings.
"Scenarios such as these come up all of the time," Goodman said, "which is why comparing search share to click share is misleading when you are trying to rectify the numbers from different companies with different data sets without taking all of these behavioral scenarios into account."
Is comScore's data invalid and a reason for webmasters to be concerned about the validity of their search share results? No. But Conductor's findings are also pretty consistent with what many webmasters find when it comes to actual click-throughs to a site from Google search results.
However, until there is a study that compares the identical metrics, we have to assume that the comScore and the Conductor data is both valid, just that it can't be compared against each other.