Stone Temple Consulting (STC) has released the first findings from its 2007 Web Analytics Shootout: a competitive comparison of 7 Web analytics vendors, whose software was run head to head across 4 different sites.
Analytics packages included in the study were Clicktracks, Google Analytics, IndexTools, Omniture SiteCatalyst, Unica Affinium NetInsight, WebSideStory HBX Analytics, and WebTrends. Data recorded included Visits, Unique Visitors, Page Views, as well as a variety of content groups. The traffic data varied significantly between packages, which STC President Eric Enge attributes to factors like implementation decisions, and first-party vs. third-party cookies and their deletion rates.
While none of them can be said to be more accurate than the others, any analytics program, implemented properly, can provide useful traffic data that can be compared to itself to produce useful results, Enge said.
“As Jim Sterne is fond of saying, if your yardstick measures 39 inches instead of 36 inches, it’s still great to have a measurement tool,” Enge writes in the report. “The yardstick will still help you measure changes with a great deal of accuracy. So if tomorrow your 39 inch yardstick tells you that you are at 1 yard and 1 inch (i.e., 40 inches), you know you have made some progress.”
In addition, the report was able to highlight tendencies of each package to report high or low. For instance, HBX Analytics had a tendency to report lower traffic numbers, while Clicktracks and Google Analytics tended to report higher. These were not absolute in all cases, cautions Enge, but they do give you something to compare to understand the nature of the yardstick you’re using, he said.
The various packages measured pageviews more consistently, since that metric is not as affected by the way the software is set up, or by cookie deletion among visitors.
The Interim Report also contains an analysis of third party cookie deletion rates compared to first party cookie deletion rates, showing that third party cookies get deleted at a rate that’s 13 percentage points higher than first party cookies. So if first party cookies get deleted 3% of the time, third party cookies would be deleted 16% of the time. Cookie deletion can greatly affect accuracy in counting unique visitors, and visits, and understanding whether or not a user is a repeat visitor.
There is much more granular data to be found in the report, which is available at the Stone Temple Consulting site. STC is also working on a final report from the study, to be published in July, which will include an analysis of the features,
functions, and usability of the 7 packages.