What started as a great idea for linkbait turned into a comprehensive analysis of Web analytics packages rivaling research from the largest firms.
Stone Temple Consulting has published the 55-page 2007 Web Analytics Shootout, the results of a nine-month study of seven top Web analytics packages on four sites. The report looks at performance, accuracy, and capabilities of Clicktracks, Google Analytics, IndexTools, Omniture SiteCatalyst, Unica Affinium NetInsight, Visual Sciences' HBX Analytics, and WebTrends.
The report is not intended to identify a "best" package, but rather to identify the strengths and weaknesses of the various applications so that potential buyers can understand how their own site structure might work with or against them, according to Eric Enge, president of Stone Temple Consulting. A major section of the report is a qualitative comparison of the various packages to help webmasters find the right fit for their situation.
"We've exposed a lot of things that show the scope of differences between the packages, and to start to show which application might be right for which circumstances," Enge said. "The report comes from nine months of living with the packages, as well as extensive discussions with vendors."
The idea began with an idea Rand Fishkin wrote about on the SEOmoz blog. Enge answered the call for someone to compare analytics packages, and spent the next nine months immersed in the project. Enge published an interim report at the eMetrics Summit in May. He revealed some of the findings in the final report at an Issues in Analytics panel at Search Engine Strategies San Jose last month, and published the final report last week.
He has also posted several "10 cool things" lists gleaned from his conversations with the analytics vendors themselves. So far, he has lists covering HBX Analytics, NetInsight, Clicktracks, and IndexTools.
Enge started with the assumption that all of the packages would have some inaccuracies, and would show some variance between their results based on the differences in the packages and their implementations. The goal of the study was to find out how inaccurate they were, what was contributing to the inaccuracy, and how it could be minimized.
As it turns out, there is a great deal of variance between analytics packages measuring traffic on the same site, as much as 20 percent in some cases. Those differences can be caused by the counting method of each package, as well as the impact of many implementation decisions made by the user. For example, the definition of daily unique visitors may include all visitors in the past 24 hours, or only those in a given calendar day.
"If you're pursuing high-value analytics, you need to understand what your sources of variance are," he said. One way to help do that is to run multiple packages at the same time, at least initially, to see where the trouble spots may be. For webmasters, the key lesson is to spend a significant amount of time in planning an implementation beforehand, and to look for areas where issues may arise early on, so that they can be addressed, Enge said.
While there is variance between packages, that does not mean that the results they deliver are not useful, Enge said. As long as you understand what is being measured, and not comparing one package's results directly to another's, any analytics package can deliver useful results.
"As Jim Sterne is fond of saying, if your yardstick measures 39 inches instead of 36 inches, it's still great to have a measurement tool," Enge said.
The difficulty comes when someone using a 39-inch yardstick wants to compare results with someone using a 36-inch yardstick, such as when companies are partnering or acquiring one another, or when an agency or vendor is comparing results with your own. In those cases, the two need to calibrate their analytics, so that they are talking about the same numbers, he said.
We report the top search marketing news daily at the Search Engine Watch Blog. You'll find more news from around the Web below.
- Time-Based SEM Analysis, SEM Angel
- Web Analytics for Facebook: Applications, FBML, and Facebook Engagement?, Web Analytics Demystified
- Big Brands - Big Issues - Small Summary, YOUmoz
- The Anatomy Of Compelling Search Ad Copy, Search Engine Land
- Why You Should Embrace the New Social Media News Release, Search Engine Guide
- Top 21 Signs You Need a Break From SEO (2007 version), Small Business SEM
- All I Really Need To Know About Search Marketing I Learned In Kindergarten, Search Engine Land
- Video Model Yet to Be Determined for Yellow Book and Other Local Ad Providers, ClickZ
- An Old Trademark Suit is Settled in Google's Favor, ClickZ
- Following your money in Facebook, Inside Facebook
- A Bill of Rights for Users of the Social Web, Marc Canter
- College.com: Returning To Facebook’s Roots, TechCrunch
- Videos: Internet Law with John W. Dozier, Jr. (video 7:50), Web Marketing Today
- MSN China Wants To Be Out Of Joint Venture, Unofficial SEO Blog
- No, Junior, You Can't Be Our SEO Team Leader (Yet), Search Engine Land