Getting the measure of Web analytics: The case of the 39-inch yardstick.

Want to boost site traffic 150% overnight? Well, if you’re measuring traffic with Unica NetInsight, switch to ClickTracks. Or if you want an even bigger leap in the figures, install Google Analytics and switch off your HBX Analytics software.

If only life were so simple. The truth is, even ignoring the dubious logic (not to mention ethics) of such a move, in the world of web analytics even the level of inaccuracy is unpredictable.

Eric Enge of StoneTemple Consulting recently released his 2007 Web Analytics Shootout Final Report. It's a formidable chart- and table-laden document detailing the often startling findings emerging from a comparative investigation into major web analytics platforms.

The study, which took several months, involved setting up and running multiple analytics packages simultaneously on four participating sites, and then comparing the reporting of key site metrics. The analytics platforms coming under the microscope were:

• Clicktracks
• Google Analytics
• IndexTools
• Unica Affinium NetInsight
• Visual Sciences’ HBX Analytics (aka WebSideStory)
• Omniture SiteCatalyst
• WebTrends

Although the main thread of the report — that all analytics packages are not created equal and do not measure equally well — will come as no surprise to web analytics professionals, StoneTemple’s systematic appraisal goes a long way towards studying and quantifying the variances between analytics solutions, and their causes.

Web analytics findings

Among the key findings:

• Web analytics packages, installed on the same web site and configured the same way, can produce radically different numbers. The StoneTemple study reveals that in some cases the analytics package showing the highest numbers reported 150% more traffic than the package reporting the least traffic.
• The biggest source of error in analytics is implementation error. StoneTemple likens a web analytics implementation to a software development project, requiring the same tough standards of scrutiny and testing if it is to produce credible reports.
• The positioning within the web page of the JavaScript used by web analytics software is critical. Tracking code at the bottom of a page can get missed as users hurry through the site. The speed at which your tracking code executes also has a significant effect on measurement accuracy: for example, an execution delay of 1.4 seconds can result in the loss of 2-4% of measurement data.
• Different views on what is being counted, specifically how a  user ‘session’ is defined, can have a profound effect on reported numbers. Measurements of page views, on the other hand, tend to vary less between analytics packages.
• The differences in measurement make comparing traffic between sites, and certain other key metrics, ‘an almost fruitless exercise’.
• Using the principle that even a flawed measuring tool is better than no measurement at all, the study concludes that it’s important to understand what your analytics installation is good at, and what it isn’t, and to then focus on the ‘good’ numbers. A/B and multivariate testing, PPC campaign optimisation, organic SEO campaign optimisation and the segmenting of visitor traffic are all areas where, in the study’s view, even an imperfect analytics tool is very worthwhile.

While much of the report’s depth of detail will be of greatest interest to analytics professionals, some insights cannot be ignored by any web professional:

• Certain web analytics packages do tend to ‘under-read’ or ‘over-read’, and while this may not matter too much for any one site — after all, relative measurements will still be meaningful — it can make simple comparisons between websites somewhat treacherous. Be careful, too, with how you interpret the traffic figures reported by web publishers trying to sell you advertising: even without their knowledge, the figures could be over-optimistic — or conversely, you could be getting a lot better value than your publisher realises.
• Even when you know that a particular site of interest has the same analytics software as your own, the StoneTemple study clearly shows that the way the software is installed can have a significant skewing effect one way or the other.
• Have you just completed a major migration from one content management system to another? Changes, planned or incidental, in where the tracking code now falls within each web page can have you scratching your head when you look at the post-migration traffic figures.
• If you’ve recently changed analytics packages, or are planning to, take the time to understand the metrics changes that will come so that you’re not caught out by abrupt shifts in reported uniques or page views.

Let’s agree to disagree: Just how much can analytics reporting differ?

Interpreting these figures: “Under the operating and installation conditions of Site X, the biggest difference in the reported metric was between Analytics Package A and Analytics Package B, which differed by the percentage shown.” - Raw data source: www.stonetemple.com