The Challenges of Measuring SEO Success, Part 1

Over the past five years, I’ve had to explain the state of an SEO campaign at hundreds of meetings. During that time, both the data that was made available to me to analyze, and the repercussions of what that data means, has changed dramatically. In addition, I’ve learned a lot about the challenges of explaining to non-SEO folks why we do what we do, and how the data that we see and what it says drives ongoing SEO strategy.

SEO Success Measurement Challenges

In the old days, one simply looked at total organic traffic from month to month as a primary metric of success. Couple that with a rankings report across “x” number of keywords and you have what served as a “typical” SEO report for most people.

As we have become more sophisticated over the years, we now correlate rankings and traffic so we can identify which ranking changes actually had an effect on traffic. This prevents SEOs from needlessly spending effort on addressing changes that don’t have an effect on bottom line.

One of the biggest challenges in any SEO campaign is how to provide maximum ROI for a client on an ongoing basis. Those decisions need to be data-driven, but we need to be measuring the right metrics.

Brand vs. Non-Brand Keywords

So in addition to correlating specific ranking changes with organic referring traffic levels, and in order to understand SEO performance, we need to segment brand versus non-brand keywords. Doing so allows us to understand whether the overall traffic levels changed as a result of better saturation across generic terms (indicating more market share exposure) or if more awareness and demand has been generated in the marketplace as a result of non-organic search online marketing and offline marketing efforts.

Understanding traffic variances in lieu of changes in demand to brand-related terms and correlating those changes with marketing efforts allows us to better understand real SEO performance, and also can be used as additional data to evaluate the relative success of all marketing initiatives (by comparing which events have the biggest impact on brand related organic search).

For example, it isn’t uncommon to see surges in brand-related search due to large TV ad campaigns, appearances on radio or television shows, press releases with a lot of exposure, company acquisitions or major newsworthy events, significant industry promotions, and several other potential marketing activities and events. Actually, spikes in brand-related search volume can be used as a KPI in branding campaigns that aren’t even focused on SEO.

Long Tail Performance

In addition to understanding the difference between brand and non-brand related traffic, it’s important to understand the long tail performance of high priority keywords within each of these two segments. Activities performed in the “Continuous Phase,” which are targeted toward a particular keyword phrase, will often have an effect on longer tail permutations of the same phrase.

So if you target “dog food” in your optimization efforts, you’re also helping to optimize for all relating permutations of “dog food” like “dog food for sale,” “discount dog food,” or other permutations found on your site. Therefore, you need to evaluate the long tail performance of your most important brand and non-brand words — especially those targeted by your ongoing optimization efforts to truly understand the impact that those efforts are having on performance.

Year-Over-Year Comparison

So now we’re measuring overall organic traffic, segmenting it by brand and non-brand related traffic, and within each segment we then analyze the long tail performance of high priority keywords. Then we compare these numbers to year-over-year numbers to understand performance.

Comparing month-over-month numbers is useful to understand trends and identify abnormal patterns, but performance needs to be measured from a year-over-year standpoint to account for seasonality. Much the same as most organizations, compare year-over-year sales numbers by month to evaluate performance and growth.

These year-over-year comparisons can be challenging in the event that you have a client or your business changes content a lot. A dramatic shift in campaign keyword focus makes year-over-year numbers difficult to measure against one another because the top referring keywords may be very different and therefore the comparison really doesn’t measure performance across the same metrics.

Another challenge is that for large sites, traffic fluctuations can occur across a number of terms that were outside the scope of the campaign (or weren’t targeted by ongoing efforts). These fluctuations can sometimes overshadow other important gains made across high priority keywords where traffic volume isn’t necessarily more important than quality based on conversion numbers. For this reason, it’s helpful to further refine our reporting within the long tail brand and non-brand segments to a “campaign level.”

Organizing related keyword groups into campaigns, similar to paid search campaigns, helps us understand SEO performance within high priority keyword segments that may not be evident for large sites with tons of data. Especially in situations where the amount of resources that is being spent on ongoing SEO activities only affects a small segment of keywords.

Correlating rankings to traffic, organizing organic traffic referring keywords into two groups, brand and non-brand, and then measuring the long tail performance of keywords within those groups and breaking them into related campaigns is an essential part of reporting and understanding SEO success. But most of the process that we have discussed is focused on only half of the equation, which is how to report on traffic from an SEO perspective.

Organic Search Rankings

And while traffic is the bottom line in SEO, traffic levels aren’t always the only indicator of success. The other half of the equation is organic search rankings.

Demand across any keyword or group of keywords can dramatically change from month to month and year to year based on market conditions. For example, it isn’t surprising that with the current economy, many shopping-oriented keyword phrases have less volume in June 2010 than they did a year prior.

Traffic variances often don’t tell the full story on whether rankings have increased, so it’s important to understand traffic patterns, and it must be done in context with search rankings. But measuring search rankings has gotten progressively harder over the years because localization and personalization vary search results based on the user.

Next time, we’ll examine the recent Google Webmaster tools data set and its profound implications for measuring search rankings, as well as reporting and understanding SEO in general.