Google Panda & Penguin: A New Way for SEOs to Measure True Impact

panda-penguinEach time Google rolls out an update or refresh of their infamous Panda or Penguin algorithms, SEOs go wild. Anecdotal stories and panic spread like wildfire across blogs, forums, chat rooms and social media networks. The impact of these updates is measurable on individual websites simply through analysis of web traffic.

More challenging, though, is getting a bigger picture of the impact, beyond the percentage of queries Google says are affected in their blog post announcements or tweeted “weather updates.”

Clay Cazier, senior director of SEO Strategy with PM Digital, offers a methodology for measuring the aggregate downstream impact of what he calls the Google Zoo, though he admits it isn't yet perfect.

“I’m not a statistician, so I’m sure there are going to be questions around the data. I think I solved most of the issues, but let this be the first step toward a more objective step toward a better look at the downstream impact, beyond opinion surveys. ” Cazier told SEW.

So what is it?

Cazier’s Impact Assessment compares year-over-year (YoY) and month-over-month (MoM) drops in organic click volume on Google, segmented by verticals. He developed a custom quantifier dubbed Google Organic Click Turbulence, or GOCT, using comScore Search Planner data, to measure negative changes in Google organic clicks.

An Interesting Predicament: Are SEOs Sabotaging Their Own Success?

The purpose of his research was to determine whether Panda and Penguin actually had the negative impact reported by SEOs. Early in 2012, digital marketers were surveyed to determine which of Google’s search changes had affected their business. Fifty-four percent voted for Panda. In May, 65 percent of SEOs reported less traffic after April’s Penguin update.

Do opinion-based surveys reveal the true state of search after an algorithm change, though? “A desire to measure the perceived, negative effect of Google’s updates vs. the true, statistical effect is the impetus for this whitepaper,” says Cazier. He hopes the GOCT quantifier can be discussed and refined by the SEO community.

His motivation for undertaking the analysis is a relevant question: “Given the fact that Google updates impacted at maximum 12-13 percent of U.S. searches, how is it that 40 percent of SEOs and website owners are reporting an impact?” he said.

“This fear and doubt Google has put into organic with these updates has certainly resulted in increases in paid activity. There may be an echo-chamber effect, where activity in forums and on blogs results in decision-makers moving budget to paid,” said Cazier. “Google certainly has no reason to correct these perceptions, so SEOs and marketers need a way to measure the actual effect of an update on their own.”

This raises an interesting question, one Cazier is currently looking into. Are panic-stricken SEOs their own worst enemies when it comes to Penguin and Panda updates? A lack of confidence in organic could be causing decision-makers to reallocate budget away from SEO, to paid search.

The next time you read or write a misinterpreted “Jarring and Jolting” doom and gloom post or cry that the sky is falling, consider the effect on the SEO industry as a whole. If that’s not inspiration enough to hold your tongue or look deeper, consider the loss of faith in your own abilities on the part of the people who pay you.

How Cazier Developed the GOCT Quantifier

ComScore provided the data used to measure organic clicks, from a global cross-section of about a million U.S. internet users who had given comScore permission to track their browsing and transaction behavior.

To measure the negative impact of a Google update on a specific vertical, such as travel, says Cazier, one could either look at rankings or click volume. He chose to focus on click volume as a more accurate measure of the bigger picture.

Cazier then removed non-US TLDs from their dataset and used comScore’s “category definition” to determine which websites composed a vertical. Profiles for each vertical range are comprised of URL sets ranging from 14 to 233 URLs.

Cazier explains:

In our quest to establish a quantifier to measure the impact of Google updates, it’s tempting to simply count and sum each negative move within year over year (YoY) and month over month (MoM) Google click data to produce a formula like negative YoY months + negative MoM months = quantifier but a introduction of non-Google organic click data is necessary.

Looking at the difference in Google YoY and non-Google YoY rates of organic click change will yield a plus/minus factor that focuses the YoY and MoM data on extra-ordinary changes, not variances within Google due to seasonal consumer search patterns or data issues.

With this in mind, let us establish a few useful definitions: Cross-SE YoY Differential: The difference in Google vs. non-Google, organic YoY clicks Cross-SE MoM Differential: The difference in Google vs. non-Google, organic MoM clicks If the Cross-SE YoY Differential is Google-negative it is a “YoY Loss” If the Cross-SE MoM Differential is Google-negative it is a “MoM Loss”.

A count of “YoY Losses” and “MoM Losses” are what should be summed and quantified.

He did also account for MoM drops due to seasonality.

How Cazier’s Impact Assessment Works

In his report, Cazier explains how it works using the following example:

google-panda-penguin-sample-goct

From the report:

In the example above, only the items in bold red, large font are counted as YoY or MoM Losses for a total Google Organic Click Turbulence score of 8. We will see that is a very low number = a vertical with “calm” Google organic click traffic.

  • Cross-SE YoY Diff for Jul 11 through Sep 11 was not counted because the overall Google YoY organic click growth was positive.
  • Cross-SE YoY Diff for May 12 through Jun 12 was not counted because the non-Google organic YoY click decrease was greater than the Google organic YoY click decrease.
  • Cross-SE MoM Diff was tallied the same way.

Quantification of each vertical’s GOTC should be compared across verticals, Cazier explained. Verticals with a comparatively higher GOCT had greater negative click growth than those with lower scores. Measurement of this quantifier against those of sites independently verified as impacted by Panda 2.5 and Penguin 1.0 can provide the upper GOCT threshold.

Penguin 1.0 and Panda 2.5 - Not As Bad As Reported?

Cazier analyzed Google organic and non-Google organic clicks across key verticals from February 2011 to present, with clicks graphed from January 2010 to provide YoY data. His complete analysis in each vertical is available in the full report.

He admits that the strength of the study and his conclusions may be affected by “undiscovered data issues or through creative manipulation of the sites that compose a vertical.” The GOCT serves as a “line in the sand” from which SEOs can move forward, he notes.

Cazier reports:

  • Google’s Penguin 1.0/Panda 3.5-3.6 release in April 2012 negatively impacted 50 percent the study’s 12 groups. March 2012’s release of Panda 3.4 appears to have a similarly-scaled negative impact.
  • Government, education and B2B verticals show little organic click volatility. With their baseline set at 6-8 GOCT, we can quickly look to see that the real estate, retail (general) and retail consumer electronics verticals were also relatively untouched.
  • The news, retail apparel and retail department store verticals saw moderate Google Organic Click Turbulence with the most impactful Google updates appearing to be Panda 3.4-3.6 and Penguin 1.0
  • The Panda 2.5 and Penguin 1.0 reference groups set the expectation that a GOCT score > 20 means the vertical was hit particularly hard by Google updates. There is one vertical in particular that appears to have been hardest hit by Google’s updates across the board: travel.

So what of the surveys mentioned earlier, where more than 40 percent of SEOs believed their site traffic had been harmed by Google Penguin or Panda?

“Regrettably, we cannot make an apples-to-apples comparison of these percentages versus YoY or MoM declines,” Cazier reported. “It’s unavoidable to conclude that either some polling bias occurred, or there’s an undue, oversized negative perception of Google’s updates.”

Cazier’s report includes a list of unresolved questions and he encourages SEOs to take part in the conversation.

”I wanted to provide a basis for people to know whether the world is truly collapsing around them and they can make better decisions around organic and paid,” Cazier told us. “I mapped out paid for these updates as well and it’s crazy, you can see these huge spikes in paid search around Google Panda and Penguin updates, that aren’t accounted for by seasonality.” That report on paid search activity in relation to Google updates is due out later this year.

What do you think of Cazier’s Impact Assessment method using Google Organic Click Turbulence as a measure of the overall impact of an algorithm update? Perhaps more importantly, how could you use these insights in your business? Share your thoughts and suggestions in the comments.