Working With Google Scholar -- And More Approved Cloaking

Leigh Dodds provides a great rundown in his Google Scholar piece about one aggregator's experience in getting content prepared for entry into the new Google Scholar service.

My favorite part was this:

The second issue was to ensure that the crawler got the full text so they could work their on the full content rather than just the titles and abstracts. A bit of sleight-of-hand at our end ensured that the crawler got what it needed but with the URLs in the Google index being a suitable entry point for an end user.

That sleight-of-hand is almost certainly cloaking, showing an end user something different than what the crawler saw. Cloaking, of course, is against Google's published policies for webmasters.

As I've covered before in the situations of cloaking allowed at Google for NPR and Google Scholar, this type of cloaking is helpful to searchers. It's good cloaking. I have to stress, Dodds and the other Google Scholar participants are doing nothing wrong. They are working directly with Google, with Google's full approval, in a way that Google rightly feels will help searchers.

Nevertheless, Google's failure to update its policy continues to make it sound hypocritical. Telling general web publishers not to cloak, then having your Google Scholar participants talk about "sleight-of-hand" is a mixed message.

As I blogged earlier, it's long overdue for Google's policy on cloaking to be updated, to eliminate this mixed message. Simple changes like shown in bold below would be enough:

The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site without permission. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they'll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking without our permission, if we feel it is harmful to our search rankings.

If you're a Search Engine Watch member, Google & The Approved Cloaking Problem takes an even longer look at the issue, not just about the needed definition change, but also the fact that general web publishers are long-overdue for some of the special assistances being given to merchants, book and scholarly publishers by Google.

FYI, I came across the Dodds article via the new On Google Scholar blog, a nice resource for those wanting to track things about the new service.

About the author

Danny Sullivan was the founder and editor of Search Engine Watch from June 1997 until November 2006.

To contact current Search Engine Watch editorial staff, please click here.