Google's rolling out a new system where ad landing pages will be automatically spidered by a new AdsBot. The content of landing pages will help determine the quality of an ad campaign. That quality score, along with the amount you are willing to pay, is then used to determine an ad's AdRank, the position where an ad will appear in the results. A high quality score means you can rank higher even if you pay less than others. And not participating in the new spidering system can hurt your AdRank.
What's the deal? Didn't Google already spider landing pages as part of the announcement back in December that landing page content would be assessed? To my understanding from Google, only if the AdSense spider had seen the page for ad content placement purposes or if regular Googlebot had already indexed the page for inclusion in the web search index. If the page wasn't already visible to these or perhaps some other Google spiders, or had been specifically blocked from spidering, then AdWords couldn't assess it.
Sometime in the coming weeks, a new AdsBot crawler will be grabbing all landing pages independently of AdSense, Googlebot or other Google spiders. Can you still block being spidered? Yes. But if you do so, Google AdWords will consider you a "non-participating advertiser" in the review process. As a result, you'll take a ding on your overall AdWords quality score.
From new information about the change:
While you can exclude your site from review, this will provide us with little information about your landing page's quality and relevance. Therefore, if you restrict AdWords from visiting your landing pages, you will experience a drop in Quality Scores for your related keywords. (This will cause higher minimum bid requirements for any landing page for which you've restricted access.)
That page also explains how to block AdsBot from getting your pages, how the visits won't cost you money even though AdsBot is following your ad links and how blocking or allowing AdsBot to your site will have no impact on what Googlebot thinks about it in terms of ranking it for free, organic results.
For Search Engine Watch members, the longer version of this article covers more on the change from my talk with Google during a visit there last week, such as how it is designed to improve relevancy and ease concerns that users (rather than advertisers) might be harmed by search arbitrage.
Want to comment or discuss? Visit our Search Engine Watch Forum thread, AdWords To Begin Crawling Landing Pages & Analyzing For AdRank.