Google's Matt Cutts Confirms AdSense Bot Helping Googlebot With Indexing
Matt Cutts, who is
speaking
at this week’s PubCon,
confirmed that the AdSense mediapartners bot is doing double duty by not
only targeting ads for AdSense but also
indexing for
the regular Google search database, in a bandwidth saving move.
Matt also noted that there is no advantage to being indexed by one bot or the
other, however, those cloaking content and serving different pages to each bot
could run into problems in the search index. More details on
JenSense.
Postscript From Danny: Matt
adds more about this on his blog, and it’s a super-important clarification:
Pages with AdSense will not be indexed more frequently. It’s literally just a
crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted
the same page, we?d retrieve the page from the crawl cache. But there’s no boost
at all in rankings if you’re in AdSense or Google News. You don?t get any more
pages crawled either.
In other words, there are two big issues with the AdSense crawler helping
Googlebot:
Matt’s saying that no to both cases. There is no ranking boost. As for fast
indexing, no to that as well. The AdSense bot is simply refreshing the cached
copy of your page — but the copy in the index, what people are searching on,
won’t be updated.
That brings up an entirely new point. It means that Google is now potentially
presenting its results as fresher than they are.
What you search on is in the index, not the cache, as I’ve explained. So if a
page changes and the index isn’t updated — only the cache — then Google won’t
know about those changes to help with searching.
Matt also noted that there is no advantage to being indexed by one bot or the other, however, those cloaking content and serving different pages to each bot could run into problems in the search index. More details on JenSense.
For searchers, the date on the cache is a useful way to know if Google’s
index is updated. Now, you can’t tell. For site owners, the cache has been a
useful way to know that Google has indeed updated your pages. Now it no longer
serves that function.