Matt Cutts, who is speaking at this week's PubCon, confirmed that the AdSense mediapartners bot is doing double duty by not only targeting ads for AdSense but also indexing for the regular Google search database, in a bandwidth saving move.
Matt also noted that there is no advantage to being indexed by one bot or the other, however, those cloaking content and serving different pages to each bot could run into problems in the search index. More details on JenSense.
Postscript From Danny: Matt adds more about this on his blog, and it's a super-important clarification:
Pages with AdSense will not be indexed more frequently. It's literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we?d retrieve the page from the crawl cache. But there's no boost at all in rankings if you're in AdSense or Google News. You don?t get any more pages crawled either.
In other words, there are two big issues with the AdSense crawler helping Googlebot:
- Since the AdSense crawler swoops in fast, it could be a way for people to
effectively get fast inclusion of their pages. Just add AdSense, wait for the
AdSense bot to fly in, and you're set.
- Is having the AdSense crawler likely to get you a RANKING boost, in addition to getting INDEXED faster. I've capitalized both words to stress them, as a reminder that being in the index isn't the same as ranking well for a query.
Matt's saying that no to both cases. There is no ranking boost. As for fast indexing, no to that as well. The AdSense bot is simply refreshing the cached copy of your page -- but the copy in the index, what people are searching on, won't be updated.
That brings up an entirely new point. It means that Google is now potentially presenting its results as fresher than they are.
What you search on is in the index, not the cache, as I've explained. So if a page changes and the index isn't updated -- only the cache -- then Google won't know about those changes to help with searching.Matt also noted that there is no advantage to being indexed by one bot or the other, however, those cloaking content and serving different pages to each bot could run into problems in the search index. More details on JenSense.
For searchers, the date on the cache is a useful way to know if Google's index is updated. Now, you can't tell. For site owners, the cache has been a useful way to know that Google has indeed updated your pages. Now it no longer serves that function.