Google's Matt Cutts Confirms AdSense Bot Helping Googlebot With Indexing

Matt Cutts, who is speaking at this week's PubCon, confirmed that the AdSense mediapartners bot is doing double duty by not only targeting ads for AdSense but also indexing for the regular Google search database, in a bandwidth saving move.

Matt also noted that there is no advantage to being indexed by one bot or the other, however, those cloaking content and serving different pages to each bot could run into problems in the search index. More details on JenSense.

Postscript From Danny: Matt adds more about this on his blog, and it's a super-important clarification:

Pages with AdSense will not be indexed more frequently. It's literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we?d retrieve the page from the crawl cache. But there's no boost at all in rankings if you're in AdSense or Google News. You don?t get any more pages crawled either.

In other words, there are two big issues with the AdSense crawler helping Googlebot:

  1. Since the AdSense crawler swoops in fast, it could be a way for people to effectively get fast inclusion of their pages. Just add AdSense, wait for the AdSense bot to fly in, and you're set.
     
  2. Is having the AdSense crawler likely to get you a RANKING boost, in addition to getting INDEXED faster. I've capitalized both words to stress them, as a reminder that being in the index isn't the same as ranking well for a query.

Matt's saying that no to both cases. There is no ranking boost. As for fast indexing, no to that as well. The AdSense bot is simply refreshing the cached copy of your page -- but the copy in the index, what people are searching on, won't be updated.

That brings up an entirely new point. It means that Google is now potentially presenting its results as fresher than they are.

What you search on is in the index, not the cache, as I've explained. So if a page changes and the index isn't updated -- only the cache -- then Google won't know about those changes to help with searching.

Matt also noted that there is no advantage to being indexed by one bot or the other, however, those cloaking content and serving different pages to each bot could run into problems in the search index. More details on JenSense.

For searchers, the date on the cache is a useful way to know if Google's index is updated. Now, you can't tell. For site owners, the cache has been a useful way to know that Google has indeed updated your pages. Now it no longer serves that function.

About the author

Jennifer Slegg began as a freelance writer, and turned to search engine optimization and writing content for the web in 1998. She has created numerous content-rich sites in niche markets and works with many clients on content creation, strategy, and monetization. She writes about many search industry and social media topics on her blog, JenniferSlegg.com and is a frequent speaker at search industry conferences on SEO, content marketing and content monetization. Acknowledged as the leading expert on the Google AdSense contextual advertising program, she runs JenSense, a blog dealing exclusively with contextual advertising. She is also the founder and editor of The SEM Post. She is known by many as her handle Jenstar on various webmaster forums.