Google Webmaster Tools Gain Crawl Charts, Enhanced Crawl Rate & Image Labeler Support

Google Reporting Current Crawl Rate

Learn more about Googlebot's crawl of your site and more! at the Official Google Webmaster Central Blog covers new features Google has added, visual charts to show Googlebot's crawling activity, expanded crawl rate support, inclusion in the image search labeling program and number of URLs submitted. I talked with the Google Webmaster Central team earlier this week, and here are a few more details on some of the features.

To see Googlebot activity reports, go to Google Webmaster Tools, choose one of the sites you've verified, then pick the "Crawl rate" option on the Diagnostics tab. You'll get a chart showing how many pages Google has crawled per day over the past three months. For example, here's what it looks like for the Search Engine Watch Blog:

Google Reporting Pages Indexed

It's interesting to see visually how Google has backed off the number of requests over time. There's nothing I've done to do this, but it may reflect Google getting smarter about the fact that it doesn't need to revisit every page on the site so often. It could also be due to our server being less responsive (see below).

You can also see kilobytes downloaded per day, as well as the time spent downloading a page in milliseconds. The chart on that for us is really revealing:

Google Reporting Server Response Time

You can see that our response time nearly doubled at the end of July. That's exactly when we left our servers at Jupitermedia, our old publisher, and switched to new ones with Incisive, our current publisher. Despite the slower time, I haven't noticed any drop in traffic from Google, so the slower responsiveness -- while not good -- hasn't been damaging. But if you did see a plunge in traffic, a chart like this might help you visually realize what might be wrong directly from Google.

Google Reporting Current Crawl Rate

At the bottom of the Crawl rate page is the ability to set how fast you want Google to crawl your site. This was introduced back in August, but now it's available to everyone using Google Webmaster Tools, not just some. In addition, Google has simplified the options from five to just three, Faster, Normal and Slower. Google said feedback suggested fewer options would be easier to understand.

Crawl rate still doesn't guarantee that Google will hit your server faster or slower than normal, even if you request it. But Google said it is much more responsive to these requests now. In fact, it is so responsive that you need to renew your choice every 90 days. That's to prevent someone authorized on your account from telling Google to slam your server, then leaving and Google continuing to do that forevermore.

Also on the Diagnostics tab, you'll find an Enhanced Image Search option. What's that about? For now, it simply means that images from your site will be available to those using the Google Image Labeler system, which we wrote about last month: Google Images Labeler: Google's Challenge To Flickr?

Not all images from Google Images are currently added to Google Image Labeler. Google said it currently uses a subset of pictures that it feels site owners would be amenable to having labeled. This new feature lets you explicitly tell Google you'd like to have your pictures play in the new program. More on this is covered in the help page about enhanced image search.

Finally, if you submit a sitemap to Google, it will now tell you the number of pages submitted in that sitemap. Why care? Apparently, at least one person did and requested the feature. As Google explains in that blog post, this person generated a sitemap automatically and so had no idea how many URLs he was spitting out in it. Now he -- and others -- can know.

About the author

Danny Sullivan was the founder and editor of Search Engine Watch from June 1997 until November 2006.

To contact current Search Engine Watch editorial staff, please click here.