Learn more about Googlebot’s crawl of your site and more! at the Official
Google Webmaster Central Blog covers new features Google has added, visual
charts to show Googlebot’s crawling activity, expanded crawl rate support,
inclusion in the image search labeling program and number of URLs submitted. I
talked with the Google Webmaster Central team earlier this week, and here are a
few more details on some of the features.
To see Googlebot activity reports, go to
Google Webmaster Tools,
choose one of the sites you’ve verified, then pick the "Crawl rate" option on
the Diagnostics tab. You’ll get a chart showing how many pages Google has
crawled per day over the past three months. For example, here’s what it looks
like for the Search Engine Watch Blog:
It’s interesting to see visually how Google has backed off the number of
requests over time. There’s nothing I’ve done to do this, but it may reflect
Google getting smarter about the fact that it doesn’t need to revisit every page
on the site so often. It could also be due to our server being less responsive
You can also see kilobytes downloaded per day, as well as the time spent
downloading a page in milliseconds. The chart on that for us is really
You can see that our response time nearly doubled at the end of July. That’s
exactly when we
left our servers at Jupitermedia, our old publisher, and switched to new
Incisive, our current publisher. Despite the slower time, I haven’t noticed any
drop in traffic from Google, so the slower responsiveness — while not good —
hasn’t been damaging. But if you did see a plunge in traffic, a chart like this
might help you visually realize what might be wrong directly from Google.
At the bottom of the Crawl rate page is the ability to set how fast you
want Google to crawl your site. This was
in August, but now it’s available to everyone using Google Webmaster Tools,
not just some. In addition, Google has simplified the options from five to just
three, Faster, Normal and Slower. Google said feedback suggested fewer options
would be easier to understand.
Crawl rate still doesn’t guarantee that Google will hit your server faster or
slower than normal, even if you request it. But Google said it is much more
responsive to these requests now. In fact, it is so responsive that you need to renew
your choice every 90 days. That’s to prevent someone authorized on your account
from telling Google to slam your server, then leaving and Google continuing to
do that forevermore.
Also on the Diagnostics tab, you’ll find an Enhanced Image Search option.
What’s that about? For now, it simply means that images from your site will be
available to those using the
Google Image Labeler system, which we wrote about last month:
Labeler: Google’s Challenge To Flickr?
Not all images from Google Images are currently added to Google Image
Labeler. Google said it currently uses a subset of pictures that it feels site
owners would be amenable to having labeled. This new feature lets you explicitly
tell Google you’d like to have your pictures play in the new program. More on
this is covered in the help
page about enhanced image search.
Finally, if you submit a sitemap to Google, it will now tell you the number
of pages submitted in that sitemap. Why care? Apparently, at least one person
did and requested the feature. As Google explains in that blog post, this person
generated a sitemap automatically and so had no idea how many URLs he was spitting
out in it. Now he — and others — can know.