Very nice. Wondering how a search engine will process your robots.txt file? Google now provides a way to check on that through the Google Sitemaps program. More stats and analysis of robots.txt files from the official Inside Google Sitemaps blog explains more.
Overall, I'm thrilled with the new tool. I'd like to see the other search engines add similar ones. Even better, I'd like to see them all come together on creating an enhanced and more standardized robots.txt standard. Consider:
wildcards, but others don't.
Yahoo allow crawl delays (but don't define minimum or maximum values).
Google does not.
- Ask & Google have ALLOW commands that no others support
Postscript: Matt Cutts from Google has some good comments over here, pointing out Google also has an allow command (I've updated my list above) and further in comments to the post, explaining why they don't support crawl-delay yet because of concerns it might be set too low by mistake by some webmasters.
Upcoming Webinar: PPC Pause and Reflections for 2013
Thursday, December 12 - 2013 was a major turning point in search advertising. With Google's Enhanced Campaigns and Bing's innovative Smart Search capabilities in Windows 8.1, now is a great time to pause, reflect, and plan for the new year. Webinar attendance is free. Sign up today!