SEONew robots.txt Tester Launches in Google Webmaster Tools

New robots.txt Tester Launches in Google Webmaster Tools

The robots.txt testing tool in Google Webmaster Tools has just received an update to highlight errors causing Google not to crawl pages on your website, let you edit your file, test if URLs are blocked, and also view older robots.txt file versions.

The robots.txt testing tool in Google Webmaster Tools has just received an update to highlight errors causing Google not to crawl pages on your website, let you edit your file, test if URLs are blocked, and let you view older versions of your file.

If Google isn’t crawling a page or part of your site, the robots.txt tester, located under the Crawl section of Google Webmaster Tools, will now let you test whether there’s an issue in your file that’s blocking Google. (This section of GWT used to be known as Blocked URLs.)

Google Webmaster Tools Robots.txt Tester

“To guide your way through complicated directives, it will highlight the specific one that led to the final decision,” wrote Asaph Arnon, Webmaster Tools team on the Google Webmaster Central blog. “You can make changes in the file and test those too, you’ll just need to upload the new version of the file to your server afterwards to make the changes take effect.

“Additionally, you’ll be able to review older versions of your robots.txt file, and see when access issues block us from crawling,” he continued. “For example, if Googlebot sees a 500 server error for the robots.txt file, we’ll generally pause further crawling of the website.”

In announcing the news on Google+, Google’s John Mueller wrote that even if you think your robots.txt is fine, you should check for errors or warnings.

“Some of these issues can be subtle and easy to miss,” he wrote. “While you’re at it, also double-check how the important pages of your site render with Googlebot, and if you’re accidentally blocking any JS or CSS files from crawling.”

Resources

The 2023 B2B Superpowers Index

whitepaper | Analytics The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing

whitepaper | Analytics Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook

whitepaper | Digital Marketing The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

whitepaper | Digital Marketing Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y