A great thread emerged yesterday at WebmasterWorld.com about how someone managed to apparently remove the home pages of Microsoft and Adobe from Google. But was it true?
Yes, Google says. The company sent this statement today:
“We can confirm that less than 10 websites were inadvertently removed from Google’s index for several hours [Thursday”. All of these sites have been restored and are accessible through a Google search. The removal occurred as the result of an outside attempt to abuse Google’s automated web page removal tool — a free service we provide webmasters who would like to remove web pages they own from Google’s index. Upon discovering this bug, we fixed it immediately. We will also perform a thorough analysis to ensure additional web pages were not inappropriately removed.”
The thread was started by a person who claimed he removed the pages to highlight the fact that his own home page had been removed, apparently by a competitor.
Google’s automated removal tool ordinarily should only work if certain a web site either has a robots.txt file or a meta robots tag in place that serves to verify that a page should be removed. The bug allowed the pages to be removed, despite the fact that no such verification was in place.
In somewhat related news, Ben Edelman has posted a story about how he tracked down apparent cloaking by WhenU, causing that company to have pages removed from Google and Yahoo. WhenU blames a third party company for the act.