Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis “On Bots” was released at https://drunkmenworkhere.org/219. The article has an analysis and visualization of the behavior of search robots. The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The study was conducted on a single site, so I am not sure how statistically valid it is over all sites on the Web. Just take a look at the overall results to see how much of a hog Yahoo is.
Search Bot Behavior
Want SEW higher in your Google results?Add as a preferred source
More in Industry
View moreUpdates to Google Webmaster Tools Make Sites More Crawlable
Google has updated its suite of Webmaster Tools to increase reportage of which site resources are blocked from crawling. This new transparency could mean a boost in rankings.
Google Fighting German Publishers Over Copyright
Google is firing back at German lawmakers, who are bowing to the demands of major publishers with draft legislation that would make search engines pay a license to display news headlines. Do they even realize what they’re asking for?
Google Slowed Crawl Rate to Aid Websites Protesting SOPA
Bing may have seen a bit of a boost in the fresh results department as a result of their slightly faster crawl rate on SOPA Blackout day, while GoogleBot slowed down substantially. Plus, a new infographic shows just how many users were affected.