Search Bot Behavior

Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis “On Bots” was released at http://drunkmenworkhere.org/219. The article has an analysis and visualization of the behavior of search robots. The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The study was conducted on a single site, so I am not sure how statistically valid it is over all sites on the Web. Just take a look at the overall results to see how much of a hog Yahoo is.

Related reading

The word PREPARED is written on a blackboard with the UN crossed out. A hand is underlining it.
A hand holding a transparent piece of plastic or glass, with the Google logo superimposed onto it.
Simple Share Buttons