Search Bot Behavior

Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis “On Bots” was released at http://drunkmenworkhere.org/219. The article has an analysis and visualization of the behavior of search robots. The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The study was conducted on a single site, so I am not sure how statistically valid it is over all sites on the Web. Just take a look at the overall results to see how much of a hog Yahoo is.

Related reading

Image showing front and back of the rumoured iPhone 8 in blush gold colour, which is a kind of peach brown. A caption in the bottom right corner attributes the image to Benjamin Geskin.
Simple Share Buttons