Search Bot Behavior

Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis “On Bots” was released at The article has an analysis and visualization of the behavior of search robots. The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The study was conducted on a single site, so I am not sure how statistically valid it is over all sites on the Web. Just take a look at the overall results to see how much of a hog Yahoo is.

Related reading

Robot sitting on a bunch of books. Contains clipping path
A picture of the 'Springfield Shopper' newspaper from The Simpsons, bearing the headline "LOCAL MAN LOSES PANTS, LIFE"
A screenshot of visual search on Pinterest. On the left is a picture of a copper angle-poise lamp, with the words 'Visually similar results' above it. Down the right-hand side are a number of pins showing similar lamps.
Simple Share Buttons