Search Bot Behavior

Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis “On Bots” was released at http://drunkmenworkhere.org/219. The article has an analysis and visualization of the behavior of search robots. The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The study was conducted on a single site, so I am not sure how statistically valid it is over all sites on the Web. Just take a look at the overall results to see how much of a hog Yahoo is.

Related reading

the simpsons news
marketing-cloud-1-360x270
A pair of yellow cartoon hands holds the newspaper The Springfield Shopper, which bears the headline SLEEP IMPORTANT, SAY EXPERTS.
facebook at work
Simple Share Buttons