SEO News

Robots Msnbot

RSS
  1. Giving Links Away

    If you want the instructions to apply to all spiders, you can use an asterisk on the first line: User-agent: * You can also identify specific spiders to allow or disallow, such as googlebot, Yahoo's slurp, or Microsoft's MSNbot.

  2. Search Bot Behavior

    The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The article has an analysis and visualization of the behavior of search robots. Ever wonder how spider/bots/crawlers behave?