SEO News

Spider Msnbot

RSS
  1. Giving Links Away

    If you want the instructions to apply to all spiders, you can use an asterisk on the first line: User-agent: * You can also identify specific spiders to allow or disallow, such as googlebot, Yahoo's slurp, or Microsoft's MSNbot.

  2. Search Bot Behavior

    The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis "On Bots" was released at http://drunkmenworkhere...