Support Blog Yahoo Slurp

Yahoo Adds Support for Page-Level Exclusion Tags for Non-HTML DocsYahoo is giving webmasters more control over page-level directives to its Slurp crawler for non-HTML files. Understanding Bids and Budgets, Yahoo Search Marketing Blog

Is Slurp Behaving Badly? We went back into Coremetrics to find the traffic we could support. Extreme Local Search Optimization Tactics, Natural Search Blog He also spends more on paid search with Google, though Yahoo is still productive from a cost...

The Yahoo Search Blog announced that Yahoo's web crawler, aka Yahoo Slurp, now supports wildcards in the robots.txt file. Many more details at the Yahoo Search Blog. The two parameters that Yahoo now supports include the "*" and the "$.

Since Google now is able to crawl and index URLs with parameters, i.e.dynamic URLs - they have removed the line that reads "Don't use " META NAME="ROBOTS" CONTENT="NOODP" (stops MSN, Google and Yahoo from using ODP directory) or META NAME="Slurp...

Scan the IP Address column, and you'll see how Yahoo's Slurp spider is in many, many different threads all at once. Since rogue spiders are ignoring robots.txt, it doesn't then matter for there to be some type of universal agreement to have a...

Finally, signs of transition are beginning to appear in the form of Inktomi's promotion at MSN Search, a new MSN Search beta and the Inktomi crawler being rebranded Yahoo Slurp. Meanwhile, Yahoo is developing new support for people to take RSS feeds.