Yahoo! Slurp 3.0 Replaces Yahoo! Slurp Search Engine Crawler


The official launch of the new version of Yahoo! Search crawler comes as no surprise to members of Digital Point forums, readers of Search Engine Watch and attendees of SES New York, who saw Yahoo Chief Scientist Andrew Tomkins' keynote speech on the future of search.

Yahoo! Slurp 3.0 won't change the content Yahoo crawls on your site: the new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for Yahoo! Slurp. The primary difference will be in your log files where you'll see Slurp 3.0.

Yahoo shared that the new bot (phased rollout over several weeks) will start crawling from a different and smaller set of IP addresses. No change of location, though. Yahoo! Slurp 3.0 will originate from the domain. So any reverse DNS checks to ID Yahoo's crawler will still work.

The big change: Yahoo's recommendation to stop using IP-based recognition and use reverse DNS identification instead.

Here's why:

Yahoo warns you might see a drop in crawl and coverage if you're using IP-based recognition. Best bet: switch to reverse DNS-based identification of Yahoo! Slurp if you're using any other method. The current set of IPs will disappear from your log files in the next few weeks.

Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.

About the author

Kevin Heisler, formerly the executive editor of Search Engine Watch, is a search and advertising industry veteran. His former roles include VP, strategic accounts for integrated digital marketing firm 360i; director of business development for Didit Search Marketing; and search industry analyst at Jupiter Research.

To contact current Search Engine Watch editorial staff, please click here.