Yahoo! Slurp 3.0 Replaces Yahoo! Slurp Search Engine Crawler


The official launch of the new version of Yahoo! Search crawler comes as no surprise to members of Digital Point forums, readers of Search Engine Watch and attendees of SES New York, who saw Yahoo Chief Scientist Andrew Tomkinskeynote speech on the future of search.

Yahoo! Slurp 3.0 won’t change the content Yahoo crawls on your site: the new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for Yahoo! Slurp. The primary difference will be in your log files where you’ll see Slurp 3.0.

Yahoo shared that the new bot (phased rollout over several weeks) will start crawling from a different and smaller set of IP addresses. No change of location, though. Yahoo! Slurp 3.0 will originate from the domain. So any reverse DNS checks to ID Yahoo’s crawler will still work.

The big change: Yahoo’s recommendation to stop using IP-based recognition and use reverse DNS identification instead.

Here’s why:

Yahoo warns you might see a drop in crawl and coverage if you’re using IP-based recognition. Best bet: switch to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method. The current set of IPs will disappear from your log files in the next few weeks.

Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.

Related reading

Google Sandbox Is it still affecting new sites in 2019 search tools updates competitive analysis
Decommissioning Jet Two charts proving Walmart planned to ground Jet all along