SEO News

Yahoo! Slurp 3.0 Replaces Yahoo! Slurp Search Engine Crawler

by , Comments


The official launch of the new version of Yahoo! Search crawler comes as no surprise to members of Digital Point forums, readers of Search Engine Watch and attendees of SES New York, who saw Yahoo Chief Scientist Andrew Tomkins' keynote speech on the future of search.

Yahoo! Slurp 3.0 won't change the content Yahoo crawls on your site: the new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for Yahoo! Slurp. The primary difference will be in your log files where you'll see Slurp 3.0.

Yahoo shared that the new bot (phased rollout over several weeks) will start crawling from a different and smaller set of IP addresses. No change of location, though. Yahoo! Slurp 3.0 will originate from the domain. So any reverse DNS checks to ID Yahoo's crawler will still work.

The big change: Yahoo's recommendation to stop using IP-based recognition and use reverse DNS identification instead.

Here's why:

Yahoo warns you might see a drop in crawl and coverage if you're using IP-based recognition. Best bet: switch to reverse DNS-based identification of Yahoo! Slurp if you're using any other method. The current set of IPs will disappear from your log files in the next few weeks.

Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.

ClickZ Live New York What's New for 2015?
You spoke, we listened! ClickZ Live New York (Mar 30-Apr 1) is back with a brand new streamlined agenda. Don't miss the latest digital marketing tips, tricks and tools that will make you re-think your strategy and revolutionize your marketing campaigns. Super Saver Rates are available now. Register today!

Recommend this story

comments powered by Disqus