The official launch of the new version of Yahoo! Search crawler comes as no surprise to members of Digital Point forums, readers of Search Engine Watch and attendees of SES New York, who saw Yahoo Chief Scientist Andrew Tomkins' keynote speech on the future of search.
Yahoo! Slurp 3.0 won't change the content Yahoo crawls on your site: the new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for Yahoo! Slurp. The primary difference will be in your log files where you'll see Slurp 3.0.
Yahoo shared that the new bot (phased rollout over several weeks) will start crawling from a different and smaller set of IP addresses. No change of location, though. Yahoo! Slurp 3.0 will originate from the crawl.yahoo.net domain. So any reverse DNS checks to ID Yahoo's crawler will still work.
The big change: Yahoo's recommendation to stop using IP-based recognition and use reverse DNS identification instead.
Yahoo warns you might see a drop in crawl and coverage if you're using IP-based recognition. Best bet: switch to reverse DNS-based identification of Yahoo! Slurp if you're using any other method. The current set of IPs will disappear from your log files in the next few weeks.
Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.
Introducing... ClickZ Live!
SES Conference & Expo has merged with ClickZ to bring you ClickZ Live! The new global conference series takes on the identity of the industry's premier digital marketing publication, ClickZ.com, and kicks off March 31-April 3 in New York City. Join the industry's leading tech-advertisers in the advertising capital of the world! Find out more ››
*Super Saver Rates expire Jan 24.