IndustryYahoo! Slurp 3.0 Replaces Yahoo! Slurp Search Engine Crawler

Yahoo! Slurp 3.0 Replaces Yahoo! Slurp Search Engine Crawler

yahoo%20slurp%203.0.jpg

The official launch of the new version of Yahoo! Search crawler comes as no surprise to members of Digital Point forums, readers of Search Engine Watch and attendees of SES New York, who saw Yahoo Chief Scientist Andrew Tomkinskeynote speech on the future of search.

Yahoo! Slurp 3.0 won’t change the content Yahoo crawls on your site: the new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for Yahoo! Slurp. The primary difference will be in your log files where you’ll see Slurp 3.0.

Yahoo shared that the new bot (phased rollout over several weeks) will start crawling from a different and smaller set of IP addresses. No change of location, though. Yahoo! Slurp 3.0 will originate from the crawl.yahoo.net domain. So any reverse DNS checks to ID Yahoo’s crawler will still work.

The big change: Yahoo’s recommendation to stop using IP-based recognition and use reverse DNS identification instead.

Here’s why:

Yahoo warns you might see a drop in crawl and coverage if you’re using IP-based recognition. Best bet: switch to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method. The current set of IPs will disappear from your log files in the next few weeks.

Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted.

Resources

The 2023 B2B Superpowers Index
whitepaper | Analytics

The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing
whitepaper | Analytics

Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook
whitepaper | Digital Marketing

The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study
whitepaper | Digital Marketing

Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y