Yahoo Slurp Adds Wildcard Support For Robots.txt

The Yahoo Search Blog announced that Yahoo’s web crawler, aka Yahoo Slurp, now supports wildcards in the robots.txt file. The two parameters that Yahoo now supports include the “*” and the “$.” The * will tell Yahoo to do a “wildcard match a sequence of characters in your URL.” The & will tell Yahoo to do a “anchor the match to the end of the URL string.” Many more details at the Yahoo Search Blog.

Related reading

A screenshot of visual search on Pinterest. On the left is a picture of a copper angle-poise lamp, with the words 'Visually similar results' above it. Down the right-hand side are a number of pins showing similar lamps.
google home
Simple Share Buttons