Yahoo Slurp Adds Wildcard Support For Robots.txt

The Yahoo Search Blog announced that Yahoo’s web crawler, aka Yahoo Slurp, now supports wildcards in the robots.txt file. The two parameters that Yahoo now supports include the “*” and the “$.” The * will tell Yahoo to do a “wildcard match a sequence of characters in your URL.” The & will tell Yahoo to do a “anchor the match to the end of the URL string.” Many more details at the Yahoo Search Blog.

Related reading

The word PREPARED is written on a blackboard with the UN crossed out. A hand is underlining it.
A hand holding a transparent piece of plastic or glass, with the Google logo superimposed onto it.
Simple Share Buttons