The Yahoo Search Blog announced that Yahoo’s web crawler, aka Yahoo Slurp, now supports wildcards in the robots.txt file. The two parameters that Yahoo now supports include the “*” and the “$.” The * will tell Yahoo to do a “wildcard match a sequence of characters in your URL.” The & will tell Yahoo to do a “anchor the match to the end of the URL string.” Many more details at the Yahoo Search Blog.
Yahoo Slurp Adds Wildcard Support For Robots.txt
Want SEW higher in your Google results?Add as a preferred source
More in Industry
View moreYahoo, Bing Flirt With New Search Interfaces
Yahoo and Bing are reportedly testing out changes to search functionality that make results more "card-like" and move top navigation below the search results box.
Bing Ads 2014 Recap
In the past year, Bing has added features like product ads, auto-tagging and online insertion orders to make it easier to advertise on the network.
Yahoo Wants Back in the Search Game
Yahoo is rumored to be planning a return to search – both organic and paid. CEO Marissa Mayer has reportedly spearheaded two projects, code-named Fast Break and Curveball, aimed at getting the company out of its search deal with Microsoft.