Internet Protocol Suite

Both Rajat Mukherjee from Yahoo and Vanessa Fox from Google stated that the best way to control their respective bots is to use the robots.txt protocol.Slurp (the name for Yahoo's bot) is an obedient bot," Said Mukherjee.