search engine crawler

robots.txt best practice guide, setting up the robots.txt file