The robots.txt file allows you to prevent search engines from indexing documents on your web server, assuming they respect the
convention. Most major search engines do. Ian McCanerin, one of our forum moderators, has posted a nice new tool
to help you generate a robots.txt file. You can also discuss it in this forum thread.
New Robots.txt Tool
More in SEO
View moreSolving the agency search intelligence gap
How can agencies fill the voids in their search intelligence? Ian O’Rourke, CEO, Adthena, and Stephen Davis, Global Product Leader for Media Intelligence, Kantar shed light on this dilemma.
The future of Google and what it means for search
Croud's Organic Strategy Director discusses the future of Google, right from Google's antitrust lawsuit to Apple as a future rival, and more.
SEO takeaways from 2020: A review of the most unusual year for search
A review of SEO and search engine updates in 2020 right from Google, Baidu, Bing, DuckDuckGo, GIPHY, and more. Plus, a bonus list of key SEO trends for 2021.