David Naylor Launches Robots.txt File Builder

I was actually outside the Hilton discussing this project less than two weeks ago with David Naylor and he has already delivered the first part of his idea.

Many people screw up their robots.txt file and deny the search engine spiders access to their sites. Dave thought it would be a great idea to create a central site that acts autonomously where people can have their robots.txt file created and stored to insure good interaction with the spiders.

His initial offering allows people to create the file and then copy and paste it into a page they can upload to their own site. Eventually Dave wants to host the pages himself and make sure the spiders correctly spider them. The site would be the central location for all spiders to get the right written file for any website.

The subtle differences between the spiders can be adapted, but Dave felt it would also be a way to get uniformity from the engines once they saw people using the site in sizable numbers.

The other ancilary benefits he was discussing was the ability to determine load times for a given site and get the spiders to visit at low traffic times so as to not overload the client's site capabilities.

I am impressed how quickly he got started on this. But then again he did share it at SES NYC with other fast to market players.... maybe he correctly guessed he better move on it quick before someone else did.

Great job so far Dave... now don't forget the rest!

About the author

Frank Watson has been involved with the Web since it started. For the past five years, he headed SEM for FXCM -- at one time one of the top 25 spenders with AdWords. He has worked with most of the major analytics companies and pioneered the ability to tie online marketing with offline conversion.

He has now started his own marketing agency, Kangamurra Media. This new venture will keep him busy when he is not editing the Search Engine Watch forums, blogging at a number of authoritative sites, and developing some interesting online community sites.

He was one of the first 100 AdWords Professionals, a Yahoo and Overture Ambassador, and a member or mod of many of the industry forums. He is also on the Click Quality Council and has worked hard to diminish click fraud.