Articles on Sunday Times

RSS

News Robot Leads To Linking, Indexing Dispute

News search engine News Index crawled The Sunday Times web site, and the paper complained. The situation was resolved peacefully, but the article highlights the possible importance of the robots.txt file where a case might go to court.. News search engine News Index crawled The Sunday Times web site, and the paper complained. The situation was resolved peacefully, but the article highlights the possible importance of the robots.txt file where a case might go to court.. 0 Comments