SEO News

Robots Txt

RSS
  1. 5 Googley SEO Terms – Do They Mean What You Think They Mean?

    Robots.txt won't prevent the URL from being indexed, just the page content. This isn't how robots.txt works. A robots.txt is used to stop a page or part of a site from being crawled and indexed, but not the URL(s) itself.

  2. Google Webmaster Tools Now Finds Smartphone Crawl Errors

    Blocked URLs: A blocked URL is when the site's robots.txt explicitly disallows crawling by Googlebot for smartphones. Typically, such smartphone-specific robots.txt disallow directives are erroneous. It can be bit complicated for websites with a...