SEO News

Search Engine Robots Spiders

RSS
  1. How To Use HTML Meta Tags

    NOTE: The robots tag may be ignored by less scrupulous spiders. NOTE: Schema tagging and rich data snippets are single-handedly the most important (and somewhat quietly announced) change to how your site interacts with the search engines and the...

  2. Search Engine Submission 101, Part 1

    In reality, Google simply searches its own index, or database of information they've collected from their army of robots (sometimes called spiders or bots). The robots' mission in life is to constantly crawl the Internet and follow every link they...

  3. SEO Site Structure 101, Part 1

    A search engine sends out agents (a.k.a.spiders, robots, crawlers) to surf the Internet and bring back what they find and deposit that information in the search engine's databases. HTML is the language that Web browsers and search engine spiders...

    Published
  4. When's an Outage More than Just an Outage?

    Make sure they don't use a module such as "mod throttle" to keep spiders from overwhelming the servers. If they're hitting too hard, explore alternatives in your robots.txt files that the search engines can see.

  5. Giving Links Away

    In the first line you specify what spiders you want to block from spidering your content. If you want the instructions to apply to all spiders, you can use an asterisk on the first line: User-agent: * You can also identify specific spiders to allow...

  6. Highlights from the SEW Blog: March 24-28, 2008

    New Google Webmaster Tool Aids Robots.txt CreationFor the coding-challenged, creating a robots.txt file for instructing search engine spiders on the preferred way to crawl your site has not always been easy.

  7. Search Headlines & Links: March 28, 2008

    New Google Webmaster Tool Aids Robots.txt CreationFor the coding-challenged, creating a robots.txt file for instructing search engine spiders on the preferred way to crawl your site has not always been easy.

  8. Google on Global Search Standards: Thanks, But No Thanks

    Placing a robots.txt file onto sites prevents Google's spiders from indexing them and displaying sites in search results. Google has politely turned down a cordial invitation to support technology that would control how the search engine lists...