Yahoo is giving webmasters more control over page-level directives to its Slurp crawler for non-HTML files.
The X-Robots-Tag is a page-level exclusion tag that is used to direct a search engine spider in how it should treat that page. Similar to the way a robots.txt file is used, or a meta tag, the X-Robots-Tag can use the NOINDEX, NOARCHIVE, NOSNIPPET, or NOFOLLOW tag to tell spiders not to index a page, not to display a cached version of a page in search results, not to display a summary of the page in search results, or not to crawl links on a page.
The difference is that the X-Robots-Tag directive is processed in the http header, so it can now be used on non-HTML pages like PDF files, Word documents, PowerPoint, video, and other file types. It can still be used on HTML pages as well.
Yahoo also gave a weather report: "Along with this change, we'll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process."
The Original Search Marketing Event is Back!
SES Denver (Oct 16) offers an intense day of learning all the critical aspects of search engine optimization (SEO) and paid search advertising (PPC). The mission of SES remains the same as it did from the start - to help you master being found on search engines. Early Bird rates available through Sept 12. Register today!