A paper presented at the 10th International Conference on Extending Database Technology conference in Munich near the end of March, Indexing Shared Content in Information Retrieval Systems (pdf), jointly authored by employees of Yahoo, Google, and IBM, discusses how to limit index sizes of search engines by reducing the amount of duplicate content contained in their indexes.
After reading it, I started considering and listing some of the problems that sites may have that could cause search engines to not index the pages of those sites, or display them in search results. My list is in a post at SEO by the Sea, titled Duplicate Content Issues and Search Engines.
This Year's Premier Digital Marketing Event is #CZLSF
ClickZ Live San Francisco (Aug 11-14) will bring together the industry's leading online marketing practitioners to deliver 4 days of educational sessions and training workshops. From Data-Driven Marketing to Social, Mobile, Display, Search and Email, the comprehensive agenda will help you maximize your marketing efforts and ROI. Register today!