One of the first steps of a successful SEO campaign is to make sure you aren’t accidently withholding site content from search engines, as discussed in “Don’t Let Small Disconnects Lead to SEO Disaster.” Once you have a good understanding of the potential dangers of robots.txt, robots meta tags, 404s, etc. that keep engines from viewing your site content you can examine the content they should be looking at and how they prefer to view it.
Simply having your content available to crawling spiders doesn’t ensure that it will be quickly or easily crawled by the search engines. Gone are the days of, “you have an HTML sitemap, right? OK, good.”
Nowadays, Google is pickier about your content delivery. They want the data in a designated format, usually XML, so they can get a better look at what’s on your site.
Universal search was a game changer. Where once “content is king,” now “all kinds of content are king.” It isn’t solely about ranking pages anymore as it is about ranking images, video, news, shopping results, locality, and now rich snippets. Geez, can somebody give us SEOs a break!
Standard XML Sitemap
If Google offers you something, then you better use it. With this in mind, your first need is a standard XML sitemap of all your “public-facing” site pages. This is elementary to most, but you'd be quite surprised by how many don't do this.
My favorite tool for this is Sitemap Writer Pro. Within this tool you can exclude all folders and pages included in your robots.txt file and create a standard XML sitemap. From here, place it on server directly off the root.
Give mention to the file at the top of your robots.txt file as well because Google stops here first to see where not to go. It’s wise to tell Google where to go. Next, go to your Google Webmaster Tools account and upload your new sitemap in the Sitemaps section.
Image, Video and News XML Sitemaps
Once you have acknowledged that you aren’t including your image or video folder in the robots.txt, (see it all the time, had to mention it), you can create an image XML sitemap and video XML sitemap. Sitemap Writer Pro tool is good because you can create all of these different formats within this one tool.
The image and video sitemaps are great ways to speak crawler, (XML) and inform Googlebot about the images on your site, their residing URL, ALT text and similarly, the pages where your videos can be found, any corresponding tags, duration information, and title headings. Give these sitemap files you generate easily discernable filenames (e.g., image-sitemap, video-sitemap.xml, news-sitemap.xml).
Unlike the standard XML sitemap, you need not mention the URL in the robots.txt but you need to upload them to the Sitemap section of Google Webmaster Tools. The news XML sitemap is created in the same fashion as the image and video sitemap.
I’m still on the fence about using a news sitemap. These aren’t necessarily needed unless it’s a news site or the site belongs to a large company that puts out multiple newsworthy releases on a very frequent basis. However, by using it you can provide XML data per news item, including stock ticker info, relevant keywords and genres, etc.
Local and Shopping XML Feeds
As mentioned above, there are still more areas of universal search and today’s search engine result that command a specific method of delivery.
One important area is the local listing. While these sit apart from any XML maps that are uploaded to the site, if you have more than 10 locations, you can create an XML based feed to upload to Google Places for Business.
Many have said that there is more of a local listing ranking preference for manual submitted listings, but I’ve yet to see this. Despite the disputed performance factor with manual/single listing creation, when you have a client with more than a 100+ locations you don’t really have the time or patience for manual submissions.
Google provides a step-by-step process for creating these XML feeds. Now that local listings are pairing with organic listings, you can’t afford to give up this free first page real estate.
Another great opportunity for distribution to Google is through a Google Shopping feed. Creating an XML feed through Google Merchant Center gives your products access to the Google Shopping portal as well as the potential to appear for relevant search queries via universal search in Google.
Again, Google gives step-by-step information on creating the XML shopping feed. Much like the standard XML sitemap you created for your site pages, the XML file will be housed on your server as well as in your Merchant Center profile where you can also indicate a daily crawl request.
Rich Snippets & Beyond
Rich snippets aren’t like the XML maps discussed above. They are considered more a microformat that allows you to provide a markup of data within your site content that will give Google the ability to present it in SERPs alongside your site listing(s).
Available snippets at the moment include those for reviews, events, recipes, people, products, and businesses. As any good SEO knows, anything you can do to stand apart from your SERP competition is another way to enjoy a better CTR.
Look for yourself in Google’s Webmaster Tools Help section on rich snippets. They provide step-by-step guidance as well as a Rich Snippet Tool to help you test your snippets.
Where are we headed you ask? To an even richer experience!
With the recent notice from the “Big 3” as to their acceptance of schema.org, Google for example is adding 100 new markup types of rich snippets. If you stand back and take a look at all of this from 30,000 foot view, we’ve learned that we have to deliver our site information to Google in a certain format.
Moving forward, we’re now being asked to mark it up a certain way within our site. Even more recently, Google is now saying they will provide a service to rework your site content on the fly.
This all reinforces the notion that you better think twice about your content delivery to the search engine crawlers because they seem to be picky eaters!