How search-engine friendly are the Top 100 U.S. newspapers on the web? Not enough. Almost a third are are fully gated, with 26 requiring sign-ups to access all content freely and three requiring paid subscriptions (see recent Bivings study). Unfortunately, these gates typically prevent search engines from finding news too.
Of course, some papers like the NYTimes.com have organic search under control. VP Marshall Simmonds has enabled crawling of gated content, so Google and Yahoo make their pages available. When visitors arrive at paid pages like archives, they see abstracts first. Last week, there were rumors about dismantling the paid TimesSelect service. If that happens, the well-regarded columnists would have no trouble attracting more traffic and interest.
However, I'm betting that most of the remaining newspapers with sign-ups aren't up to snuff. Are they enabling search engines to crawl their content and pages? Are they creating appropriate site maps and topic indices? Not that I can easily see. Even if visitors have to log in, these newsworthy resources should be indexed and searched today.
Still I have to ask: why are so many papers requiring sign-ups anyway? That commonplace "we need to profile our visitors for advertisers" answer seems a little lame these days. Instead look at some free metrics, check out specific interests from click streams, and see what new search queries emerge. These sources seem better than self-reported, incomplete data from visitors who just want access, now.
Meanwhile, the largest online newspapers accelerate their efforts to open gates. Publishers with paid access are about to risk current revenue because they project more traffic and revenue downstream. Analyst Henry Blodget thinks it's a winner for NYTimes.com and WSJ.com to run free, if the AOL transition tells them anything. Meanwhile, Pearson continues to mull all FT.com options, in their competitive ad battle with the Journal.
Sure, these largest papers have the advantages of size, resources and online authority. They don't present an apples-to-apples comparison with the local papers. Still their recent willingness to consider risks should send a strong message to their counterparts. While the Top 100 papers have distinct challenges, they are able to attract local and thus highly valuable visitors. It's worth a little time and effort to think about search optimization.
Are any webmasters or developers reading this posting? Start by opening access to the search engines, please. Let the robots in, but make sure there's a "noarchive" metatag if you have gates. Try some site mapping, to expose more hierarchy and content to the crawlers. Check out the forums, and ask specific questions there.
The Top 100 papers are investing in so many areas, including content, video and even user-generated elements. It's the least they can do, to return to a few SEO basics: attract more visitors and spur online growth.