Twitter has updated their robots.txt file to allow search engines to crawl more of the site. Specifically, Google, Bing, Yandex and others will now have access to Twitter’s search results pages, which display Promoted Tweets where available.
The modification was first noticed by The Sociable, who offered a look at Twitter’s robots.txt file from September 11th:
It now says:
The change will allow search results pages to show up in Google’s index, though user search is still disallowed. For example, the results page for a search on the Olympics becomes indexable and therefore could show up in Google search results to users seeking information on the event.
With the new indexing direction, searches for keywords like “Ford” could potentially direct Google/Bing searchers to Twitter’s internal search for the term. This is different from the way Real-Time Search operated in that users were apt to see actual tweets on the topic displayed in Google’s results. They would see Google’s ads on the search results page, then click on a Twitter result and go straight to a user page, where there are no ads. With this change, the results appearing in search engines would direct the user to a Twitter search results page, which displays a Promoted Tweet embedded in the feed (when available).
Bing already has firehose access, so they can use real-time tweets in their search results. Google used to have this level of access. Then things between the two went sideways.
Twitter and Google have had a seemingly frosty relationship since their firehose deal ended last year. Twitter had provided access to real-time tweets until the deal ended in July 2011. In October, CEO Dick Costolo spoke with John Battelle at Web 2.0 Summit and hinted that the issue keeping the two from reaching an agreement was not a monetary one.
The timing begged the question, did the deal end because Google launched their own social network? Costolo said then that Google+ and Twitter were competing in different arenas; they in features and Twitter in simplicity.
"Twitter has emerged as a vital source of this real-time information. As we've seen time again, news breaks first on Twitter. As a result, Twitter accounts and Tweets are often the most relevant (search) results. We're concerned that, as a result of Google's changes, finding this information will be much harder for everyone. We think that's bad for people, publishers, news organizations and Twitter users."
Google responded with a statement on their Google+ Page:
“We are a bit surprised by Twitter's comments about Search plus Your World, because they chose not to renew their agreement with us last summer, and since then we have observed their rel=nofollow instructions.”
Twitter’s instructions now tell Google it’s OK for them to crawl and index more of the site. But will they? It’s difficult to decipher what actually happened at the table between Google and Twitter; we may never actually know the real reason they chose not to work together in that capacity any longer. It seems the deal itself was a huge missed opportunity for Twitter, in that users were directed around seeing their ads.
This change seems designed to address that. Now that Twitter has beefed up their own search a bit, are they hoping the search engines can help them direct traffic only to areas of the site they choose, where the potential for ad revenue exists?
Twitter doesn’t have sidebar ads, so having search engines crawl and index tweets may draw in users, but doesn’t make them any money. It encourages people to look outside of Twitter for Twitter content. However, directing people only to those monetized places is an option that makes a lot of sense for Twitter. Will it work for the search engines? We’ll have to wait and see what shows up in their results.