Google Indexing Near Real Time

Matt Cutts updates us late last night about how quickly Google updates its index. He relates a bit of the history of indexing speed, and then provides some examples to show that Google is now doing “Minty Fresh Indexing”. The gist of this is that for some site’s documents can show up in minutes instead of days.

Matt goes on to provide a couple of examples, one of which uses Rich Skrenta’s blog to show that a blog post he had been reading made it into the Google index in less than 1 hour after it was posted. I have seen some of this behavior as well.

This direction is consistent with a conversation I had with Google’s Peter Norvig when I was at Google last. Peter said that Google believes the ultimate goal would be to index the content you are creating as you are creating it. Not to worry, I am not talking about big brother monitoring of your inner thoughts here, but pretty much real time indexing.

It’s impressive to me that a polling based system, i.e. a system that required the crawlers to go and find the changed content rather than being notified about new content, can have such a real time response. It will be interesting to see how this interacts with the need to fight spam and dealing with sites that have not established any level of trust.

Of course, this is a discussion of indexing, and not ranking. In the case of the Rich Skrenta blog, the example Matt provided showed that the snippet for the blog was updated for the latest post. That’s completely appropriate for content on sites such as a blog, where the latest snippet is what a user would want to see. The site and the content in particular will still need to do other things before it will be highly ranked for competitive search terms that the site did not rank for previously (such as get lots of links).