Google Out Of Index Space?

Last year, Google Watch’s Daniel Brandt argued in his Is Google Broken article that Google’s architecture couldn’t handle adding more pages to its index. A new article of the same name but by a different author brings up the allegation afresh.

Unfortunately, relying on the page count figure on Google’s home page as the new article does isn’t a good guide to whether Google has added pages or not. My impression is that Google updates that periodically by hand, not automatically — and in particular, whenever anyone else claims to be bigger.

That’s what triggered the last change, as explained more here in my article from last year: Search Engine Size Wars & Google’s Supplemental Results. I also find relying on old publications on how Google works not a reliable guide to making assumptions of the space it has now.

As for Brandt, he’s posted a new update on the situation here: Google is dying. He suggests the Google’s supplemental index is a way around Google’s alleged capacity problem. Could be — others have done the same in the past. At the same time, it’s important to remember that having more pages doesn’t necessarily mean that a search engine is better than others, as I and others have talked about in the past. A roundup of articles on this exact issue can be found on my Search Engine Sizes page.

For those who want a trip down memory lane, compare the current allegations against Google to those launched against AltaVista back in 1997, as explored in The AltaVista Size Controversy. AltaVista unfairly bore the brunt of attacks for not indexing every page from various web sites despite the fact that it indexed far more than its competitors. For all we know, Yahoo and Ask Jeeves may be indexing worse than Google, but none of the articles I’ve seen explore this issue. I wish they would, so we’d really know how Google measures up both against expectations and its competition.

When I mentioned on Brandt’s allegations last year in my newsletter, I also asked Google for comment. They denied having any problems. I asked again about the latest allegations but haven’t gotten a response back yet.

Finally, issues of lost home pages such as with Business.com certainly lend to the impression that all’s not right with Google (see the forum thread, Google malfunctioning?. But it’s also too easy to assume a Google conspiracy, as well. Feedster’s missing home page turned out to be Feedster’s own error, as explained here.