Searching BG, Before Google, And The Need To Search Better

Just when you think stupid articles that assume we couldn't search BG -- Before Google -- have come to an end, the New York Times gives us another one. Here are examples of this mistaken assumption from 2003 and 2004. Now for 2006, Searching for Dummies, an opinion piece by Edward Tenner, explains how Google and its rivals today have made searching too easy and thus people aren't leaning about proper searching strategies. I agree with Tenner in the end, but he takes a convoluted and mistaken argument to get there.

Search today is too easy? Tenner writes:

But convenience may be part of the problem. In the Web's early days, the most serious search engine was AltaVista. To use it well, a searcher had to learn how to construct a search statement, like, say, "Engelbert Humperdinck and not Las Vegas" for the opera composer rather than the contemporary singer.

No, they didn't have to do this. Plenty of people would and did type in simple queries into AltaVista and get perfectly decent results without struggle. They didn't have to know Boolean commands to do so.

Yes, search engine math commands could help then, just as they still can today. That would especially be true if a search back then on AltaVista brought back only results about popular singer Engelbert Humperdinck rather than Engelbert Humperdinck, the Austrian composer of Hansel & Gretel. But then again,
Tenner hasn't said that was the actual case. He just gives it as an example without proof. For all we know, AltaVista did back then just what the search engines today do, give you a mixture of results on both people.

Tenner credits the improvement in search simplicity as being due to link analysis.

It took practice to produce usable results. Now, thanks to brilliant programming, a simple query usually produces a first page that's at least adequate ? "satisficing," as the economist Herbert Simon called it.

The efficiency of today's search engines arises from their ability to analyze links among Web sites. Google led in ranking sites by how often they are linked to other highly ranked sites.

That's not exactly correct. Link analysis improved the quality of results but not necessarily the need to occasionally use search commands or restructure your search terms, to get results on what you're looking for.

Link analysis was the second generation jump that helped solve the real problem that AltaVista and contemporaries had, that they were getting overwhelmed by both spam and the sheer amount of content on the web. What was the best stuff? Link analysis helped make sense of an increasingly noisy environment.

So back to Tenner, supposedly because of link analysis, we can do the Engelbert Humperdinck search and get results on both people without falling back to doing something like this:

engelbert humperdinck +opera -"las vegas"

But if that's true, then argument on searching being too easy falls apart when he says this:

Search engines have the opposite problem: dispersion rather than concentration of interest. Despite constant tweaking, their formulas display irrelevant or mediocre sites on a par with truly expert ones. Despite constant tweaking, their formulas display irrelevant or mediocre sites on a par with truly expert ones.

Curious about the academic field of world history? A neophyte would find little help entering "world history" in Google. When I tried, the only article on the world history movement, from the open-source Wikipedia project, didn't appear until the fifth screen and was brief and eccentric, erroneously dating the field from the 1980's....Only on the seventh screen did I find the World History Network site, financed by the National Endowment for the Humanities, and it is not yet a good portal for beginners.

So let me get this straight. Earlier, you complained that search engines gave you concentrated results on just one thing -- Engelbert Humperdinck the popular singer. Now you complain that they are failing to give you concentrated results on just one thing -- the "academic field of world history."

Gee -- that's kind of broad. Anything in particular about the academic field -- issues on how it should be taught, biases, publications?

Apparently, a search for [world history] should bring up information about the "world history movement." I'd argue that focusing all the results just on this single subject would be a bad, bad idea. Plenty of people looking for information on world history don't want to know about that movement but instead maybe timelines, events and other information that the first page of Google results do well.

Instead, if you really want to know specifically about the world history movement, you need some query refinement tools (which Google lacks) or you need to structure your query better, ironically similar to the way Tenner said you no longer need to do things.

For example, if you're after information about the world history movement, then it's helpful to actually use all the words that describe what you want. IE:

world history movement

That gives radically different results that a search for just world history on Google. But still, none of them seem to bring up pages specifically about this movement Tenner is interested in (and neither do the first page of results at places like Yahoo, MSN or Ask).

So instead, let's look for that exact phrase. By default, most search engines will tend to give preference to pages that have the words in the order you specify, but that's not guaranteed or foolproof. Surrounding your search terms with a quote marks ensures that you get pages that have pages with words in that exact order. So we enter:

"world history movement"

That immediately brings back only 103 results -- an incredibly tiny amount and suggesting that the world history movement isn't as widespread and commonplace on the web as Tenner thinks. That brings me back to one of my other top search tips. It's not always on the web. Want to know about the world history movement? One good thing might be to talk with a world history teacher!

Annoyingly at Google, one possibly good article on the topic isn't accessible, because it come from Google Scholar, where this academic paper is allowed to be cloaked. That makes it searchable but not actually viewable except to those with paid subscriptions. Another potentially good page goes to a 404 not found error. Other pages have references to this movement but no real explanation of it.

How about the others? The same quoted search on Yahoo gives me 55 matches, but nothing really great in the first two pages of results. MSN had 43 results and Ask had 34. Ask got me to a page describing a book about the movement -- and going back, I see that was also listed at Google and Yahoo in their first pages of results.

Hmm -- a book? Then perhaps it's not that I need to change my query but rather tap into a more specialized database like Google Book Search. So I try "world history movement" there and get a short mention of it in this page and a bit also in this book.

I'm still not satisfied. How about academic search engines, such as recently featured in SearchDay? After all, Tenner said this was an academic term. Part of searching better is understanding when to leave behind the Swiss Army Knife of web search for a more specialized resource. Let's try Google Scholar for "world history movement".

Oh, disappointment. There's that cloaked page again that I can't read without a paid subscription. There's another dead page. There's another restricted page, and another and another. Frankly, your layperson interested in this material is better off going to a reference librarian. And a reference librarian is better off probably tapping into non-Google tools to search this material.

OK, one last stab. I tried OAIster for world history movement and came away with nothing.

In the end, perhaps there is that perfect page about the world history movement out on the web. Then again, maybe the blame isn't on link analysis or a lack of searching skills in this particular case. Instead, this might be a great example of how web search doesn't find everything because not everything is on the web.

That brings me to finally agreeing with Tenner, with his closing point that student (and people in general) need to understand how to better structure queries, how to make use of different tools and even how to stop using search engines and fall back on other resources (such as professional librarians). Search engines are great in many ways, but they aren't, nor ever have been, perfect tools either BG or AG.