More On Query Refinement, The Human Scale Problem & Creating The Search Dialog

Yesterday, I wrote of how search engines could do a better job of query refinement and indeed did so in the past, especially because there was more human involvement in the search process. That drew out Jim Lanzone, senior vice president of search properties at Ask Jeeves, who sent me an email raising a good point that humans haven't gone away just because of the expense. Humans also have a "scale" problem. More comments from Jim on that are below, along with a further look at the scale issue and the need for the Google Generation to rediscover query refinement.

I agree with both Jim and Christopher Coulter, who commented on Robert Scoble's reference to my blog post:

Yeah well the Yahoo Human-Edited model couldn't scale, so you get Google Pagerank automational noised chaos. It's back to 1996 all over again. And the best 'search' is with a database

Indeed, humans haven't scaled well in terms of helping us gather content from across the web. Crawlers do a great job of that. I used a library metaphor on my ODP Founder Comments & Moving Past Directories post earlier this year to explain why directories, after some promise, went away in the face of crawlers.

In short, imagine you go into a library and can use one of two card catalogs to find books on a topic you are interested in:

  • The human-compiled card catalog looks only at book titles and short human-written descriptions of the books, maybe 25-75 words in all.
  • The crawler-compiled card catalog will let you scan every word on every page of every book in the library.

The crawler-built catalog is far more comprehensive. It's also far more up-to-date. Remember, in the library of the web, the books often rewrite themselves or add pages in the way books in a physical library do not. Humans simply can't keep up with that activity.

The key, of course, is that the crawler service isn't just comprehensive but relevant. It will find not just all the matching pages but often rank them so you are getting the very best ones.

While humans don't scale well in the info gathering and retrieval side, they can play a role. More on that in a moment, but first, here's what Jim said in response to my post:

To say that the problem with human editors was due to it being "expensive" is true, but I don?t think it goes far enough to explain the problem.

Sure, Ask had great relevancy, but only for a single-digit percentage of the overall query stream. That is not how people search, and neither you or I or any number of Web Search Universities is going to change that for the vast majority of searchers.

Algorithmic search was the only solution to that problem because only an index of billions of pages could meet the user need that exists across the long tail of rare queries.

At its peak, the Ask Jeeves "knowledgebase," as it was known internally, matched on about 85% of searches. That was a lot. However, it was only picked 20-25 percent of the time, despite having premium placement at the top of the page.

Sure, some searches resulted in far higher pick rates than this. But the vast majority did not. Therein lay the problem. And due to the exquisite overpromise made by the premise of question-answering and the butler, this had consequences for the Ask Jeeves brand.

The brand was lucky, on the other hand, to gain a foothold in the market early on, and to hold on to millions of users because of it. But at the end of the day, people use a search engine to find what they need - quickly. That foothold would have become more tenuous had we not bolstered our ability to answer searches (questions or keywords) more accurately, more quickly, and more easily, than the original Ask product could deliver.

Today, with a combination of algo search, structured data search (Smart Answers), and unique tools that help people find things faster (Zoom, Binoculars), we are delivering against user needs exponentially more than we did before.

At the end of the day, this is about playing defense, not offense, against that wide and vast stream of searches. Most people are going to do what they want with that little white box, and will not have the patience of learning how to "search better" or set up a bunch of parameters ahead of time. The minute we understood this and started intuitively responding to their searches with better results and options, the more loyal those users became to our search engine.

Again, I agree with much of what Jim says. Historically, we know that searchers don't read help documentation, don't make use of options, don't do anything much beyond put a word or two in that black hole of a search box and get sucked in to click on whatever seems like the first "normal" result that comes up. Put a bunch of refinement links/suggested searches in front of them, and they're likely to just ignore those as being "weird" and move down toward the real results.

Solutions? First and foremost, do break the habit! I've taught search classes for many years, and people are amazed that when they look a bit past the standard 10 listings, there are options and suggestions that are useful.

In fact, we'll be doing a "Pimp Your Search Engine" or "Bling Your Search Engine" series shortly via SearchDay to try and help many readers understand some of the many features that are offered to you on your own favorite search engines that habit just may have blinded you to. Get 100 results at a time! Pop open new windows from search results! Discover easy ways to refine your queries!

Beyond that, let's see the search engines do more to make use of both humans and automation. I do want a human-created knowledgebase at Ask Jeeves and elsewhere to return.

Maybe it has to be much smaller and serve only the very most popular queries. But why not make it that if I type in hotels on Ask Jeeves, the automatically-generated "Narrow Your Search" options off to the right-hand side might be determined to make more sense to show up in the main part of the results, to better help people narrow in.

Moreover, look at what I get now at Ask Jeeves in that Narrow Your Search section:

Why not step beyond the automation and for this type of broad, common search, come up with some human-generated suggestions, such as:

  • Find a hotel by price, location or other options
    Find the official site of a hotel chain
  • Find reviews of particular hotels

Those aren't perfect solutions/options, but I think they make the point. There ought to be more the search engine can do to have an actual dialog with people in the right cases.

I know, the risk is the less Google-like the results are, the more likely people are to feel uneasy or unsure about using a search engine. Well, the Google Generation needs to be smacked upside the face.

Honestly. Google has been an absolute, horrible failure in helping people refine their queries. It's no wonder people today may not even realize there are query refinement options out there beyond Google and before it existed, given that the major leader in the space hasn't made use of these. Let's see all the search engines dare to experiment more with these features and applaud them with they do.

By the way, speaking of applause, notice this over at Yahoo for hotels:

Yahoo! Shortcut - About

That's listed right above the first search result. I checked it out AFTER making my suggestions above, and you can see Yahoo's hit two of the three things I thought made sense. Great work, Yahoo! OK, cynics will say these links just help push people into Yahoo's own travel search areas. Yes, but they are good areas. People should be checking them out!

Why might you miss these links? Because they still don't look like the "regular" results that are numbered. So maybe they need to change. Maybe instead of being like this:

  1. Find your hotel at Get huge discounts on rooms at over 16,000 hotels and 800 cities. Rates for special events and sold out dates. Book online or call.
It needs to be more like this:
  1. The Yahoo Shortcut links above will take you to our special travel search areas designed to let you scan the web for hotel prices and information. You can sort by price, location, amenities and more.
  2. Find your hotel at Get huge discounts on rooms at over 16,000 hotels and 800 cities. Rates for special events and sold out dates. Book online or call.

Again, top of my head stuff. But I think it makes the point of the "dialog" being presented in a more listings-style format. Maybe, maybe, that might help users see stuff they're missing.