Voting is one of the methods that we all accept for refining search results. While voting helps from a directional standpoint, results won’t be perceived as accurate or relevant by everyone who sees them. That’s because votes come from a self-selected and thus biased sample.
The WSJ’s Numbers Guy provided a stellar example of voting bias just yesterday. He analyzed issues related to Zagat’s dining guides. Many diners rely on Zagat, but don’t depend on the votes and ratings anymore.
Zagat is a well-known brand that appeals to Baby Boomers, and it provides ratings on local dining options. These handy city guides are based on voter volunteers, and have existed for years off-line.
When Zagat turned to online voting methods, the participation levels rose and biases became more obvious. First, the ratings became inflated. Second, they started to be gamed by restaurants who even asked for votes. Zagat tries to respond and keep their results relevant for diners.
We all know that voting online presents challenges. If a subject is very popular, then brace for spamming opportunities. If a topic is more obscure, then fewer voters can sway results more easily. It seems that both played into Zagat biases.
People still appreciate Zagat guides anyway:
* “It’s actually a pretty good guide for a first scan of places to go.”
* “The older generation relies on it.”
* “It’s good for out-of-towners.”
* “Something to consider…your mileage may vary.”
Yet actions and behaviors may speak louder than votes. One Denver diner put it best, observing that “usually the best sign an eating place is OK, is if there is a line out the door on Friday/Saturday night.”
A New Yorker claims that “a restaurant that is popular and gets many votes does not mean it’s better. Remember Al Yeganeh’s [which] was the inspiration for Seinfeld’s Soup Nazi?”
Keep this dining example in mind, when you are relying on the most popular or highest ranked results. Or when you are checking out results from social sites as well. There has to be some better way to understand what’s best for you.