Finally, personalized search results seem to have arrived for the masses. Personalized search has long been promised as an important next step for increasing relevancy. Now it comes not from Google or Yahoo but instead from tiny Eurekster, which opens to the general public today. Prior to this, the site had been a beta test involving only a few hundred people over the past couple of months.
Personalized search? The concept has been that by knowing some things about you, a search engine might refine your results to make them more relevant. A teenager searching for music might get different matches than a senior citizen. A man looking for flowers might see different listings than a woman.
Eurekster's twist on this concept is to provide personalized results based not on who you are but who you know. Friends, colleagues and anyone in your Eurekster social network will influence the type of results you see.
"Word of mouth is the most common way we filter information in real life, and Eurekster amplifies this everyday process to deliver search results that matter most to users and their friends and contacts," said Grant Ryan, Eurekster's chief executive officer.
Search Filtered By Friends
For example, take a search for "thunderbirds," the popular Gerry Anderson television show that I loved as a kid and that my children are now enjoying on DVD. Without refinement, Eurekster brings back a variety of results, leading off with the US Air Force Thunderbirds flight team web site.
There's nothing wrong with the US Air Force site coming up first. After all, it's certainly relevant for this search. But I'm thinking of the sci-fi series. Scrolling down, I find the official site for the show listed in position three. By clicking on the site, my interest in it is registered by Eurekster.
Now picture that one of my friends does the same "thunderbirds" search. They'll see different results, based on my actions. The Thundersbirds TV show site moves up to position one, with a little Eurekster "e" icon next to it. That's to alert them that someone in their network likes this site. Since most of my friends are into sci-fi, this search will probably feel more relevant to them.
Return Of Clickthrough Refinement
Why didn't clickthrough refinement take off? Spam clicks were a key drawback. If all it takes is a click to move your site to the top of the list, then people will click on themselves.
Indeed, that's exactly what happened. I can recall Direct Hit originally saying the basic spam protection they had in place at launch was deemed sufficient. A year later, Direct Hit told me they were into their seventh revision of protection.
Another issue is that link analysis provided much better relevancy than clickthrough refinement. Both Direct Hit and Google came to public attention at the same time in 1998. Google's use of link analysis delivered on the promise of better relevancy, while Direct Hit's use of clickthrough measurements didn't hold up, over time.
Why might Eurekster and clickthrough measurements succeed today? For one, link analysis itself is facing problems. Link spammers and others overtly manipulate links. Links are also created naturally in ways much different than in the past, polluting their usefulness in search. Personalization poses a potential next leap forward -- and clickthrough measurement can provide that.
In addition, past uses of clickthrough measurements never delivered personalized results by default. Anyone was allowed to influence the results that everyone else saw. In Eurekster's system, only those within your search network can directly influence you. This effectively creates hundreds, thousands and even millions of different possible results for the same search.
Click spammers suddenly face many different "fronts" in the war to be in the top ten, and they only get to fight in that war by invitation -- if someone they know asks them to be part of their network. Eurekster assumes "friends don't spam friends," and it's a pretty safe assumption.
"The things you do can impact your friends, but you wouldn't be putting a whole lot of spam, because the only people you'll be affecting are people you know and trust," Ryan said.
What Gets Counted?
For a page to be boosted, a person needs to click on it from the search results and stay at the page for at least a minute. If they do this, then Eurekster assumes the page was of good quality and gives it a boost for the search terms where it was originally found.
How does Eurekster know how long you spent viewing a page? It guesses. As with previous clickthrough systems, it watches to see when you return to your original search list and click on something else or conduct a new search. If either happens within a minute, then a site isn't credited for boosting.
Let's return to my "thunderbirds" search as an example. Assume I had visited one of the pages listed in the results but was dissatisfied immediately with what I saw. I might click the back button on my browser to get back to my original search results.
As soon as I clicked on another result, Eurekster could measure the time between my first click and the second. That time would be how long I spent visiting the first site I went to. If less than a minute, the site gets no credit.
Similarly, I might return and conduct an entirely new search. Now Eurekster would measure the time between my first click and my next search -- knowing who I am, since I would be logged in.
What about those who click from results and open new windows? That's the way I normally surf results. In this case, it's no different than as described above. If I go back to the window containing my original results and click on something, or do a new search, then Eurekster can do the time measurement.
Finally, what if I don't go back for some time after selecting a page? Then Eurekster will assume I was at the page for at least a minute, giving it a boost.
Spreading The Credit
As mentioned, my actions should only directly influence other people in my network. However, my actions can spread to other networks, assuming there's a link between them.
For instance, let's say I have a friend named Michelle in my network. Anything I do directly impacts what Michelle sees and vice versa. However, Michelle is also connected with another network of people I don't know. She forms a bridge between my network and this other one. Eurekster will allow some learning to move across due to this bridge.
"We have an advanced way to determine how results propagate around the network, so people beyond yours can learn," Ryan said.
But what about the idea that you don't want clicks to go outside a network to avoid spam problems? The first defense remains that any clicks have the most direct impact on your own network. However, Eurekster can also monitor whether a particular person seems to be providing bad click information and take counter-actions, if so.
"We can build up a profile of how your suggestions have been doing. If none of your friends like them, you can build up an understanding if someone is trying to do something that is not good," Ryan said.
Ultimately, Eurekster understands that spam may still happen, no matter what systems are in place. However, they hope that personalization makes the challenge much more a deterrent.
"We suspect this will be quite a difficult thing to spam, not that people won't be trying," Ryan said.
The potential of using your friends or colleagues is enormous. Imagine Eurekster being used by all the employees of a medical research firm, where many might do similar medical-related queries. With Eurekster, all the employees can be linked together and benefit from the searches and selections made by their colleagues.
Libraries are another institution that might latch on to the Eurekster concept. Librarians are constantly asked by patrons for assistance. Eurekster would allow librarians to collaborate invisibly with each other and share what they've found to be the best for various queries.
There are downsides. Not all of my friends have the same interests as me. In addition, as my social network grows -- because my friends invite their friends and so on -- commonalities that are useful get diluted.
Some type of categorization will almost certainly be necessary. I can imagine dividing my network into subgroups, for example. All my colleagues who cover search would be in one group, my fellow youth group volunteers in another, my friends who live in California in their own and my friends who live in the UK in a fourth. Depending on the search I do, I might want to share and refine my search in some way with a particular group, to benefit from a particular commonality.
Such functionality is coming over the next few months, Eurekster says, as well as the ability to group contacts by interests, such as sports or business. In that way, you can also get a sense of what's hot by those who are into particular subjects.
"You can see what the search results are like with different hats on and do that different categorization," Ryan said.
Sharing Searches, Sites
Eurekster does more than refine your results. It can also show you the things that other people in your network are looking for. So when I searched for "thunderbirds," anyone one else in my network sees that query get listed in the "Your Friends Recent Searches" area in the right-hand column of the Eurekster search results page.
This can be pretty cool. When I saw "Firewire backups on sale" show up in that area, I knew one of my friends was looking for a Firewire backup solution. As it turns out, I'd just bought a similar solution of my own. So, I clicked on the small envelope icon that appeared next to the query. This let me email the person, to say I had info they might be interested in.
My network is incredibly small -- all of three people right now. I had a pretty good idea who had done this search, but I wanted to just test out the functionality. The idea behind it is more of the opposite. If you see a search that someone has done, you can get in touch with that person perhaps to discover what they found.
The email feature only appears for those in your direct network or extended network (where other networks are linked to yours by a common contact). Don't want to get emailed? Then you can opt-out using your account preferences.
I actually thought it might be useful to have an option where you can choose to let anyone email you, even if they aren't part of your network. To me, it could be a great way to build up new people, by spotting queries that you have in common. It could come, but Eurekster is keeping things limited, for the moment.
"As always, when you start a service, you try to set the default to what you think people would like," Ryan said.
Of course, what your friends look for is supposed to help you more directly, by influencing the actual listings you see when you search for the same thing. So, you needn't email for help. Just search for the same thing that you've seen someone else look for. Any site they liked, as determined by Eurekster, will be flagged with that "e" icon.
Eurekster will also display "Your Friends Recent Sites," which are sites those in your network have recently been to. It also keeps track of the top queries and sites visited by those in your network, as well.
While I love my friends, I don't want them seeing everything I search for. And in a small network like mine, it's pretty easy for others to guess who might have looked for something.
To protect privacy, Eurekster makes it possible to do a "private search" by checking a box below the search box. Do this, and your search will not be shared with others. Should you forget, you can click on a small X symbol that shows up next to any queries you've done, to remove them.
Both opt-out features are welcomed, but I'd still prefer that the default action be that all searches are private. The downside to this is that if people don't remember to share their searches when appropriate, the network doesn't function as it should. However, I think that by making all searches public by default, people will inevitably share material they don't want out there.
It's important to note that your actual name is never displayed alongside a query or a site that was visited. In addition, the larger your network gets, the more anonymous you become. Say you have 30 or more people in your network. Guessing at who did a particular search or visited a particular site becomes harder. Also, a porn filter does prevent porn queries from being displayed, a further bit of protection against some possibly embarrassing queries.
Aside from privacy from your friends, what about privacy in general? Yes, you will get a cookie with a unique user ID, which is the main issue that some have had with other search engines such as Google. Unlike Google and other major search engines, the cookie doesn't last for years. It expires after a year, if you don't revisit Eurekster.
Of course, being cookied with a unique ID or cookied for a long time isn't necessarily a problem, and my Search Privacy At Google & Other Search Engines article from last year covers this in depth. Instead, Eurekster brings in the entirely different issue also addressed in that article, that you are a known, registered individual (rather than an anonymous cookie ID) and linked to particular searches. That makes it much easier for your data to potentially be abused.
The How Eurekster Works page provides some assurances about privacy. The company says it won't reveal who did a particular search or visited a particular site. However, I suspect it would do this if legally compelled to. The policy also doesn't address how long your searches are retained. Some like Google Watch and the EFF believe such data should be destroyed on a regular basis.
Ultimately, anyone with serious concerns about privacy may shy away from the service, something Eurekster recognizes. In order to work properly, it does need to monitor what you search for and what you visit. The company hopes the promise of anonymity and extra tools to explicitly keep some things private will satisfy most concerns.
"To do personalization, we have to have some information, and we give you the ability to delete that information to make it private. In the future, we will give users even more tools by which to further control their data," Ryan said.
Google Replacement Or Enhancement?
Eurekster is not a replacement for Google, Inktomi or other search engines that crawl the web. Indeed, Eurekster has no listings of its own. Instead, it works to refine the results that are provided by others.
When you search on Eurekster, the listings are coming from Yahoo-owned AllTheWeb, which is an excellent search engine. Eurekster then alters those results depending on the choices made by your social network.
Eurekster can also do this for any other search engine, as well. AllTheWeb is being used because the company has an agreement with to use its listings (and also carries Yahoo-owned Overture paid listings, which appear under the Sponsored Search Results heading). Eurekster could strike an agreement with other companies and do the same on their own sites.
"This doesn't replace the algorithmic [crawler-based” results," Ryan said. "We want to partner with anyone and everyone. We see it as an additional layer that adds value."
The company says some search companies have expressed interest, though it wouldn't name any. Eurekster's sister company SLI has a relationship with some smaller search companies, such as Comet Systems, Excite Networks and CNET's Search.com to provide related searches technology. So, perhaps we'll see some movement with these partners first.
Eurekster also sees its system as something that may be of interest to social network web sites. These are places such as LinkedIn, Friendster, Tribe and Rhyz. They let you meet people based on who you already know, and they've been gaining recent attention from the press and surfing public.
"There's lots of people who want to grow their own networks, so we're kind of network agnostic," Ryan said. "We're happy to work with anyone's private network."
Incidentally, it was rumored that Google tried to buy Friendster last year. While that doesn't seemed to have panned out, Google perhaps is still interested in starting a social network of its own. Evidence? Try a search for social networks on Google and notice that an ad for jobs at Google comes up. Is it recruiting? If so, I didn't find any jobs explicitly mentioning this.
If not social networking, we certainly know that Google has a number of personalization technologies at its disposal. Google bought Kaltix last year, a start-up company of three people that attracted attention by promising undisclosed advances in personalized search. Back in 2001, Google acquired Outride, another firm experimenting with personalized search results.
Choose Your Engine?
One feature I'd love to see is the ability to use Eurekster with your favorite search engine. I think it would help in getting people to try it. Those comfortable with Google, for example, might want to run Eurekster "on top" of Google's results.
Such a thing will require Eurekster to strike new agreements, and it doesn't have one with Google, for the moment. But the company did express interest in my idea that perhaps it could let users make use of their own Google API keys, in order to legitimately call Google's results into Eurekster. If this happens, I'll pass along more details.
Will Personalization Fly?
I started out by calling Eurekster tiny, but the company has a wealth of search refinement experience. Go back in time to GlobalBrain, which was a promising search technology that I wrote about in 1998. GlobalBrain was founded by brothers Grant and Shaun Ryan. Snap -- then later called NBCi -- bought GlobalBrain but failed to take the technology forward as the company pulled back from search in the wake of the dotcom crash.
The Ryans bought back the GlobalBrain technology and founded SLI Systems. As mentioned, SLI systems has been using it so far to provide related search functionality to a number of minor search sites, handling 300 million queries per month, the company says. Now SLI is one of the cofounding companies behind Eurekster (the other being social networking company RealContacts, also founded by the Ryans).
Will the second time out be lucky for the Ryans? It remains to be seen. But one thing's for certain. Unlike back in 1999, the majors are showing a real interest in personalized search.
Google (see News.com), Yahoo (see Wall Street Journal) and AOL (see Search Engine Watch) have all made public statements that they view personalization as a vital step forward in improving search -- and as mentioned earlier, Google even owns at least two companies, Kaltix and Outride, that were involved with it.
Eurekster is first out of the block for 2004, but we'll likely see the others follow, in some way. Past issues about privacy and user resistance will remain, (see Google May Get Personal & Searching for the personal touch), but there appears the will on the part of the majors to solve these issues.