Wisenut, the Google Killer? Nah...

There's been a lot of clamor in the media lately over which search engine will be the "next" Google. It makes for good copy -- the scrappy underdog vs. the reigning champ, David vs. Goliath, St. George vs. the dragon, yada yada yada. Entertaining as the stories about the presumptive "Google killer" are, virtually all miss the most important point. Which is... well, let's save that for the end. Meanwhile, let's take a look at Wisenut, one of the new search engines that's supposedly going to topple Google from its "throne" as the king of web search.

Wisenut is two things, really -- a new search engine that makes its official debut today, and search technology company founded by Yeogirl Yun, co-founder and former CTO of mySimon. mySimon was one of the first online comparative shopping services, and is now part of C|NET.

Wisenut the search engine is actually very good, and those of you who swear by Google alone will be pleasantly surprised by this new alternative. Wisenut uses an algorithm similar to Google's PageRank to compute relevance, examining the web's link structure and popularity of pages to help determine the best results for your query.

If Wisenut's so similar, why bother with it rather than just using Google? Several reasons.

First, Wisenut offers some neat goodies that can help you refine your search. Much like another newcomer, Vivisimo, Wisenut automatically categorizes results into "wiseguides" that are semantically related to the words in your query. You can open up a wiseguide category by clicking its link, or, alternately, click on an icon next to the category to automatically conduct a new search using the category as your query.

Like many other search engines, Wisenut offers links for related queries, and clusters results from individual sites. Wisenut's clustering is unique, though. Instead of the usual "more results" from this site link, Wisenut lists the exact number of pages on a site that it has determined are relevant to your query.

If you're a fan of Google's cached pages, WiseNut offers a feature called "Sneak-a-Peek" that let's you get a preview of a page without leaving the WiseNut's result page. "It eliminates a lot of 'mouse traffic' between the results page and the back button," says Amy Suzanne King, Wisenut's Communications Manager.

Finally, Wisenut has launched a new salvo in the search engine size wars, claiming that they've indexed the full text of 850 million web pages. Not quite up to Google's recently announced billion-plus full text page index, but an impressive number nonetheless, and one that's likely to provide a competitive prod to the other major crawlers to increase their own respective sizes.

Wisenut has stirred up some controversy with its size claims. At one point, the Wisenut's home page boasted nearly 1.5 billion URLs, but this is actually the number of unique URLs the company has identified, but not necessarily crawled and indexed, according to Wisenut CEO Yun.

With today's launch, Wisenut's crawler is capable of fetching more than 100 million URLs per day, using just twelve machines. This suggests that Wisenut's index will likely be quite fresh, presuming the company keeps the crawler constantly active.

The crawler algorithm is highly scalable, says Yun, requiring just a simple addition of processing power to both potentially increase crawling frequency and to keep up with the growth of the web. The crawler uses a patent-pending hash algorithm that can determine within a millisecond whether a URL it encounters is new and should be added to the crawl queue.

Webmasters can submit their site to Wisenut using the submit URL form. The form also allows you to include comments about your site, including information about its content and purpose, but it's not clear whether this is important. The instructions state clearly that the information you provide will not influence how your site is indexed. Nonetheless, submitting an abstract of your site using your own keywords and language probably won't hurt your ranking, either.

Like Google, Wisenut relies heavily on link analysis to calculate relevance, so a good link-building strategy will likely help your odds of being found in Wisenut. In addition to link analysis, Wisenut also examines context-sensitive information for each keyword. In particular, it examines the words in anchor text of URLs, and in the text surrounding URLs.

"We recompute the popularity of a web page based on the keyword," says Wisenut's Yun. "Ranking is very dynamic based on the keyword the user enters." This suggests that traditional optimization techniques will also help improve relevance rankings in Wisenut.

Wisenut's indexing technology has been designed to scale up to 1 trillion URLs. While this may seem like overkill given the current estimates of 2 to 4 billion pages on the web, it means that Wisenut is serious about staying in the game as the web continues to grow.

So, why isn't Wisenut the Google killer? Quite simply because the company doesn't want to be. Wisenut's public search engine is just one part of their overall business, expected to generate only about 20% of revenues. Wisenut's main emphasis will be on providing site search to large web sites, and enterprise search for intranets.

This is where the real money and profit potential for search technology lies, and these markets dwarf public web search services. This is the important point that a surprising number of journalists and pundits tend to overlook. Namely, that search in fact is a real, profit-making business, and that there is plenty of room for more than just a single "king." IDC Analyst Susan Feldman recently estimated that the search technology sector will experience nearly a 50% compounded annual growth rate over the next five years -- and that's likely a conservative number.

Bottom line, Wisenut is a promising new search engine that deserves a place in your search toolkit. It's not a Google killer -- but no other emerging search technology is either, despite media hoohaw to the contrary. When Google takes a fall it will likely be from a self-inflicted blunder akin to those that caused one-time leaders AltaVista, Infoseek, HotBot and Lycos to stumble. But given their track record, it's not likely we're going to see a major misstep from the folks in the Googleplex any time soon.


Wisenut Search Engine White Paper
http://www.wisenut.com/pdf/WISEnutWhitePaper.pdf (link initiates PDF download)
More information about Wisenut technology can be found in the Wisenut Search Engine White Paper.

Wisenut Site Submission Form
Wisenut says you need only submit your top-level URL and its crawler will automatically find and index the rest of the site. You can also include comments about your site, including information about its content and purpose.

Make Room For Teoma
Just as consolidation in the search engine space seemed inevitable, new player Teoma has stepped up with an impressive debut of its new search service that leverages links to provide relevant results.

The End for Search Engines?
Scarcely six months ago, some pundits were lamenting the end of all search engines on the web. In this article, Danny Sullivan looks at the cyclical nature of the web search industry and how shortsighted "analysis" is often simply wrong.

About the author

Chris Sherman is a frequent contributor to several information industry journals. He's written several books, including The McGraw-Hill CD ROM Handbook and The Invisible Web: Uncovering Information Sources Search Engines Can't See, co-authored with Gary Price. Chris has written about search and search engines since 1994, when he developed online searching tutorials for several clients. From 1998 to 2001, he was About.com's Web Search Guide.