Microsoft's MSN Search To Build Crawler-Based Search Engine

A longer, more detailed version of this article is
available to Search Engine Watch members.
Click here to learn more about becoming a member

In April, news emerged that Microsoft intended to make a huge new investment in web search. Now signs of that investment are appearing. Microsoft's MSN Search site posted a large list of jobs in May, then drew much attention last month when official information about its own search spider "MSNBOT" was posted to the public.

So what's the future for MSN Search? The details are still being determined, the service says. However, building its own crawler-based solution to gather editorial listings is seen as a key element needed to win in the search sweepstakes.

"We view it as a three horse race between ourselves, Yahoo and Google, with Google in the lead," said Lisa Gurry, a group product manager with MSN who's working with the search team. "Across the board, we're in a good place with our search engine, but there are good opportunities to increase the relevancy."

MSN Search is already one of the most popular search sites on the web. But in terms of brand recognition, it's an entirely different story. Google is so tightly associated with search that some use its name as a synonym for search. In contrast, even MSN admits some people may not realize how they ended up doing a search at its service.

"We think we have a lot of work to do to build equity in our MSN Search brand," said Gurry. "It doesn't have the brand recognition that Yahoo and Google have, and that's an area we hope to address over the next year or so, to raise the visibility."

Microsoft: Search Monopolist Or Counterbalance?

MSN's plans to fight harder in the search space don't please everyone. Microsoft's dominance on the computer front is well known. Most personal computers run some version of the Microsoft Windows operating system, and the company commands a large share of certain types of software, such as word processing programs. Some have assumed that Microsoft will use its deep pockets to gain a similar dominance when it comes to web search.

Indeed, fear of this has inspired the Boycott Microsoft Search page, a campaign to enlist bloggers to block Microsoft's crawler from accessing their web pages.

"Microsoft is building a web search engine, and they intend for it to become the industry standard. Given Microsoft's track record during the browser wars, there is every reason to believe the company will again use its monopoly power to eliminate competition by building a web search service into the next version of Windows," the boycott page warns.

Scary sounding stuff. But let's hearken back to 1997, the very first time it emerged that Microsoft planned to launch its "own" search engine.

"Excite and Yahoo may sound like the happiest places on the web, but their party is about to get crashed. For more than six months, a team at Microsoft has been working on its own search engine/directory, code-named Yukon. The company should have a beta version up by October," wrote Time magazine, in August 1997.

Nearly five years later after that launch, Microsoft has hardly crashed Yahoo's party. Excite did lose the portal wars, but that wasn't because Microsoft had a killer search engine. Despite all of Microsoft's money, not to mention Yahoo's, it was Google that came out of nowhere to become the dominant search player.

In fact, Google's share of the search audience is so large that some believe it should be regulated for having a near monopoly. If anything, Microsoft's move will for this contingent make Microsoft a possible monopoly-breaker, rather than maker.

The Built-In Search Advantage

Aside from the Boycott Microsoft Search page, others like and an anonymous MSN Search candidate cited by Scripting News have suggested that Microsoft will use its control of the Windows operating system to deliver an audience to its search engine in a way that Google and other competitors cannot do.

This isn't some future possibility. This is already a fact and has been for ages. The majority of web surfers use Microsoft's Internet Explorer browser, which has direct integration into MSN Search in a variety of ways. This "built-in search advantage" is how MSN currently gets most of its audience.

"The bulk of our traffic is driven to us from our IE integration and then from the MSN home page," said Gurry.

Gurry said that Microsoft would like to raise the profile of this search integration, because some Internet Explorer users aren't aware it is there. In other words, some might push the Internet Explorer search button, get results back from MSN Search yet never realize that it was the Microsoft search engine that helped them. Others may not realize that there's any integration at all, such as the ability to query MSN Search directly from the browser address bar.

"People don't necessarily understand the tie-in," Gurry said.

Integration changes might add to Microsoft's traffic. However, there's every reason to expect that many people will also continue to choose Google, Yahoo or other search engines in droves, just as they have despite years of Microsoft already having a built-in advantage.

Search is not like software. You don't install it on your desktop, nor does switching search engines cause any serious need to "relearn" commands or reformat data. The only way Microsoft can lock people into its search engine is to literally prevent them from navigating to other search engines, a fairly dramatic move that public opinion would never allow.

Overall, Microsoft's built-in advantage has helped it be one of the remaining major players in the search sweepstakes. It's the ticket that has allowed it to compete, but it's a ticket that's also been punched. To win, to really push aside Google and Yahoo, Microsoft will need more than this.

An End To Outsourcing

What more can Microsoft do? Improve the quality of its search is the obvious answer. However, MSN Search has actually had pretty good quality over the years, a fact that I think has gone largely unrecognized by many. Nevertheless, Microsoft believes things can get even better.

"As we've taken close look at search, we've asked, how can we improve the experience? Across the board, about 50 to 70 percent of queries go unanswered. That indicates to us that there's a lot of growth yet to come in the search category," said Gurry, who explained that the high failure rate is based on Microsoft's own internal research. "We've felt we should really develop our own [crawler-based” search engine to try and solve this problem."

That "close look" at search happened in March, when the company finished a review of where it was going in search and decided that outsourcing no longer made sense.

"We looked at all options, whether it be potential acquisition or continue to outsource or ultimately the right move we decided, to bring it in house," Gurry said.

It's a decision that echoes the earlier move Yahoo made in December, to purchase Inktomi.

"Given how important search is to our businesses, we really needed to control our own destiny in this space and not be dependent on any one third-party provider," said Jeff Weiner in January, Yahoo's senior vice president of search and marketplace, explaining his company's decision.

The moves by Yahoo and MSN represent a fast swing back to search engines owning technology, rather than leasing it.

Back in 1995, it was normal for the popular search sites to have their own internal solutions that powered their main search results. But by the end of last year, the majority of the most popular search sites, Yahoo, MSN and AOL, were outsourcing for their main results. Google alone among the mega-search sites owned its technology.

Now the situation is in full reverse. Yahoo owns its own crawler-technology and almost certainly will shift to using it in the near future. When MSN's technology is ready, we'll see the company make its own change. AOL is likely to remains the sole standout for outsourcing.

Building The Crawler

When can we expect to see MSN's own crawler listings appear? MSN will only say no immediate switchover is planned.

"I can't give a specific time frame, but it's not going to be something that happens overnight," said Gurry. "We're not making any big changes in the short term. The site will likely look quite a bit different down the road, but we're talking a fair amount out."

Despite the hype about the crawler emerging in June, it's actually been out on the web since at least April, under the agent name of MicrosoftPrototypeCrawler, based on what appears to be the first sighting posted at The excitement in June came because general information about the crawler, which also underwent a name change to MSNBOT, was spotted as being offered to the public.

How about that call to block the crawler? Live for two weeks now, less than 100 sites have signed on to the cause. Gurry hopes even the few that have enlisted will reconsider.

"Being associated with Microsoft is good in some ways and can have some negatives for some people, because it is a big company," Gurry said. But she added, "A lot of people come to MSN because it is a good site and we deliver a good service to consumers....the best way we can do that [with the new search engine” is having all the web sites out there in the process, participating with the MSN crawler."

A longer, more detailed version of this article is
available to Search Engine Watch members.
Click here to learn more about becoming a member