Microsoft's MSN Search To Build Crawler-Based Search Engine
Microsoft's MSN Search is still determining how it will evolve, but a key feature is running its own crawler-based search engine in house.
Microsoft's MSN Search is still determining how it will evolve, but a key feature is running its own crawler-based search engine in house.
In April, news emerged that Microsoft intended to make a huge new investment in web search. Now signs of that investment are appearing. Microsoft’s MSN Search site posted a large list of jobs in May, then drew much attention last month when official information about its own search spider “MSNBOT” was posted to the public.
So what’s the future for MSN Search? The details are still being determined, the service says. However, building its own crawler-based solution to gather editorial listings is seen as a key element needed to win in the search sweepstakes.
“We view it as a three horse race between ourselves, Yahoo and Google, with Google in the lead,” said Lisa Gurry, a group product manager with MSN who’s working with the search team. “Across the board, we’re in a good place with our search engine, but there are good opportunities to increase the relevancy.”
MSN Search is already one of the most popular search sites on the web. But in terms of brand recognition, it’s an entirely different story. Google is so tightly associated with search that some use its name as a synonym for search. In contrast, even MSN admits some people may not realize how they ended up doing a search at its service.
“We think we have a lot of work to do to build equity in our MSN Search brand,” said Gurry. “It doesn’t have the brand recognition that Yahoo and Google have, and that’s an area we hope to address over the next year or so, to raise the visibility.”
One means of doing this may be marketing MSN’s search capabilities, something the service has never done before, Gurry said. Search might also be made more prevalent across the entire MSN network, she added.
MSN’s plans to fight harder in the search space don’t please everyone. Microsoft’s dominance on the computer front is well known. Most personal computers run some version of the Microsoft Windows operating system, and the company commands a large share of certain types of software, such as word processing programs. Some have assumed that Microsoft will use its deep pockets to gain a similar dominance when it comes to web search.
Indeed, fear of this has inspired the Boycott Microsoft Search page, a campaign to enlist bloggers to block Microsoft’s crawler from accessing their web pages.
“Microsoft is building a web search engine, and they intend for it to become the industry standard. Given Microsoft’s track record during the browser wars, there is every reason to believe the company will again use its monopoly power to eliminate competition by building a web search service into the next version of Windows,” the boycott page warns.
Scary sounding stuff. But let’s hearken back to 1997, the very first time it emerged that Microsoft planned to launch its “own” search engine.
“Excite and Yahoo may sound like the happiest places on the web, but their party is about to get crashed. For more than six months, a team at Microsoft has been working on its own search engine/directory, code-named Yukon. The company should have a beta version up by October,” wrote Time magazine, in August 1997.
Nearly five years later after that launch, Microsoft has hardly crashed Yahoo’s party. Excite did lose the portal wars, but that wasn’t because Microsoft had a killer search engine. Despite all of Microsoft’s money, not to mention Yahoo’s, it was Google that came out of nowhere to become the dominant search player.
In fact, Google’s share of the search audience is so large that some believe it should be regulated for having a near monopoly. If anything, Microsoft’s move will for this contingent make Microsoft a possible monopoly-breaker, rather than maker.
Aside from the Boycott Microsoft Search page, others like News.com and an anonymous MSN Search candidate cited by Scripting News have suggested that Microsoft will use its control of the Windows operating system to deliver an audience to its search engine in a way that Google and other competitors cannot do.
This isn’t some future possibility. This is already a fact and has been for ages. The majority of web surfers use Microsoft’s Internet Explorer browser, which has direct integration into MSN Search in a variety of ways. This “built-in search advantage” is how MSN currently gets most of its audience.
“The bulk of our traffic is driven to us from our IE integration and then from the MSN home page,” said Gurry.
Gurry said that Microsoft would like to raise the profile of this search integration, because some Internet Explorer users aren’t aware it is there. In other words, some might push the Internet Explorer search button, get results back from MSN Search yet never realize that it was the Microsoft search engine that helped them. Others may not realize that there’s any integration at all, such as the ability to query MSN Search directly from the browser address bar.
“People don’t necessarily understand the tie-in,” Gurry said.
To solve this, an actual MSN toolbar might be created, perhaps similar to Yahoo Companion or the Google Toolbar, Gurry said. The toolbar might make search capabilities more prominent as well as help build the MSN brand, since the MSN logo would always be present.
“The toolbar would give you presence across other sites, regardless of the site you are on, whether it is MSN or Yahoo. Yahoo’s toolbar is consistently on your desktop, including their search bar,” Gurry said.
Integration changes might add to Microsoft’s traffic. However, there’s every reason to expect that many people will also continue to choose Google, Yahoo or other search engines in droves, just as they have despite years of Microsoft already having a built-in advantage.
Search is not like software. You don’t install it on your desktop, nor does switching search engines cause any serious need to “relearn” commands or reformat data. The only way Microsoft can lock people into its search engine is to literally prevent them from navigating to other search engines, a fairly dramatic move that public opinion would never allow.
Overall, Microsoft’s built-in advantage has helped it be one of the remaining major players in the search sweepstakes. It’s the ticket that has allowed it to compete, but it’s a ticket that’s also been punched. To win, to really push aside Google and Yahoo, Microsoft will need more than this.
What more can Microsoft do? Improve the quality of its search is the obvious answer. However, MSN Search has actually had pretty good quality over the years, a fact that I think has gone largely unrecognized by many. Nevertheless, Microsoft believes things can get even better.
“As we’ve taken close look at search, we’ve asked, how can we improve the experience? Across the board, about 50 to 70 percent of queries go unanswered. That indicates to us that there’s a lot of growth yet to come in the search category,” said Gurry, who explained that the high failure rate is based on Microsoft’s own internal research. “We’ve felt we should really develop our own [crawler-based” search engine to try and solve this problem.”
That “close look” at search happened in March, when the company finished a review of where it was going in search and decided that outsourcing no longer made sense.
“We looked at all options, whether it be potential acquisition or continue to outsource or ultimately the right move we decided, to bring it in house,” Gurry said.
It’s a decision that echoes the earlier move Yahoo made in December, to purchase Inktomi.
“Given how important search is to our businesses, we really needed to control our own destiny in this space and not be dependent on any one third-party provider,” said Jeff Weiner in January, Yahoo’s senior vice president of search and marketplace, explaining his company’s decision.
The moves by Yahoo and MSN represent a fast swing back to search engines owning technology, rather than leasing it.
Back in 1995, it was normal for the popular search sites to have their own internal solutions that powered their main search results. But by the end of last year, the majority of the most popular search sites, Yahoo, MSN and AOL, were outsourcing for their main results. Google alone among the mega-search sites owned its technology.
Now the situation is in full reverse. Yahoo owns its own crawler-technology and almost certainly will shift to using it in the near future. When MSN’s technology is ready, we’ll see the company make its own change. AOL is likely to remains the sole standout for outsourcing.
Microsoft has a huge amount of experience in running a search engine but ironically none at all in operating a crawler to gather the listings that power that search engine. It’s going to be playing serious catch-up in the quest to have a crawler competitive with Google.
“That’s why you’ve seen a large list of jobs out there. We recognize that we have a lot of skilled staff today, but if were going to be able to truly be able to deliver a best of class service, we need to ramp up considerably. It isn’t just in specific areas such as indexing the web but to generally have a bigger team to help support the effort,” Gurry said.
But with both Google and Overture also seeking search candidates — and both owning some well-established search technologies — will Microsoft find the people it needs?
“We think the people are there. We think Microsoft is a valuable company to work for. It’s a unique opportunity to have the Microsoft resources and support, as well as the ability to be an innovator and create some new technology, and to also know you have a strong distribution like MSN,” Gurry said.
Microsoft’s not completely alone, of course. Overture has been happy to tell me and others that they’re working closely with Microsoft to help them make the technological leap.
That might sound pretty odd, given that Overture has its own crawler-based product in the works, a blend of AltaVista and AllTheWeb technologies. Why help MSN when you’d prefer they rent this?
The answer is probably because by being MSN’s best friend, as Forbes recently put it, MSN will build its own crawler but not decide to also take paid listings in-house. That’s far more important to Overture to preserve, as opposed to gaining a contract to do crawling.
It’s also notable just how unequivocal Microsoft is that its partnership with Overture is solid ground.
“At this point of time, we’re very happy with our partnership with Overture. We’ve seen a great return on that partnership and have seen our adverting revenue skyrocket,” said Gurry. “That’s working well, and we don’t plan to make any changes on that front.”
Such statements have been standard fare since SoundView Technology analyst Jordan Rohan caught wind of Microsoft’s plans via an internal memo to increase its spending on search. Rohan speculated that by mid-2005, MSN would have developed its own internal paid listings program.
Overture and Microsoft dismissed the report. In fact, Microsoft’s continued strong support for Overture since then contrasts starkly against Overture’s other major partner, Yahoo, which is happy to say that it has a “flexible” arrangement with Overture.
Overall, I’m left feeling the agreement behind the scenes is one where Overture has agreed to help Microsoft on the crawler-front in return for Microsoft maintaining a strong public support for the Overture partnership on paid listings.
That support doesn’t mean that the paid listings relationship is eternally set in stone, of course. This year, Microsoft has decided it needs to bring editorial crawling in-house. It could very well at some future date decide paid listings should also be brought in-house, as well.
Other companies can attest that befriending Microsoft does not spell long-term success, and it’s a risk even Overture says it understands.
“We don’t just have our heads in the sand that that could happen,” said Overture chief operating officer Jaynie Studenmund, when we spoke about the issue in May.
Ultimately, it will be up to Overture or another paid listing provider to continually show Microsoft that it makes more sense to outsource paid listings than take them internal. With all the work Microsoft has to do on the crawler-front, it’s not hard to imagine that Overture has secured its position, for now.
How about MSN’s other search partners, Yahoo-owned Inktomi and LookSmart?
Inktomi provides MSN with its secondary results. If a search fails to hit matches from MSN’s own editor picks or those supplied by LookSmart, Inktomi kicks in with its own crawler-based listings.
It was widely predicted after Yahoo purchased Inktomi that MSN would seek a new crawler partner, rather than see revenue go to rival Yahoo. Now that MSN is building its own crawler, it seems likely that Inktomi will last until the new crawler is ready to take over, then be dropped.
“It’s a pretty fair assumption. Once we develop what we believe to be a superior algorithmic [crawler-based” engine, the need to partner with Inktomi won’t be there,” said Gurry.
MSN’s deal to use Inktomi results lasts through December 2005. However, the service has also said that the agreement can be terminated before then, if MSN wants.
What about LookSmart, which currently serves as MSN’s partner for its main results through a deal lasting until December 2003?
“We think that the future of delivering the best search experience is likely to be technology driven, so I don’t have anything specific to say on the LookSmart relationship but that we’ll evaluate it,” Gurry said.
Sound like LookSmart might be on the way out? When asked, LookSmart stressed that they are very much still talking about future roles with Microsoft.
“The global relationship between LookSmart and MSN is deep and complex. We are currently in the midst of renegotiating the relationship, and as you can imagine, that takes time. MSN and LookSmart have agreed not to comment on the negotiations while they are in progress,” said LookSmart vice president Dakota Sullivan.
And after my follow-up on the issue with LookSmart, Gurry added some stronger support for the company:
“While we do think that developing better technology will be critical to improving the overall search experience, we also are pleased with our partnership with LookSmart and look forward to working with them on future opportunities as our respective businesses evolve,” Gurry said.
When can we expect to see MSN’s own crawler listings appear? MSN will only say no immediate switchover is planned.
“I can’t give a specific time frame, but it’s not going to be something that happens overnight,” said Gurry. “We’re not making any big changes in the short term. The site will likely look quite a bit different down the road, but we’re talking a fair amount out.”
So in the near term, expect MSN Search to keep following the model it has used for ages: matches from its own human-edited choices, then LookSmart’s human-compiled results and then crawler-based results from Inktomi (as described more on the How MSN Search Works page).
As for the crawler, it’s probably best not to worry much about it. Though it is live and gathering pages, it’s uncertain yet how exactly things will evolve.
“The technology may indeed end up becoming the basis of our own search engine, but we dont know specifically if it may show up,” Gurry said. “We are actually building up our own database and library of documents. It goes beyond looking and documents and actually indexing them. That said, it is very much initial technology that may or may not be commercial technology that we introduce to consumers.”
Despite the hype about the crawler emerging in June, it’s actually been out on the web since at least April, under the agent name of MicrosoftPrototypeCrawler, based on what appears to be the first sighting posted at WebmasterWorld.com. The excitement in June came because general information about the crawler, which also underwent a name change to MSNBOT, was spotted as being offered to the public.
How about that call to block the crawler? Live for two weeks now, less than 100 sites have signed on to the cause. Gurry hopes even the few that have enlisted will reconsider.
“Being associated with Microsoft is good in some ways and can have some negatives for some people, because it is a big company,” Gurry said. But she added, “A lot of people come to MSN because it is a good site and we deliver a good service to consumers….the best way we can do that [with the new search engine” is having all the web sites out there in the process, participating with the MSN crawler.”
Returning to the Microsoft-Overture partnership, it’s interesting to hear how both companies seem to be singing from the same songbook, in terms of how poor the current state of search is said to be.
When Overture announced its acquisition of AllTheWeb, the company said in its press release that even the best search engines return what users want only half the time. Microsoft is now saying the same — 50 to 70 percent of queries go unfulfilled.
These are surprising statistics to hear, because they suggest the state of web search is terrible. Yet back in 2000, the NPD Search & Portal Study found a search success rate (finding what you want every time or most of the time) to be 81 percent, across a range of search engines. At Google, it was up to 97 percent. Even at Overture, then called GoTo, the rate was a high 82 percent.
With such high success rates, how can it be that three years later, everything is worse? That goes against all the anecdotal evidence. Google has not grown in popularity because people find it only useful half the time or worse.
For its part, Microsoft says its figures came out of internal testing.
“The research was conducted by us, talking with consumers and running tests to determine successfulness of search queries,” Gurry said. “They did a search and didn’t feel they found what they were looking for.”
Personally, I think the success at searching is much higher than the figures being bandied about by Overture and MSN. Yet I don’t disagree with the overall attitude that plenty of changes in search are yet to come. Overture’s party line at the moment is to push the analogy of search technology being like 8-track tapes. CDs and MP3s for the search world are yet to come.
Absolutely! In a few years, the idea that we depend mainly on a web page index as the primary repository for answers will probably seem absurd. There are so many different types of data that can be tapped into, and hopefully search engines will have learned to do this intelligently on the fly.
That’s just one of many ways search will probably change, but no one knows for certain. Even Google cofounder and technology president Sergey Brin can’t foresee that future.
“It’s kind of hard to predict, because we are very experimental ourselves,” he said, when we spoke in May about the future of Google. “I certainly like to think that on our site, in the way people find information, that we’d have something really new and different that they hadn’t considered before or seen before.”
Overture’s — and Microsoft’s — interest in portraying search as still new and developing is clear. They don’t want the world to believe that Google has won the game. Nor has Google won. The search game is constantly changing, and only a foolish player will fail to keep evolving.