In late September, Search Engine Watch spoke with Microsoft to learn more about Bing’s new back-end search infrastructure, one component of which is a search indexing technology known as “Tiger.” Now, two additional pieces have been uncovered: a cloud storage and computational engine called Cosmos, and its parallel querying language, SCOPE.
Microsoft told SEW that Tiger, designed to improve efficient information retrieval and reduce company costs, began rolling out in August and will be completed by the end of the year. Tiger uses Solid State Disk (SSD) technology to improve Bing’s process for checking its index against search queries. This isn’t entirely unexpected, as Google upped the game with their Caffeine update last year.
As for Cosmos, it has been described by former Microsoft platform architect Pat Helland as “some of the plumbing for Bing. Helland wrote about the project recently in his blog post announcing his departure from Bing.
“It stores hundreds of petabytes of data on tens of thousands of computers,”Helland said. “Large scale batch processing using Dryad with a high-level language called SCOPE on top of it.”
A Microsoft job posting for a Senior Software Development Engineer describes what they hope to accomplish:
“The index pipeline team owns the complete backend processing for Bing web search. We aim to redefine real-time search and push the envelope on how fast any page anywhere in the world can be indexed. We write software from the ground-up, running across thousands of servers, managing petabytes of data. Our software has to reliably reprocess billions of web documents every day, ensuring that every document gets crawled, joined with the appropriate datasets and then indexed with the correct features. We are chartered with complicated problems such as finding, crawling, processing and serving any interesting and emerging web page in a matter of seconds; It doesn’t matter if it’s a new New-York times article, an posting to Facebook or an update to someone’s personal blog, we want that page in the index the moment it’s available.”
The job posting also says that the engineer will go on to work in “the critical area of Ads,” building the infrastructure for real-time ad processing and delivery for all adCenter properties.
Microsoft has lost billions on Bing, with the online services division losing the company $9 billion since 2007. In September, President of Online Services Qi Lu told financial analysts they plan to bring Bing to profitability over the course of several years by expanding how a search engine functions by better connecting users to the information they need from the search page.