SEO News
Search

Search Wikia Launches Open Source, Distributed Crawler

newcomb-kevin
by , Comments

Wikia, the commercial site led by Wikipedia founder Jimmy Wales, is taking the next step toward launching its open source, human-assisted Web search tool, Search Wikia.

Speaking at the O'Reilly Open Source Convention (OSCON), Wales announced that Wikia has acquired Grub, the distributed search spidering technology previously owned by LookSmart, and will begin using it to build an index for the Search Wikia project.

Grub, which LookSmart bought in January 2003 for $1.4 million, is a distributed crawling service that LookSmart had implemented as a screensaver that would use idle CPU time on a user's PC to crawl the Web. The data was used to supplement its own centralized crawler's indexing efforts.

Wikia will immediately release Grub to the open source community, and make both the crawler and source code available at Grub.org. Users who download the application can run it either as a screensaver or a background process while other applications are running.

Specifics of the deal were not revealed, though it is part of a larger advertising deal between Wikia and LookSmart which was announced last week.

Under the deal, LookSmart will provide text and display ads in Wikia's freely hosted wiki communities, and eventually on the Search Wikia site, Wales said. Ads will be sold by Wikia on either a cost-per-click (CPC) or cost-per-thousand impressions (CPM) model. Inventory not sold by Wikia will be back-filled by ads from LookSmart's distributed ad network.

Share your thoughts on Search Wikia in the SEW Forums.


ClickZ Live New York What's New for 2015?
You spoke, we listened! ClickZ Live New York (Mar 30-Apr 1) is back with a brand new streamlined agenda. Don't miss the latest digital marketing tips, tricks and tools that will make you re-think your strategy and revolutionize your marketing campaigns. Super Saver Rates are available now. Register today!

Recommend this story

comments powered by Disqus