SEO News
Search

Search Wikia Launches Open Source, Distributed Crawler

newcomb-kevin
by , Comments

Wikia, the commercial site led by Wikipedia founder Jimmy Wales, is taking the next step toward launching its open source, human-assisted Web search tool, Search Wikia.

Speaking at the O'Reilly Open Source Convention (OSCON), Wales announced that Wikia has acquired Grub, the distributed search spidering technology previously owned by LookSmart, and will begin using it to build an index for the Search Wikia project.

Grub, which LookSmart bought in January 2003 for $1.4 million, is a distributed crawling service that LookSmart had implemented as a screensaver that would use idle CPU time on a user's PC to crawl the Web. The data was used to supplement its own centralized crawler's indexing efforts.

Wikia will immediately release Grub to the open source community, and make both the crawler and source code available at Grub.org. Users who download the application can run it either as a screensaver or a background process while other applications are running.

Specifics of the deal were not revealed, though it is part of a larger advertising deal between Wikia and LookSmart which was announced last week.

Under the deal, LookSmart will provide text and display ads in Wikia's freely hosted wiki communities, and eventually on the Search Wikia site, Wales said. Ads will be sold by Wikia on either a cost-per-click (CPC) or cost-per-thousand impressions (CPM) model. Inventory not sold by Wikia will be back-filled by ads from LookSmart's distributed ad network.

Share your thoughts on Search Wikia in the SEW Forums.


The Original Search Marketing Event is Back!
SES DenverSES Denver (Oct 16) offers an intense day of learning all the critical aspects of search engine optimization (SEO) and paid search advertising (PPC). The mission of SES remains the same as it did from the start - to help you master being found on search engines. Early Bird rates extended through Sept 19. Register today!

Recommend this story

comments powered by Disqus