Within Grub's site stats, there's a "Wikia Search" team that crawled around 918k URLs so far. Site stats about Grub members tell a more complete story, as the top 100 members crawled 350 million URLs so far.
Berk showed a short demonstration of the distributed crawling capability of Grub. They all stated that simple URLs were more attractive to crawlers and may even encourage spider to crawl deeper on sites.
A look at Grub, the company and technology that LookSmart purchased earlier this year to do distributed crawling of the web. Should we use dashes or underscores in our URLs to separate keywords? Please note that long URLs may break into two lines...
The Grub crawler visits a list of essentially random URLs sent down from a central server. The standard Grub interface shows you two graphs, displaying your bandwidth "history" and the number of URLs crawled per minute.
The standard Grub interface shows you two graphs, displaying your bandwidth "history" and the number of URLs crawled per minute. The Grub crawler visits a list of essentially random URLs sent down from a central server.
Grub Joins Forces with LookSmart Grub, March 17, 2003 http://www.grub.org/html/news.php? LookSmart has acquired distributed crawling company Grub. Please note that long URLs may break into two lines in some mail readers.
Grub Joins Forces with LookSmart Grub, March 17, 2003 http://www.grub.org/html/news.php? LookSmart has acquired distributed crawling company Grub. Please note that long URLs may break into two lines in some mail readers.
