Articles on spidering

RSS

Google Launches Robots.txt File Checker; Now We Need Robots.txt Standardization

Google now provides a way to check on that through the Google Sitemaps program. A real life example of how nice this is, along with a plea that the robots.txt standard needs to become, well, more standard. Google now provides a way to check on that through the Google Sitemaps program. A real life example of how nice this is, along with a plea that the robots.txt standard needs to become, well, more standard. 0 Comments

Google Adds More Fresh Pages, Changes Robots.txt & 403 Errors, Gains iWon

Over the past month, Google has stepped up the number of pages that it is spidering on a daily basis in an effort to increase the freshness of its database. The search engine has also changed the way it reacts if given a 403 forbidden error message when asking for a robots.txt file and has picked up iWon as a new customer for its search results. Over the past month, Google has stepped up the number of pages that it is spidering on a daily basis in an effort to increase the freshness of its database. The search engine has also changed the way it reacts if given a 403 forbidden error message when asking for a robots.txt file and has picked up iWon as a new customer for its search results. 0 Comments

Lycos & FAST Launch Paid Inclusion

After months of beta testing, FAST has finally opened its paid inclusion program to the general public. However, FAST is not selling inclusion into its web index directly. Instead, FAST is allowing its search partners to resell the service to web site owners and site promoters. After months of beta testing, FAST has finally opened its paid inclusion program to the general public. However, FAST is not selling inclusion into its web index directly. Instead, FAST is allowing its search partners to resell the service to web site owners and site promoters. 0 Comments

AltaVista Opens Paid Inclusion

After months of talking about it, AltaVista finally unveiled a paid inclusion program last week. Similar to the program Inktomi began offering last September, this guarantees that submitted pages will be listed within AltaVista's web page index in a prompt manner and revisited on a regular basis. After months of talking about it, AltaVista finally unveiled a paid inclusion program last week. Similar to the program Inktomi began offering last September, this guarantees that submitted pages will be listed within AltaVista's web page index in a prompt manner and revisited on a regular basis. 0 Comments

Position Pro Manages Submissions, Optimization Efforts

When you are working with a site that has a large number of new or changing pages, it would be ideal to have a tool that can do a deep submit to the various search engines, according to their different limits. That's exactly what Position Pro has offered since the middle of 1999. It's one of the few decent deep submit services I've seen worth recommending. When you are working with a site that has a large number of new or changing pages, it would be ideal to have a tool that can do a deep submit to the various search engines, according to their different limits. That's exactly what Position Pro has offered since the middle of 1999. It's one of the few decent deep submit services I've seen worth recommending. 0 Comments

Lycos Partners More Closely With FAST

Lycos has finally committed to using FAST Search's results to power the "Web Sites" section of its search results. Over the past two months, Lycos experimented with using results from its own crawler, from Inktomi and from FAST Search in this section. Now, FAST Search, in which Lycos has a significant investment, has been given the nod. The change does not impact Inktomi's deal to power results at Lycos-owned HotBot. Lycos has finally committed to using FAST Search's results to power the "Web Sites" section of its search results. Over the past two months, Lycos experimented with using results from its own crawler, from Inktomi and from FAST Search in this section. Now, FAST Search, in which Lycos has a significant investment, has been given the nod. The change does not impact Inktomi's deal to power results at Lycos-owned HotBot. 0 Comments

Bye To Go's Instant Add Feature

Go (Infoseek) has dropped its instant Add URL feature. Previously, any pagedirectly submitted would appear within a day or two. Now that time frame hasbeen dropped back to a week, and a faster add won't be coming back, says JanPederson, Go's Director of Search. Moreover, because of on-going engineeringworks, new submits will take longer than a week to appear, in the short term. Onthe other hand, the spidering changes that Go has underway will allow it to morein-depth crawling of sites than in the past, Pederson said. Expect a follow upin a coming newsletter Go (Infoseek) has dropped its instant Add URL feature. Previously, any pagedirectly submitted would appear within a day or two. Now that time frame hasbeen dropped back to a week, and a faster add won't be coming back, says JanPederson, Go's Director of Search. Moreover, because of on-going engineeringworks, new submits will take longer than a week to appear, in the short term. Onthe other hand, the spidering changes that Go has underway will allow it to morein-depth crawling of sites than in the past, Pederson said. Expect a follow upin a coming newsletter 0 Comments

Submitting To Lycos

Forget everything you ever learned about submitting to Lycos, because thesituation is now radically different. First, it is imperative that you get yoursite listed within the Open Directory, since that's now powering the mainlistings at Lycos. However, it's probably best to do this via Lycos rather thandirectly at the Open Directory itself, for the moment. This will make more senseafter we go through the submission tips below.To start, imagine the top search term you'd like to be found for. Now searchfor it at Lycos Forget everything you ever learned about submitting to Lycos, because thesituation is now radically different. First, it is imperative that you get yoursite listed within the Open Directory, since that's now powering the mainlistings at Lycos. However, it's probably best to do this via Lycos rather thandirectly at the Open Directory itself, for the moment. This will make more senseafter we go through the submission tips below.To start, imagine the top search term you'd like to be found for. Now searchfor it at Lycos 0 Comments

Infoseek Check URL Service Suffers Bugs

Infoseek's Check URL service is currently not working, causing some people to believe that their pages aren't being listed, when they are. Infoseek is hoping to fix the problem soon, though there is no date as to when this may happen, according to Infoseek's Nilo Zaratan, who oversees the spidering system. In the meantime, there is an easy workaround. See the link below for the "Checking Your URL" page, which explains how to use the url: command at Infoseek to find your pages Infoseek's Check URL service is currently not working, causing some people to believe that their pages aren't being listed, when they are. Infoseek is hoping to fix the problem soon, though there is no date as to when this may happen, according to Infoseek's Nilo Zaratan, who oversees the spidering system. In the meantime, there is an easy workaround. See the link below for the "Checking Your URL" page, which explains how to use the url: command at Infoseek to find your pages 0 Comments

Infoseek Instant Add Gone For Good

Infoseek Instant Add Gone For Good From The Search Engine Update August 19, 1998 Infoseek says that its "instant add" feature will not be returning to the service. The feature allowed any page submitted via the Add URL form to appear in the Infoseek index within minutes Infoseek Instant Add Gone For Good From The Search Engine Update August 19, 1998 Infoseek says that its "instant add" feature will not be returning to the service. The feature allowed any page submitted via the Add URL form to appear in the Infoseek index within minutes 0 Comments

Infoseek Institutes Listing Delay

Infoseek is no longer listing sites submitted to it within minutes. Instead, it now takes between one to three days for pages to appear. Infoseek is no longer listing sites submitted to it within minutes. Instead, it now takes between one to three days for pages to appear. 0 Comments

Patent Wars!

Lycos and Infoseek both made announcements regarding search engine patents, making analysts wonder if royalty suits may be coming. Lycos and Infoseek both made announcements regarding search engine patents, making analysts wonder if royalty suits may be coming. 0 Comments