By Danny Sullivan , October 14, 1998
 The Search Engine Watch SpiderSpotting Chart On this chart, * is a wildcard, so *.infoseek.com Updated: October 14, 1998 |
||
| Search Engine | Agent Names | Host Names |
| AltaVista (normal spider) |
Scooter/2.0 G.R.A.B. X2.0 Scooter/1.0 scooter@pa.dec.com |
scooter.pa-x.dec.com scooter*.av.pa-x.dec.com such as: scooter3.av.pa-x.dec.com |
| AltaVista (instant spider) |
Scooter/1.0 | add-url.altavista.digital.com ww2.altavista.digital.com |
| Euroseek | Arachnoidea (arachnoidea@euroseek.com) | *.euroseek.net such as: infra.euroseek.net |
| Excite (mega spider) |
ArchitextSpider | crawl*.atext.com such as: crawl2.atext.com |
| Excite (fresh spider) |
ArchitextSpider | crimpshrine.atext.com |
| Fireball (German search engine) |
KIT-Fireball/2.0 | heavymetal.fireball.de |
| Google (Experimental search engine) |
BackRub/2.1 backrub@google.stanford.edu http://google.stanford.edu/ | *.stanford.edu such as: hake.stanford.edu |
| Inktomi (powers HotBot, others) |
Slurp/2.0
(slurp@inktomi.com; http://www.inktomi.com/slurp.html) |
*.inktomi.com such as: j2001.inktomi.com or j10.inktomi.com |
| Infoseek (normal spider) |
InfoSeek Sidewinder/0.9 | *.infoseek.com such as: wilbur-bbn.infoseek.com or IP number such as: 204.162.98.90 |
| Infoseek (instant spider) |
Mozilla/3.01 (Win95; I) | as above |
| Lycos (regular spider) |
Lycos_Spider_(T-Rex) | lycosidae.lycos.com
or *.pgh.lycos.com such as: spider3.srv.pgh.lycos.com |
| Lycos (Add URL spider) |
Lycos_Spider_(T-Rex) | *.sjc.lycos.com such as: sjc-fe4-1.sjc.lycos.com |
| Northern Light | Gulliver/1.2 | taz.northernlight.com |
| WebCrawler | Served by Excite spiders | Served by Excite spiders |
| More Resources | ||
SpiderSpotting SpiderHunter.com A huge collection of resources devoted to
tracking spiders. You Search
Engine Spider IP Addresses The Web Robots Database BotWatch Configuration File |
||