SearchDay is our free daily e-mail summarizing the day's search marketing news. See example.

11) Crawler Name Report

By Danny Sullivan, Search Engine Watch, Mar 1, 1997

Crawler Report

First request analyzed: Saturday, March 01, 1997 06:21 AM

Last request analyzed: Sunday, June 01, 1997 06:55 PM

Crawler Agent Name
Browser	Total Requests	Visits
InfoSeek Sidewinder/0.9	205	110
WiseWire-Widow-1.0	181	98
linecker/1.0	89	89
Lycos_Spider_(T-Rex)/3.0	41	39
SiteTech-Rover	39	39
BackRub/2.0 backrub@pcd.stanford.edu http://backrub.stanford.edu/	33	31
WiseWire-Widow-Beta-2	30	30
ArchitextSpider	24	24
CyberSpyder Link Test/2.0 alpha 4a	25	24
Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)	20	19
Web21 CustomCrawl bert@web21.com	18	18
MOMspider/1.00 libwww-perl/0.40	16	16
Lycos_Spider_(T-Rex)/1.0	37	15
NetMechanic	14	14
MS Web Ctrl	13	12
Netscape-Catalog-Robot/1.0	12	12
Scooter/1.0 scooter@pa.dec.com	9	9
fido/1.0 Harvest/1.4.pl2	8	8
Microsoft IE 3.x	7	7
WebCrawler/3.0_Robot libwww/5.0a	11	7
etsai@netscape.com	6	6
CERN-LineMode/unspecified libwww/2.17	10	5
CyberSpyder Link Test/2.0 alpha 3b1	5	5
Gulliver/1.1	5	5
CyberSpyder Link Test/2.0 beta 2	4	4
Netscape 3.x	4	4
TestApp/1.0 libwww/5.0a	4	4
Folio_Retriever	3	3
PyBot/1.0 python/1.4	4	3
WebZone/15	6	3
HTTPGET/1.0 GlimpseHTTP/3.0	2	2
ia_archiver/1.6	2	2
Infoseek Robot 1.17	2	2
Libertech-Rover	2	2
LinkLint	3	2
LMTASpider/OTWR:002p116 libwww/2.17	2	2
Netscape-Catalog-Robot/1.01	2	2
Road Runner: ImageScape Robot (lim@cs.leidenuniv.nl)	4	2
SiteTech Rover	2	2
TITAN/2.0	2	2
Wget/1.4.2	2	2
WiseWire-Widow-Alpha	2	2
Arachnoidea	1	1
CyberSpyder Link Test/2.0 beta 1a	1	1
CyberSpyder Link Test/2.0 beta 1a (geneseib@ccp.com)	1	1
CyberSpyder Link Test/2.0 beta 1a (iwilsker@ih2000.net)	1	1
CyberSpyder Link Test/2.0 beta 1a (jay.nicol@cominco.com)	1	1
CyberSpyder Link Test/2.0 beta 2 (btphelps@officeiq.com)	1	1
CyberSpyder Link Test/2.0 beta 3 (bigeye@pobox.com)	1	1
CyberSpyder Link Test/2.0 beta 3 (jbacon@arts.state.tx.us)	1	1
CyberSpyder Link Test/2.0 beta 4 (baywood@memes.com)	1	1
Flipper/1.1 libwww/5.0a	1	1
fly/6.01 libwww/4.0D	1	1
FooBar/0.0 via Harvest Cache version 3.0pl4-Solaris	1	1
FreeCrawl	1	1
Industry Canada Bot (0.1 Alpha)	2	1
iSpi/2.0	1	1
Java102	2	1
KIT_Fireball/1.1 libwww/5.0a	1	1
LMTASSpider/OTWR:002p116 libwww/2.17	1	1
Microsoft IE 2.x	1	1
Mozilla/iSpi-2.0-Semio	1	1
MuscatFerret	1	1
NetCarta_WebMapper/7<;:?>=BA@E (end-user robot-please see http://www.netcarta.com/wpindex.htm for more info)	1	1
NetCarta_WebMapper/kmokxxurtvr (end-user robot-please see http://www.netcarta.com/wpindex.htm for more info)	1	1
RoboDude/1.0x1	1	1
Roverbot	3	1
Squirrel/OTWR:002p116 libwww/	1	1
Superewe/v2.0	1	1
WebCrawler/3.0 Robot libwww/5.0a	1	1

Crawler Host Name
Site IP	Total Requests	Visits
tower.york.ac.uk	89	89
galore-bbn.infoseek.com	154	66
spider2.wisewire.com	139	60
208.215.47.178	54	54
205.187.9.85	23	22
tarantula.av.pa-x.dec.com	17	17
crimpshrine.atext.com	16	16
lycosidae.lycos.com	36	14
www.netmechanic2.com	14	14
163.234.32.2	11	11
backfire.eccosys.com	11	11
spider3.srv.pgh.lycos.com	11	10
cero.stanford.edu	10	9
scooter.pa-x.dec.com	9	9
crawler1.anzwers.ozemail.net	10	9
crawl3.atext.com	8	8
backdraft-bbn.infoseek.com	8	8
205.187.9.70	8	8
206.64.113.4	8	8
spider6.srv.pgh.lycos.com	8	8
hake.stanford.edu	7	7
j10.inktomi.com	7	7
spider5.srv.pgh.lycos.com	8	7
blue-bbn.infoseek.com	6	6
204.192.112.115	10	6
southpaw-bbn.infoseek.com	7	6
sole.stanford.edu	6	6
h-205-217-229-125.netscape.com	6	6
spider2.srv.pgh.lycos.com	6	6
ricis.cl.uh.edu	10	5
204.162.98.11	7	5
204.29.186.36	5	5
spider1.wisewire.com	5	5
203.254.168.202	5	5
production.blazenet.net	5	5
spider7.srv.pgh.lycos.com	5	5
backdraft.eccosys.com	4	4
204.29.186.44	4	4
tornado.northernlight.com	4	4
entreworld.org	4	4
deep.webcrawler.com	8	4
205.130.217.230	4	4
cunner.stanford.edu	3	3
welles.library.nwu.edu	3	3
h-205-217-251-244.netscape.com	3	3
dutyfree.proxy.lucent.com	3	3
toucan.webcrawler.com	3	3
pyros.cs.uoregon.edu	3	3
fig.leba.net	3	3
index1-gui.server.virgin.net	3	3
gather.cnri.reston.va.us	4	3
cliffy.lfwc.lockheed.com	3	3
durgon.stanford.edu	3	2
199.75.88.101	2	2
amberjack.stanford.edu	2	2
mullet.stanford.edu	2	2
204.162.34.253	2	2
205.130.230.37	2	2
140.122.190.96	2	2
204.192.112.178	2	2
206.232.236.23	2	2
echo.cica.fr	2	2
beta.alphaspace3.com	4	2
homer-bbn.infoseek.com	2	2
151.143.230.63	2	2
cust124.max2.new-york.ny.ms.uu.net	2	2
j8.inktomi.com	2	2
nat-20-145.wisewire.com	2	2
c298106-a.twsn1.md.home.com	3	2
ppp10-18.lbcktx.onramp.net	2	2
al.infoseek.com	6	2
207.122.100.132	2	2
spider4.srv.pgh.lycos.com	2	2
nlptitan1.navi.ntt.co.jp	2	2
jaring.nmhu.edu	1	1
ws-koe1.win-ip.dfn.de	1	1
h-205-217-250-253.netscape.com	1	1
142.230.199.215	1	1
j11.inktomi.com	1	1
207.86.163.68	1	1
cust62.max2.new-york.ny.ms.uu.net	1	1
panorama.nus.sg	1	1
firestone.archive.org	1	1
207.122.100.103	1	1
207.122.100.142	1	1
roadkill.webcrawler.com	1	1
202.96.61.66	2	1
cust19.max4.new-york.ny.ms.uu.net	1	1
eg-019-079.eglin.af.mil	1	1
ppp2709.on.sympatico.ca	1	1
canopus.inel.gov	1	1
imt23.pi.net	1	1
ts5ip173.cadvision.com	1	1
204.29.186.41	1	1
dial_14.sanet.com.br	1	1
memes.com	1	1
204.91.2.208	1	1
205.253.6.164	1	1
204.252.47.2	1	1
netscoop.cts.com	1	1
www.semio.com	1	1
206.232.236.120	1	1
seven.eccosys.com	1	1
ciebot04.uco.es	1	1
arachnophilia.alta-vista.net	1	1
207.49.155.247	2	1
www.lycos.com	1	1
demonet.opentext.com	1	1
shellx.best.com	1	1
140.158.225.28	1	1
proweb.cica.fr	1	1
203.65.188.17	1	1
kuban.csc.liv.ac.uk	1	1
dialup037.leuven.eunet.be	1	1
funnel55.btx.dtag.de	1	1
207.122.100.141	1	1
bos-ip-3-90.ziplink.net	1	1
anx-gar0101.deltanet.com	1	1
pm4_89.pionet.net	1	1
whowhere.com	1	1
heavymetal.cs.tu-berlin.de	1	1
195.84.82.7	1	1
ae052.du.pipex.com	1	1
ppp179-stk0.sirius.com	1	1
ts47ip176.cadvision.com	1	1
p4.haifa2.actcom.co.il	1	1
sterling.alexa.com	1	1
p8.haifa2.actcom.co.il	1	1
heinlein.empirical.com	1	1
ind156b.wi.leidenuniv.nl	2	1
ind134a.wi.leidenuniv.nl	2	1
enda05.usnet.us.net	2	1
strike.infoseek.com	1	1
flr-jk.dbc.bib.dk	1	1
dialin1-6.ndirect.co.uk	1	1
ppp09.bis.com.br	1	1
golem.ub.fu-berlin.de	1	1
max4-6.cei.net	1	1
support02.peg.apc.org	1	1
ns3.cybersoft.co.uk	1	1
interpix.com	1	1
automation.ipa.ro	1	1
207.19.249.100	1	1
eagle1.cc.gasou.edu	1	1
www1.uk.mdis.com	2	1
196.22.74.117	1	1
sakku-ppp46.arctic.ca	1	1
www-5.one.net	1	1
fnord.frontnet.pp.se	1	1
cust50.max39.new-york.ny.ms.uu.net	1	1
testament.webtv.net	1	1
ts017p14.pop4a.netvision.net.il	1	1
dnice.bridge.net	1	1
tlv-shark119.artaxia.com	1	1
local26.danbbs.dk	1	1
local32.danbbs.dk	1	1
193.252.152.248	1	1
orion.aye.net	1	1
ic1.netal.com	1	1
193.189.227.22	1	1
208.213.91.10	1	1
h-204-29-186-36.netscape.com	1	1
ppp10-14.lbcktx.onramp.net	1	1
cust4.max13.philadelphia.pa.ms.uu.net	1	1
cable230.blazenet.net	1	1
199.72.39.19	1	1
p4.haifa1.actcom.co.il	1	1
roverbot.com	3	1
rcy3.info.isdnet.fr	1	1
ppp-206-170-122-14.sndg02.pacbell.net	1	1
user-37kbo1k.dialup.mindspring.com	1	1
206.213.84.26	2	1
200.5.127.25	1	1
bathtub.nikel.ulst.ac.uk	1	1
preview.sitetech.com	1	1
user232.icor.fr	4	1
dip034-1.hamburg.on-line.de	1	1
207.79.34.5	1	1
crawler.ic.gc.ca	2	1
dyn-140.vin.oleane.com	1	1
180.orlando-006.fl.dial-access.att.net	1	1
max1-22.ystel.fr	1	1
spider1.srv.pgh.lycos.com	1	1
async10.pinrt3.telecom.at	1	1
cust84.max23.new-york.ny.ms.uu.net	1	1
pm3_183.pionet.net	1	1
dyn2125a.dialin.rad.net.id	1	1
dialup3.1.ccp.com	1	1
ocaxp1.cc.oberlin.edu	1	1
testament.artemis.com	1	1
ludwik.uni-paderborn.de	1	1
202.54.4.102	2	1
bracewel.demon.co.uk	1	1
beastie-ppp9.knoware.nl	1	1
info.nic.surfnet.nl	1	1
202.54.3.241	9	1
204.65.45.30	1	1
195.84.82.5	1	1
dat-cw.dbc.bib.dk	1	1
pm5-44.image.dk	1	1
ppp10-4.lbcktx.onramp.net	1	1
genova6-63.tin.it	1	1
slc-ut5-17.ix.netcom.com	1	1
129.247.185.83	1	1
osiris.waikato.ac.nz	1	1
sundog.demon.co.uk	1	1
infov58.vnet.es	2	1
194.216.219.72	1	1
206.185.8.9	1	1
aecsun1.ca.uky.edu	1	1
taz.northernlight.com	1	1