SpiderSpotting: When A Search Engine, Robot Or Crawler Visits
Offline browsers also come to call, such as NetCarta's WebMapper. Search engines send out what are called spiders, crawlers or robots to visit your site and gather web pages. These robots leave traces behind in your access logs, just as an ordinary...
