WebmasterWorld Bans Spiders From Crawling

Wow. Brett Tabke drops the nuclear bomb of
banning all spiders from
WebmasterWorld. He explains
here that heavy
rogue spidering is the reason behind the move. Members worry in the thread that
as pages drop out of search engines, it will become difficult to impossible to
find anything at WebmasterWorld, which
self-admittedly
lacks good site search.

Brett figures he’s got 60 days until pages drop from places like Google to
get an alternative search solution in place. That seems optimistic to me.
WebmasterWorld is a prominent site and should get getting revisited on a
sub-daily basis. If search engines are hitting that robots.txt ban repeatedly,
they ought to be dropping those pages in short order, or they aren’t very good
search engines. I mean, can you imagine the irony of Google and Yahoo getting
pilloried on WebmasterWorld for taking so long to drop pages after they were
told to do so after the ban was put into place?

A separate issue is the potential loss of search traffic. We have had the odd
site from time-to-time declare that it might
ban Google or
Microsoft
because of opposition to those companies, and we’ve certainly had companies ban
all spiders for other reasons. But in one bold move, WebmasterWorld suddenly is
about to become a big giant test case about what happens to a site if it cuts
itself off from the oxygen of search results — an incredible irony when so many
come to the site looking specifically on how to gain more search traffic.

Realistically, any established site should be able to ride out having no
search traffic at all. WebmasterWorld has plenty of people who will seek it out
directly, plus referral links from other sites will keep traffic going and
perhaps even growing. But search has been estimated to drive anywhere between
7 to
13 percent of new
visitors to a web site, visitors who after they arrive continue to come back. I
wouldn’t want to roll the dice against losing them.

It’ll be interesting to see if WebmasterWorld really sticks with this ban or
seeks other ways of getting its content into the major search engine without
spidering, such as via Google Base or Yahoo’s paid inclusion programs, for
example.

Related reading

hp
A picture of the first slide from Greg Gifford's Brighton SEO presentation, featuring a movie still of a man in a racing car with the words "Marketing to Local Businesses" across it in orange, and below, "Moving beyond local SEO to win the visibility race" in white text.
google mobile
RIM-162_launched_from_USS_Carl_Vinson_(CVN-70)_July_2010