Search Engines and Web Server Issues

A special report from the Search Engine Strategies 2003 Conference, August 18-21, San Jose, CA.

A longer version of this article that goes into detail about topics such as static vs. dynamic web pages, search-friendly URL structure, and virtual servers and hosting issues is available to Search Engine Watch members.
Click here to learn more about becoming a member

What is the best way to move a site from one server to another without affecting search engine visibility? What if your company decides to use a different domain name? After a site redesign, how can you communicate to the search engines that old URLs should be redirected to new URLs without affecting positioning? A panel of experts addressed these and other server-specific issues.

Giving Search Engines Access To Your Site

“It sounds really simple, but you must pay your ISP bill,” said Matt Cutts, software engineer at Google. “You will not believe some of the things I see that people do, or don’t do. If Googlebot cannot index your page at first, it will try back a few days later. Maybe if you have a server error, such as a 500-level error like your server is choking, Googlebot will wait for that.”

Since search engine visibility is crucial for most online businesses, web site owners need to ensure that their servers are functional 24 hours a day, 7 days a week. A search engine spider can visit a site at any time. If a search engine crawler requests a page from a server and that server is down for any reason, that page might not appear in a search engine index until the crawler visits the site again.

Checking your robots.txt file to ensure that you are allowing Google in to index your site, said Cutts. Check for typos, syntax, and missing spaces. “Everybody should have this file, even if it’s an empty file,” he said.

Cutts also recommended checking the syntax of your meta tags. “There are a lot of different ways you can use them,” he explained. “NOINDEX means, do not index the content on this page. NOFOLLOW means do not follow the links on this page. And NOARCHIVE has a special meaning for Google: don’t show the cached page only.”

There are other ways to control how easily search engines can crawl your site. “A site map is something people overlook all the time,” said Cutts. “Likewise, see how many links down you need to surf to get to deep pages on your site. The farther you have to go down, the harder it will be for Google to find those pages as well.”

Password protection is another way of insuring that search engines do not crawl sensitive content. “Sometimes we’ll get a letter from a bank or a university saying that they didn’t mean to leave private information up on the public Web where people and search engine spiders could find it,” said Cutts. “Password protection keeps that from happening.”

Site Redesigns And Search Engine Indexing

If you have redesigned your site or updated it with new technology, there are ways of communicating to the search engines that the URLs have been modified. Webmasters can use a temporary redirect, which is a 302 command, or a permanent redirect, which is a 301 command.

“The 301 and 302 refer to those little status messages that appear any time a person or spider tries to retrieve the page. It’s something to be aware of if you’ve moved from an old domain to a new domain,” Cutts explained. “You’ll want to put a 301 redirect on the old domain to the new domain. It will make sure your visitors get redirected correctly to your new site, and it will make sure you still keep credit for your links on the search engines.”

“A 302 temporary redirect tells Googlebot, ‘Okay, go here for now, but try again later because it may not be that way later,'” he further explained. “If it’s going to be that way for good and in the same location, then do a permanent one — a 301 redirect.”

Sometimes, webmasters want to switch IP addresses. In order to do that effectively, make sure that both servers are serving up the content that the search engines ask for before completely making the switch. “At least for a day,” Cutts recommended.

Most search engines have a natural URL-removal tool that eliminates dead links from the search engine index. If you find a URL in a search engine index that is not supposed to be there, you can remove it yourself via a form without having to speak to a customer service representative.

According to Cutts, competitors will not be able to sabotage your site by attempting to remove pages through the URL-removal tool. “What we do is we check several times over the course of two to three days to make sure the page is completely gone on the site,” he explained. “Plus, we have safeguards in place to authenticate that you really did request the removal.”

Google Webmaster Information

Google Remove URL Form

Robots Exclusion Protocol

Shari Thurow is the Marketing Director at Grantastic Designs, Inc. and the author of the book Search Engine Visibility. She has been designing and promoting web sites since 1995 for businesses in a wide range of fields.

A longer version of this article that goes into detail about topics such as static vs. dynamic web pages, search-friendly URL structure, and virtual servers and hosting issues is available to Search Engine Watch members.
Click here to learn more about becoming a member

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication’s search facility, which most have, and search for the headline.

Microsoft to keep using Overture results Nov 18 2003 1:15PM GMT kicks off a TV ad campaign Nov 18 2003 9:58AM GMT
Microsoft testing news search service Nov 18 2003 9:22AM GMT
Microsoft Tests Answer to Google News
Internet News Nov 18 2003 7:06AM GMT
Investor search engine launched
Sunday Times South Africa Nov 18 2003 3:12AM GMT
Microsoft music store to open next year
CNET Nov 17 2003 9:03PM GMT
ICANN Wins a Round in Wait List Case
PC Magazine Nov 17 2003 8:10PM GMT
Google launches battle for search dominance
Sydney Morning Herald Nov 17 2003 6:49PM GMT
The Missing Keyword Of Search Engine Marketing: Multimedia
Media Post Nov 17 2003 12:28PM GMT
Yahoo back in X-rated business
San Francisco Chronicle Nov 17 2003 11:19AM GMT
Yahoo moves to block pop-ups Nov 17 2003 8:45AM GMT
Gates: We never talked to Google
USA Today Nov 17 2003 7:33AM GMT
Garage Gadget Wins Digital Copyright Case
San Jose Mercury News Nov 16 2003 10:27PM GMT
powered by

Related reading

Simple Share Buttons