Dynamic URLs and Crawlers

Matt Cutts puts up another post – this time providing a glossary of URL terms. At the end of the post he talks about the difference between static URLs and dynamic URLs, and correctly tells us: “A dynamic url is a document that requires the webserver to do some computation before returning the web document”.

He also notes that many people define a dynamic URL as being one with parameters on the end of it. This is an outgrowth of the fact that many dynamic pages are generated using the parameters at indices into a database, to help the web application find the right content to display. However, parameters on the end of the URL do not necessarily mean that the page is dynamic. In addition, the lack of parameters does not necessarily mean that the page is not dynamic. These are truly separate things.

Related to this, there has long been a myth out there that search engine crawlers don’t like dynamic web pages. A better way to say it is that search engine crawlers don’t really like lots of parameters on URLs. For one thing, the parameters may be confused with session IDs, and if that happens your crawl is going to get all messed up.

When we sites with lots of parameters on the URL, we often urge them to remap those URLs into something that looks more folder based (e.g. ?id=1234&prodid=4321 get mapped into /auto-parts/carburetors, or something along those lines). This is easily done on Apache servers using your .htaccess file.

Dynamic web pages are perfectly fine. All that is really required is that any time you to go a given URL, your web application will serve up the same content over and over again. As long as this is the case, and your URL is clean looking, the crawler will have no problem with your dynamic web page.