September 26, 2008

SEW Experts: Common Problems with 404 Error Pages

Misconfigured 404 error pages can affect your search engine rankings. The last thing you need in the Google index is an error page, but even worse are multiple URLs that were mistyped being returned as "OK." In today's Search Marketing Crossfire column, "Common Problems with 404 Error Pages," Chris Boggs and Frank Watson outline some best practices for creating custom 404 pages.

Posted by Kevin Newcomb at 12:00 AM | Permalink | Comments (0)

September 16, 2008

SEW Experts: Hosting Issues and SEO

When it comes to server issues, many of us would rather not get into the down-and-dirty details. In today's organic search engine optimization column, "Hosting Issues and SEO," Mark Jackson reminds us that knowing the basics of server response codes -- and when to use them correctly -- is critical for any competitive Webmaster or search marketer.

Posted by Kevin Newcomb at 12:00 AM | Permalink | Comments (1)

May 6, 2008

SEW Experts: Conducting a Redirect Audit on Your Web Site - Part 2

Knowing which pages on your site are returning errors, and which ones are being redirected can help you pinpoint issues to address. In today's Organic Search Engine Optimization column, "Conducting a Redirect Audit on Your Web Site - Part 2," Mark Jackson shows how performing a redirect audit can help you get to the bottom of those problems.

Posted by Kevin Newcomb at 12:00 AM | Permalink

April 29, 2008

SEW Experts: Conducting a Redirect Audit on Your Web Site

A redirect audit looks at the server redirects that are happening on your site, and which sites are sending visitors to links on your site that are being redirected. It also looks at 404 errors (file not found), as well as other server status codes appearing in your site's log files. In today's Organic Search Engine Optimization column, "Conducting a Redirect Audit on Your Web Site," Mark Jackson shows you how a redirect audit can take care of many issues that search engines might be having with your Web site, and may help you recover visitors you may be losing due to technical issues.

Posted by Kevin Newcomb at 12:00 AM | Permalink

March 18, 2008

SEW Experts: To Rewrite or Not to Rewrite...That is the Question

While URL rewriting can have SEO benefits, it can also cause more SEO problems if it's done hastily, or incorrectly. In today's Organic Search Engine Optimization column, "Dynamic URLs: To Rewrite or Not to Rewrite...That is the Question," Mark Jackson outlines the pros and cons to consider before making a change.

Posted by Kevin Newcomb at 12:00 AM | Permalink

February 1, 2008

SEW Experts: Rewriting URLs: SEO for CMS, E-Commerce, and Dynamic Sites

There's as much confusion and controversy surrounding URL rewriting as Darwin's theory of evolution. In today's SEM Crossfire column, "Rewriting URLs: SEO for CMS, E-Commerce, and Dynamic Sites," Chris Boggs says that just as Darwin's "survival of the fittest" means species most suited for their environment will adapt and survive, URLs need to adapt too.

Posted by Kevin Newcomb at 12:00 AM | Permalink

January 29, 2008

SEW Experts: SEO Millionaire: Who Wants to Be One? - Part 2

Last week, we challenged SEOs to identify which of the big three travel sites has a canonical issue? In today's au Natural column, "SEO Millionaire: Who Wants to Be One? - Part 2," Mark Jackson provides the answer, and responds to voluminous reader response.

Posted by Kevin Newcomb at 12:00 AM | Permalink

December 4, 2007

SEW Experts: Choosing the Best Domains for Search Engine Visibility

What's in a name? Search engine visibility starts with buying the best domains. In today's au Natural column, "How to Choose the Best Domains for Search Engine Visibility," Mark Jackson explains why choosing the right domains, and creating a domain redirect strategy, can be a valuable move in your SEO plan.

Posted by Kevin Newcomb at 12:00 AM | Permalink

February 19, 2007

The 301 Redirect Headache

Moving Web pages that have been indexed by search engines to a new URL via a 301 permanent redirect can cause serious dips in traffic once the search engines discover that the old page has moved. This is a headache that must be planned for whenever anyone considers changing the addresses of their Web pages, for whatever reason.

Barry Schwartz at Search Engine Roundtable last week highlighted a discussion at Google Groups in which Google's Adam Lasnik claimed that it only takes them a "couple of weeks for things to smooth out."

Unfortunately, although Google seems to be able to index pages within a few weeks, the past rankings for those pages are not updated in any time close to that, especially for competitive terms, without some additional effort. The primary method to speed this process seems to be gaining links from authority sites to the new URL, in a rapid fashion. Funny because that also seems to be the way out of what some call the fictional Google Sandbox.

The "trick" often used in order to try to lessen the severity of the old pages' loss of rankings is to employ a 302 redirect instead. This causes Google to keep the listing of the previous page within its index, and often in the same position within the rankings.

Some SEO's recommend using this tactic while "building up value" of the new pages through the form of new links that lead to the new URL. Although I have personally seen this work, I would recommend using 301s right away for all pages. It is nice to do this "off-season" if you are a seasonal kind of business, but unfortunately that isn't the case for every one.

I have always hoped to find a larger sample of case studies which show that Google can perform faster than what people consistently forecast as 3 months before original rankings are regained, if ever. Unfortunately, according to our engineers, the clients we have worked with have rarely seen this rapidity in bounce-back-ability. I asked Barry who agreed that 3 months was much closer to the norm.

So will Google allow users of the Webmaster Central portal to maybe jump line when it comes to regaining lost rankings due to URL rewrite or move? Will its algorithm ever speed this process out or will it stay like this to help avoid accidentally ranking pages which have considerable content changes.

Either way, moving to new URLs is something that will cause headaches no matter what, it seems. Everyone involved with the Web site should know that before the redirects are implemented, and other means of driving traffic to the pages should be considered as a top gap. These means include, but are not limited to the use of Paid Search, traditional marketing, as well as banner placement on well-trafficked sites.

Posted by Chris Boggs at 12:14 PM | Permalink

April 11, 2006

DNS Cache Poisoning & Hijacking Search Results

I reported earlier this morning that some scam artists are using DNS Cache Poisoning to redirect your PPC and organic listings to other pages. First reports of this come from WebmasterWorld, where it has been documented that an advertiser was targeted and exploited. Basically, the attacker will poison your DNS cache (if it is possible) and then "redirect a popular search engine to a pop-up ad site," or something similar.

SEO Consultants has a very detailed write up on this vulnerability. You should immediately check your domain names at DNS Reports to ensure you pass for "Open DNS servers" test.

Posted by Barry Schwartz at 8:55 AM | Permalink

November 23, 2005

Getting Search Engine Love From Affiliate Links

A long time issue with those who ponder using affiliates is whether paying for affiliate links will rob them of link juice with the search engines. If the affiliate links make use of tracking codes or redirection, will search engines count these as much as "real" links?

SEO Friendly Affiliate Systems from Greg Boser is a great tutorial on how affiliate links work in various flavors and how to get the ones that will help you with search engines (short answer, use 301 redirection), assuming your affiliates buy into the idea. As Greg explains, giving you the link juice means they might be less likely to show up themselves.

Want to comment or discuss? Visit our Search Engine Watch Forum thread, Affiliate URLs - 301s vs 302s.

Posted by Danny Sullivan at 10:08 AM | Permalink

October 21, 2005

The MSN PageRank 2 Controversy & Search Engines Needing To Offer Domain Management Tools

As part of the current Google update underway, it's been noticed that MSN now has a PageRank score of 2. What's the deal, Google -- decide to pull a little love away from MSN? Not so, says Google's Matt Cutts -- they're actually a PR8. Then why do you see a PR2 score when you go to MSN? Let's break it down, as well as revisit the oft-desired need for search engines to allow site owners to tell them directly which domains should be treated as the same.

Visit MSN at http://www.msn.com with the Google Toolbar installed, and you'll see a PR2 score reported.
MSN.com down to pagerank 2! at WebmasterWorld has some of the discussion this sparked, where the anonymous GoogleGuy from Google puts the blame on redirection that MSN is doing. Look at http://msn.com, and you'll see a PR8 score is reported, we're told.
OK, but if you try that, you get redirected to http://www.msn.com, where it's still PR2.
Answer? You need to get the Google PageRank score for msn.com in another way than trying to reach that page, since the redirection will send you to what's technically a different page, the home page of www.msn.com
How? Google's Matt Cutts, posting over Threadwatch and sounding pretty in sync with GoogleGuy, explains that msn.com is a PR8 site and points to the Future PageRank checker at SEO Tools as a way to see this. (At this point, you're asking "Isn't Matt Cutts GoogleGuy?" For the record, Matt's never publicly laid claim to being GoogleGuy. But since Matt's more active on commenting with things these days, I think it's well time that GoogleGuy step forward with a real name, so that if they are one and the same, there's isn't confusion that two different people are talking. Honestly, at some point we'll have someone citing GoogleGuy, then someone citing Matt against GoogleGuy, which is absurd if they are the same. I and many others do know the real identity of GoogleGuy. I think it's well time everyone knows and hope GoogleGuy will step forward).
Run a test for msn.com using the checker, and you'll get a list of the PageRank scores reported by various Google datacenters, including the server that feeds the toolbar display. All of them are PR8.
Again, you can't see these scores showing up in your browser when trying to go to msn.com, because you get redirected to www.msn.com, which has a different PR score.

All this brings us back to the issue of redirection. MSN is doing a 302 temporary redirect from msn.com to www.msn.com, which can confuse search engines into knowing if they should be treated at the same site. A 301 permanent redirect would be preferred.

But more preferred than that, life would be a lot easier if site owners could simply register all the various domains that may resolve to their "main" domain with Google and other search engines, rather than them having to guess.

People have been wanting this for ages. C'mon Google and Yahoo! You've both made moves to let us submit sitemaps and URLs to be crawled. Let's get with it so we can register domains with you and how they should be treated through some type of program. It so long overdue. That's especially so given that after the last indexing summit, as I've written, the search engines were unable to unify on any common treatment of dealing with redirects.

Posted by Danny Sullivan at 9:21 AM | Permalink

October 17, 2005

Matt Cutts Gives Tips On Moving Sites & Keeping Google Happy

Google's Matt Cutts has moved his blog to a new host, and he shares some advice on how to keep Google happy if you have to do a similar move here. The last step is most important. Once you are sure Googlebots are visiting the new site and not the old, it's safe to shut things down. He also touches on what do to if you need to change domains (short answer, 301 redirect from old to new). By the way, Retaining Traffic after a Web Site Redesign from SearchDay last year has some related tips you may find useful.

Posted by Danny Sullivan at 7:27 PM | Permalink

September 29, 2005

Listings Hijacked At MSN, With A Little Help From Google

Google 302 and MSN from Dave Naylor is chock full of badness on the parts of both Google and MSN, showing how Google redirections are causing it to hijack listings in MSN's search results. Dave gives you the short rundown. Here's the spelled out version, and thanks for his help in assembling it.

Look at this search result at MSN UK for batman animated bean bag.
See how the first result is for this page at Kids UK?
Now look at the URL MSN UK lists for that page: http://groups.google.co.uk/froogle_url?q= http://www.kidsuk.co.uk/shop/catalog/ Batman-Bedding-p-1-c-1288.html %3Fsource%3Dfroogle&fr=AJrr2tQq23-_SJjef Mma5wwNUyhA6FBUGEdlEBymj9jJAAAAAAAAAAA
See the bold part? That shows that MSN believes this page is hosted at groups.google.co.uk.
What's happening is over at Froogle UK, all links you click on there are redirected out of Google and to the destination sites, but...
Google is using 302 temporary redirection, which is causing MSN to let it "hijack" these listings.
To be clear, MSN is NOT listing a Google page, even though the page has a Google URL. Look at the cached copy of that page, and you can see that it is the same page as at Kids UK. But Google has control over the URL in MSN's results.
In other words, should Google lose its mind, it could at any point send MSN a cloaked version of the Kids UK page and likely maintain the ranking while showing human visitors something else entirely. Kids UK is not in control of that listing on MSN, even though it currently leads to the Kids UK site. It's been hijacked by Google! If Google were using a 301 redirection, however, this shouldn't be happening.
Side point. If this is a Froogle UK thing, why does that URL say GROUPS.google.co.uk? Google UK has some domain madness going on. Visit the home page. Click the Froogle link to get this page. Now click the Groups link to reach this page. Notice now how even though you are in Google Groups, the the froogle.co.uk domain is what shows in your address bar. That shouldn't be happening. Other mix-ups like this are leading to the confusion.
Hey! What's MSN doing crawling Froogle anyway? The robots.txt file there should be keeping it out, right? Sure. But if some site has made copies of Froogle results, scraped the content as fodder for a fake blog or something else to attract traffic, MSN might crawl that and thus see the Froogle redirections.

Overall, a nice demonstration of why MSN needs to consider how it handles redirection. My Revisiting Hijacking & Redirects: Moving To A Solution story gives you more background on the hijacking situation as it especially has impacted Google.

I also wrote that story as a lead in for our Indexing Summit 2 session as SES San Jose that was held last month, to see if we could get a standard solution to handing redirection and eliminate these type of problems. I was planning to finally write up what happened at that session next week, and I still will, promise. But here's the summary:

Yahoo: We have a solution (as described in my article) that seems to work.
Google: Matt Cutts wants to use the Yahoo solution but the engineer overseeing how redirections are handled says they've solved it another way. Matt said if you still see it happening, report it to Google, and then he's got some ammunition to say "I told you so" and get the Yahoo solution going. It's been reported at least once already. Bacon polenta on Matt's blog explains that and more important, gives updated instructions on how to report a hijacking in Google's listings.
Ask Jeeves: Thinks it has a handle on the situation and doesn't need to follow the Yahoo solution.
MSN: Didn't take part in the summit.

Want to discuss or comment? Visit our forum thread, Google Hijacks Batman Room Decor Listing At MSN!

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Posted by Danny Sullivan at 1:42 PM | Permalink

September 28, 2005

Yahoo Paid Inclusion Redirection & Hijacking Confusion

Paid Inclusion Making Yahoo Results Seem Hijacked? looks at the confusing situation one of our forum moderators Jeff Martin found when looking at some listings in Yahoo. They redirected through Business.com until winding up at sites that had nothing to do with that B2B search engine. Jeff also describes more here. What's up? I posted my thoughts in the forum thread. To me, it looks like Yahoo is taking a paid inclusion feed from Business.com -- hence why the click redirects through Business.com before hitting the destination sites. In other words, buy some listings in Business.com, and those are distributed also through Yahoo. That's a long-time tactic. LookSmart long did the same.

Posted by Danny Sullivan at 7:33 AM | Permalink

September 7, 2005

Redirection Resources

Trying to understand more about redirection as it applies to seach marketing? Todd over at Stuntdubl lists a few resources you may want to check out.

Posted by Danny Sullivan at 9:05 AM | Permalink

August 1, 2005

Revisiting Hijacking & Redirects: Moving To A Solution

Cast your mind back to May, when I wrote about the Google AdSense page getting "hijacked" by another site for searches on adsense and google adsense. Why hadn't Google yet solved this problem, I asked, especially when it was something that competitor Yahoo seemed to have cracked?

I talked with Google about the issue in more depth a couple of days later. In short, they're aware that the issue needs to be solved, but they're looking for the best way forward and would like your help.

Moreover, it's something we're going to explore at the Indexing Summit 2 at the SES San Jose show next week, to se if there's an across the board solution for all the major search engines. So any advice and suggestions you have on the topic are welcomed. More on how to contribute is below.

Redirection - From Old To New

To understand the issue, let's revisit the basics. What are redirects, and how are they supposed to operate?

It would be great if pages always had the same URL forever and ever, but that's often not the case. People get new domain names, reorganize their web sites or use a new content delivery system. Those are just some of many reasons that URLs can change.

Change your URL, and people -- or search engines -- can't find the page when they look in the old location. Redirection is the solution. With redirection, a site owner can automatically send requests for the page at the old location to the new one. Think of it as mail forwarding or call forwarding for web pages.

Redirection can be done through a meta refresh tag, but I'm not going to get into that here. Just understand that meta tag redirection is a page-based method of pointing from one place to another. In other words, it's something you put on the actual web page.

Instead, I'm going to instead focus on server-side redirection, where your web server does all the work of pointing from one place to another.

When your server does a redirection (or responds to anything), it sends out a little status code about the request it's processed. You know how when you try to reach a web site that's gone, and you sometimes get a "404 Not Found" error? That's a type of status code a server sends, which in turn may trigger the appearance of that "Not Found" page.

With redirection, there are two main codes that go out, corresponding to the type of redirection happening. These are 301 and 302, numbers you've probably heard if you've been reading about redirection and hijacking issues. Let's look further and what they represent and how things are supposed to go.

301 Permanent Redirect

The W3C guidelines for a 301 "permanent redirect" say that this is for use when a page has been permanently moved and you want people to record the new address in place of the old one.

In other words, say you change domains from superdupersite.com to reallycoolhotsite.com. You want people and search engines to know that the new domain should be used in place of the old one. You'd do a 301 permanent redirect like this:

superdupersite.com --- 301 Permanent Redirect ---> reallycoolhotsite.com

Do that, and it's the URL pointed at (that I've highlighted in bold) that should be retained for use in, say, search listings.

302 Temporary Redirect

The W3C guidelines for a 302 "temporary redirect," as it's commonly though inaccurately referred to, say that this is for use when a page has been temporarily moved to a new location and you want people to KEEP the old address rather than use the new one.

In other words, say one day superdupersite.com gets Slashdotted or receives heavy traffic, too much to keep serving up things as normal. You decide to temporarily send people off to a mirror site, mirror.superdupersite.com. You want people and search engines to reach the new location but not record the address as a permanent change. You'd do a 302 temporary redirect like this:

superdupersite.com --- 302 Temporary Redirect ---> mirror.superdupersite.com

Do that, and it's the site doing the pointing whose URL should be retained for use in search listings.

What Yahoo Does

Those are the official "rules," to some degree, on how redirects are supposed to be handled. However, following them has caused problems with the search engines.

In reaction, Yahoo made changes last year in how it handles redirects, as this slide (PDF file) illustrates. Exactly what Yahoo records will differ from what the W3C suggests, depending on the situation. I'll break that down below.

Yahoo: Redirects Between Domain

The first situation is for redirects between two DIFFERENT web sites or domains. Redirects work like this:

301 Permanent Redirect source-domain --> target-domain = target-domain URL kept and used for listings
302 Temporary Redirect source-domain --> target-domain = target-domain URL kept and used for listings

In the examples above, things work exactly the same. If the "source-domain" (the site doing the redirection) redirects in any way to "target-domain" (the site getting the traffic), the target-domain URL is kept.

This solves any hijacking problem. You can't point at someone else and possibly, as with Google (and likely MSN and Ask Jeeves) somehow get your own URL listed rather than the site you're pointing at. Or more important, somone can't redirect to you and manage to get their own URL listed instead of yours.

Here's what Yahoo's Eric Baldeschwieler, director of software development, emailed me about the change:

We decided to handle all cross domain redirects as permanent redirects to remove possibilities for abuse. We were able to avoid the "hijacking" problem. Also, the webmaster community was vocal in its desire to have good permanent redirect support and we have received very positive feedback on this change

Yahoo: Redirects Within A Domain, Root Pages

Things are more complicated at Yahoo when you redirect within your own web site. First the situation with 301 permanent redirects and root pages or "home" pages:

root-page --> deep-page = rootpage kept and used for listings

What's the logic here? Baldeschwieler said:

When a user searches for a domain, we would like to return a domain root page. This motivates our exceptional handling of domain root pages. We put this in place because it reduces the number of user complaints.

Let's use Amazon as an example. Say you go to Yahoo and search for Amazon by name. You want to reach the Amazon home page, in all likelihood. Type in Amazon.com into your browser to see what the home page URL looks like. It should be something similar to this:

http://www.amazon.com/exec/obidos/subst/home/home.html/LOTSOFNUMBERS

Why isn't the address of the Amazon home page just www.amazon.com? Technically, it is. Enter that address, just the domain name, and the "root" page loads up, the default page the server sends out if you don't give it a specific page. But Amazon redirects requests for that page deep within its site and appends a number for tracking purposes.

Now URLs that show in search results are important to users. Studies have shown that they rely on them for making choices. If you want to reach the Amazon home page, it makes a lot of sense to show a nice, short URL rather than the redirection URL that Amazon uses. That's what Yahoo does with this change. Do a search for Amazon, and you'll see the URL shown is:

www.amazon.com

Yahoo does this by technically breaking the rules, but to me, it's a good reason to break them (and hopefully the "rules" will officially change for search engines).

In contrast, Google follows the rules and so the domain it lists for a search on Amazon looks like this (FYI, the tracking numbers aren't shown because as Google can't be cookied, I believe Amazon doesn't generate a unique code for its spider:

www.amazon.com/exec/obidos/subst/home/home.html

You can see the difference. There are some rare occasions when someone might redirect from their home page and want the "deep" page URL to be used. Yahoo admits this and says perhaps other mechanisms will be found to help solve that. But for the most part, I think this "rule breaking" is a good idea.

Yahoo: Redirects Within A Domain, Deep Pages

Now clear your mind of home pages. What if you want to redirect from one "deep" page within a web site to another deep page. Yahoo does this if you do a 301 permanent redirect:

deep-page --> other-deep-page = the page directed to is used for listings

That makes sense (and follows the rules). If you are redirecting from deep pages within a domain, chances are you really do want the "new" address used in listings.

What if you don't want the new address recorded? Easy. Just use a 302 temporary redirect and it follows this logic:

deep-page --> other-deep-page = the redirect is NOT used for listings

As this is happening WITHIN your domain, Yahoo feels it can follow the rules in this case without risk that someone else is trying to "hijack" your listing, as happens with cross-domain redirection.

What Google Does

How does Google handle redirects? With 301 permanent redirects, it uses the URL of whatever is pointed at. There are no potential hijacking problems with this.

With 302 temporary redirects, technically a search engine should keep the URL of the page doing the pointing as explained. If Google actually did this, the hijacking situation would be far worse than it is now. Anyone could point at anyone else and potentially hijack listings. And what if you encounter multiple sites all temporarily redirecting to another site? Which of the "pointing" domains should be kept?

To date, Google has primarily relied on PageRank to help sort the situation out. It generally assumes that the page with the highest PageRank score is the URL that should be used in search listings. Fair to say, this generally works out to be the case.

With Google's much publicized situation in May, there were a number of very unusual glitches that caused its AdSense page to end up having a lower PageRank score than the person pointing at it. Nevertheless, with so many sites out there, such unusual situations can add up.

Solving The Problem

Back to my original question. Why not do what Yahoo does? Perhaps that's what Google will do. Google said when I spoke with it after the May incident that it wanted to spend the next few months getting feedback and experimenting with what seems to be the best solution. Indeed, it has been doing this.

What's so hard? Consider this situation:

Someone registers a domain name -- myname.com
Someone temporarily redirects from that domain to the home page of their blog, such as radio.weblogs.com/30383482/

Ideally, you'd want to keep the actual domain name used for listings, at least for the home page, rather than pointing at that big giant URL. Following the "rules" for a 302 permanent redirect allows this. And the Yahoo solution doesn't work because this is a cross-domain situation.

Dave Naylor, one of our SEW Forum moderators, recently summed it up even better from someone who was redirecting between two domains:

A long long time ago in a distant valley a young search engineer decides that the Google surfers would be better off seeing www.johnnybladeproductions.com rather than seeing johnnybladeproductions.home.att.net, and for a long time everyone was happy.. the webmaster was happy because Google showed his www. and not the free host. Then people realized that pointing a domain with a 302 would replace other people's URLs...and the whole 302 hijack problems started.

The "everyone was happy days" may have changed but not the fact that there can be unusual situations that come up requiring some thought on how to do redirects. Google wants to explore these more before settling on changes.

Ideally, the other search engines would look at them and all come together with a standard. Indeed, while Yahoo has made changes, those could change again. Emailed Yahoo's Baldeschwieler:

Setting redirect handling policy requires balancing the needs of our users and webmasters. The first priority is to answer our users' queries. We also strive to give webmasters as much control as possible of their contents' representation in our index. Hence we are as open, transparent and standards compliant as possible.

I'd like to emphasize that redirect policy handling is not something we have set in stone. We continue to tune our approach as our systems improve and we receive feedback on our current policies. When we last reviewed our policy, we set out a set of principles which have guided our recent redirect handling decisions. Our current implementation choices were driven by the following principles.

Fix the Problems NOW
Don't wait for the perfect solution, work on creating one and remain flexible.
Pay attention to community feedback and user experience, not just standards when designing solutions.
Webmasters and tool vendors often ignore or reinterpret existing standards

Where possible, conform to standards and give control to webmasters. In cases where the above principles don't suggest a divergence from the standard, we conform to it. In all cases, keep web masters informed of how we are handling redirects.

More Background & Your Feedback

For more on redirection issues, I highly recommend Claus Schmidt's Page Hijack: The 302 Exploit, Redirects and Google. He has lots of information there and has played a major role in helping people understand the problem that redirections can cause. You can also see:

Those are blog posts that reference past forum discussions and information from across the web on the topic, which has just continued to grow over the past year and a half.

In this article, I've focused on the situation with Google and Yahoo. I haven't yet have the chance to talk with Ask Jeeves and Microsoft on the topic, but I hoped this would be a good starting point to go forward.

In particular, I hope you can help. Please provide your comments, suggestions and unusual situations you think need to be considered over in this forum thread: Indexing Summit 2: Give Your Feedback On Handling Redirects. Google especially is looking for that feedback. Perhaps a content-location header solution makes sense. Perhaps there are other solutions people have out there.

I'll be mining that thread for possible solutions that we'll explore at Indexing Summit 2, which will be happening at the next Search Engine Strategies show in San Jose this August. Redirection is one of the two key subjects of that. So please speak up now in the thread, so I can bring your voice out along with the others at the actual summit next month.

Posted by Danny Sullivan at 1:03 PM | Permalink

May 26, 2005

Google Regains Its Hijacked Listing; This Was A Big Deal, Folks!

Two days after it appeared, Google has finally managed to get its hijacked listing restored for queries on adsense and google adsense. Two days! And this to correct a problem it has been told about for over year, a problem it largely dismissed as not being a big issue?

It looks bad, coming days after the recent song-and-dance at the Google Factory Tour about how much energy is supposedly expended on core search and ads. Here's a personalized home page, but don't worry, we're not a portal, Google said.

Funny, this type of inattention is exactly what made people get turned off from the portals of the past, when they lost focus on search quality. Yahoo seems to have fixed this redirect hijacking problem, but Google is still struggling with it?

The JenSense blog has a comment from the person's whose page usurped Google, saying he never meant to do it. His page with a meta redirect to the Google AdSense site was part of a general system of linking to sites that may change addresses.

It's actually pretty reasonable. I still have lots of links leading to GoTo.com, direct links I can only fix if I go through and change them one by one. Instead, this person says he always links to pages in his site that use meta refresh to push you elsewhere. So to link to GoTo, he'd link to his GoTo page. Then if the address of GoTo changed (as it did to Overture.com), then he only needs to change one page.

There are more elegant ways to do this, of course. In fact, many directories will run all links through redirect scripts, for easier management or tracking reasons.

In fact, this was the crux of a Threadwatch discussion back in March: Millions of Pages Google Hijacked via Open Directory feed, which in turn led to a Slashdot thread about it. That in turn brought Google out in the form of GoogleGuy to strongly dispute the claim.

To be fair, GoogleGuy was correct in part, as I emailed Nick Wilson from Threadwatch as we discussed the story:

The fact that this site's taken the ODP and 302 redirected doesn't mean that all or even millions of those sites listed in the ODP have been hijacked. Heck, you and I both wouldn't still show up in Google, if that were the case. Everything I've seen so far seems to indicate this really may only happen to sites with a smaller PR than the page pointing at them, and even then not necessarily in the case.

But that didn't mean it was a non-issue, not at all, as I also wrote:

Having said all that, I've no doubt hundreds, maybe thousands of pages might have been grabbed in this way. And as cornwall says, hard to understand why this site thinks a 302 is necessary other than to hijack this way. There's no doubt some people are being harmed, it is an issue, and it's amazing that Google still isn't solving it.

That same day, I went back to Google about getting an official line on the situation, but we never got a discussion set-up. I've done the same this week, but with the holiday weekend coming up in the US, folks who know are gone. The key question I have hanging out remains: if Yahoo solved the situation, why is Google taking so long?

It's also well worth revisiting the hijack situation raised in March to see the spin Google's put on it, and how that's come back to knock them upside the head. GoogleGuy posted this on the Slashdot thread (I've insert a link to more material):

Here's the skinny on "302 hijacking" from my point of view, and why you pretty much only hear about it on search engine optimizer sites and webmaster forums. When you see two copies of a url or site (or you see redirects from one site to another), you have to choose a canonical url. There are lots of ways to make that choice, but it often boils down to wanting to choose the url with the most reputation. PageRank is a pretty good proxy for reputation, and incorporating PageRank into the decision for the canonical url helps to choose the right url.

A lot of sites that try to spam search engine indices get caught, and their PageRank goes lower and lower as their reputation suffers. We do a very good job of picking canonical urls for normal sites; sites with their PageRank going toward zero are more likely to have a different canonical url picked, though, and to a webmaster I understand that it can look like "hijacking" even though the base cause is usually your reputation declining. For a long time, it was hard to get anyone to report canonicalization problems, because the site that got "hijacked" would be free-cheap-texas-holdem-plus-viagra-and-payday-loa ns-as-well.com type sites. In fact, I had to offer to ignore the spamminess of any reported sites in order to get people to send in any real data.

But even though I suspected that this issue affected very few sites, we still wanted to collect feedback to see how big of a problem it was, and to see if we could improve our url canonicalization. So starting a while ago, we offered a way to report "302 hijacking" to Google; I mentioned the method on several webmaster forums. You contact user support and use the keyword "canonicalpage" in your report. Then I created a little mailing list with some engineers on it, and user support passes on emails that meet the criteria to the mailing list.

So how much reports has all this work (including posting multiple times on lots of webmaster boards to request data) gotten me? The last time I checked, it was under 30. Not a million pages. Not even a hundred reports. Under 30. Don't get me wrong, we're still looking at how we can do better: one engineer proposed a way that might help these sites, and he's got a testset of sites that would be affected by changes in how we canonicalized urls. A few of us have been looking through it to see if we can improve things, but please know that this is not a wildfire issue that will result in the web melting down.

In short, it's not that big of a deal. It impacts relatively few sites. And I agree, that seemed to be the case to me when I started hearing about this last year. It was coming up on SEO forums, but I wasn't hearing it from my own readership in any significant way. It really didn't seem to impact a ton of people.

Nevertheless, it was a problem. And as the situation with Google has now demonstrated, it wasn't a problem that only hits spammy sites or webmasters you might think "deserved" it anyway. It hit Google.

Say it again. It hit Google. Google got its own listing hijacked. I thought I'd seen huge irony in March when WordPress spammed Google after pledging right on the Google Blog to help fight spam or when Google banned one if its own pages for cloaking. But this takes the cake. Google's redirect bug bites Google itself.

That's search quality? That's a class product? That's a laser-like focus on search, to be aware of a problem for over a year, then let it run and run and run until it hits your own site? And then take two days to solve it? And the fix almost certainly isn't one that's been applied across the index as a whole? No, this was a major, huge embarrassing failure.

For more on this recent hijacking issue, see our past post, Google's Own Listing Gets Hijacked. To understand the situation more, see Page Hijack: The 302 Exploit, Redirects and Google from Claus Schmidt, who deserves major kudos as someone who sat down to explain the situation to everyone earlier this year, not to mention his posts on forums before that.

Want to discuss? Please drop by our Google AdSense Page Highjacked forum thread.

Posted by Danny Sullivan at 8:46 AM | Permalink

May 24, 2005

Google's Own Listing Gets Hijacked

I spent yesterday sending four or five emails with a reader who couldn't believe that search listings on Google actually could get hijacked. Yes, I told him -- they do. Today, there was a great example to show him. Google itself had one of its own listings hijacked.

SEO DotComicide spotted this and posted with a screenshot. JenSense also has a screenshot and a longer look here. I'll do a summary below.

Search for adsense or google adsense on Google. What currently comes up first is this:

Google AdSense ... Google AdSense is a fast and easy way for website publishers of all sizes to display relevant Google ads on their website's content pages and earn money ... www.all-in-one-business.com/adsense/ - 16k - Cached - Similar pages

Notice the URL:

www.all-in-one-business.com/adsense/

That's definitely not Google's! Yet if you click on it, you end up at the official Google page, located here:

https://www.google.com/adsense/

What's happening? The URL that's listed is using a meta redirect command like this:

META HTTP-EQUIV="Refresh" CONTENT="0; URL=https://www.google.com/adsense/"

That fast redirect is being treated by Google as if it is a 302 temporary redirect. And in that case, it may substitute the page being pointed at by the redirect with the address of the page doing the pointing.

Claus Schmidt did a bang-up explaining this in great detail back in March. See his Page Hijack: The 302 Exploit, Redirects and Google for more on it.

FYI, Yahoo seems to have solved this problem by going against the way redirects are officially supposed to be treated. Barry Schwartz explains more at the end of this post: Redirects and Rewriting

I heard from someone out at SES Toronto that despite this, Yahoo was still having problems similar to Google with redirection hijacking. However, I talked with Claus today, and he's not heard of this coming back as a problem. Neither have I. Which begs the question, if Yahoo can do it, why can't Google?

For more background, see links in these past posts:

Want to discuss? You'll find chatter at Threadwatch and WebmasterWorld, as well as at our own Search Engine Watch Forums, in the Google AdSense Page Highjacked thread. That thread also covers how another site has managed to hijack the backlinks to Google itself. Instant PR 10 site!

Postscript: Google sent me this comment yesterday: "We are aware of the problem and working to remedy the situation."

Posted by Danny Sullivan at 12:09 PM | Permalink

May 17, 2005

Google Ranking Alternative Domain Tops For Searches On Google

The Google.com does not rank #1 for Google thread in our Search Engine Watch Forums looks at how Google has recently been returning not the URL of its own home page first in response to a search on Google but instead a redirected URL. In other words, rather than getting:

You sometimes get this URL ranked first in a search for google.

https://desktop.google.com/

That looks like Google Desktop is being top ranked -- but note the https prefix. Rather than this being the Google Desktop site at http://desktop.google.com, it's instead a secure server that redirects to Google.

This has been an off-and-on situation from at least the beginning of this month, when the thread began. Google Blogoscoped noted the return of it happening again yesterday. Today, we seem back to "normal." Meanwhile, see the second comment at Google Blogoscoped in response to its story. It's a nice explanation of how this confusion is apparently caused by how SSL requests are treated if a certificate is declined.

Posted by Danny Sullivan at 10:26 AM | Permalink

April 20, 2005

Google Addressing 302 Redirection Hijacking Issues

Google Tackles the 302 Redirect Issue at Search Engine Roundtable highlights comments from GoogleGuy in this WebmasterWorld thread that Google is making changes to help ease concerns that by using redirection, others might hijack your listings in Google. Page Hijack: The 302 Exploit, Redirects and Google from Claus Schmidt provides excellent background on the issue, and does some revisiting in this Threadwatch post: Google's 302 problem solved? Also see Google's Redirect Hijacking Problem Gets Slashdotted and Redirection Problems With Google, Yahoo for more background and links.

Posted by Danny Sullivan at 9:41 AM | Permalink

March 15, 2005

Google's Redirect Hijacking Problem Gets Slashdotted

I've mentioned Google's problem with 302 redirects possibly "hijacking" someone else's page in passing before, always meaning to do a longer article explaining the situation more. The new Page Hijack: The 302 Exploit, Redirects and Google up from Claus Schmidt looks to do an admirable job of explaining the situation, so check that out if you want to come up to speed.

The key thing is that the situation seems to really only impact you if someone does a redirect from a URL with a higher PageRank score than the target page. But that can be done, and the situation has long been a worry to some webmasters.

Now Slashdot has discovered it: Google 302 Exploit Knocks Sites Out. Perhaps that attention might finally prompt Google to solve the problem, where sadly ample discussion on search forums and at search conferences for ages hasn't.

The problem has been discussed on Webmaster World for almost a year in so many threads I can't even begin to point to the originating one. Dupe content checker - 302's - Page Jacking - Meta Refreshes from September 2004 is just one example. Your site is at risk from hijackers is more recent, having started at the end of February.

Our own Come on Google, Fix it !!! thread covers the same subject, outlinks to some key Webmaster World threads and has comments from GoogleGuy on the subject. He says -- as Google itself has told me before -- that they see it as more a minor threat than a reality hitting many sites.

In fact, that thread gained some infamy when GoogleGuy went further and offered an amnesty to anyone worried about spam concerns, if they'd forward example URLs of hijacking. The problem is that while many complain about hijacking, they are also hesitant to have their sites examined. But it can happen. I've seen it. Others have as well.

You might also check out our Let's Test Hijacking A Google Listing, Duplicate Content Penalty Timespan or 302 Redirect, Duplicate Content Penalty Timespan or 302 Redirect and Google Indexing 302 Source URLs Under the Targeted Site threads. Still want more? Kuro5hin Discover Google Hijacking - Bless 'em... lists some discussions at Threadwatch.

That post got kicked off over this write-up at Kuro5hin: Google and the Mysterious Case of the 1969 Pagejackers, also worth a read, and with a huge compilation of further Webmaster World threads at the bottom.

Posted by Danny Sullivan at 2:41 PM | Permalink

January 27, 2005

Redesign & Retain Rankings

Time to redesign your web site? Matt Bailey at Search Engine Guide offers some tips on doing that and maintain your search traffic in Planning Ahead for an Effective Redesign.

Posted by Danny Sullivan at 9:01 AM | Permalink

September 20, 2004

Redirection Problems With Google, Yahoo

A month after it was raised during a session of the Search Engine Strategies show, and even longer since it was raised on various search forums, a bug allowing people to hijack listings at Google continues. Pandia has a nice summary: Spammers hijack web site listings in Google. Discussion in our forums here: MIA in Google? Google bug allows 3rd party hijacking.

Meanwhile, another long-standing problem with redirections, this time on Yahoo's side, also continues. More in our forums: Yahoo 301 Redirect indexing problem

I'm planning a longer look at both issues, for the near future.

Posted by Danny Sullivan at 9:05 AM | Permalink | Comments (0)

See More Posts From:

This Week | This Month

Incisive Interactive Marketing LLC. 2009 All rights reserved.

Legal Notices, Licensing, Reprints & Permissions

Privacy Policy
Affiliates

var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); var pageTracker = _gat._getTracker("UA-564586-7"); pageTracker._setDomainName(".searchenginewatch.com"); pageTracker._trackPageview(); window.collarity_appid = "incmedia"; //> //>

Senior Digital Planner
U.S. International Media Los Angeles, United States

Senior Search Analyst
U.S. International Media Los Angeles, United States New York, United States

Webmaster - Marketing
West Virginia School of Osteopathic Medicine Lewisburg, United States

Web Marketing Manager
Harvard Business Publishing Watertown, United States

0