SEO Guide Expands, But Ratings Upset Some Firms
Late last year, I reviewed MarketingSherpa's "Buyer's Guide to Search Engine Optimization & Positioning Firms." Since that time, the first edition of the guide has undergone two further updates and a completely new second edition is on the way.
The updates to the first edition have continued to build the guide into a resource that I'd highly recommend to anyone seeking a search engine marketing firm. However, they've also given rise to some serious concerns from some search engine marketing companies.
The guide was first released last September and contained reviews of 24 US-based search engine marketing firms, with a promise to add over 30 additional firms by March 2002.
The second update came out in October, listing two new US firms and 17 UK-based companies. However, a much more important change was the addition of a "Best Practices" ranking chart. This gave companies a letter grade showing how well they scored against the Buyer's Guide's "Best Practices" criteria. The grades were maintained when the latest update listing additional firms in the US, Canada, Australia and the UK came out at the end of February.
Grading Search Engine Marketing Firms
MarketingSherpa said the grades were added to make it easier for the guide's readers to identify which firms followed practices that might cause them trouble with the search engines, a major worry for potential buyers of SEM services.
"You dont want to get your company bounced from the search engines," said the guide's new managing editor, Alexis Gutzman.
Companies earned points for following good practices that MarketingSherpa deemed unlikely to jeopardize what it calls "client safety" on search engines. For example, a company that said link building or keyword research was important to the work they performed gained points.
Bad practices, such as depending heavily on cloaking or mirror sites, cost points. Companies earning the most points earned an A, with the scale slipping down to the ultimate failing grade of an F.
A problem with the ratings is that MarketingSherpa's Best Practices criteria are not an official industry standard. Indeed, the search engine marketing industry lacks any real "official" standards about what's "right" and "wrong."
Certainly some things would be commonly accepted as "bad." For example, anyone trying to mislead a search engine about the true content of their web pages is not doing acceptable work. All the major search engines would agree with this, as would any reputable SEM firm.
On other issues, the consensus is more muddy. Many dislike the use of "doorway" pages, but those that employ them might argue that they are the only way to represent some web sites that refuse to alter image-heavy pages.
Then there's cloaking, which certainly is bad, right? Not at all. At Google, yes it is, though people still get away with it there. However, AltaVista, Inktomi and FAST all accept XML feeds, which is a form of cloaking. Non-XML but cloaked pages have also been allowed through all of these services' paid inclusion programs. Given this, a blanket decree that cloaking is a bad practice makes no sense, if the company is performing cloaking that a particular search engine explicitly allows.
Guide Creates De Facto Standards
Of course, it's perfectly acceptable for MarketingSherpa to create its own guidelines that it recommends to readers in lieu of official industry standards. Consumer magazines are a good example of this. They decide what they think is important for their readers, then rate products according to those criteria.
Indeed, despite cries throughout the SEM industry that official standards are needed, divisions within the industry mean that they aren't likely to come anytime soon, if at all. Given this, MarketingSherpa's best practices criteria have essentially become de facto standards to fill the gap. Those reviewed in the guide either meet the standards or face the embarrassment of a bad grade.
For example, the pressure caused the firm More Visibility to report to MarketingSherpa that they were no longer going to cloak, which means the company's grade may rise when rereviewed for the second edition of the guide. No doubt other companies are also considering changes to meet standards that could help their grades.
Lawsuit Involving Ratings
Of course, a bad grade can be more than embarrassing. It potentially could ruin someone's business. It was this concern that prompted firm International Crossing to file suit against both MarketingSherpa and the guide's chief US researcher, Anthony Muller.
As part of its review, MarketingSherpa had asked an International Crossing sales person if the company cloaked, among other questions about practices and tactics. The reply was that the company did not. However, when MarketingSherpa checked on one of International Crossing's listings for a client in Google, the page MarketingSherpa saw looked different than the page shown in Google's page cache, which is what Google's spider would have seen.
Because of the differences, MarketingSherpa wrote that International Crossing had lied during its interview, an action that caused the company to get an automatic failing grade, when it was included in the guide's third release.
"Upon investigation into the ranking they gave, it is blatantly obvious using our methods that they are cloaking EVEN THOUGH they said they do not cloak. The result is instant failure since no other answers can hold any relevant information. If a company lies to us (SEO EXPERTS) on a survey, it is a fairly good indicator that the same might happen when a potential client contacts the company who is generally familiar with SEO," the MarketingSherpa report said, in the update released on February 26.
However, International Crossing denied that it was cloaking. The company was showing the same page to spiders and humans but using frames in way so that some portions would not be visible to humans. This was important, because the MarketingSherpa report defined cloaking narrowly as using IP detection to show different pages to spiders versus human beings.
"Cloaking is a program that detects the IP address of whoever is visiting your site to discern if the visitor is a search engine spider. If it's a spider, the program will show a page that's been created especially to impress spiders. Human surfers will never see that version of the page," the report says.
Concerned about its reputation -- and already having had one client question its tactics within hours after the report came out -- International Crossing filed suit the day the report was released, claiming defamation, trade libel and unfair competition. The company sought to have the allegations that it cloaked and lied about cloaking withdrawn and wanted to be paid for damages.
Two days later, on February 28, MarketingSherpa had retracted its statement, saying that after talking with International Crossing, it had now realized they were not cloaking.
"Our original score of International Crossing...was based on our understanding that they were using 'cloaking,' which is commonly defined as delivering one page to a visitor and a different page to the search engines. After consultation with International Crossing, it is clear that we misunderstood their technology, and they are not using cloaking. We have no reason to believe that the technology they are using would lead to any of their clients being delisted from search engines. Therefore we have revised the Guide to reflect the correct score for International Crossing, which is based on the interview we conducted with them about their practices and our own research of their results. Please note that our research bore out the accuracy of the information they provided us. We regret any confusion this has caused," MarketingSherpa wrote, in a special email to those with access to the guide.
As a result of the retraction, International Crossing dropped its action against MarketingSherpa. However, MarketingSherpa stressed that its retraction was not done as part of a legal settlement.
"The retraction we issued, less than 24 hours after International Crossing brought their problem to our attention, was issued completely voluntarily as have been other updates we've done to the Guide. We want our readers to have the best information possible. This was not a quid pro quo -- we did not have any legal agreement with IC at that time," said MarketingSherpa editor Anne Holland.
International Crossing is also no longer included in the report.
Suit Against Researcher
While the case against MarketingSherpa has been dropped, International Crossing is still taking action against researcher Muller.
"As an author, he should be held liable for the due diligence he does," said Jeffrey Pruitt, International Crossing's director of business development, about pursuing the report's primary author, in addition to its publisher.
Pruitt said the company wanted an apology separately from Muller for being called liars. He was also surprised no one called International Crossing to reverify the situation before the company was said to be lying and upset that the interview with International Crossing was done with a sales person, rather than a technical person.
"He [Muller” didnt talk to anyone about how we do our technology," Pruitt said. Had a technical person been interviewed, Pruitt says that person might have understood that International Crossing's frame-based delivery system could have been confused as IP-based cloaking.
"He could have said, 'No, we dont cloak, heres what we do'," Pruitt said.
For his part, Muller says his legal response will be that more than just the cloaking issue formed his final decision in the comments about International Crossing and that on the cloaking issue itself, the MarketingSherpa guide does not include the definitive line on what's considered cloaking.
"It basically just gives one example of cloaking that a buyer could fairly easily understand, but it is by no means meant to be the Webster's Dictionary definition of cloaking. The main issue is that a page that is one way for a search engine and another for the end user is cloaking and can be perceived as spam depending on the search engine it is on," Muller said.
Muller also said that he didn't perform the actual best practices interview with the International Crossing representative but regardless of whether it was a sales or technical person, that person should have been able to answer the question.
"If the person being interviewed wasn't capable of answering our questions completely, our attention should have been brought to it. A potential client or buyer would expect the same when calling and speaking with a companies representative," Muller said.
Holland added that International Crossing itself was responsible for who was interviewed.
"International Crossing -- as all SEOP firms included -- got to decide who on their team would do the interview. They picked their 'expert' to answer questions, we didn't. We just called whoever they told us to call," Holland said.
Marketer Reviewing Marketers
That Anthony Muller is embroiled in a lawsuit over the ratings is ironic, in that both he and MarketingSherpa agree that Muller didn't want to add ratings to the guide.
"Anthony felt his hand was forced into the rating scale," said Gutzman.
When the plan to rate was pushed through, Muller's firm ZenHits-- which is included in the report -- was the only one not graded. Muller asked for this to be done to ease concerns that ZenHits would have an advantage over other firms, since Muller was involved with the report.
That's not enough to save Muller from criticism, however. Other firms can still view him with suspicion, as International Crossing does.
"The main problem I have with Anthony Muller is that he is rating other SEOs, yet at the same time he owns a competing firm. It is difficult for a industry to set standards when those writing about them have competing interest. You would not see this in a mature industry like accounting and/or finance. I know we are not the only SEO firm that feels this way," Pruitt said.
To further reduce criticism, Muller is no longer taking on new SEO work but rather providing only consulting services, where he evaluates web sites, offers usability and search engine friendliness suggestions, and helps companies in finding SEO firms if they require further assistance.
"We no longer perform SEO in the way that a client would come off the street and say, 'Can you get me to the top?.' It's more like, 'Let me take a look at what you have, what you need, what you can handle in house and make sure we get the right company to do the rest," he said.
Muller also said that if he refers clients to other SEOs for work, he does not accept a referral fee. In addition, in the second edition of the guide, Muller said his firm will not be included, since his business model has changed.
Damaged By Being Left Out?
While International Crossing was upset by how it was going to be listed in the MarketingSherpa guide, Oneupweb president Lisa Wehr launched a PR battle last month over how the guide is impacting firms like hers that are not included within it.
Wehr claims that SEM firms left out of the guide are by implication seen as disreputable or less than worthy than those which are included, especially given the way the MarketingSherpa report has advertised itself.
"Who determines who's reputable? Are firms not included, not reputable? This appears to be the implication. How can anyone possibly state that these firms are the 'leading' firms when only a very small percentage of companies have been interviewed?," Wehr writes, in an editorial posted at her company's web site.
In her editorial, Wehr lists three different ads that MarketingSherpa has run, which include statements such as "Which 64 firms are best in the US and the UK" and "Which 55 firms are named among the 'Most Reputable'?"
"This guide is suggesting it has the best and most reputable," Wehr said, who says she's lost potential clients who asked why her company wasn't in it. "I can attribute loss of business to the fact that we were not included."
How New Firms Are Added
Wehr was especially upset that her company was passed over for inclusion three times by the guide. However, as far as MarketingSherpa is concerned, Wehr's firm missed out only once.
The confusion has to do with the "first edition" of the guide being released over several months in three different updates. Each update added more firms, but MarketingSherpa says these firms were those initially identified through its polling of several marketing mailing lists.
A "second edition" is in production now, and it is expected to include about 200 new firms, when released in the Spring of this year. These are all companies that asked to be included since the guide first came out through the close of the "nomination" period on Feb. 15 or which were identified in other ways as important to include.
"Everybody who asked to be included by February 15th who is a SEOP firm is being included. It's very democratic," said Holland.
MarketingSherpa is also accepting nominations for additional companies to be included in the second update of the second edition, due to be published in the summer. If you'd like your firm to be included, see the URL at the end of this article.
Thanks But No Thanks
Don't expect to see Wehr's firm included in the second edition. While Oneupweb was scheduled to be reviewed, Wehr says she doesn't trust or respect the rating system that's now in place.
"They did call me and told me they were ready to do my interview, and I told them that I declined at this point in time," Wehr said. "I dont want to be included in a guide that I dont think represents the industry fairly," she said. "Why do I want to put my entire livelihood at risk by having them review me, when I dont even respect the people doing it," she added.
MarketingSherpa suggests that Oneupweb might not want to be included because it is fearful of a bad review.
"It is not clear what Oneupweb's actual reasons are for withdrawing from the second edition. While they cite 'lack of trust,' they might also simply be afraid that we'll give them a bad ranking based on their actual practices versus what search engines and other experts maintain are best practices," said Holland.
By the way, for a short time MarketingSherpa was considering publishing a list of companies that were nominated but which declined to be included in the guide. However, the company decided it wouldn't be helpful to its readers and so has abandoned that idea.
Refining The Ratings
Given all the flak MarketingSherpa is taking over the ratings, you'd think the company would consider dropping them from the guide. However, the original goal of the ratings -- to help consumers know if they can trust a company not to get them banned in a search engine -- would be lost. To that end, MarketingSherpa says that it's going to stick with ratings but plans further enhancements, to try and make them more fair.
For example, cloaking wouldn't be seen as a client safety penalty, if you do it on a search engine that allows it in certain forms. In addition, the goal is to make the client safety grade only one of several that would be listed in an aggregate chart. The chart would also show key services offered and pricing.
MarketingSherpa is also planning to expand the list of sample clients to three for each firm and will include comments they receive from the companies' clients, pro or con. Indeed, MarketingSherpa has been actively surveying now for client comments.
MarketingSherpa also said that it does want to hear from companies that feel they've been wronged in the ratings. It has corrected scores for some companies when, upon further review, it was found they deserved better scores than were initially given.
"Boy were we sorry, and we fixed it," said Gutzman. "In every case, we had it fixed in 24 hours."
Focus On Capabilities
As for myself, I don't disagree at all with many of the "Best Practices" that are outlined in the report. However, the difficulty in assessing exact standards makes me wary of wanting to see everything boiled down to simple letter grades.
Instead, my own suggestion is that the guide somehow provide an at-a-glance guide to the various things the companies say they do. Then one could easily spot companies that perform international submission, provide linking service, depend on doorways, depend on informational pages and so on. From that, users could make their own choices.
For example, imagine a chart like this:
Company Intl? Manual Auto Paid Direct Link Cloak Price AlphaSub No 5 6 5 10 10 0 $$ SuperList Yes 4 6 9 0 0 10 $$$
Such a chart would let you know the particular features a firm offers, such as "Intl?" for International Submission, "Paid" for paid listings placement and "Link" for link building.
You could also see the emphasis that firms place on certain tactics, such as "Manual" submission versus use of "Auto" submission tools, or "Cloak" for use of cloaking. (In the MarkeingSherpa report, 10 is most emphasis, 1 is least, in the guide).
Users could make their own decision about whether certain tactics are a problem, based on advice from the guide, then spot the firms they like by scanning the chart. It's more work for the consumer, but they'll be better educated, and it might be more fair to some of the firms, as well.
MarketingSherpa says the difficulty with this approach is that the guide is scheduled to grow by another 200 or so companies. "How do you fit 200 companies on one chart," Gutzman said.
Need For Accepted Standards
Both Oneupweb and International Crossing -- which have objections to the existing MarketingSherpa guide -- said they saw value in some type of guide that could help consumers make safe choices about choosing a search engine marketing vendor. But how do you establish the criteria about what's safe? Both companies wanted some type of panel of experts to be involved.
"I would certainly be interested in being included when a fair grading system could be implemented. I dont what that is. I would like to see a group of people involved in doing it whose opinions I respected," Wehr said.
Of course, what's to prevent that panel of people from finding themselves sued, if a company doesn't like how they got rated according to standards that were established? International Crossing said it thinks such suits aren't likely to happen, if the panel has industry support.
"MarketingSherpa is writing about the industry but they havent been accepted by the players in the industry," said Pruitt. If a panel was somehow approved, he explained, it would offer more protection. "You cant just go sue them because of a statement they make, because theyve been accepted."
Involving Search Engines
An ideal solution in starting the creation of such a panel would be for the search engines themselves to rate or at least declare whether a company has been banned by them for some reason. After all, they are the ones who ultimately set standards, in that they decide what's acceptable for their listings.
Certainly all the major search engines have their own internal "blacklists" of companies and web sites that they consider to have spammed them. They've even discussed sharing such lists among themselves or with the public. However, the search engines also worry that someone might sue them for making such public statements. As a result, to date, none of them have moved forward on this front.
Given this situation, consumers seeking help with search engine marketing may continue to have to sort out for themselves whether a company is trustworthy or not. The best advice here would be to, as always, check references.
It's important to note that the search engines have been involved to some degree with helping build the MarketingSherpa report, the company says -- along wide advice from others.
"We have also updated the survey and rankings system to reflect new practices in SEO and better gather clear non-biased data that consumers can use to select the right firm for their needs," said Holland. "These changes and updates have been based on input from the search engines themselves, input from recognized industry experts, input from consumers who buy the guide, and last but not least, input from SEOP firms."
I started out the review by saying that MarketingSherpa's guide is a resource that I'd highly recommend to those seeking a search engine marketing firm. It's not perfect, but for a consumer, it provides an excellent starting place in understanding what types of services are offered by search engine marketing companies and for getting an initial list of firms to contact.
The rating system, if it is to be retained, does need to be overhauled so that the companies reviewed feel it treats them fairly. This doesn't mean that everyone gets top scores, of course. However, an emphasis on client lists and customer comments may be a move in the right direction.
One hope would be that the search engines themselves could organize a system to let MarketingSherpa and others who will no doubt produce similar guides to get some feedback directly from them. Perhaps the search engines could review a particular company's work and give the nod to say whether they'd had problems in the past or not. The entire industry could be helped by this.
Consumers considering the guide should understand that is not a guide to the best search engine marketing firms out there. There are no doubt good firms in the guide, but there are also plenty of good firms not in it.
Inclusion in the guide shouldn't be seen as a stamp of approval, nor should you immediately consider a company that is not listed as having done something wrong. Instead, use some of the good advice that the guide gives about choosing a firm, in order to review companies that you are interested in yourself.
For those running search engine marketing firms, should you seek to be included? Certainly many of the firms I've spoken with who are listed in the guide say they've had substantial increases in business, and MarketingSherpa confirms that many in the guide are seeing lots of new clients. Your business could go up, if you are listed.
Of course, it's also possible that you'll get a bad review. Consider the criteria that MarketingSherpa rates on, then try to measure yourself. If you think you'd do badly, it may be that the wise decision is to pass on being reviewed. Alternatively, you might consider changing the tactics you use.
Buyer's Guide to Search Engine Optimization & Positioning Firms
Information about the guide and the ability to purchase it online, from MarketingSherpa. FYI, the odd URL is not an affiliate link -- simply the format MarketingSherpa uses for its pages. Search Engine Watch does not benefit if you purchase a report.
I had problems getting to the store site when reverifying links before posting this article. If you also have problems, try visiting the MarketingSherpa site itself, to see if they post a different link to reach the report form.
Surplus Of Search Engine Marketing Reports
The Search Engine Report, Oct. 2, 2001
Includes the last review of the Buyer's Guide. Search Engine Watch members: use the link at the top of the article to reach the extended review.
Desperately Seeking Search Engine Marketing Standards
The Search Engine Report, Nov. 5, 2001
Examines how different philosophies among marketers and a lack of guidance from the search engines themselves make this a difficult challenge. Search Engine Watch members: use the link at the top of the article to reach the extended article.
Ethical Search Engine Marketing Committee
WAIM, Feb. 16, 2002
The World Association of Internet Marketers is trying to move forward with creating standards for search engine marketing, through the creation of a new committee.
The MarketingSherpa Guide to SEO Debate
Oneupweb, March 2002
Oneupweb's Lisa Wehr outlines her concerns with the MarketingSherpa report. Includes responses sent by MarketingSherpa, as well as comments from other firms in the industry -- pro and con -- about the guide.
MarketingSherpa SEOP Guide Nomination Form
Firms interested in being reviewed for the guide's third edition can nominate themselves here.
MarketingSherpa SEOP Guide Client Comments
Been the client of a search engine marketing firm? You can contribute your comments about particular firms to MarketingSherpa via this form.
iProspect Ranks #1 Among Search Engine Optimization Firms;
Earns the Only A+ Rating in Industry-Wide Study
iProspect, March 12, 2002
iProspect is more than happy with the MarketingSherpa ratings, having earned the only A+ among the firms reviewed, as it explained in a press release last month.
New online listing guide to those offering SEO services, planned to launch in June 2002.
Interview With Edward Lewis, Creator of The SEO Consultants Directory
The Search Light, March 20, 2002
More details about the planned SEO Consultants Directory.
Behind the Scenes at the SEO Industry's First Buyer's Guide: Part 1
SEO Today, Dec. 19, 2001
Explains how the Buyer's Guide came to be and how ratings were added to the second update.
Behind the Scenes at the SEO Industry's First Buyer's Guide: Part 2
SEO Today, Jan. 10, 2002
Covers surprises the guide found about a significant number of firms failing to follow what it considers best practices and other findings.
Introducing... ClickZ Live!
SES Conference & Expo has merged with ClickZ to bring you ClickZ Live! The new global conference series takes on the identity of the industry's premier digital marketing publication, ClickZ.com, and kicks off March 31-April 3 in New York City. Join the industry's leading tech-advertisers in the advertising capital of the world! Find out more ››
*Super Saver Rates expire Jan 24.