SEO Guide Expands, But Ratings Upset Some Firms

A longer, more detailed version of this article is
available to Search Engine Watch members.
Click here to learn more about becoming a member

Late last year, I reviewed MarketingSherpa's "Buyer's Guide to Search Engine Optimization & Positioning Firms." Since that time, the first edition of the guide has undergone two further updates and a completely new second edition is on the way.

The updates to the first edition have continued to build the guide into a resource that I'd highly recommend to anyone seeking a search engine marketing firm. However, they've also given rise to some serious concerns from some search engine marketing companies.

The guide was first released last September and contained reviews of 24 US-based search engine marketing firms, with a promise to add over 30 additional firms by March 2002.

The second update came out in October, listing two new US firms and 17 UK-based companies. However, a much more important change was the addition of a "Best Practices" ranking chart. This gave companies a letter grade showing how well they scored against the Buyer's Guide's "Best Practices" criteria. The grades were maintained when the latest update listing additional firms in the US, Canada, Australia and the UK came out at the end of February.

Grading Search Engine Marketing Firms

MarketingSherpa said the grades were added to make it easier for the guide's readers to identify which firms followed practices that might cause them trouble with the search engines, a major worry for potential buyers of SEM services.

"You dont want to get your company bounced from the search engines," said the guide's new managing editor, Alexis Gutzman.

Companies earned points for following good practices that MarketingSherpa deemed unlikely to jeopardize what it calls "client safety" on search engines. For example, a company that said link building or keyword research was important to the work they performed gained points.

Bad practices, such as depending heavily on cloaking or mirror sites, cost points. Companies earning the most points earned an A, with the scale slipping down to the ultimate failing grade of an F.

A problem with the ratings is that MarketingSherpa's Best Practices criteria are not an official industry standard. Indeed, the search engine marketing industry lacks any real "official" standards about what's "right" and "wrong."

Guide Creates De Facto Standards

Of course, it's perfectly acceptable for MarketingSherpa to create its own guidelines that it recommends to readers in lieu of official industry standards. Consumer magazines are a good example of this. They decide what they think is important for their readers, then rate products according to those criteria.

Indeed, despite cries throughout the SEM industry that official standards are needed, divisions within the industry mean that they aren't likely to come anytime soon, if at all. Given this, MarketingSherpa's best practices criteria have essentially become de facto standards to fill the gap. Those reviewed in the guide either meet the standards or face the embarrassment of a bad grade.

Lawsuit Involving Ratings

Of course, a bad grade can be more than embarrassing. It potentially could ruin someone's business. It was this concern that prompted firm International Crossing to file suit against both MarketingSherpa and the guide's chief US researcher, Anthony Muller.

As part of its review, MarketingSherpa had asked an International Crossing sales person if the company cloaked, among other questions about practices and tactics. The reply was that the company did not. However, when MarketingSherpa checked on one of International Crossing's listings for a client in Google, the page MarketingSherpa saw looked different than the page shown in Google's page cache, which is what Google's spider would have seen.

Because of the differences, MarketingSherpa wrote that International Crossing had lied during its interview, an action that caused the company to get an automatic failing grade, when it was included in the guide's third release.

However, International Crossing denied that it was cloaking. The company was showing the same page to spiders and humans but using frames in way so that some portions would not be visible to humans. This was important, because the MarketingSherpa report defined cloaking narrowly as using IP detection to show different pages to spiders versus human beings.

Concerned about its reputation -- and already having had one client question its tactics within hours after the report came out -- International Crossing filed suit the day the report was released, claiming defamation, trade libel and unfair competition. The company sought to have the allegations that it cloaked and lied about cloaking withdrawn and wanted to be paid for damages.

Two days later, on February 28, MarketingSherpa had retracted its statement, saying that after talking with International Crossing, it had now realized they were not cloaking.

As a result of the retraction, International Crossing dropped its action against MarketingSherpa. However, MarketingSherpa stressed that its retraction was not done as part of a legal settlement.

"The retraction we issued, less than 24 hours after International Crossing brought their problem to our attention, was issued completely voluntarily as have been other updates we've done to the Guide. We want our readers to have the best information possible. This was not a quid pro quo -- we did not have any legal agreement with IC at that time," said MarketingSherpa editor Anne Holland.

International Crossing is also no longer included in the report.

Marketer Reviewing Marketers

While the case against MarketingSherpa has been dropped, International Crossing is still taking action against researcher Muller.

That Anthony Muller is embroiled in a lawsuit over the ratings is ironic, in that both he and MarketingSherpa agree that Muller didn't want to add ratings to the guide.

"Anthony felt his hand was forced into the rating scale," said Gutzman.

When the plan to rate was pushed through, Muller's firm ZenHits-- which is included in the report -- was the only one not graded. Muller asked for this to be done to ease concerns that ZenHits would have an advantage over other firms, since Muller was involved with the report.

That's not enough to save Muller from criticism, however. Other firms can still view him with suspicion, as International Crossing does.

"The main problem I have with Anthony Muller is that he is rating other SEOs, yet at the same time he owns a competing firm. It is difficult for a industry to set standards when those writing about them have competing interest. You would not see this in a mature industry like accounting and/or finance. I know we are not the only SEO firm that feels this way," Pruitt said.

To further reduce criticism, Muller is no longer taking on new SEO work but rather providing only consulting services, where he evaluates web sites, offers usability and search engine friendliness suggestions, and helps companies in finding SEO firms if they require further assistance.

"We no longer perform SEO in the way that a client would come off the street and say, 'Can you get me to the top?.' It's more like, 'Let me take a look at what you have, what you need, what you can handle in house and make sure we get the right company to do the rest," he said.

Muller also said that if he refers clients to other SEOs for work, he does not accept a referral fee. In addition, in the second edition of the guide, Muller said his firm will not be included, since his business model has changed.

Damaged By Being Left Out?

While International Crossing was upset by how it was going to be listed in the MarketingSherpa guide, Oneupweb president Lisa Wehr launched a PR battle last month over how the guide is impacting firms like hers that are not included within it.

Wehr claims that SEM firms left out of the guide are by implication seen as disreputable or less than worthy than those which are included, especially given the way the MarketingSherpa report has advertised itself.

"Who determines who's reputable? Are firms not included, not reputable? This appears to be the implication. How can anyone possibly state that these firms are the 'leading' firms when only a very small percentage of companies have been interviewed?," Wehr writes, in an editorial posted at her company's web site.

In her editorial, Wehr lists three different ads that MarketingSherpa has run, which include statements such as "Which 64 firms are best in the US and the UK" and "Which 55 firms are named among the 'Most Reputable'?"

"This guide is suggesting it has the best and most reputable," Wehr said, who says she's lost potential clients who asked why her company wasn't in it. "I can attribute loss of business to the fact that we were not included."

How New Firms Are Added

Wehr was especially upset that her company was passed over for inclusion three times by the guide. However, as far as MarketingSherpa is concerned, Wehr's firm missed out only once.

The confusion has to do with the "first edition" of the guide being released over several months in three different updates. Each update added more firms, but MarketingSherpa says these firms were those initially identified through its polling of several marketing mailing lists.

A "second edition" is in production now, and it is expected to include about 200 new firms, when released in the Spring of this year. These are all companies that asked to be included since the guide first came out through the close of the "nomination" period on Feb. 15 or which were identified in other ways as important to include.

"Everybody who asked to be included by February 15th who is a SEOP firm is being included. It's very democratic," said Holland.

MarketingSherpa is also accepting nominations for additional companies to be included in the second update of the second edition, due to be published in the summer. If you'd like your firm to be included, see the URL at the end of this article.

Thanks But No Thanks

Don't expect to see Wehr's firm included in the second edition. While Oneupweb was scheduled to be reviewed, Wehr says she doesn't trust or respect the rating system that's now in place.

"They did call me and told me they were ready to do my interview, and I told them that I declined at this point in time," Wehr said. "I dont want to be included in a guide that I dont think represents the industry fairly," she said. "Why do I want to put my entire livelihood at risk by having them review me, when I dont even respect the people doing it," she added.

MarketingSherpa suggests that Oneupweb might not want to be included because it is fearful of a bad review.

"It is not clear what Oneupweb's actual reasons are for withdrawing from the second edition. While they cite 'lack of trust,' they might also simply be afraid that we'll give them a bad ranking based on their actual practices versus what search engines and other experts maintain are best practices," said Holland.

By the way, for a short time MarketingSherpa was considering publishing a list of companies that were nominated but which declined to be included in the guide. However, the company decided it wouldn't be helpful to its readers and so has abandoned that idea.

Refining The Ratings

Given all the flak MarketingSherpa is taking over the ratings, you'd think the company would consider dropping them from the guide. However, the original goal of the ratings -- to help consumers know if they can trust a company not to get them banned in a search engine -- would be lost. To that end, MarketingSherpa says that it's going to stick with ratings but plans further enhancements, to try and make them more fair.

For example, cloaking wouldn't be seen as a client safety penalty, if you do it on a search engine that allows it in certain forms. In addition, the goal is to make the client safety grade only one of several that would be listed in an aggregate chart. The chart would also show key services offered and pricing.

MarketingSherpa is also planning to expand the list of sample clients to three for each firm and will include comments they receive from the companies' clients, pro or con. Indeed, MarketingSherpa has been actively surveying now for client comments.

MarketingSherpa also said that it does want to hear from companies that feel they've been wronged in the ratings. It has corrected scores for some companies when, upon further review, it was found they deserved better scores than were initially given.

"Boy were we sorry, and we fixed it," said Gutzman. "In every case, we had it fixed in 24 hours."

Need For Accepted Standards

Both Oneupweb and International Crossing -- which have objections to the existing MarketingSherpa guide -- said they saw value in some type of guide that could help consumers make safe choices about choosing a search engine marketing vendor. But how do you establish the criteria about what's safe? Both companies wanted some type of panel of experts to be involved.

"I would certainly be interested in being included when a fair grading system could be implemented. I dont what that is. I would like to see a group of people involved in doing it whose opinions I respected," Wehr said.

Of course, what's to prevent that panel of people from finding themselves sued, if a company doesn't like how they got rated according to standards that were established? International Crossing said it thinks such suits aren't likely to happen, if the panel has industry support.

"MarketingSherpa is writing about the industry but they havent been accepted by the players in the industry," said Pruitt. If a panel was somehow approved, he explained, it would offer more protection. "You cant just go sue them because of a statement they make, because theyve been accepted."

Involving Search Engines

An ideal solution in starting the creation of such a panel would be for the search engines themselves to rate or at least declare whether a company has been banned by them for some reason. After all, they are the ones who ultimately set standards, in that they decide what's acceptable for their listings.

Certainly all the major search engines have their own internal "blacklists" of companies and web sites that they consider to have spammed them. They've even discussed sharing such lists among themselves or with the public. However, the search engines also worry that someone might sue them for making such public statements. As a result, to date, none of them have moved forward on this front.

Given this situation, consumers seeking help with search engine marketing may continue to have to sort out for themselves whether a company is trustworthy or not. The best advice here would be to, as always, check references.

It's important to note that the search engines have been involved to some degree with helping build the MarketingSherpa report, the company says -- along wide advice from others.

"We have also updated the survey and rankings system to reflect new practices in SEO and better gather clear non-biased data that consumers can use to select the right firm for their needs," said Holland. "These changes and updates have been based on input from the search engines themselves, input from recognized industry experts, input from consumers who buy the guide, and last but not least, input from SEOP firms."

Final Observations

I started out the review by saying that MarketingSherpa's guide is a resource that I'd highly recommend to those seeking a search engine marketing firm. It's not perfect, but for a consumer, it provides an excellent starting place in understanding what types of services are offered by search engine marketing companies and for getting an initial list of firms to contact.

The rating system, if it is to be retained, does need to be overhauled so that the companies reviewed feel it treats them fairly. This doesn't mean that everyone gets top scores, of course. However, an emphasis on client lists and customer comments may be a move in the right direction.

One hope would be that the search engines themselves could organize a system to let MarketingSherpa and others who will no doubt produce similar guides to get some feedback directly from them. Perhaps the search engines could review a particular company's work and give the nod to say whether they'd had problems in the past or not. The entire industry could be helped by this.

Consumers considering the guide should understand that is not a guide to the best search engine marketing firms out there. There are no doubt good firms in the guide, but there are also plenty of good firms not in it.

Inclusion in the guide shouldn't be seen as a stamp of approval, nor should you immediately consider a company that is not listed as having done something wrong. Instead, use some of the good advice that the guide gives about choosing a firm, in order to review companies that you are interested in yourself.

A longer, more detailed version of this article is
available to Search Engine Watch members.
Click here to learn more about becoming a member

Buyer's Guide to Search Engine Optimization & Positioning Firms

Information about the guide and the ability to purchase it online, from MarketingSherpa. FYI, the odd URL is not an affiliate link -- simply the format MarketingSherpa uses for its pages. Search Engine Watch does not benefit if you purchase a report.


I had problems getting to the store site when reverifying links before posting this article. If you also have problems, try visiting the MarketingSherpa site itself, to see if they post a different link to reach the report form.

Surplus Of Search Engine Marketing Reports
The Search Engine Report, Oct. 2, 2001

Includes the last review of the Buyer's Guide. Search Engine Watch members: use the link at the top of the article to reach the extended review.

Desperately Seeking Search Engine Marketing Standards
The Search Engine Report, Nov. 5, 2001

Examines how different philosophies among marketers and a lack of guidance from the search engines themselves make this a difficult challenge. Search Engine Watch members: use the link at the top of the article to reach the extended article.

Ethical Search Engine Marketing Committee
WAIM, Feb. 16, 2002

The World Association of Internet Marketers is trying to move forward with creating standards for search engine marketing, through the creation of a new committee.

International Crossing

The MarketingSherpa Guide to SEO Debate
Oneupweb, March 2002

Oneupweb's Lisa Wehr outlines her concerns with the MarketingSherpa report. Includes responses sent by MarketingSherpa, as well as comments from other firms in the industry -- pro and con -- about the guide.

MarketingSherpa SEOP Guide Nomination Form

Firms interested in being reviewed for the guide's third edition can nominate themselves here.

MarketingSherpa SEOP Guide Client Comments

Been the client of a search engine marketing firm? You can contribute your comments about particular firms to MarketingSherpa via this form.

iProspect Ranks #1 Among Search Engine Optimization Firms;
Earns the Only A+ Rating in Industry-Wide Study
iProspect, March 12, 2002

iProspect is more than happy with the MarketingSherpa ratings, having earned the only A+ among the firms reviewed, as it explained in a press release last month.

SEO Consultants

New online listing guide to those offering SEO services, planned to launch in June 2002.

Interview With Edward Lewis, Creator of The SEO Consultants Directory
The Search Light, March 20, 2002

More details about the planned SEO Consultants Directory.

Behind the Scenes at the SEO Industry's First Buyer's Guide: Part 1
SEO Today, Dec. 19, 2001

Explains how the Buyer's Guide came to be and how ratings were added to the second update.

Behind the Scenes at the SEO Industry's First Buyer's Guide: Part 2
SEO Today, Jan. 10, 2002

Covers surprises the guide found about a significant number of firms failing to follow what it considers best practices and other findings.