Researching Search Terms With WordTracker

The key to success with search engine marketing is understanding how your audience is searching for your products and services. If you fail to make use of the terms your audience searches with, you'll fail to be found by them. It's that simple.

For example, if you are selling shoes, you might assume that people will search for you and sites like yours by entering "shoes" into a search engine. That's true, but significant numbers of people will also search using the word "footwear." If you haven't made use of the word "footwear" within the web pages at your site, or within the descriptions you submit to directories such as Yahoo, then you'll have virtually no chance of ranking well for that word.

Term research is so crucial to success, yet marketers have lacked a robust tool to help them with it. That oversight has ended, thanks to the recent improvement of the WordTracker service.

WordTracker is one of only two widely-available services that let you perform research to see what people are actually searching for at a major search engine. The other is GoTo's Search Term Suggestion Tool. Without a doubt, the venerable GoTo tool has been an essential resource for any search engine marketing professional since it became available in 1998. However, the GoTo tool has several problems that the improved WordTracker service solves.

Unlike the GoTo tool, WordTracker isn't free. You can purchase a daily subscription for US $6 ranging upward to a 1 year subscription for $200. As you'll see, I think this is money well spent by any marketing professional.

Direct Feed From MetaCrawler and Dogpile

WordTracker lets you research what people are searching for on the popular meta search services of MetaCrawler and Dogpile, both of which are owned by InfoSpace. Dogpile alone had a healthy 5 percent reach of the web audience in May 2001, according to Jupiter Media Metrix, so the search data provided from it should be fairly dependable.

The data is certainly a huge improvement over what WordTracker previously depended on. The service has been running since February 1999, but before the deal gave WordTracker approved access to the Dogpile and MetaCrawler's query logs, WordTracker had to rely on mining "live" or "voyeur" search displays to gather its data. For instance, Excite "Search Voyeur" service shows a list of live searches being conducted every 30 seconds.

"We were getting data from all the search engine voyeur places, but the information wasn't too great," said Andy Mindel, one of the two partners who runs WordTracker.

One thing that skewed the data were position checkers. People running queries against the major search engines could influence the data significantly.

"We were finding that some of the words, like 'search engine positioning' especially, would get high postings but would get no traffic from it," Mindel said.

In contrast, getting a feed from meta search engines is good because few people are rank checking against them, Mindel said. WebPosition, for instance, doesn't include either Dogpile or MetaCrawler in its reports.

Through the new deal signed in June, all data that WordTracker uses comes directly from Dogpile and MetaCrawler themselves. There's access to logs going back over two months, which amounts to 350 million queries or 40 million unique search terms. The database is also kept constantly updated. Each week, a week's worth of the oldest data is removed and updated with newer information.

In the future, the database may be allowed to grow over a longer period of time. That will be a great help for those who want to research seasonal terms, such as Christmas-related searches. Since we are months away from the holiday, the amount and accuracy of Christmas-oriented searches in the database right now will naturally be very small.

Researching With WordTracker

The deal with Dogpile and MetaCrawler certainly brings WordTracker on par with the GoTo tool, with granting direct access to query logs, but what takes it beyond? For one thing, it is designed to get you thinking of related terms, rather than just subterms. "Subterms?" To understand this difference, let's compare WordTracker to the GoTo tool.

With the GoTo Suggestion Tool, you enter a word, and then you are shown how often that word and terms containing it appeared in searches on the GoTo network in a particular month. For example, if I enter "shoes," I'm told that "shoes" was searched for 174,358 in June 2001, "womens shoes" 20,978 times, "running shoes" 16,337 times and so on.

The weakness with researching this way at GoTo is that only "subterms" are displayed, terms that contain your original query. In other words, you'll only be shown top terms that actually contain the word "shoes." You'll never be told that "footwear" is a term related to shoes that also might be used by your audience.

In contrast, WordTracker truly can show you related terms rather than just subterms. When you log into the service, this happens in the first step you're guided to, the "Keyword Universe" area. Here, when you enter "shoes," WordTracker returns a list of top related terms that appear in the left-hand window of the Keyword Universe area -- shoes, footwear, boots, shoe, sandals, running shoes and so on.

WordTracker understands these are related terms in two ways. Its "lateral" option runs the search at three major engines (these can change, depending on which seem to provide the "best" results, WordTracker says) and the top 200 web pages matching the query are analyzed. It then examines these pages to determine which terms appear most often in the title and meta tags, as well as the body copy. The "thesaurus" option checks a conventional thesaurus, to find related terms. By default, both options are enabled when you use the Keyword Universe tool, and you must use one of them. I would stick with using both to get the greatest range of possible meanings.

Once you've generated a top related terms list, it's time to go shopping. Scan the list and click on a term that you think is a match to your potential audience. For instance, if you sell running shoes, then choosing "running shoes," "sneakers" or "jogging shoes" are obvious choices while "boots" or "casual shoes" are not.

When you select a particular term, a GoTo-style list of subterms will appear for your choice in the right-hand side of your screen, ranked by popularity. You'll be shown how many times each term was searched on over a two-month period, as well as a "predicted" traffic figure, which I'll cover further below. Here's an example of the top terms that appear for "running shoes," along with their two-month count:

running shoes - 3170 nike running shoes - 674 discount running shoes - 544 brooks running shoes - 285 new balance running shoes - 273 asics running shoes - 204 adidas running shoes - 156 trail running shoes - 126

Once you have the list of subterms, you should review it and select only those that are relevant to your site. From the list above, let's say this means you only choose generic terms that don't include brand names, leaving you with these:

running shoes - 3170 discount running shoes - 544 trail running shoes - 126

Now here's the real power of WordTracker over the GoTo tool. When you select each term from the list, WordTracker adds it to your term "shopping basket," which appears at the bottom of the screen. This means you can go on to do additional research, which also goes into your shopping basket. The result will be a customized list of related terms, all ranked by popularity.

For instance, going back to the list in the left-hand side of the Keyword Universe area, let's say you next select "sneakers." That brings up a subterms list on the right-hand side of the screen, where you select these choices:

sneakers - 1642 roller sneakers - 162 roller skate sneakers - 26

Finally, you click on "athletic footwear" and "jogging shoes" and add these to your basket:

sneakers - 1642 athletic footwear - 92 jogging shoes - 22

When done with all your selections, you can then move onto the Step 3 window to see a ranked list of all the terms in your shopping cart, like this:

running shoes - 3170 sneakers - 1642 discount running shoes - 544 roller sneakers - 162 trail running shoes - 126 athletic footwear - 92 roller skate sneakers - 26 jogging shoes - 22

To get a similar list using GoTo's tool would require a lot of copying, pasting and deleting of unwanted data. In contrast, it's a simple point and click operation, in WordTracker. The list, once generated, can be exported for use with other programs or emailed.

Prediction Counts

In the examples above, I've shown only the "count" figure, which is how often the term was searched for over a two-month period at Dogpile and MetaCrawler. Many site owners want some estimate on how much traffic a term generates each day, across all major search engines.

No one has these figures, but WordTracker does try to give an estimate with the "Predict" counts that it also shows for each term. It uses the share of web searches that Dogpile is estimated to generate each day as a basis of coming up with the total number of searches conducted on the web and then thus the predicted amount of searches any particular term should generate. The WordTracker help files explain this in much greater detail, if you are interested.

The WordTracker figures are only estimates and could easily be skewed if Dogpile's share of the web's searches is estimated inaccurately or if particular words are searched on more or less frequently at Dogpile than at other major search engines. More importantly, they shouldn't be taken as an estimate of how much traffic you will actually receive, for each term. That will depend on whether you rank well for the term, at which and how many search engines you rank well for it, where exactly you are listed and how well your listing attracts clicks.

"Competition" Reporting

After generating your list, you can also run a "competition" search for these terms, which means determining how many pages exist for them on various search engines, in order to discover which may be easier to target. In general, the fewer pages that appear in response to a search, the greater the odds that you can do well for it.

For example, when running a competition report against AltaVista, these results came up:

"discount running shoes" - 7998 "roller sneakers" - 1874 "running shoes" - 272 "roller skate sneakers" - 39 "sneakers" - 36 "trail running shoes" - 24 "athletic footwear" - 1 "jogging shoes" - 0.2

The number shown is the "KEI Score," which is a concept created by search engine optimization specialist Sumantra Roy. It shows the relationship between how popular a search term is and how many "competing" pages there are for that term. The higher the KEI score, the more likely you are to compete well for that term. Keep in mind that the KEI Score is only meant as a guide, not a guarantee to ranking success. Use it, if you find it helpful and ignore it, if not.

It's important to be aware that when running a competition report, WordTracker will surround your search terms by quotes, when sending them out to search engines in order to gather count figures. You can and should override this setting and get counts for pages without using quotes around your terms, in my opinion.

WordTracker uses quotes by default in order to find only pages that contain the exact phrases you are interested in, rather than those that contain only some of the words. This is designed to better represent your competition, in WordTracker's opinion. However, although major search engines will tend to list pages that contain all the terms in the order specified first, this isn't always the case. Because of this, I think doing a search exactly as a searcher does it gives you a better idea of the actual pages you are competing against.

Competition reports can also be run against paid placement search engines, such as GoTo and FindWhat. These can be really useful, because you will visually see the entire range of bids on the term for positions 1 through 20.

By the way, after running a competition report against a paid placement search engine, you can then go to the WordTracker home page, select the "View all your results" option and then use the "Pay Per Bid Value" link, which is somewhat hidden in a reverse bar at the bottom of the page. This lets you calculate how much you can afford to spend per visitor, then brings back matching positions from that price at paid placement services.

WordTracker Database Options

By default, WordTracker uses its "compressed" database, as indicated in the option box at the bottom of the Keyword Universe window. This means that all search commands such as + and - are stripped out and lower and upper case forms are combined. I think using the "precise" database is better for most people, where only capital forms are combined. This is because even though some search engines are case sensitive, you'll probably find little success in overtly trying to optimize for a particular case. Thus, combining both cases is fine -- but combining operator specific forms wouldn't be.

If you don't want case consolidation, that can be overridden using the "exact" option, and there are several others, including "simple" that combines plural terms and an adult term filter. All are fully defined in WordTracker's excellent help files. Indeed, virtually any question you might have about how the service operates seems to have an answer. There's also a downloadable manual.

The choice of so many databases is a real plus, when compared to GoTo's tool. There, plural forms and different cases are all combined, with no ability to separate them out. That's fine when you want to simplify your bid management, but it's a real weakness when trying to do market research on how exactly people search for purposes of search engine optimization.

By the way, one of WordTracker's database options is to use the GoTo terms database itself. This means that you'll be able to research terms using GoTo's data plus have many of the advantages to using WordTracker's tools.

Other WordTracker Features

Any work you do can be saved as accessed via the "Keyword Project" option, from WordTracker's home page. You can have up to seven projects saved, and should you need room, you can always export data from projects that you haven't used for a while.

On the home page, you'll also find a "Full Search" option. This puts you on a fast track to do the term research as already described. You enter a word like "shoes," then you'll be shown related terms. After you check the relevant ones, a full list of subterms for those related words will be displayed. You can then eliminate irrelevant terms from this mega list, rather than doing the elimination as part of the process of building the list.

The "Multiple Search" option from the home page lets you dump a list of terms you are interested in into a box, to see how popular they are in relation to each other. This is perfect for those who have data from log analysis tools such as WebTrends. You can take the terms you are already getting found for, see how popular they are plus use them as a base for finding other terms you may wish to target.

The "Top 1000 reports" option shows you the top 1000 terms search on Dogpile and MetaCrawler.

Database Verification

Earlier, I'd mentioned that WordTracker had found problems with mining live search displays, such as skewing by position checkers. Another problem with live displays is "spam queries." These are automated queries done for no other reason than to show up in live search displays.

For example, let's say a company wants to promote itself to those watching a live search display, such as MetaCrawler's MetaSpy service. The company might arrange for a program to automatically send a query to MetaCrawler every few seconds, such as "best prices on viagra online." The odds are, this query would make it into the MetaSpy display. In addition, anyone who clicked on the MetaSpy link for that query would get matching results -- and you can bet the company sending the automated query has only done so if they are top ranked for it.

Even though WordTracker now gets a dedicated feed from Dogpile and MetaCrawler, it still needs to watch for spam queries like these. The company described for me a variety of techniques that watch for frequency and statistical anomalies that left me feeling confident that they were removing most of the spam.

Get It!

There's no doubt that the GoTo tool has been an invaluable resource to many search engine marketers. However, the tool was primarily designed to help GoTo's advertisers research terms they should bid on, not to help marketers understand how to optimize their web sites. Moreover, since GoTo's tool is free, it has never been enhanced to provide features beyond what are immediately useful to GoTo's advertisers.

Given all this, if you are serious about search engine optimization, then WordTracker is an essential investment, a must have resource for any search engine marketing professional. It combines both a respectable search term database with tools that make mining the information easy.


Google Zeitgeist

Brand new, this provides a look at what people are searching for at Google.

Researching Keywords

Tips on using the GoTo tool and other resources to research search terms. By the way, I'm told the GoTo tool is supposed to include counts for searches conducted anywhere GoTo links appear. That means searches at major partners such as AOL Search are included in the counts.

What People Search For

Lists many of the major "live" search displays that are available from major services.