My Decade Of Writing About Search Engines

Ten years ago today, I first starting writing publicly about search engines. If we had blogs back then, I suppose I would have been a search blogger. But we didn't. We hand-coded our HTML, walked through the snow for eight miles to FTP files to our web servers, and we liked it :)

My involvement with search engines goes back to my first year as a student at the University Of California, Irvine in 1983. No, I wasn't part of the university's highly regarded information and computer science department. Instead, I was an English major -- and a pretty bored one for my first two months, when I had to commute until getting on-campus housing.

I spent some time exploring the library, having been a big library user since I was a child. The library had a magical electronic card catalog called Melvyl (named for Melvil Dewey, who created the Dewey Decimal System). For fun, I'd do Melvyl searches for broad topics such as history, art or love, to see how many matches would come back. I could routinely crash the search routine by doing this.

The system would diligently try, telling me it would take 50, 60, 80 search cycles, and then the countdown would begin. Some searches would eventually get through all cycles and give me a matching results count. Often, the system would just give up as the countdown approached the teens.

My 1996 Study

Search engines remained fascinating to me when I reencountered them in 1995. I'd left working as a newspaper reporter to go into web development, since I didn't want to miss out what was obviously going to be the future of publishing. As the general manager of Maximized Online, my job was to help get people in the Orange County, California area online. We'd build web sites, get them publicized to search engines and other publicity venues, plus host them.

One of our clients was upset at the end of 1995 that his OC jobs site wasn't ranking tops for a search on "orange county" in WebCrawler. We didn't have a good answer to give him. We'd done the submission, made use of the meta tags the search engines said to use, but why exactly a site would rank well wasn't well known. So I decided to look into it.

I spent January through April 1996 making changes to the InfoPages directory that my company maintained, a search engine just for Orange County web resources, to see if it could rank better in a search for "orange county." I tried putting those words in the body text, the title tag, in the meta tags and also checked to see if spamming helped, if repeating the word over and over would have an impact.

I published the results online, and 10 years on, a lot of the advice remains exactly the same. Don't depend on ALT text. Don't fixate on only one or two terms, because there are many ways people will seek you -- a long tail before we had talk of long tails. Build links, because links can send you traffic. And don't fixate on getting traffic just from search engines. The conclusions from that study are below for those really interested; others can jump past for the rest of this article.

There don't seem to be any magic methods that will make a page appear at the top of every search engines' listings. There's too much fluctuation on the web for any page to claim a foothold, and all the engines handle relevancy slightly differently. However, there are some general tips that do help a page appear more relevant.

  • Have text on your home page: Search engine catalogs contain the text read from the various home pages the engines visit. If a page lacks descriptive text, then there is little chance that page will come up in the results of a search engine query. It's not enough for that text to be in graphics. It must be HTML text. Some search engines will catalog ALT text and text in comment and meta tags. To be safe, a straight HTML description is recommended.
  • Pick your keywords: Focus on the two or three keywords that you think are most crucial to your site, then ensure those words are both in your title and mentioned early on your web page. Generally, most people will already have those words present on their pages but may not also have them in page titles. Keep in mind that the keywords you consider crucial may not be exactly what users enter. Our study focused on making the InfoPages directory appear high on lists if keywords "Orange County" were entered. The lack of success with some search engines does not mean that the site isn't being found. Many people find the site by entering more words, such as "Orange County California" or "Orange County Web." The addition of just one extra word can suddenly make a site appear more relevant, and it can be impossible to anticipate what that word will be. The best bet is to focus on your chosen keywords but to also have a complete description.
  • Have links to inside pages: If there are no links to inside pages from the home page, it seems that some search engines will not fully catalog a site. Unfortunately, the most descriptive, relevant pages that are often inside pages rather than the home page. You can also try sending search engines directly to your lower levels, if they don't ordinarily go there.
  • Forget Spamming: For one thing, spamming doesn't seem to work with every search engine. Ethically, the content of most web pages ought to be enough for search engines to determine relevancy without webmasters having to resort to repeating keywords for no reason other than to try and "beat" other web pages. The stakes will simply keep rising, and users will also begin to hate sites that undertake these measures. Efforts would be better spent on networking and alternative forms of publicity described below.
  • Network: If your site fails to make the top ten lists, then get together with those that do. Perhaps some might be considered "competitors," but others might be happy to link to your site in return for a link back. After all, your site may appear first when slightly different keywords are used. Links are what the web was built on, and they remain one of the best ways for people to find your site.
  • Relax: Search engines are a primary way people look for web sites, but they are not the only way. People also find sites through word-of-mouth, traditional advertising, the traditional media, newsgroup postings, web directories and links from other sites. Many times, these alternative forms are far more effective draws than are search engines. The audience you want may be visiting to a site that you can partner with, or reading a magazine that you've never informed of your site. Do the simple things to best make your site relevant to search engines, then concentrate on the other areas.

A Webmaster's Guide To Search Engines

Along with the study, I also published a collection of documents called "A Webmaster's Guide To Search Engines." My goal was to help site owners better understand the essentials of being found plus identify which search engines really mattered. Knowing who mattered was crucial when you'd have some search engines like Galaxy forcing your through a three part, multiple question submission process to be included in their directory. Was spending all that time worthwhile? (For Galaxy, the answer was no!).

The guide provided links to the FAQs of each search engines, along with my own observations about whether how each search engines said it worked actually lived up to reality. There was a guide to which search engines I considered to be "major" or most important to site owners and searchers alike. I had a "Strategic Alliances & Victories" chart to show which search engines had deals with the Netscape or Internet Explorer browsers and which had gained positive reviews in magazines.

The information I published quickly generated a lot of positive feedback, both from site owners and searchers such as librarians. At the same time, the web development company I worked for closed, so that the parent firm could concentrate on web software development. I hung out my internet consultant shingle and kept maintaining the Webmaster's Guide on a part time basis, sending out a newsletter update (The Search Engine Report) twice that year, along with making further site updates.

In 1997, I moved to the UK from California, so my wife could be closer to her family. I also began spending more and more time on the site, as well as writing freelance articles on search for various publications. In the middle of the year, I rebranded the site as Search Engine Watch, which generated more attention. By the end of the year, Mecklermedia purchased the site from me, and I continued on as editor of it.

The Search Revolution

Ten years on, I remain as fascinated with search engines as ever. I've been fortunate to help chronicle the birth of an entirely new advertising medium. Equally important has been the birth of an entirely new way for people to seek out information.

I knew search engines were important when I decided to write about them. The journalist in me could see they were a good story, especially when you realized that under the hood, they weren't doing things like crawling as often as people widely believed. But a study by Keen in 2001 especially resonated with me. Search engines (as a whole -- we weren't Google obsessed yet then) were the single most likely way people would seek information.

The study was small, but the findings were still stunning. In only about five years, search engines had ousted things like friends, family, books, magazines, libraries and other perfectly good resources for seeking answers.

Some of this was bad. I'd personally watched people when doing search training spending ages trying to find a phone number, when a call to telephone information would have found much faster. Old but still useful search strategies were abandoned in favor of the magic search box.

Lots of this is good. Search engines remain amazing tools that get us the right answers quickly in many circumstances.

Looking Ahead

Will I still be doing this in 20 years? Almost certainly not, at least not in the daily grind format I've been doing. I'd like to keep writing about search issues, but eventually I'll move away from the regular day-to-day coverage to perhaps focus on less frequent but deeper looks at particular search issues.

I'm also thinking a lot about doing a book these days. I'd always wanted to do a book on search, indeed the exact type of history that John Battelle did a fantastic job with in The Search.

Since that's come out, I've thought more and more about doing a more personal retelling of web search history -- the evolution, developments and trends I've seen from having been in the trenches of covering them over the years.

I'd also like to do a separate one talking to various search marketers, spotlighting them and focusing on how that medium has evolved over the years and where it will be going. The most fascinating book idea remains the impact of search on our everyday lives, how people make use of them, how habits have changed, our laws are starting to account for the power of search and many related issues like that.

Someday! What I can say is that for the near future, I expect to remain working on the site and coverage as I have, bringing some of our standing content back up to date, which I know has been neglected due to the need to cover the news that continues to flow in. My original Webmaster's Guide helped many understand search engines, and I very much want to ensure Search Engine Watch remains as a leading resource doing that in the years to come.

Looking Back

I don't have a succinct list of big picture items or "high order bits" to offer. A lot of this has already been covered in things I've written, so instead I'm going to spend some time recapping pieces I think are most important below. These are either big trend pieces I've done or big shifts in the search landscape I think worth noting.

I know -- I KNOW -- I've left some things out. My apologies, if so. It's a bit easier for me to cover all the things I've written that what me or Chris Sherman both have done, and he's clearly covered tons himself. Plus, skimming through 10 years worth of writings means I'll accidentally miss stuff. If you want to go poking yourself, I'll give more tips after the summary.

The biggest overall theme in doing the recap is how that big old wheel keeps spinning around and around, with people often buying hype because they don't remember things have come before -- or marketers making errors because they don't understand issues that were explored already in the past.

I've definitely felt myself getting more and more jaded. Part of that's bad, because there are cool, new things that I don't want to be blinded to. But then again, you go through the list below and tell me if you don't emerge feeling a big jaded about some ideas and concepts that are retreads.






  • WebTop Search Rage Study: Study sez we depend on search engines so much now that we'll get enraged if they don't give us the right answers quickly.
  • Bush's Dubious Victory At Google: Google's link system takes another blow as the number one result for motherf***r is the official George W. Bush campaign web site.
  • The End For Search Engines?: A look at why so many search engines were on the ropes or dead by this year but how paid listings gave hope and life for new ones.
  • Internet Top Information Resource, Study Finds: Search engines and the internet found to be the top information resource.
  • Being Search Boxed To Death: Google and other search engines today are struggling with how to best tell people about the many vertical search results they have. It's a long-standing problem, as this story looks at.
  • Google Acquires Deja Newsgroup Service: Google gets its first real heavy dose of criticism after Deja junkies freak out when Usenet features get lost temporarily after the acquisition.
  • Search Engine Marketing Finally Getting Respect: Despite search being an essential marketing strategy for years, finally more traditional outlets begin to "get it" and give search marketing some resepct.
  • Avoiding The Search Gap: Search is one of the most popular web activities, yet most sites don't get the majority of their traffic from search? What gives? It's the search gap and key to understanding how search makes an introduction that can lead to a lifetime of visits.
  • Fourth Time Lucky For AltaVista?: "AltaVista could give Madonna a run for her money in the changing your image game. Earlier this month, the service once again significantly changed its look and feel, the fourth such redesign in just over a year." That was 2001. And in 2000, there were another four designs, rather than just improving the results. Perhaps they should have changed the name to MSN Windows AltaVista Live Search.
  • Make Room For Teoma: A different twist on link analysis, but more important, a new search voice worth taking seriously -- and Ask Jeeves did, buying it later that year.
  • Consumer Group Asks FTC To Investigate Search Ads : Perhaps those "featured listings" popping up everywhere through payment should be better labeled as ads, the FTC is asked.
  • Desperately Seeking Search Engine Marketing Standards : Another push for standards in SEO falls into issues of knowing what exactly the rules are.
  • Search Engine Marketing: You Like It, You Really Like It -: I suggest the term "search engine marketing" as an umbrella term to encompass search engine optimization (the act of getting better free listings on search engines) and search advertising (paying for listings). Readers like it. FYI, today I use the term "search marketing" a bit more.



  • Yahoo To Buy Inktomi: Hmm. Maybe Google's not our friend and we should own our own crawler-based technology to protect ourselves. So thinks and acts Yahoo.
  • Overture To Buy AltaVista: Hmm. Maybe Google offering both paid and unpaid results really is an advantage and we should own some technology to produce crawler-based editorial results. So thinks and acts Overture.
  • Overture To Buy FAST Web Search Division: Hmm. Maybe we should by FAST as well and keep competitors from getting the technology.
  • Ending The Debate Over Cloaking: Perhaps we could skip past defining spam by techniques and look instead at intent and results? Techniques like cloaking doesn't always indicate a harmful results for search engines -- and some of them certainly allow it with approval.
  • Google Throws Hat Into The Contextual Advertising Ring: Later to be called AdSense, Google starts an entirely new economy of bloggers and publishers depending on its ads -- plus ironically begins funding a lot of the same spam that screws up its search results.
  • Google And The Big Brother Nomination: Is the Do No Evil company really evil in terms of spying on us? A long, long look at accusations and verdicts.
  • Search Privacy At Google & Other Search Engines: Search privacy wasn't (and isn't) just a Google issue, as this article explained.
  • Coping With GDS, The Google Dance Syndrome: An entire generation of search marketers was now online knowing nothing but Google Google Google and freaking over each "Google Dance" that altered the results.
  • Yahoo To Buy Overture: Hmm. Maybe we should own our own search ad system, thinks and acts Yahoo.
  • Microsoft's MSN Search To Build Crawler-Based Search Engine: Hmm. Maybe Google's going to eat us for lunch, so we should own our own search technology. So thinks and acts Microsoft.
  • Searching With Invisible Tabs: How search engines are going to automatically deliver the right vertical search results, even if you don't click on the right "tab."
  • Florida Google Dance Resources: A huge Google Dance shakes the results for site owners large and small, and an entire new generation of search marketers gets reeducated about how search engines are fickle creatures that you should never, ever build your entire business around.



2006 (To Date)

As I said earlier, I know I've missed stuff. Unfortunately, there's no easy way for me to see everything written on Search Engine Watch over the years in one single list. For those who wish to explore, the Search Engine Report archives are probably the best thing to review. Each month, there's an issue of the Search Engine Report that recaps virtually everything of importance that was published on the site. You can also see the SearchDay archives and the SEW Blog archives, though material from both of those places is integrated into Search Engine Report mailings.


Finally, some thanks....

  • Ken Spreitzer, my college friend who brought me on to do his web development company, getting me firmly going along a path that led to Search Engine Watch. Plus thanks to all those I worked with at Maximized Online: Tom, Joachim, Steve, Geeman (5 minutes!) and Michelle. I learned tons from all of them.
  • Glenn Fleischman, former moderator of the now defunct Internet Marketing Discussion List that helped educate me and many others about internet marketing when it was just developing.
  • Eric Ward, who freely provided guidance to many people looking to learn more about link building in the early days, including myself. Plus, he was Search Engine Watch's very first subscriber, coughing up a donation when I asked people to consider the site as shareware and help support it.
  • Jupitermedia (which acquired the site as Mecklermedia, changed into, then INT Media until taking its current name). Having someone take over the advertising and technical side of Search Engine Watch was a huge relief, allowing me to focus firmly on the editorial. Chris Elwell in particular was a rock of support and wisdom over the years. Incisive Media now owns the site, of course, and remains as welcomed in leaving me to focus on the editorial.
  • The search engines, and in particular the many employees who I've spoken with over the years, who have shared ideas, thoughts, theories and more. I won't name any individuals, simply because I fear I'd inadvertently leave some off -- plus it would be a huge list. You all play such an important role in this still developing field; keep doing great stuff.
  • The search marketers, who remain largely unsung heroes. You get people found; you get held to unbelievably high measuring standards yet still deliver; you help subsidize and make possible the search engines we all depend on. You also serve as a check-and-balance on the search engines. If they're playing favorites or doing something odd, you know it and spread the word.
  • The searchers, and in particular those who read the site to learn to search better. Your questions have inspired articles; please keep them coming. You're also what this is all about -- delivering up content to serve you better.
  • Search Engine Watch readers. There have been plenty of long days when I've written a long piece about some issue, then wondered if anyone cares. Then out of the blue, I'll get a word of thanks -- and that means the world to a writer.
  • Search Engine Watch members. For the longest time, I asked people to consider SEW as if it were a shareware site. A few sent in donations, then even more did when I opened a special members-only area. During the dotcom downturn, members helped ensure the survival of the site. We thrived because you helped fund us directly. Thanks to all of those who've shown support in this way, and I look forward to expanding benefits for you in the coming months.
  • The Search Engine Watch team. Chris Sherman's been my partner in search crime here on SEW since 2001, and I couldn't have been luckier in having him come aboard. He's been a great person to work with and produces wonderful content day in and day out. Elisabeth Osmeloski helped build our SEW Forums up from nothing to nearly 10,000 members now, and I'm excited to have her launching into doing more with the site overall. Gary Price is no longer part of our line-up, and we do still miss him -- but Barry Schwartz has been welcomed and doing a fantastic job. Plus my thanks to Jennifer Slegg and now Brian Smith for picking up their areas on our SEW Blog, along our many SEW Forums moderators who help keep things running over there.
  • My wife and my boys, who've missed me through too many dinners while I've had to finish and article or a newsletter or had to see me disappear to take a call about some last-minute product launch.

Want to comment or discuss? There's a thread going at our Search Engine Watch Forums.