My Decade Of Writing About Search Engines

Ten years ago today, I first starting writing publicly about search engines.
If we had blogs back then, I suppose I would have been a search blogger. But we
didn’t. We hand-coded our HTML, walked through the snow for eight miles to FTP
files to our web servers, and we liked it :)

My involvement with search engines goes back to my first year as a student
at the University Of California, Irvine in 1983. No, I wasn’t part of the
university’s highly regarded information and computer science department.
Instead, I was an English major — and a pretty bored one for my first two
months, when I had to commute until getting on-campus housing.

I spent some time exploring the library, having been a big library user since
I was a child. The library had a magical electronic card catalog called
Melvyl (named
for Melvil Dewey, who created the Dewey Decimal System). For fun, I’d do Melvyl
searches for broad topics such as history, art or love, to see how many matches
would come back. I could routinely crash the search routine by doing this.

The system would diligently try, telling me it would take 50, 60, 80 search
cycles, and then the countdown would begin. Some searches would eventually get
through all cycles and give me a matching results count. Often, the system would
just give up as the countdown approached the teens.

My 1996 Study

Search engines remained fascinating to me when I reencountered them in 1995.
I’d left working as a newspaper reporter to go into web development, since I
didn’t want to miss out what was obviously going to be the future of publishing.
As the general manager of Maximized Online, my job was to help get people in the
Orange County, California area online. We’d build web sites, get them publicized
to search engines and other publicity venues, plus host them.

One of our clients was upset at the end of 1995 that his OC jobs site wasn’t
ranking tops for a search on "orange county" in WebCrawler. We didn’t have a
good answer to give him. We’d done the submission, made use of the meta tags the
search engines said to use, but why exactly a site would rank well wasn’t well
known. So I decided to look into it.

I spent January through April 1996 making changes to the InfoPages directory
that my company maintained, a search engine just for Orange County web
resources, to see if it could rank better in a search for "orange county." I
tried putting those words in the body text, the title tag, in the meta tags and
also checked to see if spamming helped, if repeating the word over and over
would have an impact.

I published the results online, and 10 years on, a lot of the advice remains
exactly the same. Don’t depend on ALT text. Don’t fixate on only one or two
terms, because there are many ways people will seek you — a long tail before we
had talk of long
tails
. Build links, because links can send you traffic. And don’t fixate on
getting traffic just from search engines. The conclusions from that study are below for those
really interested; others can jump past for the rest of this article.

There don’t seem to be any magic methods that will make a page appear at
the top of every search engines’ listings. There’s too much fluctuation on the
web for any page to claim a foothold, and all the engines handle relevancy
slightly differently. However, there are some general tips that do help a page
appear more relevant.

  • Have text on your home page: Search engine catalogs contain the
    text read from the various home pages the engines visit. If a page lacks
    descriptive text, then there is little chance that page will come up in the
    results of a search engine query. It’s not enough for that text to be in
    graphics. It must be HTML text. Some search engines will catalog ALT text
    and text in comment and meta tags. To be safe, a straight HTML description
    is recommended.
     
  • Pick your keywords: Focus on the two or three keywords that you
    think are most crucial to your site, then ensure those words are both in
    your title and mentioned early on your web page. Generally, most people will
    already have those words present on their pages but may not also have them
    in page titles. Keep in mind that the keywords you consider crucial may not
    be exactly what users enter. Our study focused on making the InfoPages
    directory appear high on lists if keywords "Orange County" were entered. The
    lack of success with some search engines does not mean that the site isn’t
    being found. Many people find the site by entering more words, such as
    "Orange County California" or "Orange County Web." The addition of just one
    extra word can suddenly make a site appear more relevant, and it can be
    impossible to anticipate what that word will be. The best bet is to focus on
    your chosen keywords but to also have a complete description.
     
  • Have links to inside pages: If there are no links to inside pages
    from the home page, it seems that some search engines will not fully catalog
    a site. Unfortunately, the most descriptive, relevant pages that are often
    inside pages rather than the home page. You can also try sending search
    engines directly to your lower levels, if they don’t ordinarily go there.
     
  • Forget Spamming: For one thing, spamming doesn’t seem to work
    with every search engine. Ethically, the content of most web pages ought to
    be enough for search engines to determine relevancy without webmasters
    having to resort to repeating keywords for no reason other than to try and
    "beat" other web pages. The stakes will simply keep rising, and users will
    also begin to hate sites that undertake these measures. Efforts would be
    better spent on networking and alternative forms of publicity described
    below.
     
  • Network: If your site fails to make the top ten lists, then get
    together with those that do. Perhaps some might be considered "competitors,"
    but others might be happy to link to your site in return for a link back.
    After all, your site may appear first when slightly different keywords are
    used. Links are what the web was built on, and they remain one of the best
    ways for people to find your site.
     
  • Relax: Search engines are a primary way people look for
    web sites, but they are not the only way. People also find sites
    through word-of-mouth, traditional advertising, the traditional media,
    newsgroup postings, web directories and links from other sites. Many times,
    these alternative forms are far more effective draws than are search
    engines. The audience you want may be visiting to a site that you can
    partner with, or reading a magazine that you’ve never informed of your site.
    Do the simple things to best make your site relevant to search engines, then
    concentrate on the other areas.

A Webmaster’s Guide To Search Engines

Along with the study, I also published a collection of documents called "A
Webmaster’s Guide To Search Engines." My goal was to help site owners better
understand the essentials of being found plus identify which search engines really
mattered. Knowing who mattered was crucial when you’d have some search engines
like Galaxy forcing your through a three part, multiple question submission
process to be included in their directory. Was spending all that time worthwhile? (For
Galaxy, the answer was no!).

The guide provided links to the FAQs of each search engines, along with my own
observations about whether how each search engines said it worked actually lived
up to reality. There was a guide to which search engines I considered to be
"major" or most important to site owners and searchers alike. I had a
"Strategic Alliances & Victories" chart to show which search engines had deals
with the Netscape or Internet Explorer browsers and which had gained positive
reviews in magazines.

The information I published quickly generated a lot of positive feedback,
both from site owners and searchers such as librarians. At the same time, the
web development company I worked for closed, so that the parent firm could
concentrate on web software development. I hung out my internet consultant
shingle and kept maintaining the Webmaster’s Guide on a part time basis, sending
out a newsletter update (The Search Engine Report)
twice that
year
, along with making further site updates.

In 1997, I moved to the UK from California, so my wife could be closer to her
family. I also began spending more and more time on the site, as well as writing
freelance articles on search for various publications. In the middle of the
year, I rebranded the site as Search Engine Watch, which generated more
attention. By the end of the year, Mecklermedia purchased the site from me, and
I continued on as editor of it.

The Search Revolution

Ten years on, I remain as fascinated with search engines as ever. I’ve been
fortunate to help chronicle the birth of an entirely new advertising medium.
Equally important has been the birth of an entirely new way for people to seek
out information.

I knew search engines were important when I decided to write about them. The
journalist in me could see they were a good story, especially when you realized
that under the hood, they weren’t doing things like crawling as often as people
widely believed. But a
study by
Keen in 2001 especially resonated with me. Search engines (as a whole — we
weren’t Google obsessed yet then) were the single most likely way people would
seek information.

The study was small, but the findings were still stunning. In only about five
years, search engines had ousted things like friends, family, books, magazines,
libraries and other perfectly good resources for seeking answers.

Some of this was bad. I’d personally watched people when doing search
training spending ages trying to find a phone number, when a call to telephone
information would have found much faster. Old but still useful search strategies
were abandoned in favor of the magic search box.

Lots of this is good. Search engines remain amazing tools that get us the
right answers quickly in many circumstances.

Looking Ahead

Will I still be doing this in 20 years? Almost certainly not, at least not in
the daily grind format I’ve been doing. I’d like to keep writing about
search issues, but eventually I’ll move away from the regular day-to-day
coverage to perhaps focus on less frequent but deeper looks at particular search issues.

I’m also thinking a lot about doing a book these days. I’d always wanted to do a book on search, indeed the exact type of
history that
John Battelle did a
fantastic job
with
in The Search.

Since that’s come out, I’ve thought more and more about
doing a more personal retelling of web search history — the evolution, developments and trends
I’ve seen from having been in the trenches of covering them over the years.

I’d
also like to do a separate one talking to various search marketers, spotlighting
them and focusing on how that medium has evolved over the years and where it
will be going. The most fascinating book idea remains the impact of search on our
everyday lives, how people make use of them, how habits have changed, our
laws are starting to account for the power of search and many related issues
like that.

Someday! What I can say is that for the near future, I expect to remain
working on the site and coverage as I have, bringing some of our standing
content back up to date, which I know has been neglected due to the need to
cover the news that continues to flow in. My original Webmaster’s Guide helped
many understand search engines, and I very much want to
ensure Search Engine Watch remains as a leading resource doing that in the years
to come.

Looking Back

I don’t have a succinct list of big picture items or "high order bits" to offer.
A lot of this has already been covered in things I’ve
written, so instead I’m going to spend some time recapping pieces I think are
most important below. These are either big trend pieces I’ve done or big shifts
in the search landscape I think worth noting.

I know — I KNOW — I’ve left some things out. My apologies, if so. It’s a
bit easier for me to cover all the things I’ve written that what me or Chris
Sherman both have done, and he’s clearly covered tons himself. Plus, skimming
through 10 years worth of writings means I’ll accidentally miss stuff. If you
want to go poking yourself, I’ll give more tips after the summary.

The biggest overall theme in doing the recap is how that big old wheel keeps
spinning around and around, with people often buying hype because they don’t
remember things have come before — or marketers making errors because they
don’t understand issues that were explored already in the past.

I’ve definitely felt myself getting more and more jaded. Part of that’s bad,
because there are cool, new things that I don’t want to be blinded to. But then
again, you go through the list below and tell me if you don’t emerge feeling a
big jaded about some ideas and concepts that are retreads.

1997

1998

1999

2000

2001

  • WebTop
    Search Rage Study
    : Study sez we depend on search engines so much
    now that we’ll get enraged if they don’t give us the right answers quickly.
     
  • Bush’s
    Dubious Victory At Google
    : Google’s link system takes another blow as the
    number one result for motherf***r is the official George W. Bush campaign web
    site.
     
  • The End
    For Search Engines?
    : A look at why so many search engines were on the
    ropes or dead by this year but how paid listings gave hope and life for new
    ones.
     
  • Internet
    Top Information Resource, Study Finds
    : Search engines and the internet
    found to be the top information resource.
     
  • Being
    Search Boxed To Death
    : Google and other search engines today are
    struggling with how to best tell people about the many vertical search results
    they have. It’s a long-standing problem, as this story looks at.
     
  • Google
    Acquires Deja Newsgroup Service
    : Google gets its first real heavy dose of
    criticism after Deja junkies freak out when Usenet features get lost
    temporarily after the acquisition.
     
  • Search
    Engine Marketing Finally Getting Respect
    : Despite search being an
    essential marketing strategy for years, finally more traditional outlets begin
    to "get it" and give search marketing some resepct.
     
  • Avoiding
    The Search Gap
    : Search is one of the most popular web activities, yet most
    sites don’t get the majority of their traffic from search? What gives? It’s
    the search gap and key to understanding how search makes an introduction that
    can lead to a lifetime of visits.
     
  • Fourth
    Time Lucky For AltaVista?
    : "AltaVista could give Madonna a run for her
    money in the changing your image game. Earlier this month, the service once
    again significantly changed its look and feel, the fourth such redesign in
    just over a year." That was 2001. And in 2000, there were another four
    designs, rather than just improving the results. Perhaps they should have
    changed the name to MSN Windows AltaVista Live Search.
     
  • Make Room
    For Teoma
    : A different twist on link analysis, but more important, a new
    search voice worth taking seriously — and Ask Jeeves did,
    buying
    it later that year.
     
  • Consumer
    Group Asks FTC To Investigate Search Ads
    : Perhaps those "featured
    listings" popping up everywhere through payment should be better labeled as
    ads, the FTC is asked.
     

  • Desperately Seeking Search Engine Marketing Standards
    : Another push for
    standards in SEO falls into issues of knowing what exactly the rules are.
     
  • Search
    Engine Marketing: You Like It, You Really Like It
    -: I suggest the term
    "search engine marketing" as an umbrella term to encompass search engine
    optimization (the act of getting better free listings on search engines) and
    search advertising (paying for listings). Readers like it. FYI, today I use
    the term "search marketing" a bit more.

2002

2003

  • Yahoo To
    Buy Inktomi
    : Hmm. Maybe Google’s not our friend and we should own our own
    crawler-based technology to protect ourselves. So thinks and acts Yahoo.
     
  • Overture
    To Buy AltaVista
    : Hmm. Maybe Google offering both paid and unpaid results
    really is an advantage and we should own some technology to produce
    crawler-based editorial results. So thinks and acts Overture.
     
  • Overture
    To Buy FAST Web Search Division
    : Hmm. Maybe we should by FAST as well and
    keep competitors from getting the technology.
     
  • Ending The
    Debate Over Cloaking
    : Perhaps we could skip past defining spam by
    techniques and look instead at intent and results? Techniques like cloaking
    doesn’t always indicate a harmful results for search engines — and some of
    them certainly allow it with approval.
     
  • Google
    Throws Hat Into The Contextual Advertising Ring
    : Later to be called
    AdSense, Google starts an entirely new economy of bloggers and publishers
    depending on its ads — plus ironically begins funding a lot of the same spam
    that screws up its search results.
     
  • Google And
    The Big Brother Nomination
    : Is the Do No Evil company really evil in terms
    of spying on us? A long, long look at accusations and verdicts.
     
  • Search
    Privacy At Google & Other Search Engines
    : Search privacy wasn’t (and
    isn’t) just a Google issue, as this article explained.
     
  • Coping
    With GDS, The Google Dance Syndrome
    : An entire generation of search
    marketers was now online knowing nothing but Google Google Google and freaking
    over each "Google Dance" that altered the results.
     
  • Yahoo To
    Buy Overture
    : Hmm. Maybe we should own our own search ad system, thinks
    and acts Yahoo.
     

  • Microsoft’s MSN Search To Build Crawler-Based Search Engine
    : Hmm. Maybe
    Google’s going to eat us for lunch, so we should own our own search
    technology. So thinks and acts Microsoft.
     
  • Searching
    With Invisible Tabs
    : How search engines are going to automatically deliver
    the right vertical search results, even if you don’t click on the right "tab."
     
  • Florida
    Google Dance Resources
    : A huge Google Dance shakes the results for site
    owners large and small, and an entire new generation of search marketers gets
    reeducated about how search engines are fickle creatures that you should
    never, ever build your entire business around.

2004

2005

2006 (To Date)

As I said earlier, I know I’ve missed stuff. Unfortunately, there’s no easy
way for me to see everything written on Search Engine Watch over the years in
one single list. For those who wish to explore, the
Search Engine
Report archives
are probably the best thing to review. Each month, there’s
an issue of the Search Engine Report that recaps virtually everything of
importance that was published on the site. You can also see the
SearchDay archives
and the SEW Blog
archives
, though material from both of those places is integrated into
Search Engine Report mailings.

Thanks

Finally, some thanks….

  • Ken Spreitzer, my college
    friend who brought me on to do his web development company, getting me firmly
    going along a path that led to Search Engine Watch. Plus thanks to all those I
    worked with at Maximized Online: Tom, Joachim, Steve, Geeman (5 minutes!) and
    Michelle. I learned tons from all of them.
     
  • Glenn Fleischman,
    former moderator of the now defunct Internet Marketing Discussion List that
    helped educate me and many others about internet marketing when it was just
    developing.
     
  • Eric Ward, who freely provided
    guidance to many people looking to learn more about link building in the early
    days, including myself. Plus, he was Search Engine Watch’s very first
    subscriber, coughing up a donation when I asked people to consider the site as
    shareware and help support it.
     
  • Jupitermedia (which acquired
    the site as Mecklermedia, changed into Internet.com, then INT Media until
    taking its current name). Having someone take over the advertising and
    technical side of Search Engine Watch was a huge relief, allowing me to focus
    firmly on the editorial. Chris Elwell in particular was a rock of support and
    wisdom over the years. Incisive Media now owns the site, of course, and
    remains as welcomed in leaving me to focus on the editorial.
     
  • The search engines, and in particular the many employees who I’ve spoken
    with over the years, who have shared ideas, thoughts, theories and more. I
    won’t name any individuals, simply because I fear I’d inadvertently leave some
    off — plus it would be a huge list. You all play such an important role in
    this still developing field; keep doing great stuff.
     
  • The search marketers, who remain largely unsung heroes. You get people
    found; you get held to unbelievably high measuring standards yet still
    deliver; you help subsidize and make possible the search engines we all depend
    on. You also serve as a check-and-balance on the search engines. If they’re
    playing favorites or doing something odd, you know it and spread the word.
     
  • The searchers, and in particular those who read the site to learn to
    search better. Your questions have inspired articles; please keep them coming.
    You’re also what this is all about — delivering up content to serve you
    better.
     
  • Search Engine Watch readers. There have been plenty of long days when I’ve
    written a long piece about some issue, then wondered if anyone cares. Then out
    of the blue, I’ll get a word of thanks — and that means the world to a
    writer.
     
  • Search Engine
    Watch members
    . For the longest time, I asked people to consider SEW as if
    it were a shareware site. A few sent in donations, then even more did when I
    opened a special members-only area. During the dotcom downturn, members helped
    ensure the survival of the site. We thrived because you helped fund us
    directly. Thanks to all of those who’ve shown support in this way, and I look
    forward to expanding benefits for you in the coming months.
     
  • The
    Search Engine Watch team
    .
    Chris
    Sherman’s
    been my partner in search crime here on SEW since 2001, and I
    couldn’t have been luckier in having him come aboard. He’s been a great person
    to work with and produces wonderful content day in and day out.

    Elisabeth
    Osmeloski
    helped build our SEW Forums up from nothing to nearly 10,000
    members now, and I’m excited to have her launching into doing more with the
    site overall. Gary Price is no longer part of our line-up, and we do still
    miss him — but Barry Schwartz has been welcomed and doing a fantastic job.
    Plus my thanks to Jennifer Slegg and now Brian Smith for picking up their
    areas on our SEW Blog, along our many SEW Forums
    moderators
    who help keep things running over there.
     

  • My wife and my boys, who’ve missed me through too many dinners while I’ve
    had to finish and article or a newsletter or had to see me disappear to take a
    call about some last-minute product launch.

Want to comment or discuss? There’s a
thread
going at our Search Engine Watch Forums.