IndustryDeath Of A Meta Tag

Death Of A Meta Tag

Now supported by only one major crawler-based search engine -- Inktomi -- the value of adding meta keywords tags to pages seems little worth the time.

Traffick.com’s Andrew Goodman wrote recently in an essay about meta tags, “If somebody would just declare the end of the metatag era, full stop, it would make it easier on everyone.”

I’m happy to oblige, at least in the case of the meta keywords tag. Now supported by only one major crawler-based search engine — Inktomi — the value of adding meta keywords tags to pages seems little worth the time. In my opinion, the meta keywords tag is dead, dead, dead. And like Andrew, good riddance, I say!

The Rise & Fall Of The Meta Keywords Tag

For those unaware, the meta keyword tag is a way to insert text into an HTML page that is not visible when the page is viewed through a browser. Some search engines have read the content of the tag and associated the words within it along with the page’s regular body copy.

The first major crawler-based search engines to use the meta keywords tag were Infoseek and AltaVista. It’s unclear which one provided support first, but both were offering it in early 1996. When Inktomi launched in mid-1996 through the HotBot search engine, it also provided support for the tag. Lycos did the same in mid-1997, taking support up to four out of the seven major crawlers at the time (Excite, WebCrawler and Northern Light did not provide support).

The ascendancy of the tag did not last after 1997. Experience with the tag has showed it to be a spam magnet. Some web site owners would insert misleading words about their pages or use excessive repetition of words in hopes of tricking the crawlers about relevancy. For this reason, Excite (which also owned WebCrawler) resisted added support. Lycos quietly dropped its support of the tag in 1998, and newer search engines such as Google and FAST never added support at all.

After Infoseek (Go.com) closed in 2000, the meta keywords tag was left with only two major supporters: AltaVista and Inktomi. Now Inktomi remains the only one, with AltaVista having dropped its support in July, the company says.

“In the past we have indexed the meta keywords tag but have found that the high incidence of keyword repetition and spam made it an unreliable indication of site content and quality. We do continue to look at this issue, and may re-include them if the perceived quality improves over time,” said Jon Glick, AltaVista’s director of internet search.

As for Inktomi, the search engine has no immediate plans to follow AltaVista’s lead:

“The meta keywords value is just one of many factors in our ranking equation, and we’ve never given too much weight to it. That said, we will continue to use it as long as our relevance modeling shows that it adds value,” said Ken Norton, director of product marketing for Inktomi’s web search division.

I’m certainly not crying over the decline of the meta keywords tag. It’s always been a confusing issue for site owners. Should I use commas between words in the tag or not? How many times can I repeat a word on the page without getting banned? If I don’t list a term in the tag, does that mean my page won’t show up? Those are common questions consistently raised over the years and represent time wasted worrying about a page element that a minority of crawlers supported — and for those that did, an element that was assigned little if any ranking boost.

Indeed, my advice about the meta keywords tag for ages has been simple. For those running large web sites or short on time, don’t worry about it. The stress and time involved in trying to craft a tag was not worth it, in terms of the minor benefit it might bring. It is far more important for site owners to instead concentrate on creating good title tags for their pages, a key page element that has consistently shown it can help with ranking across all major crawlers.

Now I can make my advice about the meta keywords tag even easier. Just don’t use the tag at all! Obviously, if you personally find it or believe it to be useful, keep doing so. But I suspect it’s just a waste of time, for most people.

Meta Description Tag Under Pressure But Surviving

In the case of the meta description tag, support of it has also weakened, and that’s unfortunate. You can put the blame on Google, though its own system of using “snippets” to describe web pages deserves high praise, as well.

When you search at Google, the search engine automatically creates what it calls snippets to describe the pages it lists. This snippet is created by looking for the first occurrence of the terms you searched for on the page and then extracting a few words from around that search. For example, in a search for “helium balloons,” you get this description for the top page (I’m leaving the title in as well, for readability reasons):

Howstuffworks “How Helium Balloons Work”
… How Helium Balloons Work by Marshall Brain Tell a friend about this
article! … There is something incredibly neat about helium balloons! …

This is very useful for the user, because it shows them precisely the context on the page where their search term occurs. That can be helpful in determining whether a page is a potential match for what they want.

Of course, meta tags can be useful, as well. It’s true that there is the potential for page authors to be misleading with these. However, the incidence of this seems relatively low. In fact, it’s one reason why Excite (when it still was crawling the web), eventually added support for the meta description tag. And in the example above, if Google had used the meta description tag, this succinct and helpful description would have appeared:

Howstuffworks “How Helium Balloons Work”
Helium balloons tend to fascinate adults and children alike (and it’s not
just the Donald Duck voice thing, though that is a big draw). Learn all
about helium and why it floats!

In my view, it’s nice to see how an author describes their page, in addition to seeing a snippet based on what you searched for. Both are valuable perspectives which should be preserved. That’s something Google’s major competitors are trying to do. For instance, both AltaVista and Teoma will blend the two, as with this example from AltaVista:

Howstuffworks “How Helium Balloons Work”
Helium balloons tend to fascinate adults and children alike (and it’s not
just the Donald … Much Does the Earth Weigh” –> Teachers! –> –> How
Helium Balloons Work by Marshall Brain Tell a friend …

Unfortunately, such a mixture isn’t particularly satisfying. Instead, I love the approach used by FAST, which shows both a snippet and then the meta description tag content in a separate “Description” area:

Howstuffworks “How Helium Balloons Work”
… view your points! How Helium Balloons Work by Marshall Brain …
Introduction to How Helium Balloons Work Floating in … incredibly
neat about helium balloons! If you buy one at … Introduction to How
Helium Balloons Work Floating in …
Description: Helium balloons tend to fascinate adults and children alike
(and it’s not just the Donald Duck voice thing, though that is a big draw).
Learn all about helium and why it floats!

The main improvement I’d suggest might be to swap the order, leading off with the meta description tag and making the snippet the second element, perhaps calling it out as “Extract” or even “Your Terms On The Page.” Why not say “Snippet?” The term is actually Google’s jargon for its descriptions, so I suspect the other search engines probably wouldn’t want to adopt it.

As for Inktomi, the company is currently shifting how it lists web pages. The company plans to rollout its own snippet-style descriptions by the end of the year and will use these “Smart Abstracts,” as it calls them, when it decides they make the most sense. In other cases, a meta description tag might be used, if available.

As you can see, the meta description tag — unlike the meta keywords tag — is definitely not dead. It remains well worth it for site owners to make use of the tag, and doing so needn’t be a time consuming process. My recommendation for those in a hurry is to take the first sentence or two from the beginning of the body copy of your home page. Quite often those will provide a good summary of what your page is about.

Google And Descriptions

Let’s take a closer look at how pages are listed at the various major crawlers, because it isn’t always a simple case of the meta description tag versus snippets. Our tour will start with Google. Here’s a summary of how descriptions are formed, in order of priority:

  1. Snippet & ODP Description
  2. Snippet
  3. Meta Tag

Whenever possible, Google prefers to list pages using both a snippet and the description of the page, if it is listed in the Open Directory. For instance, look at this description, for the top site that came up for “helium balloons:”

Howstuffworks “How Helium Balloons Work”
How Helium Balloons Work by Marshall Brain Tell a friend about this
article! There is something incredibly neat about helium balloons!
Description: Explanation of lifting capabilities of helium, hydrogen and hot air balloons, from How Stuff Works.
Category: Science > Physics > Classical Mechanics
www.howstuffworks.com/helium.htm – 25k – 28 Sep 2002 – Similar pages

The first line is the page’s title, of course. The next two lines are the “snippet,” the first text found on the page that contains the search terms “helium balloons,” then the second sentence on the page containing those terms.

After that, the “Description” line appears. This is because the page is also listed in the Open Directory, and so the description listed by the editors of the Open Directory is shown. In fact, if you click on the “Category” link, you’ll see that the page is listed in Google’s version of the Open Directory.

Unfortunately, you have little ability to control either of these major descriptive elements. The snippet that Google will display for a page is based on what someone searches for — so if a page is relevant for a variety of different terms, the snippet will constantly change. The only advice here is that to ensure that the first few lines of your pages contain the key terms you think the page should be found for and ensure that those lines help describe the page well. That might produce a better snippet.

As for the Open Directory description, if you don’t like what’s shown there, you’ll have to contact the editor of the Open Directory category and request a change. This is really only likely to happen if there’s a good, factual reason to make such a change.

If a page isn’t listed with the Open Directory, then no “Description” line will be displayed. Finally, in some rare cases, Google will make use of a meta description tag if there’s no substantial text on a page.

AltaVista And Descriptions

At AltaVista, descriptions are formed in this order:

  1. Custom Tagline
  2. Meta Tag
  3. Meta Tag + Snippet
  4. Snippet

At AltaVista, the most surefire way to get the description you want is via the paid inclusion program. For any page you’ve gotten listed via paid inclusion, you can choose to purchase an optional “Custom Tagline” through the Listing Enhancements program. This will let the page description be exactly as you’d like. The big downside, of course, is price. It will cost anywhere from $38 to $78 per year to get a single page included in AltaVista, then another $72 to $100 to add a custom tagline to that page. So you’ll be spending between $110 to $178 per page that you want described exactly as you’d prefer.

Want to save money? Then the meta description tag is your next best bet. If it is present, AltaVista will always make use of it. However, how much of it may vary.

In some cases, AltaVista will use all of your tag. In other situations, it will use part of your meta tag and then append a snippet to make the rest of the description. For example, with the helium balloon example:

Howstuffworks “How Helium Balloons Work”
Helium balloons tend to fascinate adults and children alike (and it’s not
just the Donald … Much Does the Earth Weigh” –> Teachers! –> –> How
Helium Balloons Work by Marshall Brain Tell a friend …

The meta tag portion lasts until the first ellipse appears (the … portion). The remaining portion is pulled from the text near the first words on the page.

If you don’t use a meta tag, then the entire description will be a snippet, or what AltaVista calls a “dynamic abstract.” Also see the article below for a close look at abstracts with AltaVista.

Teoma And Descriptions

At Teoma, descriptions are made in this order:

  1. Snippet + Meta Tag
  2. Snippet
  3. ODP Description

As you can see, it is very similar to AltaVista. If a meta tag is used, Teoma will use the content of the meta tag for the first part of the page description then append the remaining portion with a snippet. And, if you don’t use a meta tag, then a snippet will be extracted from the page. Finally, if there is no body copy, then a description from the Open Directory will be used, if the page is listed there.

By the way, things may change at the end of this month. Teoma says it expects to shift to fully dynamic descriptions, rather than a meta tag and snippet blend. However, text from within the meta tag might be used to form a snippet — so if your meta description tag contains the words the page is most likely to appear for, then portions of the meta description tag are more likely to be used.

FAST (AllTheWeb.com) And Descriptions

At FAST’s AllTheWeb.com site, summaries are formed like this:

  1. Snippet & Meta Tag
  2. Snippet & ODP Description
  3. Snippet

At AllTheWeb.com, the first portion of the description will always be drawn from the content of the web page itself, snippet-style. Below this portion, as mentioned, may be a second block of descriptive text that begins with the word “Description.” This block will appear if a meta description tag is used and will use the text of the tag. Otherwise, if a page is listed in the Open Directory, then the Open Directory description will be used if the page DOES NOT use a meta description tag.

FAST Descriptions At Lycos

Now that you are all clear about how FAST does descriptions for AllTheWeb.com, just to confuse you, things work like this for the FAST-powered results at Lycos.com:

  1. Meta Tag
  2. Snippet

In other words, the description will come from the meta tag, if one is used. If not, then a snippet will be created.

Inktomi Descriptions On MSN Search

Most people are likely to encounter your page descriptions on Inktomi via MSN Search. However, explaining what’s happening with these descriptions is complicated, given that Inktomi is currently changing its system. At the moment, things work like this:

  1. Meta Tag For Paid Inclusion Customers
  2. LookSmart Description
  3. Meta Tag OR First Text

If you are a paid inclusion customer with Inktomi, then you get complete control over your description — and that’s reasonable, considering you are paying them to have the page listed. Indeed, unlike AltaVista, Inktomi doesn’t charge you a premium to have the description you want. Just make sure your paid inclusion pages have meta tags on them, and your meta tags will automatically take priority. Inktomi also says that it is considering some new ways of allowing paid inclusion customers to control their page appearances, so I’ll bring more details as these are released.

If you aren’t a paid inclusion customer, then the next issue to consider is whether your page is listed with LookSmart. If so, then your LookSmart description will be used even if you have a meta tag on it.

Finally, if your page isn’t listed in LookSmart, then either your meta tag or the first few lines of text on the page will be used to make a description. Inktomi says that it automatically chooses what it considers to be best, so it’s difficult to predict what will come up.

Making that prediction is going to get even harder in the future. By the end of the year, the description process will look like this:

  1. Meta Tag For Paid Inclusion Customers
  2. LookSmart OR Meta Tag OR First Text OR Snippet

The new system has two important changes. First, Inktomi will begin presenting snippet-like descriptions, rather than using the first text on the page. Second, a page might have any of four possible descriptions used. For example, a page listed in LookSmart and with meta tags might get a description used from either of those sources OR from the first text on its page OR have a snippet-style description. The choice will be made dynamically, by Inktomi.

How To Use HTML Meta Tags
https://www.searchenginewatch.com/webmasters/meta.html

More in-depth coverage of the three major meta tags that have been used by crawler-based search engines: the meta keywords, description and robots tags. I haven’t changed the section that says do use the meta keywords tag to don’ bother using it — but if you’ve just read this article, then you already know this!

Image Search Faces Renewed Legal Challenge
The Search Engine Update, August 22, 2001
https://www.searchenginewatch.com/sereport/01/08-images.html

The meta robots tag also has some special provisions that can be used to stop image indexing.

An End to Metatags
Traffick.com, Sept. 4, 2002
http://www.traffick.com/article.asp?aID=102
http://www.traffick.com/article.asp?aID=105

Andrew Goodman”s column on the devaluation of meta tags, which is in two parts.

Re: HTML 2.0 spec and the META element
www-html mailing list, June 12, 1994
http://lists.w3.org/Archives/Public/www-html/1994Jun/0041.html

And how did we end up with the three major meta tags supported by search engines, anyway? I’ve spent far too long trying to track down the exact origins without success. However, I’ll tell the tale as best I can figure. It starts with this posting, the earliest mention I can find about the need for adding meta data to documents. This proposal for the HTML 2.0 specifications does talk about a “keywords” tag as being one possible option, though exactly why and for what a keywords tag was intended is not clear. Bear in mind that WebCrawler and Lycos were the only major crawlers operating at this time, only months old, and neither supported such a tag or were pushing the W3C to create one for them.

HTML 3.0: Head And Related Elements
W3C, March 1995
http://www.w3.org/MarkUp/html3/dochead.html

Early the next year, the draft of HTML 3 included the provision of meta tags of any type (something that was also being integrated into the final guidelines for HTML 2 — but at this time, people were already looking beyond that.

Meta Tag – proposal (suggestions ???)
www-html mailing list, http://lists.w3.org/Archives/Public/www-html/1995Nov/0047.html

Here we get some more specific suggestions on how meta tags might be constructed and used.

Spidering Birds Of A Feather Report
W3C Distributed Indexing/Searching Workshop, May 28-19, 1996
http://www.w3.org/pub/WWW/Search/9605-Indexing-Workshop/ReportOutcomes/Spidering.txt

And finally, we have some proposals come out of the W3C where the search engines themselves were involved. Definitions for the meta robots and description tags were set, but the meta keywords tag was NOT defined. Indeed, at this point, AltaVista and Infoseek had already posted their own independent instructions on using a meta keywords tag for their search engines (and neither required commas between keywords).

Distributed Indexing/Searching Workshop
W3C , May 28-19, 1996
http://www.w3.org/Search/9605-Indexing-Workshop/

Home page for the workshop where the above “Spidering Birds Of A Feather” workshop was held. This workshop was unique in being the first and last time, to my knowledge, that representatives from the major crawler-based search engines ever got together to agree on some technical specifications that they all could share.

Notes on helping search engines index your Web site
W3C HTML 4.1 Specification, Dec. 24, 1999
http://www.w3.org/TR/html401/appendix/notes.html#recs

The current HTML specifications from the W3C have an appendix on coding for search engines. However, the major crawlers were not involved with creating these guidelines nor do they necessarily follow them. For instance, the major crawlers do not rely on the language meta tag to determine adocument’s language — indeed, they’ve generally said this information is undependable, when I’ve queried them on it in the past. Also, “comma-separated” keywords in a meta keywords tag have not been required by the major crawlers, and as mentioned, Inktomi is now the only one supporting this tag. I also don’t believe the use of the link element makes any difference to the crawlers. As for the meta robots tag, as mentioned about image indexing, various major crawlers have added their own variations to the tag without gaining W3C approval nor coordinating efforts among themselves.

W3C Semantic Web
http://www.w3.org/2001/sw/

The Semantic Web is a visionary concept of how the web will function better through greater interconnectedness of web pages. Yes, pages are already connected via links, but in the Semantic Web, information will flow between them as necessary and needed, as well. Unfortunately, the portion of this future depending on human-provided meta data is simply not going to happen, not where the web-wide search engines are involved. Experience has consistently shown that they don’t trust meta data, both because some people are explicitly misleading with it while others neglect to add it or add incorrect information. Case in point — there is a meta author tag that could be added to documents, and my HTML editor will do this automatically for me. However, should my wife sit at my computer and make a document, I’ll still be listed as the author unless she changes the settings. Many will fail to do this.

Dublin Core Metadata Initiative
http://dublincore.org/

Dublin Core tags are part of the Semantic Web’ve also been used in other ways, as well. No major crawler supports the tag, but intranet and enterprise software does.

Resources

The 2023 B2B Superpowers Index

whitepaper | Analytics The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing

whitepaper | Analytics Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook

whitepaper | Digital Marketing The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

whitepaper | Digital Marketing Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y