Yahoo: Delays Expected

Yahoo: Delays Expected

By Danny Sullivan, Editor
The Search Engine Report, Sept. 3, 1997

NOTE: This is part of the Yahoo Special Report.

Yahoo's strength and success as a guide comes from humans. Unlike the search engines, Yahoo does not spider web sites to build listings automatically, using technology. Instead, human editors provide the brain power and intuition needed to classify the web's many offerings.

Through this human effort, Yahoo has become the de-facto Dewey Decimal System for categorizing web sites. If you are looking for something, you'll usually find a relevant subject in Yahoo that lists web resources.

Unfortunately, humans are also Yahoo's weakness. As submissions increase, the guide must either bring on new editors to process listings, let the backlog grow or simply accept a smaller percentage of listings as in the past.

Currently, it's the third choice that is being followed. A smaller percentage is getting in.

"We love submissions, and we only want to encourage them," said Srinija Srinivasan Yahoo's Ontological Yahoo, or Director of Surfing, who oversees the staff of 40+ editors who build the guide. "However, we also have to accept that we can't get to everything." Instead, she says, "We would like to think the best of what's out there is in Yahoo."

Even though every site submitted is not listed, Yahoo says that it is still providing a useful service to those searching for information.

"The users are finding what they need," Srinivasan said. "That is our first and foremost goal."

For the most part, she's right. Most users are going to come away from Yahoo finding something that they are looking for. To them, whether everything is listed remains irrelevant.

It's as if you go into a library to get a book on camping and find twenty different titles on the shelf. If you're happy with those books, you probably don't care that another four aren't available.

The situation is far different for web site owners and content providers. They're acutely aware when some books are missing. That's because they publish what is, or isn't, on the shelf. Those not on the shelf have no chance of being read.

One survey respondent writes, "Yahoo holds the power to so many Internet businesses."

It's somewhat ironic that a guide with such a populist feel, edited by sincere and friendly staff, and born from the Internet's early non-commercial roots would produce such a fearful sounding statement. But it also points out the reality of how important being listed in Yahoo is to businesses, as well as other web sites across the net.

Furthermore, statements like that aren't good for the bottom line, at least according to Yahoo's 1996 annual report, filed last March:

"The company's future success also depends in part upon the timely processing of Web site listings submitted by users and web content providers, which have increased substantially in recent periods. The company has from time to time experienced significant delays in the processing of submissions, and further delays could have a material adverse effect on the company's goodwill among web users and content providers, and on the company's business."

With a declaration like that, shouldn't Yahoo spend some of its millions to hire more editors immediately? Why not double its staff, rather than risk losing its good will?

"The web grows faster than we do, and we couldn't possible scale personnel to match the rate in which new web sites that are coming along" Srinivasan said. And she adds, "With unlimited resources, I wouldn't even do that."

One of the biggest challenges is that the surfers all communicate with each other and exist in a culture where they understand how to categorize information in the same way. "There's got to be some similar subjectivity that we are all applying," Srinivasan said. "Otherwise, we would fail to have a useful guide."

Clearly more people will be brought on, and Yahoo continues to look at ways to improve or even semi-automate the listing process. But to date, it hasn't found anything it feels does a good enough job. And the shear number of submissions continues to be a challenge.

"I'm not sure any process change is going to solve the volume process," Srinivasan said. "The fact that we're inundated with requests isn't going to go away."

That means the existing situation of sites facing listing delays or simply not being listed is going to continue for some time.

"I think it's going to be more or less status quo for the near future," Srinivasan said. "Fortunately or unfortunately, the system that we have in place works reasonably well. We're pleased with the content that coming in," Srinivasan said, adding that they do get good feedback from many webmasters.

To be fair, Yahoo is not adding fewer sites. In fact, it probably adds more sites to the guide than it has in the past. The problem is that there are so many sites that that Yahoo is adding a smaller percentage of submissions.

Nor is Yahoo the only major guide to the web having trouble keeping up, as discussed more below. But Yahoo faces particular pressure to be complete. It's the market leader, by far the most popular guide currently used. But beyond that, Yahoo is the closest thing to an official phone book that the web has.

Today, web sites are becoming as common as phone numbers. Everyone has them, and everyone needs to be listed. While there are many places to submit, Yahoo is the most important. It is the one people turn to most. And when a web site can't get it, it's as if the phone company is refusing to publish its number.

Of course, Yahoo may not feel it has this responsibility. It is not a public utility. It is a privately-run guide to the web, and it can publish whatever it wants, however it wants.

"Our primary goal is to satisfy the users, not the listers," Srinivasan said. "We're not trying to be their advertising vehicle. The listing isn't trying to satisfy that."

But almost in the same breath, Srinivasan recognizes the importance of Yahoo to site owners.

"The individual business is the fastest growing submission. Every florist, every dry cleaner wants to be in. That poses a new challenge," she said. "Especially for the businesses, we are trying to find ways to get to that next florist and the dry cleaner."

What about a paying for listings? Not necessarily to improve a position, or to influence the description, but simply to help speed up the process? "That opens a humongous can of worms," Srinivasan says, in terms of accountability and perception.

(Note: Yahoo now offers this. See Yahoo Opens Express Submission Service, from the March 1999 Search Engine Report)

Yahoo recognizes the challenge before it, but there remains no easy solution about to emerge. Meanwhile, Yahoo always has the ultimate out to defend charges that it doesn't list all that's out there.

"We rely on the search engines to give everything," Srinivasan said.

At the bottom of every page in Yahoo are outward bound links to search engines. Click on any of them, and your query will be sent to the search engine of your choice. In addition, Yahoo automatically sends queries to its partner AltaVista, should it fail to find a match within its own listings.

Unfortunately, the search engines, AltaVista included, are no longer promising to index everything. The "we index it all" attitude that was popular in 1996 is changing into a "we index the best" attitude as 1997 progresses.

Despite the reliance on technology, spidering everything out there has proven hard. And like Yahoo, the search engines argue that spidering every last web page is unnecessary. Users are finding information, and so the search engines are doing their jobs. This is covered in more depth in the How Big Are The Search Engines article, within this web site.

But unlike Yahoo, a growing number of search engines have added a "instant indexing" capability to help accommodate site owners. Directly submit a page to AltaVista, Infoseek and now HotBot, and it will be listed within minutes to two days, depending on the service, and assuming the page is not a spamming attempt.

Instant indexers immediately appease site owners. They can see that their pages are included. The pages may not be ranked well, but at least they have a presence. They can also serve to help information get posted fast. "We had memorial sites for Princess Diana that were up in a few hours," said Infoseek search product manager Sue LaChance Porter.

As Yahoo is not a search engine, it cannot add the same instant indexing service. But it competes with search engines, and so it will face even more pressure from site owners frustrated with delays of weeks, versus delays of days or less elsewhere.

Even if Yahoo does not improve the percentage of sites listed, or the speed at which sites get in, it will at least need to reform the submission procedure and keep webmasters better informed. Peer pressure demands it.

(Note: Yahoo made major changes to its submission process after this was originally written and later added an email support address. See Yahoo Opens Express Submission Service, from the March 1999 Search Engine Report, for more information.)

Every other search service but Excite offers a way to check on whether a URL is listed, and Excite is working on that. Yahoo will be the lone exception among the major search services that leaves site owners wondering.

It's worth noting that site owners are also users, and when they can't find their sites, they assume that Yahoo is not as complete as the search engines. In turn, they may surmise that Yahoo must not be as good.

"I have given up on Yahoo and exclusively use Infoseek for all my searches since I do not believe Yahoo lists all sites submitted," one survey respondent wrote.

The degree to which every site is listed may not be a fair way to measure Yahoo or any web guide, but it is nonetheless a way people will rate them.

It's uncertain whether site owners could give Yahoo a bad reputation by complaining it is incomplete. If Yahoo continues to list the prominent and "important" books on its shelves, to return to the earlier analogy, then it may stay immune to user complaints.

A recent quality survey of users of six major search engines, including Yahoo, found 96% to 97% of search engine users rating finding relevant material to be extremely or very important. Having comprehensive information was only rated 86%-90%, and having a good reputation was a low 57%-70% .

It sounds as if users are saying they don't need more, they need better. If Yahoo keeps the users happy, then criticism from site owners will be minimal. On the other hand, bad worth of mouth from any quarter is a cause of concern. And it may be that user opinion will change if they learn more about gaps or absences in listings.

Below are articles about problems submitting to Yahoo that appeared after this Search Engine Watch article was written.

Why Yahoo is Good (But May Get Worse)
Jakob Nielsen's Alertbox, Nov. 1, 1998
http://www.useit.com/alertbox/981101.html

A short but interesting look at some key strengths and weaknesses with Yahoo, ranging from the technical to the financial.

Does Yahoo Still Yahoo?
Wired, Feb. 11, 1998
http://www.wired.com/news/news/technology/story/10236.html

Nothing new here to Search Engine Watch visitors, but it is one of the first mainstream articles about problems getting listed in Yahoo.

Mining Company Illustrates Yahoo Limits
The Search Engine Report, Jan. 9, 1998
http://searchenginewatch.com/sereport/9801-miningco.html

Illustrates problems many others may face in terms of being properly categorized or described.

Is Yahoo Still Useful?
Web4Lib, Nov. 18, 1997
http://sunsite.berkeley.edu/Web4Lib/archive/9711/

Web4Lib is a mailing list that servers librarians from all over the world. One librarian expressed concerns that the service was falling behind in quality, which led to much discussion -- mostly in agreement that the service had seen better days.