US Government Search Engine Launched

US Government Search Engine Launched

From The Search Engine Report
June 2, 1999

A new search engine that focuses on information from US government sources was opened in May. Called Gov.Search, the service is jointly produced by search engine Northern Light and the U.S. Commerce Department's National Technical Information Service through a five-year agreement.

The service is unusual for the web in that searching is not free. Those wishing to use it must pay for access, which ranges from US $15 for a day pass, $30 for a monthly pass or $250 for a year. Special pricing is also available to companies and organizations that require multiple accounts.

The idea of having to pay for using the service raised some criticism when it was initially announced, causing the fee structure to be put on hold through May. Indeed, I had an initial negative reaction to the idea that a service coproduced by the US government would be charging US taxpayers, among others, to access its information. The reality is much more positive, as I'll explain below. But first, let's have a closer look at how the service works.

Northern Light has now indexed about 4 million web pages located on more than 20,000 US government servers, which also include military and some educational sites. In addition to this information, it has also indexed about 2 million specialty records from the NTIS.

"It's probably the largest collection of government web sites that are being searched at this time, to our understanding," said Leslie Ray, director of partnership marketing at Northern Light.

The NTIS has worked with Northern Light to classify the various government web servers by agency, so that you can narrow a search to within a particular branch of government. Northern Light itself has further classified documents using its own technology into subjects such as "energy" and "health care."

Thus, one can choose to search for something like "satellites" narrowed to US military sites and the subject of "missile technology." That produces radically different results than looking for "satellites" narrowed to US agencies and the subject of "space technology." It's a degree of control simply unavailable at an ordinary search service.

Anyone who needs to search through government data on a regular basis will probably find the ability to filter their searches in this way to be useful. That's what Gov.Search is banking on -- that searchers will feel that the extra organization will save them time and thus be worth paying for access. Northern Light says that so far, the signs are positive.

"We've actually had a great customer response to the site since it launched," Susan Stearns, Northern Light's director of enterprise marketing.

Now what about the concern that the service is limiting freedom of information? The opposite is true -- the fee service is actually making free access to government information better.

That's because Northern Light itself offers the ability to do a government search at its web site. It has always offered this, but now that the company is coproducing Gov.Search, it's being paid to do a deeper and more in-depth crawl of government sites than in the past. This work is intended for Gov.Search, but the deeper information can also be accessed for free through the Northern Light site.

Without Gov.Search, Northern Light would have far less incentive to do a deeper crawl. In fact, if you do a domain restricted search to just .gov sites at Northern Light, you find it has over 2.5 million pages listed, versus about 250,000 .gov pages at AltaVista and about 625,000 at Inktomi-powered MSN Search. Thus, Gov.Search has already helped all those searching for free at Northern Light.

By the way, that 2.5 million figure for Northern Light is lower than the 4 million government pages I said were indexed earlier. But don't forget that pages from military (.mil) and some educational sites (.edu) are included in the 4 million figure. There are even some government sites that end in .com, Northern Light says.

So why bother using Gov.Search? At Northern Light, you can do a search of government sites by going to the Power Search page and choosing "Government Web Sites." That's it. You can't narrow the search to particular branches of government, as described above, nor are there as many subject filters to choose from. For the casual searcher, Gov.Search is probably overkill. But for the search professional, Gov.Search may be an important new resource in your research arsenal.

One glitch I noticed was that Gov.Search was also including pages from UK ( and Australian ( government sites, probably because they use .gov in their URLs. Similar cases may also have slipped through from other countries. These sites really shouldn't be included because Gov.Search is supposed to deal only with US government web sites (and yes, in the Northern Light comparison count above, I did take measures to help mitigate for this glitch).

Gov.Search also provides access to search those 2 million specialty records I mentioned. This is material that previously couldn't be accessed through the web. "It's mostly only been available in CD-ROM, microfiche and printed form," said Northern Light's Leslie Ray. Viewing one of these records costs extra, $1 each, in most cases. Northern Light says the NTIS already charges for this information when presented in other formats.


Northern Light

Federal fee-for-search site said suspended, May 18, 1999,4,36695,00.html

The site was never suspended -- only the fee-for-search aspect was, temporarily. While the articles say critics "basted" the new service, there's only one organization being cited -- and it has nothing against Gov.Search posted at its web site.

Gov.Search Add URL form

If you run a government web site, use this form to alert Gov.Search to your server.

Google US Government Search

Google has its own US government search service. Test queries show it to be much smaller than Northern Light's index, yielding only 10 to 50 percent of Northern Light's counts. But the relevancy of some of the matches was impressive. Definitely worth a visit.


I'm still planning a proper review of iAtlas, but if the idea of filtered searches as described above excites you, iAtlas offers this ability to do this on general web documents using company criteria. For instance, you can try that "satellite" search narrowed to banking and finance companies based in Los Angeles, using its filters.