So, I slipped out a few years back to hang over at ClickZ for awhile (which was fun), but now I’m back at my old stomping ground here at Search Engine Watch.
Talking of years gone by, I was actually around when we first started saying hello to search engines. Names like WebCrawler, Excite, Infoseek, and Inktomi — all general search engines — spring to mind. Like most people at the time, I thought this was the start of long and lasting relationships. However, one-by-one, they disappeared or got gobbled up. Some of the older search engine brands are still around online, but they’re pretty much pale imitations of what they once were.
Once Google positioned itself in pole position in the early 2000s, it fended off competition from these search engines without breaking a sweat. And many in the industry believe that Google will remain invincible for the foreseeable future. Maybe.
This notion, of course, assumes that crawling the web and indexing and ranking web pages in the way that Google now does, will continue to be the primary data source that satisfies all of the information needs of the end user. But that assumption doesn’t take into account several trends: real-time data is rapidly becoming available; there’s a proliferation of mobile devices; and end users are becoming more sophisticated and demanding. Truth is, today’s search engines are seriously going to have to reinvent themselves.
Incorporating Twitter results into the mix along with sources of developing news are steps in the right direction (albeit tiny, tiny baby steps). But real-time data sources go vastly beyond tweets and Facebook updates. Just as the challenges of indexing the web was a major topic of interest to the research community back in the ’90s, the unique challenge of capturing and making sense of real-time data is the current research hot potato.
Consider the time it has taken for search engines such as Google to build trust models to cut through the noise, spam, and often totally non-authoritative content on the web. And then compare that to the “fire hose” of real-time data that needs to be indexed and updated immediately.
Move forward and beyond the technology of yesteryear for crawling the web and providing time-lagged results, search engines will need to acquire real-time data from partners that can inform their indexes the moment something is happening. Not data after the event.
I’m writing a lot about what I call the new “information providers” who will be responding directly to our long-term information needs in real time. For so long, search engines have been able to respond to short-term information needs, such as the answer to a crossword puzzle question or tour dates for a band. But satisfying the human need for information — without having to constantly ask for it in short-term bursts — is the new challenge.
Social networking is changing the way people find information. Researchers are already aware that real-time search can serve needs other than those of traditional search engines.
There’s a noticeable shift in the way people use the web. Standard web search means: click on a search result, visit a page, click on another link, visit another page. But now people spend more time receiving than searching. Much time is spent monitoring streams of data such as tweets, updates, headlines, and blog feeds.
And geo-location data, such as that from handheld and other mobile devices, provide a huge signal of relevancy when attached to a spike in the use of word or phrase in a message stream. For more than a decade, search engine optimizers have focused their attention on the activity of mindless bots scanning the web for information. But real-time search is so different from what came before it; we’re about to see a huge and transformational break away from what’s been known as general-purpose web search.
We could well be on our way to saying, “Goodbye, search engine” and “Welcome, real-time information provider.” Although someone will think of a much cooler name for the latter.