Most major search engines rely on an army of hundreds of people to create and maintain their services. Not Gigablast -- it's a high-quality search engine built and operated by sole proprietor Matt Wells.
Yes, I said sole proprietor. Gigablast is entirely Matt's work. From coding to crawling to marketing it's all Matt Wells.
Recent weeks have brought about a great deal of development at Gigablast. In August, the engine began offering access to Adobe PDF and Microsoft Word, Powerpoint and Excel content. Recently, Gigablast began offering Boolean search capabilities.
Q. Matt, can you share with us a few career highlights and a brief bio? Also so we can get up to speed with Gigablast can you share some basic statistics about the database.
Around 1988, when in high school, I designed and wrote a graphic adventure game called "Revenge of the Germs" for Radio Shack's Color Computer 2. I sold about 20 copies through Rainbow magazine. In college I developed the Artists' Den web site, a searchable, open database that allowed artists to freely add pictures and descriptions of their artwork. This project led to my employment at Infoseek where I developed core search technologies for their world-famous search engine. After leaving Infoseek in August of 2000 I began working on Gigablast as an independent project.
Gigablast currently serves about half a million queries per day now, mostly to external sites. The index size is almost 200 million pages and changing daily.
Q. When did you start Gigablast? Did you write the software code yourself?
I wrote it all from scratch in C++. It has been almost 3 years since its inception. It uses zlib for doing compression and it uses a plotting library to make administrative graphs, but other than that, I custom coded everything.
Q. Did you see a whole in the web search market that Gigablast could fill? In other words what does Gigablast do that the other engines don't. Why should a searcher use it?
The hole that I saw was a performance related hole, not a search results quality hole. I found a way to scale search more efficiently than everybody else. Theoretically, Gigablast can get by with ten times - even a hundred times - less hardware than other engines and achieve the same performance.
But as far as what is different from the user's perspective, I think Gigablast's scoring algorithms give less emphasis to links as other engines. I did this on purpose so new sites are not at as much a disadvantage as more established sites that have a large number of incoming links. This is why I chose the catch-phrase "Search the web from a different angle" to be displayed on the front page.
Another major difference is that Gigablast is the only large engine, to my knowledge, to ever do continuous updating and refreshing of the index in real-time. You can also add and update your URLs in real-time, too. I think Gigablast is the only large engine right now that allows that. Everybody else charges money and calls it "paid inclusion."
Q. Running a search engine in the period where many people think Google is IT must be difficult. How are you handling competing with Google and other large engines? On your site you mention that a portion of traffic comes from feeds from other sites - is this a big portion of your traffic?
I've been steadily improving Gigablast's relevance. Yes, it is sometimes a bummer always working in the shadow of Google, so it really makes my day when somebody tells me Gigablast gives better results. I think for a lot of queries it does, not for all of course, but those instances are almost always because of my hardware budget.
I have $$8,000 of hardware and Google has maybe around $50 million. Go figure. Since I don't have the money or resources to compete with Google I more or less rely on the dropped crumbs. I rely on the difference in my scoring algorithms, my real-time indexing features and my dirt-bottom pricing structures to differentiate Gigablast and it's products from Google. Yes, the large majority of the queries I serve come from the search feeds I supply to clients.
I haven't run into any problems yet. I've gotten some emails from someone claiming to work for the KGB telling me I need to remove a particular page from the index because it might endanger somebody. In those seemingly urgent cases I try to remove the page right away, but I'm also aware that it could be a competitor of the page being removed. The large majority of the time, however, everybody is being honest. Yes, Gigablast does follow robots.txt.
Q. From a business perspective, Gigablast carries no advertising? Is this a decision you plan to keep? How does Gigablast make money?
Money is derived from selling search services on my products page. At this point I don't think I'll put up advertisements unless I need the revenue to support Gigablast or myself.
Q. Several months ago you began a Scandinavian site that does carry advertising from e-Spotting. What is your relationship with BOP Interactive the Scandinavian company that you work with?
We have a contract with each other. I can't get into too much detail about it.
Q. As someone with operational experience and programming skills, would you care to comment about what's wrong with web search today?
There's an incredible amount of room for web search improvement today. Search is just beginning. In a few more years I think search has the potential to displace operating systems as the most complicated program space in the market. There's still a good amount of innovation in the operating system space, but it is pretty well tilled soil. I think the search sector is just beginning and has much more room to grow than the operating system sector. An operating system will allow you to write a report, the search engine of the future will write the report for you.
Q. What are your longtime goals for Gigablast? Do you see it more of a testbed for new ideas that you'll sell to others or market as enterprise search technology?
Once I finish my spell checker and some other things I really want to continue my work on some new and experimental algorithms that bring a fresh perspective to search. That is something that really interests me. My to do list is literally a quarter of a megabyte.
To keep current with the latest changes and updates at Gigablast, Matt's tech blog is where he posts the latest goings on with the engine.
NOTE: Article links often change. In case of a bad link, use the publication's search facility, which most have, and search for the headline.
Optimising Digital Marketing Campaigns with Search, Social and Analytics
At SES London (9-11 Feb) you'll get an overview of the latest tools, tips, and tactics in Paid, Owned, Earned, Integrated Media and Business Intelligence to streamline your marketing campaigns in 2015. Register by 31 October to take advantage of Early Bird Rates.