Are all Results on Search Engines Equal? A Surprising Journey Within the SERPs [Best of SEW 2010 #7”

Editor’s note: As 2010 winds down, we’re celebrating the Best of 2010, our top 10 most popular columns of the year on Search Engine Watch, as determined by our readers. Every day over the next two weeks, we’ll repost the most popular columns of the year, starting at No. 10 and counting down to No. 1 on Dec. 31. Our countdown continues today with our No. 7 column, which originally was published on August 31. Enjoy!

Editor’s Note: With the recent Yahoo-Bing merger, search engine optimization (SEO), paid search, and website conversion experts have seen the big three search engines become a two horse race in 2010. Google and Bing are the dominant players, yet it might be a mistake to think that their position is uncontested. In fact, as Google has said themselves, “competition is just one click away” and Forrester research in 2008 reported that users are in a permanent state of readiness to switch.

Pressure on these two behemoths is coming from niche players and vertical search engines. One could argue that the search space is becoming fragmented again — much like it was before 2002 when Google hit a home run with the launch of Google AdWords. Paid search has become the winning business model for startups and established companies, and 2010 is seeing the emergence of new bidding-based ad platforms that are quick to integrate with automated bid management platforms and start stealing advertising spend.

In July, Biz Stone said at the Chirp conference that Twitter was handling 800 million search queries every day. Other real-time players, such as OneRiot, Collecta, and Wowd, are establishing themselves too, with completely different interfaces, methodologies, and aims as to what they can offer users via the real-time vertical. Significantly both Twitter and OneRiot have launched paid advertising products based on real-time search trends, of which, the market changing potential of this development is underscored by Google’s own recent development of their Google Realtime algorithm.

Furthermore, Google and Bing’s dominance of “traditional” or “archival” web search is now being contested by foreign language search engines Yandex and Baidu. Both have made moves toward internationalization; Yandex now offers an English language index and Baidu has opened offices in Silicon Valley. Arguably these companies can provide a genuine threat to the status quo if they play their cards right.

Search marketing tactics in general work because of implicit behavioral tendencies among users of search engines. Users tend to focus on the top results. However, real-time search has introduced new experiences to the standard search interface, such as moving results and trending topics — are these new interfaces and result types causing a shift in user behavior?

With this in mind, Search Engine Watch wanted to know whether all results are created equal. Do these new players have a strong chance to monetize and offer true business value to their advertisers?

We teamed up with Dominik Johnson’s eye-tracking study team at Explido WebMarketing to conduct an investigation into this fragmented world of cat and mouse. We asked them to look into some questions for us and what follows is details of the experiment, their methodology, and the results.

At this point Search Engine Watch would like to give a big thank you to @dominik_johnson @vebee @mobileMat and @conversion_doc who all donated their time in support of the project. Your next beer is on me.

Dominik, over to you — Jonathan Allen, Director of Search Engine Watch

* * *

The Aim of the Experiment

My team at Explido WebMarketing and I tried to find an answer to the following questions:

  • Are all results equal?
  • Most engines have the same basic layout; however does a pixel change here and there make a difference?
  • Are universal search type results on Bing and other engines as attractive as Google?
  • Are trending topics a natural way to navigate the new breed of search engines?
  • Do Baidu and Yandex have a design edge?
  • Are moving/scrolling elements on the page effective?

We compared the following search engines:


How we analyzed the SERPs:

  • Three OS (Microsoft, Apple, Linux)
  • Four browsers (Internet Explorer, Firefox, Safari, and Chrome)
  • Keyword phrase: [iphone 4”

We deleted all cookies, and the browsers had no history or plug-ins installed. After each search query we deleted all information and started the process over again from scratch. We wanted to make sure we were conducting a fair test, with similar search results, rather than generating a snapshot, so we conducted 10 queries within every single search engine over 20 days.

The Method

At Explido, we use EyeQuant for our analysis. It is optimized to predict the initial gaze of healthy human viewers on a variety of screen sizes — what matters in the end is the image on the retina.

Regarding the task-dependency of visual attention, as one might expect, different tasks produce different gaze paths. However, the differences become most apparent after the initial scan of a site, where once familiar with the site design and layout, the vast majority of users exhibit a similar browsing pattern.

In general, model-based attention analytics can’t replace a full usability study, but it certainly provides useful hints on how to optimize that all-important “first impression” of a website in the results.

The screenshots of each single search engine include a “heatmap analysis,” “coverage analysis,” and the important “region of interest analysis” to somehow compare the different design types. By way of a simple explanation:

  • The heatmap shows the user’s eye movement, which elements are strong, and which are weak within seconds.
  • Coverage analysis is almost the same, but simplifies and illustrates where the main focus is within this short period.
  • Region of interest is quite bit different. In principle, the website generates 100 percent attention; however, there are elements stronger than 100 percent (e.g., a red button [140 percent” in comparison to a text field [40 percent”). So with this analysis we’re able to fairly compare website design factors.

So, with the explanation of methodology out of the way, let’s look at the results!


Click on images for a larger, more detailed view:











There is a huge difference between those search engines!

Via the region of interest analysis, we tried to find similar areas that are important for online marketing, in-house, and agency teams that try to achieve the best results via SEM, SEO, affiliation, and social media marketing. The diagrams show the strength and weakness of each in direct comparison to the others.

Our screenshots have a resolution of 1024×768 pixels and the whole resolution (SERP that the user notices on its screen) had on principle a perception of 100 percent. Some regions of each SERP that we analyzed go from strength to strength while others don’t.

For example, Google owns the most effective sponsored link area when compared to all others. The regions of interest are all equal, but of course based on different parts of a website, which made it much harder to answer all those questions mentioned earlier in the post.

This is what the direct comparison looks like. All areas (Logo, Search results, Searchbox and Button, and Adbox) we will analyze separately, too. We saw some points within our analysis that are strong or weak within a search engine, which is quite interesting.

Logo — category winner is Collecta:


Search results — category winner is Google:


Searchbox & Button — category winner is Collecta:


Adbox — category winner is Google:


There are many reasons for the above result. Primarily, we found universal search was a strong design factor. The other reasons, for example, are that Twitter still has no Adbox or Wolfram|Alpha wasn’t a big help if you searched for [iPhone 4”.


Some thoughts and differences between the search engines:

  • Baidu hasn’t included any pics or videos in their search results for the [iphone 4” query.
  • Bing has video results with images, related search phrases, and sharing options.
  • Collecta has an interesting design for the second result with details within the SERP, little icons within the results, and sharing options.
  • Google offers search options with filters. News results below the first results, and also image results below the first results.
  • OneRiot has images within the results with sharing options.
  • Twitter results comes up with user-icons (high trust), tweet frequency by the user, and trends on the right. Sharing options and time stamp are also good features.
  • Wolfram|Alpha has a good “description” to use.
  • Wowd has a time-stamp, sharing options, and info about the new incoming results.
  • Yahoo implemented news results on top, time stamp, search options with filters, and scattered images within the search results.
  • Yandex has little numbered icons within the results.


All results definitely aren’t equal.

  • Most engines don’t have the same basic layout, but they watch what the competitors are doing and implement some features which seems to make sense to the users (see “Google’s New SERP: Hit Or Miss?“).
  • The universal search type results on Bing and Yahoo are as attractive as Google.
  • Are trending topics a natural way to navigate a new breed of search engine? This is a hard question to answer but it looks like that trending topics are of growing importance for the search engines, especially because of mobile devices. Search-centric will be a huge topic in coming years, including voice, language, and geo-local services. SEOs will be quite busy.
  • Do Baidu and Yandex have any design edge? Well, let’s put it that way: Baidu and Yandex still model themselves on the good “old” Google design but we’ll likely see changes from them in the near future.
  • Are moving/scrolling elements on the page effective? Yes they are. Users are using them and these elements act like the search button!

Optimization is getting more complicated every day, especially on the international level, and it won’t get any easier within the next few years because of so many variations of search behavior by the users. It will be also interesting to see how the Facebook search will work out considering the social media trust factor by friends.

There isn’t one simply way to a successful search engine design and to generate user satisfaction. Despite this, the larger search engines still look similar. What are your thoughts on that?

We know there are a lot of experts worldwide, and these questions can be answered all by the Search Engine Watch readers. We would love to read your ideas and your personal findings within the comments, there’s a lot to discuss!

Related reading search tools updates competitive analysis
Decommissioning Jet Two charts proving Walmart planned to ground Jet all along
trint makes spoken words videos crawlable