Crawl, Index, Rank, Repeat: A Tactical SEO Framework (Part 3)

Part 1 of this series focused on SEO from the perspective of the search engine crawling experience, and explored the primary aspect of the crawl, index, rank methodology.

Part 2 went on to discuss the second aspect of this fundamental SEO concept, the indexation process.

This article will focus on the third aspect of the crawl, index, rank methodology: rankings.

But first, a short refresher on the assumptions put forth in the first and second articles:

Search engines crawl, index, and rank web pages. SEOs should base their tactics on these primary activities. The conclusions for SEOs working at any level boils down to these essential facts:

  • It’s all about the crawl
  • It’s all about the index
  • It’s all about the SERP

But of course, Google (and any search engine) only exists for one reason: to build a business around pleasing its users. Therefore, we must constantly remember:

  • It’s really all about the users

And also, because SEO is by nature a competitive exercise, and earning top rankings is at the expense of your competitors, we must note:

  • It’s all about the competition

Knowing this, we can initiate several tactics based on each phase of activity, which as an outcome, can help to build an overall SEO strategy. This method ensures we are pursuing SEO objectives according to the processes search engines currently employ.


Ranking is the least objective part of this equation, from the standpoint of SEOs working in the dark as to what search algorithms calculate. However, there is a way around this.

Experience is key. Those working in SEO for a long period of time begin to intuitively know which factors are propelling a site above them in SERPs, and can glance at a page (or even just its URL) and put some pretty accurate thoughts into the “whys” of its rank.

But there’s another way — one that’s even more helpful after that intuitive bit of insight is gleaned. It comes from dissecting the pages ranking above and below your property in SERPs. Doing this provides savvy SEOs with a clear view of what it will take to rank (be they external or internal links, content, site architecture factors, brand authority, etc.).

SEO all comes down to the SERP in this piece of the equation. After pulling the URLs of competitors ranking above you in search results, consider the following in your analysis and organize them into a spreadsheet.

What Causes a Page to Rank?

You’ll want data points for each of the ranking URLs (which are page-level metrics) and each of the domains (which are domain-level metrics). Both are important, but you don’t need to go overboard and consider every possible ranking factor. Just get an idea of what factors are contributing, which takes experience and a thin-slice approach.

  1. What are the pages internal and external link profiles? Now, getting all the internal link information for a specific URL on a domain you aren’t affiliated with isn’t trivial, and for these you’ll need a robust crawler. Be polite and considerate of a site and don’t attempt to hammer it with your SEO tool.For external links, make use of Yahoo Site Explorer and Linkscape. There are other ones available, but for now those are enough.
  2. As mentioned above, both page and domain links will be important, but pay extra attention to page links. Also pay close attention to the number of unique domains linking to the URL (Majestic is best for this, but tends to over-report in my experience, so take it with a grain of salt).
  3. What is the unique domain profile of these page links? The domain profile can be defined as the number, diversity, and quality of its links, crawl frequency, age, toolbar PageRank, and mozRank.
  4. What is the unique page profile of these page links? This is accounting for page-level quality as opposed to domain-level quality we identified in the step above. This is giving us more granularity on the value of the individual linking pages, as opposed to the domain’s value.
  5. Finally, you want to ask, what is the average of each of these metrics for the competitive landscape of this SERP? Where does your site fall in that spectrum? By placing each of these metrics in a column for each URL and domain, you’ll be able to sort and quickly create averages that give you an idea of where your site falls in that range.

SEO Quake is an excellent and easy tool to get a quick snapshot of key SEO metrics directly in the SERPs. Even better, the data can be easily exported into Excel. From there, a multitude of methods will allow you to visualize this data in interesting ways.

SEO Quake

SEO Quake Exportclick to enlarge


What are you left with after this analysis? The outcome will be a clear indication of what it will take to out-rank these sites in Google, and exactly where your URL sits among the competition.

The idea behind this is to “meet and beat” the sites you compete with in search results.

By undergoing a systematic analysis of SEO based on the three primary search engine activities of crawl, index, and rank, savvy SEOs can establish a framework for improving their sites that will reward them with extremely favorable rankings (provided, of course, you deserve to rank in the first place!).

You now know the way, but the hard part is doing it. So get out there and start doing it!

Related reading

Exclusive interview with Craig Campbell Golden nuggets every SEO needs to know
How to understand searcher intent to boost SEO rankings
How to master technical SEO Six areas to attack now