First Principles in Search
First principles form a core set of working assumptions, hypotheses, and key beliefs about a single area of interest.
First principles form a core set of working assumptions, hypotheses, and key beliefs about a single area of interest.
In the coming months, I’ve been invited to write on a wide variety of search-related topics for Search Engine Watch. Since this is my maiden contact with this audience, I thought it important to take a few steps back, to hover at 100,000 feet, and introduce the approach that has best served me.
I’d like to share some things I’ve learned over the past seven years of thinking about search, and working in the field, whether as a research director at Jupiter covering search technology, as founder and CTO of Open List (a vertical search platform), or in my current capacity as the lead search architect for Marchex, which acquired Open List last year.
My approach is blood simple: Let all dialogue, all thinking, and all subsequent labor derive from a set of first principles. The tactic has guided all of my work during the past seven years and my goal is to let this approach inform the dialogue I intend to start here as well.
What are first principles? I’ve adopted a relatively loose working definition: first principles form a core set of working assumptions, hypotheses, and key beliefs about a single area of interest. I evaluate the bulk of my thinking and work in their light, and if need be, those first principles can evolve over time. Having a set of fundamental precepts has always helped me keep my focus, understand the broader picture in the face of myriad detail, and kept me honest about what work really needed to be done.
Before we get any more abstract, let me share a few concrete examples of what this kind of thinking can mean to those of us who share a vested interest in the world of search.
When we founded Open List, (formerly Local-i), we held a first principle that stated that the final arbiter of relevance in vertical search was the user’s evolving context, rather than a single algorithm applied once to an atomic query. The implications for our product were clear: give users unlimited power over result sets, and allow them to drill down to the most meaningful results.
This was in early 2004, when only a handful of folks, notably Endeca on the enterprise search front, were trying to promote the benefits of faceted classification (“guided navigation”). Deep refinement became a hallmark of the product, and we built the technology and infrastructure required to support it at scale.
Not long after launching the prototype, we were encouraged by the data that showed visitors were in fact digging deep into result sets, and this and other positive feedback reinforced the validity of our first principle. The product attributes that arose from this assumption satisfied another key precept, which no doubt will merit more discussion here in the future: search results are the first, but rarely the last, step in the discovery process.
At Marchex, we’re using first principles to help guide the development of new products, both in terms of what we develop, and of where and how we apply focus. A few examples of first principles we’re using now:
All of us, whether marketers, business development folks, or technologists, operate every day at the level of the detail: requirements documents, feature sets, code and site releases, detailed tactics, marketing plans, conversion metrics, etc. But when we begin our work having taken the time to set down first principles (even if they end up being wrong), our labor is grounded in and guided by a broader view of what’s important, and why.
Let’s consider for a moment how thinking about first principles can inform our understanding of the industry as it so quickly evolves. In the past few months, there has been a great deal of commentary on Google’s decision to enable personalized search history by default for logged-in users, and the likelihood that in the future, all search results might be personalized from the outset.
What’s at stake, many argue, is the future of personalized search, and the extent to which it will supplant current models of relevance (and by extension, the implications for SEO, paid search targeting, etc.). Most of the commentary quickly boils down to two positions: pro or con, and then the implications for the businesses we operate, such as changes in SEO strategy, for example.
But to understand where we could be headed, it’s essential to see this turn as a possible shift in first principles taking place at the major engines. The single axiom shared by today’s large search engines, as detailed in Brin and Page’s landmark paper, was itself a shift away from a previously ascendant first principle in Web search relevance: beyond individual publishers of Web sites as a key source of relevancy (remember keyword stuffing and AltaVista?), and toward the community of Web site publishers and the compounded interest of intra-site link structure (Page Rank, and it’s seamy underside, the “Bomb”; or the concept of the community and hub exploited by Teoma, etc.). In the world of search technology, this change also implied a shift away from statistical algorithms and towards the study of graph structure, but that’s another story.
Seen in this light, personalized results represent a turning-upside-down of the current locus of relevance: away from publishers and communities of publishers, towards the consumers of Web sites. What we’re really looking at is a shift in the balance of power, in which SEO, which is by definition one-half publisher finesse and one-half community endeavor (“optimize your site”, then “build your page rank”), stands to lose traction.
The possibility of pervasive result personalization reflects a far deeper evolution of the social-semantic power structure: the rise, ironically, of the real “Pigeon Rank,” with Web searchers playing the role of ever more accurate birds. The issue, I’d argue, is really all about what happens when our industry itself modifies its own first principles, and what implications follow from those changes.
If we let go of the detail – in this case the potential, tactical ramifications of personalized results – and instead see the issue in the broader history of our field and how its values and key assumptions have evolved over the past decade, we begin to grasp what’s essential: how models of relevance are based on an understanding of the locus of semantic value (whether in the publisher, among the community, or in the hands of the searcher), and the power struggles that arise when those assumptions shift.
What took me almost three years to learn while at Jupiter was at its core pretty simple: spin the tiny tactical questions until you see their antecedent, more difficult ones, and then lead your inquiry from there. In that spirit, I believe that when you undertake debate about the culture, the technology, the business, or the philosophy of search based on an explicit rendering of larger questions, you go from merely commenting on and reacting to what’s going on in our field – and it comes at you fast, as we all know – to playing an active role in shaping its future.
In the coming weeks and months, we’ll look together at specific topics, in detail, from this perspective. We’ll talk about the role of habit and interface in shaping relevancy, at the reasons why search technology may have its greatest potential creating new content, at how and why human-aided approaches to enriching search are always cast as irresolvable opposites to the priesthood of the pure algorithm, and at the death of the rank and the birth of the set, among other worthy notions.
As we keep watch on the field together, let’s also try to spend our time seeing beyond the daily grind, wrapping our heads around the broader issues driving the daily news and agendas, and thereby to take a collective stab at influencing where we wind up.
Matthew Berk is Lead Search Architect for Marchex. He has previously been a research director at Jupiter covering search technology, and the founder and CTO of vertical search platform Open List, which was acquired by Marchex in May 2006.
We report the top search marketing news daily at the Search Engine Watch Blog. You’ll find more news from around the Web below.