The Precision/Exploration Continuum

First rule of analytics: you will have to make decisions with imperfect and incomplete information.

In the web analytics field, there is a constant tension between mathematical rigor and the  “eureka” moment that comes unexpectedly – the hunch that leads down a path of many more questions that need to be explored.

Despite all the (necessary and laudable) efforts placed into precise implementation of the tools, and rigorous statistical analysis of A/B testing, to succeed in this space requires the ability to make the leap to a concrete suggestion with imperfect data, versus knowing when the information you have is so incomplete as to be inadequate for the task.

In this vein, it can actually be a disadvantage in some ways to come from a very mathematically-focused education, since in the world of numbers it is possible to reach a state of perfect or near-perfect proof of something. In the real, messy, day-to-day world of business, however, many complicating factors are likely to arise, and “making all the data perfect” is unlikely to happen anywhere – there is nearly always room for improvement, or past issues that need to be accounted for. That’s the nature of placing analytics on top of live web data, which is prone to complications at the server, internet provider, browser, individual machine, site platform, back-end database, and a number of additional levels, not to mention implementations changing hands over time and human error.

To navigate this, I suggest:

  1.  Spending the first few years in rigorous study of major analytics platforms and learning the “ideal” state.
  2.  Get in a place where you can look at many different implementations to see how things play out in the real world and see different scenarios.
  3.  Throughout, follow your hunches and support them with the data that you can, but when you just “know”, use that as fodder for an A/B test or similar that allows you to fully prove out your theory.
  4.  Don’t be paralyzed by too many small details of an implementation – break the problems or suggestions into major buckets.

Of course, it may not be possible to order your career in precisely this way, but the general idea is that in order to be nimble at the speed the business is nimble, you will have to make decisions with imperfect and incomplete information. The best way to develop your analytics intuition is spending time building a “vocabulary” of scenarios in your mind so that when you encounter a new one, you have a palette to paint from.

Happy analyzing!