More data does not always bring deeper insights, and the confusion between having good information, making a good decision, and taking the best action creates more folly than foresight.
A strategy is a narrative about what you do, how you (intend to) do it, and what you expect to accomplish.
Analytics are a way to describe systematically gathered data that support our understanding of this narrative in practice.
The quality of the analytics should translate into solid support for strategic decision-making. Except it often doesn’t.
Over my career, I’ve seen only a loose connection between analytics and strategic excellence. Why? It’s all about the numbers, not the ones you might think.
Social Numbers & Complex Situations
Among the most significant challenges with good measurement is balancing the competing demands of quality, fidelity, and use. Quality metrics capture what we want in a way that is accessible and understandable. Fidelity refers to our ability to consistently measure what we wish to over time and in a manner that allows us to trust that the data gathered at different time points or situations represents the same thing. Use is about whether the data will enable us to take meaningful action based on what it tells us.
That’s a lot to ask of our data. It’s why there are professional associations for evaluation and research. It’s often why many people train for years to learn the science and the art of research.
What is often not taught in graduate school and certificate training programs is the social aspect of data. Data is a social thing — it’s not just numbers. What they mean, how they are gathered, and their representation over time is highly variable and subject to many things. For example, consider data collected on distributed or remote work policies gathered in January 2020 and what it might mean in April 2020. The entire context of work — and social life — changed between that time.
When we look at situations with high flux, the meaning-making behind our data requires more work, time, and attention. That is where many organizations find themselves. That’s where many people find themselves, too.
This is where I find some folly in analytics: too little time spent on placing data in context. That also means evaluating how well your data is fit-for-purpose: is it still measuring what you think it is, and can you trust it in supporting your strategic decision-making?
That’s only the beginning.
Measuring What Matters
Donald Campbell, a psychologist and prolific scholar of research methods, is famous for a “law” created in his name.
Campbell’s law states that the more critical a metric is in social decision-making, the more likely it is to be manipulated – and the lower the quality, fidelity, and use of the data as a result. In other words: we tend to create situations to fit our data, rather than let our data support our decision-making.
It’s the equivalent of studying for the test, not studying to learn material that the test is meant to reflect. We want kids to be good at mathematics, not good at taking math tests. We want our employees to perform at their best, not just parrot back terms and policies on a qualification exam.
Great surveys, observations, interviews, and data gathering require attention to metrics and what those metrics mean in practice. That’s why it’s not good enough to ‘throw up a survey’ online and hope you get the data you want. Even if the survey is excellent, without the social sensemaking built into the process, it’s unlikely to yield valuable anything beyond just a bunch of data.
Strategic Data Use
If you are dealing with straightforward situations, data can be simple and used simply. Are you interested in documenting the attendance for an event? Count the number of people who show up. If you’re interested in why those people showed up, why attendance was higher (or lower) than before, or why certain people showed up over others – then you are in a more complicated or complex situation.
All of this requires interpretation, sensemaking, and consideration beyond simple reports. If we’re going to make a difference in shaping our strategy — our intentions, plans, and outcomes — then we need the right data, used right. That requires these sensemaking processes and developmental design considerations built in.
What are you hiring your data to do for you? This is a designer’s question. But the answer will tell you if you’re looking for numbers for their own sake or whether you’ve created the conditions for those metrics to do much more.
Let’s chat if you want to change the narrative of your strategic plans; I can help.
Image credits: Clark Van Der Beken on Unsplash
