Many teams proudly claim their decisions are data-driven, yet often have significant blind spots in their process. The scientific appearance of data can create false confidence, leading teams to overlook crucial gaps. Data-driven processes and mindsets are tricky to establish, as data are complex at every stage – from collection to analysis to presentation.
It’s so difficult to get it right that a friend of mine, a scientific expert, makes a living dismantling opposing arguments in court. You can throw a hundred lawyers at him, but he still finds cracks in the data behind studies, products, or insurance claims to invalidate cases.
There are countless pitfalls in building data-driven decision processes, but I’ll share three of my favorite repeat offenders:
The Angle-Shy Selfie Camera: “It only photographs your good side”
The 1.5x Zoom Microscope: “Thanks, but it doesn’t show you much”
The Swiss Army Porcupine: “Good luck using it without cutting yourself”
The Angle-Shy Selfie Camera
Like an Instagrammer posting only their good side, it is easy to collect data that tell only half the story.
I once asked a software organization suffering from “too-many-tools” pollution to collect data on all the tools in use, so we could find the minimum mix that balanced developer happiness and maintainability.
After the assigned team collected the data through interviews and code repository scraping, we noticed that a lot of repositories used a particularly niche tool, and that tool ranked pretty high on the list. After digging more, it turned out that all these repositories were owned by the same team that collected the data. The data was skewed in favor of this team, not reflecting actual usage frequency or team adoption. If we hadn’t looked at the data from a different angle, we would have caused a lot of developer frustration.
You need to keep an objective mind when researching and gathering data. Often, you’ll find things you don’t like. Even so, shoot selfies of both your good and bad sides – The result is still prettier than shooting yourself in the foot.
The 1.5x Zoom Microscope
Because collecting the right data is tough, it’s tempting to gather only what’s easy to collect. It may seem cheaper at the time, but in reality, this “laziness” will cost more to manage. It’s like building a microscope that only zooms 1.5x when you need a 100x – same materials, different capabilities.
A combined top-down and bottom-up approach is a good idea here.
Top-down: Set a small number of key KPIs that truly reflect your product, business, and mission. Do you want to be world-class? Aim for one primary KPI, like Netflix's SPS. The world is too complex for you to track through proxy metrics. Avoid them and zoom into data that genuinely represents your intentions.
Bottom-up: Implement proper instrumentation for your product. Maybe your human processes, too? This is complex and time-consuming, often requiring specialized software. Integrating even existing solutions is a long-term project, not a quick fix, so be thoughtful about it. Don’t underestimate the investment required.
Would you rather spend your budget building a pretty magnifying glass or a high-end microscope?
The Swiss Army Porcupine
Imagine a dashboard displaying all your data, akin to an open Swiss Army knife. It's overwhelming and you may cut yourself. Others, especially your busy boss, who seek quick insights, might focus on irrelevant details. Best case, they don’t understand the data and ignore you. Worst case, using out-of-context information, they come up with a “brilliant” idea you'll be called upon to implement and march yourself into a black hole.
As Nassim Nicholas Taleb elegantly puts it:
“More data means more information, but it also means more false information.”
And
“More data—such as paying attention to the eye colors of the people around when crossing the street—can make you miss the big truck.”
So before you splay open the Swiss Army knife like a porcupine, think:
What story are you trying to tell?
What hypothesis are you proving with evidence?
Without clear focus, abundant data can lead to either inaction or misguided action. When everything is possible and interesting, either nothing gets done or the shiny thing gets done. So, don't sit on the porcupine. It's not comfortable.
The bottom line
Graduating from gut decisions to data-driven ones doesn’t automatically lead to better decisions. Be cautious about how you incorporate data into your process. The key is to communicate a clear, unbiased, meaningful signal to yourself first, and then help transmit an equally high-quality signal to others.
Comentarios