October 7, 2019
The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.
— F. Scott Fitzgerald
The Paradox of Data 🔗︎
Data-driven decision making has been a major business management trend over the past decade, despite its common-sensical nature. A gut reaction to an executive saying “We use facts to justify the decisions we make about millions of dollars of investment” might be “I don’t understand why you are wasting time telling me obvious things”. Almost everyone, however, can come up with examples from their lived experience where this didn’t occur - frequently, a manager who vetoed a proposal based on uninformed intuition. This “gut feeling” form of decision making has become increasingly unfashionable, however, as modern success stories, such as Optimizely’s work on the Obama2012 campaign and the vaunted ad ecosystems of Google and Facebook, have progressively disrupted and displaced companies throughout the economy through their use of data. Demand for analytics solutions, experimentation systems, and data scientists have skyrocketed in the hopes that this data alchemy can work for any company.
Despite the mindshare that it has won, it’s hard to actually define what a data-driven culture is. Most frequently, it is evangelized in terms of what it isn’t (“Managers won’t be flying by the seat of their intuition pants!”), which sets up a spectrum of contrasts: from Intuition-driven to Data-driven. Naturally, because we’ve had bad experiences with intuition-driven decision making, data-driven is seen as the superior decision making modality, which sets us up for the denouement: Some companies will take data-drivenness to an extreme, attempting to exorcise all forms of intuition, and attempting to divine corporate strategy from reams and reams of metrics. As they get lost in the noise, what was once an unalloyed good has now become a hindrance.
Winning The Paradox 🔗︎
The fundamental mistake that happens here is that we’ve set up Data-driven and Intuition-driven as exclusionary contrasts, and allowed our focus to zero in on “Data-driven” as the primary concern, when the root issue is actually Decision Making. Indeed, the inference chain we’ve forgotten is that being data-driven helps us make better decisions, and it’s making better decisions that matters. Your focus, then, should be on drawing the best of all techniques, whether it’s data, intuition, or decision pre-commitment, and melding them together in order to achieve the best possible outcomes.
We’re surrounded with mental frameworks that present these contrasts with seemingly inevitable tradeoffs. Some examples:
- Incremental delivery vs. Big-Bang/Waterfall planning
- Process vs. Results Orientation
- Top-Down vs. Bottoms-Up decision making
- Autonomous Teams vs. Command-and-Control
At the most basic level, one of the options will be generally presented as the “better” choice. In modern software organizations, I’d wager that “Incremental Delivery”, “Process Orientation”, “Bottoms-Up Decision Making”, and “Autonomous Teams” are the ends of their respective spectra that seem more appealing. This is entirely fair. Software Engineering, as a discipline, is unique enough that a prescriptive set of methodological choices is likely to be “generally correct” regardless of the organization.
Generally, though, we come to realize that these are spectra for a reason. There are advantages and disadvantages to each side, which entails making the right choice for your current context, and then being aware enough of the downsides so that you can properly mitigate them to the best of your ability.
Broader enlightenment comes when we are able to step beyond the frameworks as the binary choices that are presented to us. When we consider the question of “Incremental Delivery” vs. “Big Bang Planning”, the underlying problem that we’re concerned with is “How do we decide what we should do at any given moment?” The reality is that an excessive focus on incrementalism can result in strategic schizophrenia, but an excessive focus on future planning can result in the inability to react to a failing strategy. When you can step outside of the dichotomy you’re presented with and instead solve the underlying problem by using the full spectrum of tools at your disposal to meld a solution tailored to your context - you’ve won.