Winning The Paradox

January 5, 2020

The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.

— F. Scott Fitzgerald

Paradox 🔗︎

Human nature seems to dictate that we think in binary fashion. We love to contrast our ideas with their antithesis - a spectrum with two poles in conflict. Any number of frameworks can be cast in this light:

  • Incremental delivery vs. Big-Bang/Waterfall planning
  • Process vs. Results Orientation
  • Data-driven vs. Intuitive
  • Top-Down vs. Bottoms-Up decision making
  • Autonomous Teams vs. Command-and-Control
  • Monoliths vs. Microservices

At the most basic level, we think of one of these options as the “better” choice. In software organizations as of 2020, you’ll probably see more respect go to “Incremental Delivery”, “Process Orientation”, “Data-Driven, Bottoms-Up, Autonomous Decision Making”, and “Microservices”. This is actually a reasonably fair thing to do - software engineering, as a discipline, has a unique enough set of characteristics that we can probably make fairly broad methodological choices and be “generally correct”.

At the next level of thinking, we come to better appreciate that these are spectra - there are advantages and disadvantages to each side, and we try to choose an approach most appropriate for our current context with enough awareness of the downsides so that we can attempt to mitigate the most consequential cons.

Losing The Paradox 🔗︎

When we think about our problems with these simplified frameworks, we often fail to truly assess what our root problems are. It’s easy to fall victim to fads and to implement action that doesn’t truly address our needs. Losing the paradox occurs when you decide to “shift to the other end” of some spectrum, only to find that this is harder than simply saying “make it so” and you end up realizing none of the benefits while materializing all of the disadvantages.

A common one is the desire for organizations to be “data driven” - it’s trendy, high-performing companies seem to be doing it, so managers are telling their organizations that they need to be “data driven”. If you don’t fundamentally realize, however, that this is about making sure your decision making accurately reflects “The Truth” and assessing where your organization currently has issues with this, then all that’s going to happen is you’re going to spend a bunch of money on analysts and they’ll generate a bunch of vanity metrics that don’t actually influence the decision making that occurs.

This happens with individuals too - you might be chronically stressed about work and realize that you need to engage in better self-care. However, if you step away and keep thinking about all the work that’s piling up - you’re in a place where you’re still feeling stressed and you aren’t making progress against your backlog.

Winning The Paradox 🔗︎

Broader enlightenment comes when we step beyond the choices that are presented to us. Winning the Paradox requires understanding that complex problems have more than a single dimension in their solution space, and the contrasting options are, in many cases, not contrasting at all if you look at them with a different perspective.

Once you’ve understood the broader solution space, and not merely a one-dimensional projection of the solution space drawn between two arbitrary options, you have infinitely more freedom to design the exact solution that best solves your problems.

Let’s consider the question of “Incremental Delivery” vs. “Waterfall Planning”, which are frequently presented as contrasts. The underlying question being asked is: “How do we decide what to do at any given moment?”, and what this contrast implicitly states is: all possible solutions for planning can be evaluated on a single dimension based on “how incremental” they are. This is obviously ludicrous, but framing the problem in this fashion induces this thought pattern implicitly, and many people fall victim to this.

In many situations, a combination of both is extremely effective - having clear long-term goals and a strategic structure, coupled with implementation flexibility, small delivery increments, and a willingness to update the big picture based on new data. Proponents of both strategies will likely claim that this is the “true intent” of their respective strategies - and I believe both of them, which is arguably the true proof of my point: many of these spectra are false projections of the solution space into a single dimension, and you are best served by remembering your fundamental problem (“How should we effectively decide what to do?”) and surveying the full solution space to find the right option.

2020/01/12 The Paradox of Data paradox business