Winning The Paradox

January 5, 2020

The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.

— F. Scott Fitzgerald

Paradox 🔗︎

Many ideas have a binary presentation - to illustrate an idea in a compact fashion, we contrast a thesis with its antithesis. Consider:

  • Incremental delivery vs. Big-Bang/Waterfall planning
  • Process vs. Results Orientation
  • Data-driven vs. Intuitive
  • Top-Down vs. Bottoms-Up decision making
  • Autonomous Teams vs. Command-and-Control
  • Monoliths vs. Microservices

Within this basic presentation, one of these options is presented as the “generally better” choice. The logical absurdities of Waterfall planning are so easily remedied with Incremental delivery, so on and so forth. As of 2020, most software engineering organizations will likely exhibit some degree of preference for “Incremental Delivery”, “Process Orientation”, “Data-Driven, Bottoms-Up, Autonomous Decision Making”, and “Microservices”. It’s certainly plausible that software engineering, as a discipline, has a sufficient set of constraints to make a certain set of methodological practices “generally correct”.

I’d posit, however, that while presented as two opposite poles of a spectrum, many of these choices are not actually mutually exclusive, and that the visual metaphor of picking a “midpoint” in the spectrum is not the correct mental model for navigating these tradeoffs.

Losing The Paradox 🔗︎

The negative case is probably a bit easier to intuitively see.

Consider an individual who is chronically stressed about work. They may realize that better self-care is needed, and so they take some time off from their job. However, while watching TV, all they can think about is the work that’s piling up - and they end up feeling stressed anyway, while simultaneously not even making progress against their backlog. This person has chosen “Time Off” over “Work” and yet is experiencing the downsides of both simultaneously.

Another common one is the desire many organizations express to be “data driven”. High-performing companies seem to be doing it, so trend-following managers are telling their organizations that they need to be “data driven”. However, despite the money spent on data engineers and analysts, the only actual results seem to be vanity metrics that are used to confirm pre-existing intuitions, with minimal influence or improvement to actual decision making.

Winning The Paradox 🔗︎

The reality is that complex problems don’t have solutions with a single degree of freedom.

When Incremental Delivery and Waterfall are contrasted with each other, an attempt to trade off between them isn’t as simple as finding a delivery cadence that sits between the two strategies (“Incremental Waterfall”?) - having quarterly instead of annual waterfall, or monthly instead of weekly sprints, typically isn’t the degree of freedom that matters. The underlying problem being solved is: “How do we decide what to do at any given moment?”, and yet the framing of these two strategies as opposites on a spectrum causes the subtle implication that all possible solutions for planning are to be evaluated on the single dimension of “how incremental” they are.

Instead, you have to step out of the binary, single spectrum framework, and realize that each option presented to you is a nexus of different decisions, practices, and principles. Incremental Delivery prioritizes iteration, learning, failing fast. Waterfall prioritizes globally optimized efficiency, long-term goals, and planning. From your understanding of your problem, you can make a decision about the right framework to use, and if necessary, to mix and match different parts, depending on which properties are relevant to the problem you have at hand. Once you break apart the package of ideas into its constituent parts, you’ll realize that there are plenty of ways to combine aspects of these frameworks other than just decreasing the frequency of your sprints.

Many times, proponents of both poles will claim that overall effectiveness is the “true intent” of their respective strategies, which I think illustrates the underlying point: any single solution that is so general as to be capable of addressing all problems will be indistinguishable from the skill of generalized problem solving. Ultimately, you will be best served by focusing on the important dimensions of your fundamental problem, surveying the solution space to find out which frameworks actually align with your problem shape, and then thinking critically to combine them in the way that makes the most sense for you.