Developing software is a game of decision making under conditions of uncertainty.

Read it again, please: 

Developing software is a game of decision making under conditions of uncertainty.

Once you get this fact into your head, embrace it, and act upon it, I believe that you and your organization will perform much better. You have my word. 
This is really the fundamental challenge we are presented with. This is where part of the complexity comes from.
As a part of an organization developing products, every day you find yourself forced to make decisions in situations with incomplete information and uncertainty.

Let's compare this complexity to the game of chess: arguably considered the most difficult game out there. To win in chess, you need to analyze many levels deep, you need to consider many options regarding what will your opponent do in response to a specific move, since she is also a good player, you need to guess what will she think that you will do... And so on. 

The best players in the world, in addition to being knowledgeable can "see" deeper than others. 

Quick comparison to chess

The problems we are facing are similar in some ways and different in others:

  • Just like in chess many things are unknowns, but given enough time and budget we can find out. 
    For example : How long developing feature X will take, we can develop it and then use this information to estimate a feature.
  • Unlike chess, variability is huge - In Chess the board is right in front of us, the pieces remain the same, the rules do not change. How about organization and the market? Does it also comply with these constraints. Unfortunately, no.
  • Unlike chess - We do not get to see what our competition is doing. Not that it is easy to understand what your chess opponent will do, but it becomes extremely difficult when you do not have access to this information, just imagine playing chess with your opponent's pieces hidden...
    Even more than that, in the business world there are times when you are not aware who is your opponent, even worse, most likely you are against multiple invisible opponents.

Making our decisions easier

Once you understand the problem, you realize that the problem is a really complex one. 
You’re playing chess against multiple invisible players at once, with infinite types of pieces and ever changing winning conditions. That’s hard! 
So every decision you make needs to be focused on improving future decision making, gather information about whatever you can that will decrease (but never eliminate) uncertainty.
The more information you have, the less uncertainty you’ll have. And the more you do it, the better you get at it (that's called practice), and perhaps eventually you will be able to shift some of your decisions from being in the complex realm to the complicated one. (This is what the cynefin framework is about)

What can we do?

I believe that agile development is aimed at dealing with these pains exactly.
It helps us reduce uncertainty and improve decision making by reducing the feedback loop, and enable frequent data collection.
It provides guidance to create balance when choosing from the available "plays"

  • Decide something and just do it while ignoring "constraints" - Some of the greatest innovations were created this way, some of the biggest known failures also fall into this category. 
    Agile practices promote YAGNI. We try to avoid investing too much time in activities that are considered to have low ROI, stuff like estimation, long term planning etc. So, we "just do it", we estimate using quick methods that avoid diving into the details, we tend to try and plan in details for more than one iteration, we avoid doing deep requirement analysis before the development starts, and prefer that requirements will emerge during development.
  • Gathering data - Most agile practices are focused at that, trying to create transparency and clarity using time-boxes and visualizing information. The more data we have, the easier (some of) our decisions become.
  • Stop playing - There are situations in which the data (most often ignored) indicates that we are about to loose the game, while it is not always easy, it is sometimes a good decision to fold our hand.

 

May the force be with you.