+44 (0) 20 7846 0140
Contact us
Nov 23, 2018| Simon Copsey
EngineerBetter was asked to engage with a major utility company. Their software delivery operation, comprising approximately 600 developers, was underpinned by a single platform team that was responsible for deployments and for moving the organisation towards a continuous delivery mode of operation.
The platform team was dysfunctional, preventing deployments (including critical security fixes) from going live. This was receiving understandable attention from senior executives.
Simply observing the team during meetings revealed a number of symptoms:
From an outsider’s perspective, it could be tempting to assume that the team is just a bad team, and that replacing them may solve the problem. However, as we now know, reacting to the observable symptoms alone is dangerous.
We must push back the fog of war to understand the underlying problem that is driving these symptoms that become observable as behaviours of individuals.
EngineerBetter spent the first week observing and interviewing each member of the team. This allowed us to understand all the symptoms and how they interrelated.
The symptoms and their interrelation evolved over a long period of time, so was particularly difficult for people within the client organisation to identify for themselves. It became clear each symptom traced back to the same underlying root cause, a cause that initially seemed distant and unrelated to the team.
In this case, the utility company had hired an outsourcing company to write their software, for which the platform team then needed to automate the deployment. The contract with the outsourcer was defined in terms of a fixed scope, fixed cost and fixed deadlines - meaning the outsourcer could only vary quality when faced with implementation delays.
By the time the software was delivered to the platform team for deployment automation, the low quality precluded it from being automated, meaning the platform team were having to perform every deployment manually.
The platform team comprised talented engineers, who wanted to automate - not perform manual operations work - and this was one of the biggest reasons behind the team’s frustration, which bubbled up as the undesirable (though understandable) behaviours. This was no fault of the individuals, but a representation of the pressures they were facing.
“94% of the problems in business are system-driven, and only 6% are people-driven” - Deming
The root cause of incentivising the outsourcer to cut corners was also causing undesirable effects in other teams, such as a significant accumulation of bugs that needed to be fixed by a separate application maintenance team. If left untreated, this root cause would also prevent management from achieving their strategic goal of reducing annual servicing costs per customer, as this could only be achieved if continuous deployment was realised.
We were able to share this story with our client executive in a digestible form using a Current Reality Tree visualisation (a Theory of Constraints tool) along with a proposal for treating the underlying root cause, and providing temporary relief to the platform team.
Whilst many people we’d encountered in the customer organisation were aware of the symptoms, it had not been clear until that moment how the symptoms interrelated and each stemmed from the same single root cause.
Today’s organisations face many challenges to survival, the most interesting of which is also faced by the military. Military leaders use the term ‘fog of war’ to represent the constant uncertainty - or incomplete knowledge - that they face in decision making.
As military leaders are divorced from the frontline out of necessity, their direct understanding of the battlefield - the enemy, the conditions, and even of the capabilities of their own army - becomes limited in comparison to those soldiers who are embedded in the frontline. Such uncertainty adds complexity to centralised decision-making and makes it difficult for military leaders to issue orders with confidence.
This is similar to senior executives in a hierarchical organisation who must focus on long-term strategy. It is not feasible for such executives to also maintain a deep understanding of the day-to-day operations of their organisation - how work is actually being done.
As executives make decisions in an attempt to solve issues, they will often not realise the limit to their understanding of the issue and that a fog of war even exists. This not only makes it easy for them to focus on the wrong issues, but to also mobilise the wrong solutions that may actually perpetuate the true underlying problem that they are yet to uncover.
It is certainly not management’s fault: they’ve been tasked with trying to perform heart surgery from a huge distance, armed with only a magnifying glass and crutches.
The fog of war is more dominant in those large organisations which are intrinsically more complex, where no single manager can truly understand more than a small part of the whole. As Lean Enterprise suggests:
“In such situations, each manager will tend to optimise for what is visible to them and for the feedback they get, which is more or less determined by which people they interact with on a day-to-day basis. Thus each department or division optimizes for its own benefit — not because people are stupid or evil - but because they simply have insufficient visibility into the effects of their actions on the wider organization.”
It is a painful irony that it is the largest organisations today that need the most help, but no one person understands how they actually work and, therefore, how to make them better. It is a frequent pattern that consultants are called in to assist, rolling out large change programmes that are plagued with the assumptions of senior executives.
Systems Thinking is an approach for pushing back the fog of war** by assembling pieces of the puzzle. It allows individuals to understand the organisation as a whole so they can identify and focus on the biggest problem first, and mobilise a solution that will have the intended effects. To learn more about Systems Thinking, start with Thinking in Systems: A Primer, by Donella Meadows.
