Written by The Age of Enlightenment was a wave of humanist and scientific thought that spread across Europe during the 18th century, producing great advancements in many fields, particularly in science.
In recent years software development appears to have experienced its own age of enlightenment with the spread of ‘Agile’ methods taking more adaptive and iterative approaches, requiring continuous learning among its practitioners. But if that’s true, why does so much of the software and technology that touches our lives still seem to be problematic?
A brief background to Agile
My earliest experience of software development was during my electronic engineering degree in the early 1990s. We were introduced to the programming language Modula-2 to teach us problem-solving through the use of computers. The first chapter of the textbook laid out a method based on six steps:
- Define the problem
- Design a solution
- Refine the solution
- Develop a testing strategy
- Code and test the program
- Complete the documentation
This linear approach is also present in many of the formal or plan-driven approaches to software development that were in common use at that time (e.g. Waterfall or V-model). These methods are sequential: each stage must be completed before progressing to the next.
At that time computers were spreading fast through business, and the increasing pace and need for change triggered a crisis in the software industry. An ‘application delivery lag’ of up to three years between business requirement and working software was not unusual. Increasingly frustrated with the long development times, a group of like-minded software professionals came together in 2000 ‘looking for something more timely and responsive’.
Several emerging concepts and methodologies – iterative development (drawing on Deming and Boehm), the alignment of concurrent activity (influenced by Takeuchi and Nonaka’s 1986 Harvard Business Review article, ‘The New New Product Development Game’) and self-organising teams (as in Sutherland and Schwaber’s ‘scrum’ software development process) – now started to be drawn together. The result was a manifesto drawn up by the group which set out new principles of effective software development in contrast to the sequential and plan-driven approaches.
The document was named ‘The Agile Manifesto’. Agile was not intended to be a methodology but rather a mindset, within which a variety of methods and practices co-exist (SCRUM, Kanban, Scrumban, DSDM, BDD/TDD and XP among others).
A number of organisations ascribe their success to Agile. One of the best-known and most influential is Spotify, which so far has managed to keep ahead of much bigger competitors such as Google and Apple. Agile also has critics, however, who point to high-profile failures such as the Department of Work and Pensions’ Universal Credit and less than overwhelming results from the UK Government Digital Service. So what is causing some organisations to succeed and others to fail?
Common traps associated with Agile
The tools trap
When Japanese car manufacturers such as Toyota began to gain global market share at the expense of western competitors, industry experts queued up to visit Japanese production plants to learn the secrets of their success. What they observed was widespread use of technology (robotics, Andon cords), techniques (Kaizen, Kanban, just-in-time) and training (skills such as problem-solving and Six-Sigma analysis). Much of this learning crystallised into practices that fall under the ’Total Quality Management’ and ‘lean’ banners, which have been deployed not only by vehicle manufacturers but also in a wide variety of sectors and organisations. Yet outcomes have often been disappointing or even detrimental. The problem is that it was not the tools but rather the principles and thinking that were key to Toyota’s success, and these are completely missed when the improvement techniques are used ‘out of the box’.
The thinking trap
In the same way, with no understanding of the thinking behind Agile practices, the focus of teams remains fixed in conventional thinking, so Agile becomes reduced to cards and sticky notes on walls with teams having meetings standing up. In one large financial organisation I worked with, over 400 personnel had been through Agile training, yet a year later not one piece of work had been delivered in an Agile manner.
Even when teams learn to apply Agile principles in practice, they often run up against traditional management philosophies and structures based on the idea of the organisation as a machine to be controlled in all its components (including those completing the work), which is entirely at odds with Agile thinking.
The structure trap
At its heart, software development is a series of interactions and learning processes between knowledge workers that sits uncomfortably with the way projects and changes are handled in organisations. Traditional managers are unsettled by the lack of fixed plan or demands for cross-functional personnel for unpredictable periods, interfering with corporate resource planning and productivity reporting. They struggle with perceived lack of structure that leaves them feeling delivery is not in their control. In some organisations this results in an ‘Agile sandwich’, with delivery teams using Agile methods to deliver changes that are then subject to weeks or months of delay to fit in with traditional planning and release schedules. No matter how Agile the code developers, they are always at the mercy of those working to outdated industrial-age management dogma.
The ‘wrong problems’ trap
The final and most destructive trap is to have teams working Agilely on the wrong problems. In their article on improving software project management, Brendan O’Donovan and Peter Middleton show how software projects are often subject to highly unsystematic and partial selection criteria, carrying the danger that the organisation is merely getting faster at delivering the wrong thing, expensively replicating poor existing processes in ‘IT concrete’. The key to effective change lies in understanding service capability as a system and from the users’ perspective.
In a large financial organisation I worked with, an application development team was working to address a backlog of errors and bugs that were affecting an automated financial platform. Although the organisation was in the process of implementing Agile, the leader decided he first needed to better understand what was really happening in his organisation, using the Vanguard Method to do so (although to be clear the Vanguard Method is not one of Agile’s many flavours).
The team quickly learnt that it was much slower at resolving coding issues that it had imagined. The average end-to-end time from starting work to implementation of the solution was four months, but predictably could be up to 11 months. That was bad enough. Four further findings added to the shock:
- The fixes were small and relatively simple (typically no more than 10 lines of code, limited to one module and had no impact on external system interfaces)…
- … yet each issue had a massive effect on operational teams, which had to take manual action, often complicated and time-consuming, to correct the impact on a supposedly automated system. In one case, operations calculated that correcting errors in customer accounts was costing it the equivalent of the work of five fulltime employees.
- Issues had often been identified long before the development team got to them, typically sitting in a backlog for two years. The problem therefore affected the operational team for far longer than the developers were working on it.
- Perhaps most worrying of all, the controls and governance structure were showing the team’s performance as ‘green’ – ie it was meeting performance requirements with no outstanding issues.
Rather than simply using the Agile techniques they had been trained in, the leader and team decided to work to principles drawn from systems theory, in effect to determine whether the resulting outcomes were beneficial or not in a wider context. In essence they were putting scientific method to use by testing their hypothesis that ‘the new principles will improve the team’s effectiveness and therefore performance’.
Within months they had learnt how to put the new principles into practice, with effective solutions taking an average of four weeks, and predictably up to eight weeks, to implement. The change of focus and decision-making resulted in a new approach to governance and collective learning, leading to further improvements. A year on, effective solutions were predictably being implemented in less than a week, with simple scenarios being resolved in two days, as the application team and its leaders progressively identified constraints obstructing the work and took action to remove the ‘waste’. Having eliminated the backlog, the team progressed to close working with the operational team proactively to prevent backlogs from occurring.
All the improvement occurred without the use of Agile or related operating models. At the same time, some of the practices emerging from the systems principles had parallels with Agile.
We were left with a working hypothesis that better outcomes in software development are dependent not so much on using Agile practices as on the effective application of systems principles. A critical consequence of using a systems approach, as opposed to a software perspective, is that business change and IT efforts are focused on issues that demonstrably affect outcomes for customers.
Releasing the new principles into production
Unlike new software code into a computer, principles cannot simply be ‘installed’ in groups of human beings. The Italian astronomer and physicist Galileo is one of a number of historical figures who have suffered persecution for espousing new thinking that challenges the common dogma held by the hierarchy. Galileo was ordered to turn himself in to the Holy Office for trial for maintaining the belief that the earth revolves around the sun, then deemed heretical by the Catholic Church. Standard practice demanded that the accused be imprisoned and secluded during the trial.
The development of new principles, or a new manifesto, requires a change of thinking and behaving. In conceiving the Vanguard Method, one of John Seddon’s founding tenets was that effective change cannot be imposed on people but has to be enabled through study of what is happening in the work and why. Helping people to experience for themselves why change is required may take more effort than other approaches, but it is the only one that works sustainably. If training programmes for techniques such as Agile do not challenge managers’ traditional frame of reference, they will continue to manage rather than lead teams, nullifying Agile’s potential benefits. (Which is why new employees often find it easier to tune in to the Agile mindset and practices than do staff with long experience of older methods.)
Even if people do get to understand for themselves how and why Agile principles should be applied, this still leaves them short of method to identify the constraints affecting the performance of Agile teams. This is why in the earlier example the Vanguard Method was key to the remarkable improvements in cycle times that were eventually achieved.
So what?
Any organisation adopting Agile thus faces a choice: either treat it as the latest fad or accept that it needs to be set in a systems perspective. Do managers stay in the comfortable framework of current dogma? Or do they move into the enlightenment that comes from understanding the organisation as a system and dealing with change accordingly? The latter demands a change of thinking not only in those developing code, but in managers throughout the organisation, regardless of functional speciality or seniority. From my observation, successfully applying Agile requires a solid investment of leadership in:
- propagating a manifesto of principles grounded in systems theory
- enabling a learning organisation that is able to put the principles into practice
- using intervention theory to enable change in the organisation.
Naturally if you have any data or observations that enable me to improve on my hypothesis I would be very grateful to hear about them.
Richard Moir