Let me start by getting something off my chest that has been annoying me this whole millennium. Methodology is the study of method. The process you use to develop software, and the practices embedded in that process, constitute a method. The understanding of how that process results in working software, in customer value, in arguments between business analysts and QAs, and in the modification of that process to yield different outcomes, that is a methodology.
The people who came together at the Snowbird conference to discuss their lightweight methods were, by and large, methodologists, in that they had developed those methods and explored the commonalities, differences, and impacts of those methods. The people who subsequently read a book called “succeeding with Agile” and bought Jira licenses were not methodologists. They were appliers of methods.
Scrum actually embodies both method and methodology, in that it is a self-referencing method that includes process improvement as part of the process. This has caused a problem for many organisations who implement it, because they employ process-followers rather than process-definers, but try to follow a process that has them defining the process. If they are lucky, they ignore the process-improvement aspects of Scrum and follow the baseline process forever. If they are unlucky, they try to improve on the baseline.
Anyway. That is out of my system now, let us move on. There have not been any advances, either in software development method from the methodologists, or in software development methodology, for the last couple of decades. In the beginning, there was “let us run this like we run our hardware projects and our manufacturing procurement projects”. That was very quickly (surprisingly quickly, if you only read the headlines) seen not to work. By the 1980s people were recommending shifting risk left (i.e. exposing it earlier), by constructing working prototypes or “walking skeletons” early in the development process and sharing those with customers. By iteratively and incrementally delivering the project to customers. By periodically reflecting on whether following the plan was still the best outcome for the project, or whether the plan needed to change.
Like bankruptcy, Agile happened at first slowly, then all at once. This shifting left of risk and responding to change became so important to the outcomes of projects that people started to realise that they should basically reorganise their methods around correcting for what needed to change. Big, detailed design documents were deferred, or replaced entirely with lightweight sketches or even metaphors relating the planned system with some other system. Lengthy analyses of what users might do with the hypothetical systems got replaced with summaries of conversations about what users might want from the system, or at least reminders to have those conversations.
The publication of the Agile manifesto came neither at the beginning nor the end of this process, but in the middle. It reflected two things: one that methodologists across the industry realised they were all pointing in roughly the same direction by promoting these lightweight methods, and wanted to increase their impact by collectively sharing the commonalities. The other that they did not want to be called “lightweight methodologists” because that might make them sound trivial.
The process that is going on now is the wholesale adoption of Agile and lightweight methods across the industry, accelerated by the sharing of the manifesto but already in progress before it and still in progress two decades later. There has been no methodological novelty since then. You might point to DevOps, DevSecOps, DevSecTestQABAProjProdThreeBagsFullOps, or whatever the latest name is, but that is just the same story as Agile told again for people who did not hear it in 2001. Shift risks left. Work out what your goals are, and let the professionals collaborate on achieving those goals. Stop planning for getting things wrong, and slavishly following that plan.
We do not have methodology any more, we have method. We have people kind of following agile, and people kind of helping other people to kind of understand how to kind of follow agile. Technological changes (we can now write and run proofs of software in real time; we can create and deploy software without needing to know how it works using ML; we can update software multiple times a minute) have not changed the processes or the process of constructing processes.
In fact, if anything, dark scrum shows that there has been a regression in methodology. We have gone back to focusing on the processes and tools, with managers looking at Jira dashboards to understand “sprint velocity” rather than how the individuals are interacting to produce the software. They look at the comprehensive (project) documentation (in Jira), enforce contract negotiation (via sprint commitments), and follow a plan (via infinite backlogs).
This shows two things. One that the methodological advance that was Agile is not yet evenly distributed, even among those who are of the opinion that they are on board. The second that there is still scope for further methodological advance, because the explaining power of Agile is not sufficiently compelling to work for a significant majority of software organisations or sufficiently effective to point them toward successful methods.
The benefits of Agile are significantly clearer to someone who lived through the turmoil of its ascendance (one of the benefits of being a developer after 40) than they are to someone who has grown up in the code mines of dark scrum. Software is way less riskier now: most agile software projects never fail. They may well get “sunset” when they do not achieve revenue or growth goals, but they do so much more quickly and they deliver more value to more people than the typical failed software project did in the 1990s. The business and the technical folks are much more aligned on goals, and are invited to collaborate on ensuring that the next thing that happens is the most beneficial of the options available.
To some extent, anyway. Scrum itself is surprisingly Leninist in its promotion of democratic centralism. Everybody negotiates on what the sprint goals are until they get “locked in” and then everybody agrees to follow the plan instead of responding to change. In theory, this stops factional squabbles, except that the biggest drawback of Agile is the baked-in factionalism.
That drawback is the central contradiction of Agile: it promotes unity of purpose and self-organisation while also dividing participants into two camps. On the one hand, you have the Bizshevik party, who own the money and the purpose of the organisation. On the other hand, the Techschevik minority, who merely produce the value.
You will notice that there are way more Certified Scrum Masters than Certified Scrum Developers, because the Bizsheviks ensure that the power is concentrated in their party. Business people and developers must work together daily throughout the project, but that does not mean the Techsheviks get to have a say in how it is run. The sponsors, developers, and users should be able to maintain a constant pace indefinitely: as long as the sponsors are making money from the users, the developers get to work on making more.
Where is the next step in methodology? We need a methodology that creates methods focusing on what the people using (or otherwise exposed to) the software want, distinct from what the customer wants. Agile focuses on the interaction between “the business” and “the technical people”, and that is overly limiting. Of course, the business are putting up the resources to make this software happen, so they get some say in the software that gets made. But they should not get a say in how it is made, and they should not be the final arbiters in how it is used either.
We need to get past the product owner as the proxy for what “the user” wants, when they actually represent what the shareholder wants. We need to divorce the question of whether the software is successful from the question of whether the software engineers look busy. We need to get out of the situation where the people accepting the software get to decide how much refactoring or testing we do.
In other words, we need to continue the focus on professional autonomy that agile started: we need to proceed towards anarchic software development.
This clearly cannot be done within the confines of in-house software teams. No, the conflict of interest between doing good software, and demonstrating shareholder value, gets in the way. Good software is sacrificed for feature factories, perpetual busy-working, and building what some in-house “customer representative” thinks people want.
To do good software, the software doers have to be solely responsible for the software that is done. That means acting as agents, and telling the client this is what we do, and this is how it is done, take it or leave it. It means doing work so exemplary, and so consistently exemplary, that the obvious choice is to take it.
And this means that we have to take ownership of the relationship with the people who so far have been called the “user” or “customer”, but should really be thought of as “victims”. We need to take control of that relationship until we can turn the victims into beneficiaries.
The next generation on methodologies will define methods that do not make “technical people” the servants of “the business”. The primary measure of progress will not be working software that satisfies the needs of “the customer”. It will be the mutual satisfaction of the paymasters, the beneficiaries, and the engineers: such satisfaction achieved through the instrument of working software where necessary. The technical people and the business will be working together to provide something that works for them and for the software’s beneficiaries.
Of course, to succeed, such methods need to be so productive that those who are doing it the old way are clearly doing it the old way. Actually, no. This idea that everybody needs to adopt the new thing or it is a failure, that every new innovation needs to succeed at global scale, is a fallacy. All that is needed for the new methods to succeed is that some people successfully use them to make themselves and other people happy. The rest can happen slowly, and then all at once.
Cover photo by engin akyurt on Unsplash.
