Big Bang Process Release

The Big Bang Anti-Pattern

For years, perhaps decades, we have known that big bang releases do not work in software and yet many companies and, shamefully, many consultants consistently advocate for big bang releases of new process (or "Ways of Working" as they are sometimes called). It is worth repeating some of the reasons why big bang does not work for software. As far as I have seen over my working lifetime, the most important reasons are the near impossibility of testing a big bang before it goes live and the absolute impossibility of understanding what was responsible for either good or bad effects after the release.

Big Bang Testing

I often as clients who insist on some form of code / test / release cycle how many things they think can cause a problem in the next release. I once sat in a room and pointed out that they were planning on releasing 30 things. So I asked how many things could cause a problem, and therefore how many things are they carrying out their "testing" on? The answer I got was 30. "Are you sure?" I asked...

What About the Interactions?

So I changed the question to "if you release ONE thing, how many things can go wrong?" In this case, the answer is definitely one. How about if you release 2 things? "Two", they said. "I disagree", I said, "I think it is at least 3 things. It could be the first thing, or the second thing, OR it could be the unexpected, and untested, interaction between the first thing and the second thing." So let's scale that discussion up:

Things Being Released	Possible Interactions	Total Possible Causes of Problems
1	0	1
2	1	3
3	3 pairs, 1 triplet = 4	7
4	6 + 4 + 1 = 11	15
5	10 + 10 + 5 + 1 = 26	31

So in fact what we have here are successive rows of Pascals Triangle, but without the initial 1 on the left (sine that corresponds to the number of ways of choosing none of the things). So, far from being a linear progression of potential problems, we have a result that shows that the number of potential problems is of order of 2 to the power n. So if you release 10 things you aren't causing potentially 10 problems, you are causing a potential 1023 problems. For 30 different things being release, well, you can do the maths. 2^30 - 1 ~= 1,000,000,000. So no small problem to work out what went wrong and no surprise that something did go wrong. Every time.

Just to be clear, this is very much a VERY worst case scenario. If your codebase is well curated, you have plenty of tests in the right places, including plenty of unit tests, enough integration tests and an appropriate level of what you may call UI or end 2 end tests, then it is unlikely that you can accidentally cause all the possible interactions. The trouble is though, that the lack of such tests is probably why you fell into the big bang cycle of fear in the first place. The chances are that if you have all those things, you are doing continuous delivery, not big bang releases.

Post Hoc Validation

So clearly, testing is going to be an issue in any big bang release. But what about understanding if your new stuff had the effects that it was designed to have? We know that in the modern Lean enterprise we should only do stuff that has a clearly defined definition of success. In general this means you need a hypothesis that will be tested after a release. Sometimes you can use an AB style test where some users get the old version of something and others get the new version. Then you can compare the effect of the change. Other times you might decide that it is enough to compare some metric after the release with the same metric before. Whatever your methodology, you need to know that the only thing different between the samples is the thing that your hypothesis says could move that needle. 

So for pretty much the same reasons outlined above, we should be releasing one thing at a time into whatever system we are analysing so that we can be sure that it was that thing that caused the uptick (or downtick) in whatever metric is associated with our hypothesis. If I also release a load of other stuff at the same time how can I be sure what it was that caused the improvement or decline in the thing I'm measuring?

The Theory of Constraints

Assuming we are using some kind of Agile process, whether it be Scrum or Kanban or whatever, it is fair to assume that we have cards that are independent of one another flowing across our board (or through our value stream) in parallel with one another. The Theory of Constraints, as laid out originally in the Goal by Eli Goldratt, tells us that it only makes sense at any one time to fix your biggest problem, or your biggest impediment to flow. 

Imagine, for example that every story takes only one hour of developer time to be code complete but each story sits in some queue, waiting for a QA to pick it up, for 2 days. The total cycle time might be, say, 4 days per story. We only care about the total cycle time because we return value to our organisation only, and not until, we put something live. So there is no point in optimising the one hour developer time but there is every reason to work out how to reduce the 2 days of queueing time each story spends waiting for QA.

Buy-in, Desirable Outcomes and Personal Motivations

When I have seen "New Ways of Working" or "Process Remodelling" or "Scrum Reset" struggling to gain purchase I have generally observed a big lack of engagement from the people in the teams who are being affected by whatever this new thing is. Some of the causes of this lack of engagement that I have identified are:

• lack of buy in from the teams themselves ("what are those people who don't understand my job getting me do now"?)
• lack of understanding from the people involved, probably because nobody ever bothered explaining it to them, of the outcomes they should be driving towards at each stage of the process.
• lack of empathy from managers, or scrum masters, or consultants, for the motivations of those that are doing the work and therefore lack of alignment between people in the teams about what constitutes a valuable or desirable outcome.

Getting Buy-in From Teams

Some time ago an aspiring technical lead and I were having a chat. She told me that she was worried about how to make the right technical decision for her team. I asked her why she thought she needed to make a decision, surely she should be guiding the team towards making the right decision? We discussed it and I explained to her that as technical leaders we ought to have a reasonable grasp of what the best decision is at whatever point we are at. But, assuming it is a major decision that will affect all the team, we should not impose that decision on the team. Rather, we should be facilitating a discussion in the team (perhaps through tech huddles?) which will come to the right decision. Perhaps there is a chance that you didn't actually know the best way so maybe the discussion could even be educational for you and maybe the decision will be different from what you thought. Whatever, if you take the time to facilitate a discussion rather than impose something on your team, you will get buy in from them and you will find you are much better set up for success.

In my experience, therefore, it is much better to find a way to get the team to start discussing the decisions on the process and working out for themselves, with appropriate guidance, what sensible changes to the process might look like.

What Outcome Does this Process Support?

A while ago when I was working for a company going through the pain of trying to impose a top down, big bang change of process on 9 different development teams, I attended a lot of the scrum rituals and saw a lot of disengagement, boredom and straightforward cargo-culting from each of the teams during most of the rituals. After enduring a few of those meetings I wondered if people actually understood, on any level, what they were supposed to be trying to achieve in the meetings.

There was a document in existence, a PowerPoint deck, that described the different parts of the process that they were trying to implement. So I went through this deck in detail as it was the only resource I knew of that could give insight in what they were trying to do. It was very good looking, the graphics were really nice. I could see that somebody had spent a lot of time on making that document look good when it was presented. It was also pretty good at specifying what rituals should happen, when they should be happening and which people should attend which ritual. What I didn't see in this document was any kind of goal statement for the whole thing nor did I see any kind of statement of what outcomes were being driven out of each ritual in the specification. It was as if the authors had taken for granted that people would just simply understand the "why" and they only needed to know the "what" and it would all be brilliant.

The "Scrum Reboot" as it was being called was being driven by a group of Scrum Masters. Leaving aside the fact that I've never really understood what a scrum master does and the fact that most people that I've met who self identify as scrum masters are waterfall project managers who went to a training course to get Scrum certified, I went to talk to the one of them who seemed to be the first among equals. I asked him if anybody knew what outcome was being sought in each ritual. "It's in the deck" was his response. With respect, I begged, it isn't in the deck, I looked through the deck just that day specifically for it. Besides which, with all the will in the world, people rarely read such decks.

"Well, I would have said when I gave the presentation", he said. OK, fair enough, I said. But in my experience of top down change, people are rarely engaged during a presentation of what they are going to be expected to do. So there is a good chance that people will have forgotten, or perhaps never heard, the outcomes that each ritual should drive.

"Do me a favour please", I asked, "for the next week or two could you and the others take a couple of minutes at the start of each meeting to discuss with the team what the outcome is that is being supported with this meeting." I believe that if you do that, I told him, you will enjoy better engagement from the teams. A couple of weeks later, results started to improve. The CTO asked us what we had done in such a short space of time. We haven't done anything, I said, it was all down to the Scrum masters.

What Motivates You?

After the success of describing the outcomes the scrum masters were more likely to listen to us and we gave another piece of simple advice. I asked them to have one on one conversations with each of the people in their teams. There can be a few reasons why such conversations are desirable, such as:

• Give and receive feedback around anything.
• Understand what pain points the person has and whether and how you can help with them.
• Understand what their career aspirations are and whether and how you can help with them.

In this case, as well as such goals, I asked Steve if he could attempt to work out what motivates everybody. For example, some people are motivated by a possible promotion, some might be motivated by building beautiful products, still others might be motivated by a desire to avoid being shouted at by a particularly nasty person in the company hierarchy. So my ask was, find out what each person's motivation is and then explain to that person how the outcome being driven by the scrum rituals will help achieve those things. After a few more weeks we saw performance increasing again.

It Will Take Time!

If we accept that big bang process releases are a bad idea and we accept that the theory of constraints makes sense then we should accept that any kind of process change will take some time. It follows from the previous assumptions that we will be making one change at a time to our process (an attempt to relieve the biggest problem) and that we will need some time to assess whether the change had the desired effect or not.

I also strongly believe that it is necessary to not put the process cart before the outcome horse. By this I mean that I believe that any conversation about modifying the process should start by understanding what outcomes we value as a group and why those outcomes are valuable. This also will take time. "Outcome over Process" has become something of a mantra of mine over the past couple of years and I have been attempting to get teams I've been involved in to take that to heart. Unfortunately, this takes a lot of time as well. People don't just turn up to work one day, have a quick conversation about outcomes and then shout "Eureka!" and start magically understanding how better to achieve those outcomes.

My Recommendations

• Big bang process change won't work so don't try it.
• Top down dictats about change won't work. Guide your teams into coming up with "their own ideas" for how to improve their own process.
• Nothing will improve without understanding what outcomes you are driving towards.
• Attempt to fix only your biggest problem, give yourself time to assess if it worked, then fix your next biggest problem. Or do something else if your biggest problem has improved but is still your biggest problem.
• Spend time to understand and empathise with as many people as possible. Explain to them why they should be valuing the agreed team outcomes.