The Big Bang Cycle of FearI have written and talked in the past about the Big Bang Cycle of Fear (similar to its cousin, the Tech Debt Cycle). This is a cycle that has happened in countless places over the decades. Its end game - long cycle times, lots of failed releases, low automation and general fear around changing anything - is the state that I have been introduced to at the start of many engagements as a consultant, first for ThoughtWorks and now for Codurance.
The Big Bang Cycle of Fear (extended version) goes:
- There was a "bad" release that caused some production problems.
- The response is "we have to do more tests" but instead of automating tests, because the codebase is in such a state that test isolation is hard to achieve, we add more manual testing process and possibly more "testers".
- In order to be able to "test everything properly" before each release, we do a "code freeze" after a set period and then assign a set period for testing of that "release candidate".
- At the end of this testing period nobody is confident that there isn't a problem but the "regression packs" passed with only minor issues so it is probably OK. In any case, the product owners are screaming for the release of "my feature" to go ahead.
- Ostensibly to discuss the risks associated with the release and therefore whether the release should go ahead, we have a long and costly meeting with all the team leads, the product owners, some key developers, the head of engineering and possibly the CTO. In reality the purpose of this meeting is to share the blame for the failure when it comes and thus ensure that no single person is blamed and possibly fired.
- The release goes ahead.
- There is a production disaster and we go back to step one.
Understanding the Issue
More Manual Testing
More Automated Tests
How Many Things Can Go Wrong?
My Mathematical Question
If I release one thing, how many things can go wrong?
Clearly, the answer here is 1. So I followed up with another simple question:
If I release two things, how many things can go wrong?
At this point there was a bit of a murmur in the room but the general consensus was that two things could go wrong. "I beg to differ", I said, "Thing #1 can go wrong, Thing #2 can go wrong OR the unexpected interaction between Thing #1 and Thing #2 could cause a problem." Nobody argued with this because that was exactly the experience that they had been having.
So the next question I posed was, naturally:
If I release three things, how many things can go wrong?
Now, the maths starts getting a little complex because at this point each of three individual things can go wrong, Each or three interactions between pairs can go wrong or an interaction involving all three things can go wrong. So we now have 3 + 3 + 1 = 7 places to look for our problem. I could see the group starting to appreciate where I was going with this.