Pascal's Big Bang

The Big Bang Cycle of Fear

I have written and talked in the past about the Big Bang Cycle of Fear (similar to its cousin, the Tech Debt Cycle). This is a cycle that has happened in countless places over the decades. Its end game - long cycle times, lots of failed releases, low automation and general fear around changing anything - is the state that I have been introduced to at the start of many engagements as a consultant, first for ThoughtWorks and now for Codurance.

The Big Bang Cycle of Fear (extended version) goes:

1. There was a "bad" release that caused some production problems.
2. The response is "we have to do more tests" but instead of automating tests, because the codebase is in such a state that test isolation is hard to achieve, we add more manual testing process and possibly more "testers".
3. In order to be able to "test everything properly" before each release, we do a "code freeze" after a set period and then assign a set period for testing of that "release candidate".
4. At the end of this testing period nobody is confident that there isn't a problem but the "regression packs" passed with only minor issues so it is probably OK. In any case, the product owners are screaming for the release of "my feature" to go ahead.
5. Ostensibly to discuss the risks associated with the release and therefore whether the release should go ahead, we have a long and costly meeting with all the team leads, the product owners, some key developers, the head of engineering and possibly the CTO. In reality the purpose of this meeting is to share the blame for the failure when it comes and thus ensure that no single person is blamed and possibly fired.
6. The release goes ahead.
7. There is a production disaster and we go back to step one.

Understanding the Issue

The first step to solving any problem is understanding that you have a problem. I think most people involved in a BBCF would recognise that they have a problem. But would they be able to identify the probable root cause of the issue? I've recently read some books about systems thinking and one of the insights in there was that people normally look for solutions to problems at the point at which the problem manifests. I think in the case of our cycle, it isn't clear where the problem happened because it is a cycle. Thus, it might not be clear where the best place to look for a solution will be.

Of course, most organisations flounder around for months, possibly (even probably) years not understanding that they are causing their own problem. A things goes wrong in production and they blame the technology part of the organisation. Of course, they do, it is a technology problem isn't it? Sadly all too often the response of the CTO is also to not understand the issue. So instead of tackling the real cause or fixing the real problem, they decide that they have to do more testing, which manifests in some common anti-patterns.

More Manual Testing

Sometimes an organisation will hire more "testers" for their "QA team", not realising that you can't test quality in after the event. Maybe they insist on a longer code freeze to make sure that they can test everything, not understanding that by doing this, they are exponentially making it it harder to test effectively. The worst way that this gets done is when the organisation already outsources its testing (we do development in house, of course!) because it is seen as lower value, and thus they hire dozens more people at the outsourcing firm, probably several time zones distant, ensuring that any feedback loop is automatically extended by at least a working day.

More Automated Tests

A seemingly more sophisticated response is to decide that what is needed is more automated tests to cover the existing functionality. They have no people available to write these tests, or the skills needed, so they hire a "Test Automation Consultant" to write a suite of tests. This seems on the face of it to be a sensible short term expense to solve a long term problem and it may even appear at first to be effective, a classic anti pattern trait. The trouble is that these "automation test consultants" leave once they have written the tests, nobody will then maintain the tests, they will then start to fail as soon as changes are introduced. Now the company is stuck with pipelines that won't complete and which probably take hours to run because they are "end to end" tests, because that is what you asked for.

How Many Things Can Go Wrong?

A couple of years ago I was working as an adviser for a company who had undergone, and were still undergoing, this big bang cycle of fear. Instinctively it feels wrong to make a big release. The more changes there are to release, the more things can go wrong. I noticed that the testers (separate from the developers) had added tests at each new release which were by hand tests, so each successive release necessitated more testing for any subsequent release. This would have been OK, of course, if the new tests added were all automated and could be run in a reasonable amount of time. Sadly, they weren't automated and so they couldn't be run in a reasonable amount of time. So straight away they had introduced a linear scaling problem.

New Problems

The strange thing, from the perspective of the managers at this company, was that even though they had enough people to run all of these "regression packs" at each stage and thus, they thought, verify their existing functionality, they still experienced strange, unforeseen issues. "How can this be?" they asked, "when we are running all of these regression scripts?" It was clear to us that the answer was simple: over time the design of the system had degraded to the extent that there were all sorts of tight, loose and temporal couplings between the various parts of the system. Nobody had a sense for what these couplings were and nobody had any idea about what they were intended to do, let alone whether they were doing it correctly. So clearly they weren't considering enough things in their testing. They needed to consider interactions between the things.

My Mathematical Question

I went to a meeting with all of the senior managers just after another failed release in which they had released in the region of 30 different things. As usual they had gone through their ritual of blame sharing, which they called "Go / NoGo" so therefore they were safe in the knowledge that no single person in the room was going to carry any cans. In fact, the interactions between the different parts of their solution were so unpredictable that they didn't even know which "thing" had failed. They just knew the release had caused issues.

I started with a simple question:

If I release one thing, how many things can go wrong?

Clearly, the answer here is 1. So I followed up with another simple question:

If I release two things, how many things can go wrong?

At this point there was a bit of a murmur in the room but the general consensus was that two things could go wrong. "I beg to differ", I said, "Thing #1 can go wrong, Thing #2 can go wrong OR the unexpected interaction between Thing #1 and Thing #2 could cause a problem." Nobody argued with this because that was exactly the experience that they had been having.

So the next question I posed was, naturally:

If I release three things, how many things can go wrong?

Now, the maths starts getting a little complex because at this point each of three individual things can go wrong, Each or three interactions between pairs can go wrong or an interaction involving all three things can go wrong. So we now have 3 + 3 + 1 = 7 places to look for our problem. I could see the group starting to appreciate where I was going with this.

Pascal's Triangle

So naturally, my train of thought led me to think that there must be a simple formula that tells us how many things can go wrong when you release N things. At first I thought I was looking at the formula for adding the first N numbers, that is to say N(N + 1) things, or O(N²), which is bad enough, but then I realised it was even worse than that.

You may remember Pascal's Triangle from studying maths at school. My recollection of it was that it was introduced to illustrate the Binomial Theorem. I realised as I was going from 3 things to 4 things... to N things that what I was seeing was the successive rows in Pascal's triangle:

So if you start from row zero (the single 1 at the top) and number the successive rows, then row 5 contains the numbers {1, 5, 10, 10, 5, 1}. If you ignore the initial 1 (which can be regarded as representing number of cases where no things interact with no other things and thus can't be the cause of any issue) you can see that there are 5 single things that can go wrong, 10 pairs of things interacting that can go wrong, 10 triplets interacting that can go wrong, 5 sets of 4 things interacting and a single set of 5 things. So we can see that if we release 5 things, there is a potential 31 things that can cause us problems.

As well as representing the coefficients in a binomial expansion the numbers in the Nth horizontal row of Pascal's Triangle add up to 2^N so given that we know the first 1 is irrelevant for this discussion, the conclusion is that if we release N things, we are causing a potential 2^N - 1 issues and therefore need to test all of those things to be sure that our release will not fail. Even if we could rule out certain interactions and therefore reduce the overall universe of possible interactions, the number of things that can go wrong, and therefore could need to be tested, when we release N things is still O(2^N) and thus of exponential complexity. So the scary answer for my client back in 2018 was that if you as a group agree to release 30 things simultaneously then you need to test 2³⁰ - 1 things, which is just over a billion things. It is easy to see that you are stretching the limits of feasibility here and thus why your releases fail every time.

What is the Answer?

The answer is simple and, to a lot of people, obvious. Release one thing at a time. This means reduce your batch sizes until you are doing continuous delivery. Find a way to gain confidence around your changes so that they can be released as soon as they are "done", not weeks later. This could mean you do no new work (or reduced amounts of work) while you pay down technical debt, it could mean assigning developer effort away from new features but probably most importantly it HAS to include taking a long hard look at your Work in Progress (WIP) and aggressively reducing it. 

How can you do this and still deliver value steadily? Well, for a short period of time you can't. You have to accept that your rate of release of new features will reduce. BUT, ask yourself how often you have released new things without causing new problems and then ask yourself what is your REAL rate of return of value to the business? The calculation will be different for every system but ultimately if you don't gain confidence around your working system your throughput will eventually grind to a shuddering halt.

I think the desired end point is clear. We should all be practising continuous delivery which means tiny releases very often. Depending on the current state it could be easier or harder to get there or it may not even be possible without some kind of large scale software modernisation program. Hopefully most people will see that particular state coming and do something about it before it arrives.

Conclusion

The bigger your release the more likely it is to fail. The combinatorial mathematics shows this.

Use Pascal's Big Bang to demonstrate to stakeholders the fallacy of adding extra testing cycles.

To understand your real throughput consider releases as complete only when they have no remaining issues.

Reduce batch sizes aggressively until you are comfortable with continuous delivery.

The best form of cure is prevention! Don't suffer from boiling frog syndrome. If your levels of technical debt are increasing, slowing delivery and causing problems for releases, don't allow your company to fall into one one of the common anti-patterns that inevitably lead to the Big Bang Cycle of Fear.

On the Giving and Receiving of Feedback

What is Feedback?

I joined ThoughtWorks in 2015 after working at a startup for 10 years and a few other companies before that. The culture shock when I moved to ThoughtWorks from the (hero and blame) culture of my previous employer was huge. It was so huge that I struggled to adjust in my early months and at various stages in the first year was close to failing the probation period and / or quitting.

One of the weirdest things for me at the time was the culture of feedback. In all of my previous jobs personal feedback meant a once a year conversation with a manager in which the manager picked fault with your performance in the previous year and ignored anything good you had done in order to justify a miserable annual pay rise and a disappointing bonus. Feedback was thus only used for bad news, was so infrequent as to give no opportunity to improve and was strictly a one way process which, like most nasty things, only flowed downhill.

When I moved to ThoughtWorks (my current employer, Codurance, has a similar culture of feedback) my experience was very different. Firstly, in the two day induction there was a module on giving and receiving feedback. Secondly, personal feedback for the purposes of salary review was a very different thing. Instead of a manager (neither ThoughtWorks nor Codurance has the concept of reporting lines) using feedback as a way to beat down your salary expectations, the responsibility is put on the individual to gather feedback as and when you see fit in order to improve yourself in your role and possibly to support your argument for a pay raise at a later point. The key point being that it is up to you to seek feedback and also up to you how to solicit it, how to record it and how or whether to act upon it.

Instant Feedback

Instant feedback is extremely useful when used well because, at the risk of sounding obvious, it gives you a short feedback loop. Anybody familiar with Agile values should appreciate the value of short feedback loops. For example, have you ever been in some kind of feedback discussion, focussing on a large period of time, when somebody says to you something like "sometimes you can be arrogant"? This is all well and good and you might be inclined to say, or think, "I'll try to be less arrogant" but the chances are you weren't aware of being arrogant when apparently were being so. You may well, therefore, ask "can you give me an example of when you perceived me as having been arrogant?" Unfortunately, the person talking to you probably can't and therefore you are unsure what behaviour is being thus characterised and you can't act upon this feedback.

Giving Instant Feedback

I was taught to give instant feedback following three simple rules:

• Do it as early as possible after observing something.
• Tell the person what they did or said.
• Tell them how it made you feel.

As Early as Possible

This is quite obvious. The earlier feedback is given referring to a specific incident the fresher in the memory of the feedback giver and receiver it will be. Moreover, you have the readymade example so need to ask for a specific example of a general behaviour.

What Happened

This should be quite clear as well. Tell the person what they did or said that is causing you to give them some feedback. Provided you have followed the first rule of instant feedback, there should be no dispute or interpretation at this stage. You are simply stating a fact of what happened.

How it Made You Feel

This may seem a little strange but is actually extremely important. The reason why you tell the person how it made you feel is because they cannot dispute this. Only you know how something made you feel. The difference between saying "you were trying to make me look small in front of the client" and "your comment made me feel like you didn't respect me" is enormous. The first comment can easily be met with an indignant (and very possibly justified) "No I wasn't!", the second comment can only prompt further useful discussion.

My Experience

Once I was used to dynamic of receiving and giving this type of fast feedback it became hugely useful for me when I received it and hugely satisfying to give it. Yes, there were sometimes hard conversations but almost always the conversation contained at some point a phrase such as "I had no idea that could make people feel like that, thanks for letting me know, please point it out if it happens again." I felt liberated and most importantly never again took part in conversations that went "You do things that to belittle people....it's just the way you talk to people....I can't think of a specific example, but you do it all the time..." The most satisfying aspect of all was that I could see, both when I gave and received feedback like this, is how much more valuable the early, instantly actionable, feedback was compared to if we had waited some considerable time before attempting the same conversation.

The Reinforcing and The Constructive

Just to be clear, this type of feedback giving should also include positive, reinforcing feedback. I realised that just as not having a specific example for behaviours that you think should be curtailed, not having a specific example for a behaviour that should be amplified is extremely frustrating. So my habit became to frequently give small nuggets of feedback, good and bad, but always immediate and actionable. It is my belief that none of that feedback was resented or ill received.

Longer Term Feedback

Longer term feedback is a bit harder to do effectively but unfortunately it seems to be much more common than instant feedback. There are many problems with it, mainly that the passage of time will have dimmed memories of certain incidents as previously mentioned. It also may be way too late to take any useful action on the back of the feedback that has been given as it could be many months after the actions that prompted it.

Why Engage in Long Term Feedback?

Annual Pay Reviews

As I've mentioned above the most likely reason that people may be having a conversation about actions and behaviours that have occurred during some long period of time will be to support the case for pay reviews. Most companies will have an annual pay review involving all their staff in which they have some kind of allegedly fair compensation formula which will reward everybody as they deserve but which in reality will force the whole of the organisation at every level of magnification to rank the peers and force their pay awards into some kind of normalised distribution that will result in a pre determined fixed rise to the total wage bill. Have I mentioned that I'm not a fan of this type of one-way-no-discussion-allowed feedback?

Project or Milestone Retrospectives

Sometimes teams break up, or reform at least, at the end of a project or at some significant milestone. It is also possible, particularly if you are a consultant, that people will regularly rotate in and out of engagements. In recent years I haven't spent more than nine months working for the same client. So it is often seen as appropriate to give and receive some kind of retrospective feedback from the people with whom you have worked for a time and from whom you may soon be separated.

Career Progression Discussions

A third use case for a retrospective style giving and receiving of feedback is to gain some kind of view on your colleagues' perceptions of your readiness or suitability for some kind of career progression. It could be that you believe that you are deserving of a promotion and you want to verify whether you are being reasonable or not. It could be that you have certain blindspots to your own shortcomings, or even your own strengths, and therefore you want to engage your colleagues in order to understand what some of these blind spots might be.

Some organisations, in fact many, will combine the annual pay review conversation with career progression conversations.

So How do I Do It?

how do you approach the giving and receiving of long-term feedback? This is a question that wasn't really addressed in a systematic fashion in the time I was at ThoughtWorks. In fact, other than saying things like "We have a feedback culture" or "We value the giving and receiving of feedback", I haven't seen many organisations that give their people much guidance on how to give and receive periodic feedback. 

Most of the time I have seen people sending emails to a group of colleagues soliciting feedback. More often than not it will say something like "Please reply to this email with some feedback." I never found it easy to respond to such requests. Too often, the response ends up being strong in terms of praise and very weak in terms of actionable points. Sometimes people request a meeting to discuss feedback. In those circumstances it can be even harder to give anything other than bland, "well done" type feedback. I think there are a number of reasons why this is but primarily it is because it is hard for many people to to give constructive feedback. Most of us are just too nice. The other reason I think is similar to what I mentioned above in that it can be very hard to come up with useful examples (because of the passage of time) of things to improve on which makes it difficult to mention certain areas.

Make it Comfortable

Having gone through a couple of frustrating cycles of less than useful feedback I alighted almost by accident on a method that seemed to work well for me, and which I shared with many of my ThoughtWorks colleagues. Firstly, one should recognise that feedback is useful and should be regarded as a gift. Therefore, make it as easy as possible for people to give you feedback. So when I would send an email soliciting feedback, I would give the recipients the option of how they felt most comfortable with giving the feedback. I would typically use something like "I'm happy for you to answer my questions by email, or if you'd prefer, we can meet face to face in the office, over Zoom, or in a pub after work if you like."

Make it Explicit

Finally, and I think this is the key, make your request for feedback more explicit. Instead of saying "please give me some feedback on my performance", try something like:

I'm looking for some feedback on my project performance. In particular could you please consider the following questions:

• Is there anything I do as a Technical Lead [or whatever] which you think I should be doing less of?
• Is there anything that you would expect a Technical Lead to be doing that I'm not doing?
• Is there anything I could have done better in my direct interactions with you?
• Can you think of any times when you haven't got what you expected from me personally?
• What do you think I need to do more of to be considered as a Principal [or whatever the next level is]?
• Is there anything I need to stop doing to be considered as a principal?

My experience was that five or six targeted questions yielded up actionable points that I could actively work on improving or eliminating as appropriate. I would have a pool of perhaps 8 or 10 questions that I would use for a single round of feedback and would share a different subset with different people depending on how we had worked together. For example, I would be more likely to ask another of the technical leads about interactions with our stakeholders, because they were probably in a lot of the same discussions, than I would be to ask the same question to my team's graduate developer.

Conclusion

• There are two types of (very different) feedback. Instant feedback for immediate, actionable feedback on recent specific events and longer-term feedback on performance over time. Be clear on your reasons for giving and your expectations for receiving both.
• The golden rules for giving instant feedback are do it soon, stick to the facts of what happened and tell that person how their actions made you feel. Most importantly, do not attempt to speculate or second guess the recipient's motivation for their actions.
• Make it as comfortable as possible for people to have what could be an uncomfortable exchange.
• Be explicit in your request for a long term feedback.