The ProblemRecently we were working for a client whose IT function was displaying several common pathologies. They had (still have but it is improving as we are still there!) the classic inverted test pyramid, about 10 test teams, massive product backlogs that only ever grow, mistrust of their IT function in the wider company, no real understanding of continuous delivery... All of this adds up to a state of development stasis. Releases can take months and rarely add new value. Even when new features are added the uptake around the company is low to non existent.
The most disturbing aspect of this general dysfunction to me was the realisation that there are several different test teams comprising dozens of people who add little to no value to the company. Management has at some point created these layers of testers in response to poorly managed releases in an attempt to restore confidence in the release process. This has had the exact opposite effect as the release time has stretched out, more and more things go into a release, features and fixes are rushed in for fear of missing the release train and the situation only gets worse. Nobody has confidence in success therefore the test teams spend their working life not adding value to the process but ensuring that they and their immediate colleagues don't get blamed when it goes wrong.
They brought us in on an initially ill defined brief of "digital transformation". They had no clear idea of what that meant to them and it was pretty obvious to us from an early stage that a big part of our work was going to be not the software that we deliver but how we deliver it and whether we can use that example to help them change their processes and culture.
We wanted to run a session in the inception that would make them realise that they need to embrace a devops culture with regular releases, effective pipelines and rapid response to issues. We didn't have any experience of running such a session in the past because none of us have ever had to. Most businesses accept as fact that CI and build pipelines are a good thing.
We started with the notion that we wanted them to come to the "correct" conclusions. The client's staff had at various time expressed various emotions such as exasperation, scepticism and doYouNotThingWeveTriedThisIsm. So we were acutely aware that we wanted to have given the impression that whatever conclusions were reached were reached by them, not imposed by us.
The DAFF LoopAs ever, we deployed the magic whiteboard on the walls. We labelled 5 magic whiteboards with our 5 chosen bad events:
- Run-time exception (this had to be explained to some. I would name this differently if I did this again)
- Bad check-in
- Performance bottleneck
- Site unavailable
- Customer dissatisfaction with digital product
We surrounded the scenario label with a circle (borrowing heavily from OODA) and labelled this with Discover - Analyse - Fix - Finalise (which I'll call the DAFF loop).
|Discover-Analyse-Fix-Finalise for Bad Check In|
How did it End?
We were hoping that the discussion we had after each stage and the suggestions we arrived at would coalesce into some or all of continuous integration pipelines, build monitors production monitors etc, build it run it devops style mentality, customer focus and all of the other goodies that we expect in a well run organisation. Happily we found that the exercise worked out almost exactly as we'd wanted it to. With minimal prompting and poking from our team of consultants our client independently from us came to the conclusion that all of the above, and other great stuff besides, are good aspirations for their IT function to aspire to.
Obviously I don't have a massive sample size from which to draw experience but we did get a 100% success rate on guiding our client to the desirable conclusions. A very satisfying afternoon at the office. If I'm ever involved in a similar exercise I'll update this post.