Convincing a client to invest in the build pipeline with DAFF

The Problem

Recently we were working for a client whose IT function was displaying several common pathologies. They had (still have but it is improving as we are still there!) the classic inverted test pyramid, about 10 test teams, massive product backlogs that only ever grow, mistrust of their IT function in the wider company, no real understanding of continuous delivery... All of this adds up to a state of development stasis. Releases can take months and rarely add new value. Even when new features are added the uptake around the company is low to non existent.

The most disturbing aspect of this general dysfunction to me was the realisation that there are several different test teams comprising dozens of people who add little to no value to the company. Management has at some point created these layers of testers in response to poorly managed releases in an attempt to restore confidence in the release process. This has had the exact opposite effect as the release time has stretched out, more and more things go into a release, features and fixes are rushed in for fear of missing the release train and the situation only gets worse. Nobody has confidence in success therefore the test teams spend their working life not adding value to the process but ensuring that they and their immediate colleagues don't get blamed when it goes wrong.

They brought us in on an initially ill defined brief of "digital transformation". They had no clear idea of what that meant to them and it was pretty obvious to us from an early stage that a big part of our work was going to be not the software that we deliver but how we deliver it and whether we can use that example to help them change their processes and culture.

We wanted to run a session in the inception that would make them realise that they need to embrace a devops culture with regular releases, effective pipelines and rapid response to issues. We didn't have any experience of running such a session in the past because none of us have ever had to. Most businesses accept as fact that CI and build pipelines are a good thing.

We started with the notion that we wanted them to come to the "correct" conclusions. The client's staff had at various time expressed various emotions such as exasperation, scepticism and doYouNotThingWeveTriedThisIsm. So we were acutely aware that we wanted to have given the impression that whatever conclusions were reached were reached by them, not imposed by us.

The DAFF Loop

As ever, we deployed the magic whiteboard on the walls. We labelled 5 magic whiteboards with our 5 chosen bad events:

• Run-time exception (this had to be explained to some. I would name this differently if I did this again)
• Bad check-in
• Performance bottleneck
• Site unavailable
• Customer dissatisfaction with digital product

We surrounded the scenario label with a circle (borrowing heavily from OODA) and labelled this with Discover - Analyse - Fix - Finalise (which I'll call the DAFF loop).

Discover-Analyse-Fix-Finalise for Bad Check In

We then armed our workshop attendees with orange stickies and asked them to describe parts of the current process by way of attaching stickies near the appropriate part of the DAFF loop. After a suitable discussion of the current processes, and I can't remember if we rotated groups or left them where they were, we asked them to add problems with the current process in red stickies. Finally the punch line was to use yellow stickies to come up with suggestions on how to improve the current process.

How did it End?

We were hoping that the discussion we had after each stage and the suggestions we arrived at would coalesce into some or all of continuous integration pipelines, build monitors production monitors etc, build it run it devops style mentality, customer focus and all of the other goodies that we expect in a well run organisation. Happily we found that the exercise worked out almost exactly as we'd wanted it to. With minimal prompting and poking from our team of consultants our client independently from us came to the conclusion that all of the above, and other great stuff besides, are good aspirations for their IT function to aspire to.

Conclusion

Obviously I don't have a massive sample size from which to draw experience but we did get a 100% success rate on guiding our client to the desirable conclusions. A very satisfying afternoon at the office. If I'm ever involved in a similar exercise I'll update this post.