Wednesday 26 October 2016

Convincing a client to invest in the build pipeline with DAFF

The Problem

Recently we were working for a client whose IT function was displaying several common pathologies. They had (still have but it is improving as we are still there!) the classic inverted test pyramid, about 10 test teams, massive product backlogs that only ever grow, mistrust of their IT function in the wider company, no real understanding of continuous delivery... All of this adds up to a state of development stasis. Releases can take months and rarely add new value. Even when new features are added the uptake around the company is low to non existent.

The most disturbing aspect of this general dysfunction to me was the realisation that there are several different test teams comprising dozens of people who add little to no value to the company. Management has at some point created these layers of testers in response to poorly managed releases in an attempt to restore confidence in the release process. This has had the exact opposite effect as the release time has stretched out, more and more things go into a release, features and fixes are rushed in for fear of missing the release train and the situation only gets worse. Nobody has confidence in success therefore the test teams spend their working life not adding value to the process but ensuring that they and their immediate colleagues don't get blamed when it goes wrong.

They brought us in on an initially ill defined brief of "digital transformation". They had no clear idea of what that meant to them and it was pretty obvious to us from an early stage that a big part of our work was going to be not the software that we deliver but how we deliver it and whether we can use that example to help them change their processes and culture.

We wanted to run a session in the inception that would make them realise that they need to embrace a devops culture with regular releases, effective pipelines and rapid response to issues. We didn't have any experience of running such a session in the past because none of us have ever had to. Most businesses accept as fact that CI and build pipelines are a good thing.

We started with the notion that we wanted them to come to the "correct" conclusions. The client's staff had at various time expressed various emotions such as exasperation, scepticism and doYouNotThingWeveTriedThisIsm. So we were acutely aware that we wanted to have given the impression that whatever conclusions were reached were reached by them, not imposed by us.

The DAFF Loop

As ever, we deployed the magic whiteboard on the walls. We labelled 5 magic whiteboards with our 5 chosen bad events:

  • Run-time exception (this had to be explained to some. I would name this differently if I did this again)
  • Bad check-in
  • Performance bottleneck
  • Site unavailable
  • Customer dissatisfaction with digital product

We surrounded the scenario label with a circle (borrowing heavily from OODA) and labelled this with Discover - Analyse - Fix - Finalise (which I'll call the DAFF loop).

Discover-Analyse-Fix-Finalise for Bad Check In
We then armed our workshop attendees with orange stickies and asked them to describe parts of the current process by way of attaching stickies near the appropriate part of the DAFF loop. After a suitable discussion of the current processes, and I can't remember if we rotated groups or left them where they were, we asked them to add problems with the current process in red stickies. Finally the punch line was to use yellow stickies to come up with suggestions on how to improve the current process.

How did it End?

We were hoping that the discussion we had after each stage and the suggestions we arrived at would coalesce into some or all of continuous integration pipelines, build monitors production monitors etc, build it run it devops style mentality, customer focus and all of the other goodies that we expect in a well run organisation. Happily we found that the exercise worked out almost exactly as we'd wanted it to. With minimal prompting and poking from our team of consultants our client independently from us came to the conclusion that all of the above, and other great stuff besides, are good aspirations for their IT function to aspire to.


Obviously I don't have a massive sample size from which to draw experience but we did get a 100% success rate on guiding our client to the desirable conclusions. A very satisfying afternoon at the office. If I'm ever involved in a similar exercise I'll update this post.

Technical Debt Quadrant

I was reading Martin Fowler's article on the Technical Debt Quadrant, first published in 2009 recently. In it he talks about how technical debt as a term is powerful because it uses a metaphor, money, that all businesses understand.

There are, as ever, two dimensions in the quadrant - deliberate v accidental, reckless v prudent. It is easy as a software professional to see how each of the 4 types of debt in the quadrant can come about. The only section of the quadrant that requires a little thought is "accidental prudent" debt. This is explained as debt which arises because of the nature of development. We can't always know what the best design decisions are at the time that we are forced to make them. We can defer decisions until the last responsible moment but we often still will not have all of the information that we later have. Thus, prudent decisions can turn out to be sub-optimal. So at some point in the evolution of a project we can have technical debt that was prudently accumulated (because we made the best design decision given the information available at the time we were forced to make it) but accidental (because we didn't think we were taking on debt at the time).

Now, my reason for this post is the way that Martin's article ends. Essentially he says that this type of debt is difficult to explain to business because the analogy between technical debt and monetary debt breaks down. I thought about this a little while and I put forward the following analogy. In this day and age (and I accept that Martin's article was originally written in 2009, before bail-ins were a thing) it is possible to finance your company in a sensible fashion only for a financial institute to demand a bail in from its depositors (treating them as investors). Maybe I'm labouring the point a bit but would this not be a case where you incur a financial debt, or at least a liability, as a result of a prudent earlier decision?