Thursday 6 September 2018

Tech Debt Causes, Cycles and Workshop

The Tech Debt Cycle

With grateful thanks to my colleague, Vanessa Formicola, who paired with me to design this workshop and shared the facilitation.

Our current client has a lot of tech debt. And by a lot of tech debt, I mean a LOT of tech debt. 

This is a picture I knocked up for a client showcase to try to illustrate the cycle that they are perpetuating by not listening to devs and tech leads who are constantly asking to be allowed to pay down tech debt and constantly not being allowed to do it by their product people. The story of why we made this slide, and a technique that we are likely to use again, is the remainder of this post.

Technical Debt Workshop

One of the delivery teams that we work with told us that they needed to do refactor a large part of the codebase to make future development easier. So they had a debt item on their backlog "refactor this thing" which was unplayable because that part of the codebase had no tests to either assure them that it was working (before or after any refactor) or to tell them what it should do. After a discussion we agreed that there was another debt item "add tests to this thing so that we know what it does and we know that it works" that had to be played first. Thus, they were able to find a path to unlock what they wanted to unlock.

After that discussion we felt that we needed to review what people were regarding as debt, how they had it recorded, what kind of granularity was in play, whether they had a plan on how to fix things and whether they had a clear idea of what was important or valuable to fix.

We were also extremely suspicious that the product group do not understand why they should care about technical debt. We were further suspicious that the process of allocating a certain amount of time in each sprint for technical debt was being subverted unknown to the CTO and the head of product, both of whom were under the impression that this was a reasonable solution to paying down the technical debt that was in flight and working. So we wanted to illustrate why and how technical debt hurts and therefore why we should care about repaying it and not making more.

So our goals for the tech debt workshop were fourfold:
  1. Gain a common understanding of what technical debt means
  2. Try to reason out why technical debt is created and how to prevent more technical debt being created in future
  3. Raise awareness around what ill effects technical debt causes
  4. Start to capture technical debt in a sensible fashion and create a sensible process to manage it
  5. Gain alignment between developers and product managers over the importance of tech debt

The Workshop Design

We had a 2 hour session set aside for the workshop. We had a good cross section of developers in the room (a similar exercise to follow with product owners / BAs later) from junior devs up to a lead architect.

Gaining a Common Understanding

We first wanted people to think about what technical debt means to them. So we started by distributing index cards and sharpies, some of the usual tools of the ThougtWorks trade, and asked them to write down, in one or two sentences, what they think technical debt means. Pleasingly, the answers were very similar. We then discussed briefly some formal definitions of technical debt and some places (Wikipedia, Martin Fowler, Ward Cunningham etc) where further detail and ideas could be found. This part took around 10 minutes.

How Does Technical Debt get Created?

We handed out some stickies and asked the group to consider things that they, during their day to day job as developers, might do (or not do) that may lead to the creation of some new tech debt. When we framed the question we were expecting to get things like "I don't write an integration test", "I choose cut and paste re-use over generalisation" or any number of short sighted short cutting things that developers may do. We did get this type of thing in the output but we also got something else.

Why Does Tech Debt get Created?

Whether they were misunderstanding the question as we framed it, whether they were extending the scope of "you" in the "things you might do" part of the question, from just developers to everybody or whether they were simply trying to justify why they themselves, individually and collectively, had been guilty of creating new technical debt probably quite frequently, the fact was that we also got responses citing reasons why they might cut corners. We started to see some stickies citing pressure being placed on them by product, lack of resources and unrealistic deadlines. We pulled some of these out and called them "precursors" and it sparked an interesting discussion on the relationship between the product people and the developer group. This discussion surfaced a disturbing disconnect between developers and product within each scrum team. This is something that we need to look at later.

What are the Downstream Effects of Technical Debt?

We wanted the developers to think about what problems they could be causing for themselves, other developers, QAs, anybody else within the company OR end users by allowing more technical debt to be created. Our goal was to get them to think more responsibly about it. Given the revelations about the relationship with product it began to feel like we were preaching to the converted. We were already thinking at this stage that this exercise needs to be repeated with a different group.

Is There a Cycle Going on?

As we added more and more pink stickies denoting downstream ill effects we started to talk about some cause and effect relationships. We started by placing pink effects under, or near, blue causes but it was apparent that there were multiple parents and children in many cases so we started to draw arrows on the board to denote more relationships. This was an unexpected and highly valuable diversion because ultimately, amid much discussion, we realised that we were looking at a vicious cycle. In was inescapable that there was a feedback loop going on with just the awareness of debt of a small subset of the developers. This started to feel like a message we could take to the stake holders.

Managing the Technical Debt

We took a scheduled break after the pink and blue stickies. This was always the plan and my colleague, Vanessa, facilitated the next part of the workshop. The idea with the next part was to surface, understand and perhaps start to prioritise, actual debt items that could be worked on. We gave some time to them to write examples of tech debt that they know exists and that they would like the opportunity to fix. This was a lively session as well. Discussions went on about how big we were talking about (as big or as small as you like, just something you'd like to fix!), whether it was restricted to stuff they'd done or stuff that had been around forever (everything you know of please!) We coined a phrase "founders' debt" referring to the tech debt that was knowingly incurred in the early days of the company when the startup mentality meant that they didn't feel like they should care about such things. On a side note, I wish it were possible to identify the exact point where it becomes hugely important to start caring! I guess if ThoughtWorks is there helping, you've probably missed that moment.


It turned out that the debt items that got surfaced fell into 3 fairly obvious categories. Stuff to do with writing missing tests, stuff to do with large refactorings (which probably would need some missing test debt paid down as a prerequisite) and stuff to do with suggested changes to architecture.

Recording Debt

So the logical outputs from this workshop are the items of debt that need to be tackled. I have already written about a tech debt quadrant a couple of months ago (that was an earlier conversation at this same client). As far as we can see, there isn't a better way to do this. This gives us a low friction, low-fi way of recording and publicising the debt.

Culture of Adding Debt

We talked about a process whereby developers and product people discuss whether stories should go into the release process without, say, full test coverage. This is all too regular an occurrence in an environment with large piles of existing tech debt. What we managed to get to was an accord that "it would be a good idea to" have discussions about proceeding at risk, immediately capturing the debt items thus created and adding them to the debt wall. A long term aspiration would be to get to a culture where adding debt items in this way becomes some kind of shameful act to be avoided. We are very far from that place at the moment.


Discuss Things at My Level

There is still loads of confusion around technical debt. What it is, why it is important, why it should matter TO ME. It really helps with this, and most things, to get people to discuss how it will affect them. The compound interest representation of tech debt effects is useful on a high level to explain to managers why the concept should be respected but it really isn't helpful to persuade people who feel daily pain on a number of fronts. They are the ones that need to be persuaded to buy into a process, however lightweight, of doing stuff that isn't obviously directly related to their day to day work.

Make Things Visible

A lot of the conversations I had before I came to this client related to visibility of stuff. When we got here there were very few things on boards. We have changed that. We have noticed that this automatically triggers discussions and it automatically makes people talk about solutions and causes. It is a great way to get people together for a common goal. It is a great way to share success. This applies to technical debt and, frankly, any work you are doing. I can't recommend this enough. One of the constraints our client put in front of us was that the lease on this building forbids them from sticking stuff on the windows. Only internal walls can be used. We told them to buy whiteboards on wheels, at least one per team.

Alignment is always Key

Get buy in from individuals, get alignment on goals, high level and lower level. Things will only change when this happens first.