Wednesday 31 July 2019

Quantum Computing and Me

Why Quantum?

If you are one of my colleagues or close (technology) friends you'll know that I've been researching quantum computing for a while now. It all started back at Devoxx Vienna in March 2018 when I saw a great talk by Alasdair Collinson (who I've since become good friends with) entitled "The Quantum Computers are Coming". At the time of seeing Alasdair's talk my knowledge of quantum computers was close to zero. I had vague recollections of reading something about quantum key exchange in a book about codes and cyphers years ago, but I've since learnt that isn't really to do with quantum computers anyway. I was lost pretty early in the talk and I wanted to ask him some questions and also give feedback. I thought he should try and give a little more basic material at the start to try and help people along.



Alasdair's was the last talk of the day and if you watch the video, you'll hear Alasdair mention "[releasing the audience] to the beer" at the end. Immediately after that talk was the free beer party. I didn't manage to talk to him during the party but I knew the speakers were being taken for dinner that evening so I thought I would talk to him later.

Unfortunately by the time we got to the speakers' dinner either I was a little tipsy, or Alasdair was, or more likely we both were and we didn't manage to have too useful a conversation. I remember thinking to myself "how hard can it be?" I'll do some research and learn the subject myself and maybe I can do a job of making a more accessible version of this quantum computing talk malarky.

When the conference was finishing up I had a chat with one of the organisers. He told me that he was involved with a couple of conferences in Krakow and asked if I had submitted to them. I hadn't and so he asked me if I wouldn't mind submitting. The CFPs were both closing a day or two later so he urged me to submit as soon as possible. When I got home that evening I decided to submit the same talk that had got me into the Vienna conference and, on the spur of the moment, I knocked up a synopsis of a talk about quantum computers and submitted that as well. I fully expected that the Polish conferences would pick my well known talk on Microservices if they picked up anything. As it turned out, I was wrong. And a few weeks later, having not done much about learning the subject, I suddenly had around 6 weeks to prepare a talk for Devoxx Poland on a subject I knew barely anything about. The result of my learning can be viewed here.

Learning Quantum and Duncan Mortimer

A few weeks after Vienna I was lucky enough to roll off my project and have a couple of weeks on the beach. This was my opportunity to learn about quantum things. I was chatting to somebody in the kitchen in our office and one of my colleagues, Duncan Mortimer, overhearing the conversation divulged that he was interested in quantum computers. It turned out that he was an enthusiast, had studied the field at university fairly recently (he's a lot younger than me) and we agreed to pair on a talk at the ThoughtWorks Away Day. This chance encounter made all the difference as Duncan was able to help me understand enough to cobble together a coherent story for Poland with, crucially, a very basic demonstration of quantum code using Q#. If you watch that first effort of mine at talking quantum you'll see that I used Duncan as a humorous element to essentially excuse my lack of understanding at certain points in the presentation.


A ThoughtWorks Quantum Strategy

At the Away Day I chatted with the ThoughtWorks global head of technology and in the brief chat we had he mentioned that nobody in the UK (or anywhere as far as he knew) was taking the reins of quantum and moving it forward. We need a global strategy on quantum he told me. Would I like to take that on? I told him I had no idea what this means so we agreed to have a chat afterwards. I therefore took on the responsibility of trying to make ThoughtWorks "Quantum Ready". My pitch to anybody that I could engage on the subject is that at some point in the next 5 to 10 years, quantum computers will be a commercial reality in some form. When that happens we (ThoughtWorks) need to be in a position to take advantage of the new opportunities this will create.

Meetups and Conferences

Obviously I needed to know more about the subject. I had managed to go to just one talk in London about quantum computing before the first conference in Krakow. It was a great talk at Microsoft by Dr Julie Love about the Majorana Topological Qubit. They believe that this is the technology that will lead to a stable, less error prone, qubit than can currently be realised by other techniques (of which there are many) and will therefore ultimately give Microsoft an advantage. This was the first time I had gone to a meetup (other than those held at the ThoughtWorks offices) for many years and I realised a while later that the ThoughtWorks culture coupled with this new exciting field had finally combined to reawaken my interest and love for technology that had bene stifled and crushed by the toxic culture and terrible working conditions at my previous job.



Throughout the remainder of 2018 I went to many meetups organised by the London Quantum Meetup group and got to know the organisers of that group. Every time I learnt new things I was able to remove some of the cruft about my learning journey and maybe correct some of the stuff in my original presentation that I had wrong or modify the way I could talk to parts of it to reflect my increasing knowledge. Thus my presentation became a living record of how my learning moved along and I have preserved it as it was when I presented it at various points to various conferences (something I do with all my decks).

Quantum Hackathon

I started going regularly to meetups organised by the London Quantum Computing group. If you live or work in London and you are even vaguely interested in quantum computing I thoroughly recommend you go to some of these meetups, they are great. One of the organisers happened to tell me that they were trying to organise a quantum hackathon. The idea would be that we get a group of people together to work on an organic chemistry problem (a solved problem I might add, quantum computers aren't yet powerful enough to tackle the stuff that classical computers can't solve). Two companies from Cambridge, Dividiti and Riverlane provided some open source software support and IBM were on hand to give us priority access to their IBM-Q computer. The event was a great success.


Manchester Workshop

After the success of the quantum hack day I was asked by our Manchester office if I could organise something similar in Manchester. Unfortunately this wouldn't be possible because I didn't really organise the London thing, I just (through ThoughtWorks) provided the venue and the food. I did suggest that I could give a presentation and perhaps an evening workshop on how to program quantum computers. This was enthusiastically agreed to and I fixed a date with the Manchester community coordinator.

So one lunchtime, we got together in the Manchester office kitchen and I gave a long version of my conference talk (much evolved from the original) which was very well received. The audience was mainly ThoughtWorkers with a few outsiders (it was advertised as a public event) thrown in. 

The evening was a lot more nerve-wracking. The event space was full (I was told 60 people) and this was mainly external visitors. Even though we had advertised the event as "bring your own computer and play along with the presenter" nobody seemed to have read that, or at least nobody seemed to have followed that path or installed the software (Microsoft quantum developer kit) as we had asked them to. So the result was an hour of me talking people through how to use IBM-Q, a five minute break and then an hour and a half of me explaining Q# and showing demonstrations.

Shor's Algorithm

When I gave a talk at a conference in Poland in September it bombed badly. Amongst the torrent of dire feedback were some really useful comments that I determined to act upon in future. One such comment was that the demonstrations I gave were trivial and didn't really demonstrate anything that a quantum computer could do that a classical computer couldn't. This was fair and I resolved to address it.

So in the week leading up to my trip to Ukraine I found myself implementing Shor's Algorithm from first principles. The Q# samples provided by Microsoft actually has a version of Shor but I couldn't really understand it properly and, further, I felt that it was a sub-optimal implementation because the quantum computer was doing all of the work whereas it should only be used to do step 4 (the quantum period finding routine). In my mind, as well as demonstrating the Quantum Fourier Transform (QFT), implementing Shor is a great way to showcase how you should selectively pass control between your classical computer and your quantum computer, only using the (very expensive) quantum compute power for the parts of the algorithm that can't be done on a classical computer.

On the day of the talk I had a wonderfully written, test driven, example of the whole of Shor's algorithm but using a C# algorithm to find the period. All that remained was for me to write the quantum period finding routine and plug it in. Sadly, this was much easier said than done. In the end, I had to compromise and I effectively "lift and shifted" that part of the Microsoft implementation into my own code, which included an implementation of the "Fast QFT" that I didn't fully understand. You can look at my implementation (which has barely changed since I originally made it) on my Github.  

I ran it successfully in the speakers' room about an hour before my talk and sat back in satisfaction. Then I ran it again and it failed. And again. And again. And each time the failure took ages and appeared to involve some kind of .NET kernel memory overflow. Not good. When I was close to despair, it decided to work again so I took a screenshot of the results in case, as seemed likely, it failed in the subsequent talk.

Here is the slide that made it into my deck:

Andrew Bryer

In early 2019 I found myself working in the ThoughtWorks Manchester office a couple of days a week. One of the Manchester people who had reached out to me to see if I would run something quantum based in Manchester had been Andrew. When I started working in Manchester he was on the beach looking for something to do. I got talking to him and asked if he fancied doing some research in Q# and helping me out in understanding how better to implement the QFT. He was only too pleased to help out. So I lent him a book I have, Minds, Machines and the Multiverse, instructed him to read the chapter about the QFT and help me to implement it in Q# without using any library functions. It didn't take him long.

Then, I asked Andrew to look on my Github and see what I had done in my implementation of Shor and just to concentrate on implementing the quantum piece. The first thing he did, after about half an hour if memory serves me correctly, was to tell me that he had found the issue that was causing my implementation to crash and had fixed it. He then proceeded to implement the quantum period finding routine from first principals using his own QFT implementation. I thought that would take him a while, but once we'd established how Q# can be used to link any action to a control qubit (a great feature but I have no idea how it would be supported in a real quantum computer) it didn't take him long. When I was on the train home to London from Manchester that evening, I got a message from him.

So there are a couple of morals to this story. Firstly, ThoughtWorks grads are brilliant. If you have a pure software problem to solve, even if it is in a paradigm that they had never heard of until a few hours previously, they are brilliant at solving it. Secondly, quantum computers are a long way off being actually, practically useful!

Using Many Worlds...

After doing all the research up until the back end of last year and trying to understand the "real" quantum algorithms (such as Deutsch-Josza, Shor and Grover) I realised that I couldn't reason out why they were so powerful without understanding things in more depth. I started to read more books about the fundamentals of quantum physics because so many of those "basic" concepts need to be understood to grok what is going on inside of a quantum computer. This helped me to understand up to a point and certainly helped me to understand the quantum state, which is usually expressed as state vectors, often using bra and ket notation (which I'd never come across before). This work was also, for me,  an excellent refresher course in some of my long forgotten university mathematics.

But something was still missing. I remember in the early days often hearing phrases such as "the qubit is in superposition" or "...holding both the value 0 and 1" and I think I had used similar phrases myself whilst internally holding a picture of the Bloch Sphere and imagining a thing with a state entirely represented by the quantum state arrow on that Bloch Sphere. There are at least a couple of problems with holding that view in your mind.

  • Firstly, as alluded to in my previous sentence, the Bloch Sphere is really only valid to help to understand a single qubit. 
  • Secondly, we need to be able to convey that the probability of returning a 1 or a 0 is not the full picture; we need to be able to convey that any computation that the quantum algorithm will carry out will use ALL of the possible values of ALL of the qubits that take part in the computation. 
  • Thirdly qubits interact with one another and generate interference patterns which can be analysed and exploited.

Many Worlds Interpretation

One of my breakthroughs in understanding came while I was reading (I think) The Fabric of Reality possibly at the same time as reading Introducing Quantum Theory, a Graphic Guide. I had heard and read in one of his different books, that David Deutsch was a believer in the Many Worlds interpretation (MWI) of quantum physics. In fact it could be said, if I understood correctly, that he believes that the very fact that an algorithm such as Shor works, proves that the MWI is correct. I like his assertion that whilst the Copenhagen Interpretation correctly predicts the observations associated with quantum mechanics, it doesn't explain them. On the other hand, MWI predicts observations and explains them.

Young's Double Slit Experiment

Many people will remember carrying out a version of the double slit experiment at school. This was first demonstrated by Thomas Young in the early 1800s. He devised it as a way of "proving" that light is propagated by waves and not, as Newton had suggested and was the prevailing theory of the time, by particles, which Netwon had called "corpuscles". Thus, by showing that light can be made to create an interference pattern, he showed that light must travel as a wave.

But, in the early part of the 20th century, physicists were questioning again whether this view was correct. Indeed, much of the early work in the new quantum mechanics of physics indicated that light was, in fact, propagated by particles, called photons. This led to the creation of the concept of "wave particle duality". The explanation was that as light travelled the photons would interfere with one another so that explains why the interference pattern and the particulate nature or light are not inconsistent.

So far, so consistent. But then, in 1909, Geoffrey Ingram Taylor performed the experiment with low intensity light so that photons were mostly emitted and absorbed singly such that their flight paths from emission from the light source to detection by the detector did not overlap. Subsequent experiments have guaranteed that only one photon is in transit at one time. The remarkable result of these experiments was that if you allow them to run for a time so that each photon incident on the detector has its position recorded, then the interference pattern builds up over time. So interference is still happening, even though there is no other thing in flight to interfere with. The inference drawn, if you believe in MWI, is that the photon, although apparently the only photon there, is interfering with the "ghost photons" that are taking the other possible paths in all the other universes. So far this is the best explanation of what is going on in Young's double slit experiment (single photon version). The Copenhagen interpretation has no explanation for the interference patterns in the single photon version.

Making Videos

In the interests of making my talks more interesting, I revamped the "is Quantum a thing" title into "Using Many Worlds to Solve the Unsolvable". At the same time I made a video with some ThoughtWorks colleagues in which we reproduced the double slit experiment using some crude materials and a laser pointer. That video was debuted in my talk at Codemotion Amsterdam in April 2019 (but I can't find a link to that) and I used it again in Minsk in May 2019.

Adding a live video seemed to go down really well and by summer 2019 I had done a couple of lightning talks that just included the section on Shor's algorithm with a suitable level of abstraction and simplification around the QFT. They were really popular. So much so that I persuaded Devoxx Poland to give me a 40 minute slot to expand the idea and to do a bit of work around the post quantum cryptographic landscape. For that talk I wanted to do another video, this time using my daughters to help me at home. We used polarised light filters to demonstrate how light photons are polarised and thus to lead in to a talk of BB84 quantum key distribution. This longer talk has so far only aired at Devoxx Poland and to private audiences.

Right up to Date

In July 2019, I left ThoughtWorks which was very sad (as I've noted in several places). One of my sadnesses whilst still at ThoughtWorks was that I found it very difficult to get any kind of management interested in quantum computing because it just isn't commercially interesting yet, apparently. When I came to Codurance, a much smaller and more nimble consultancy, the response I got was very different. The small commercial possibilities that did not interest my previous employer are very interesting here. So hopefully I will get the support to continue my passion for learning new things in the quantum space (and giving talks about it) as well as, hope against hope, maybe we'll manage to find some real work to do in this arena before too long. No doubt I will write about it if it ever happens!

Tuesday 30 July 2019

Tech Debt Discovery and Cost-Value Quadrants

Introduction

There are various reasons why technical debt is allowed to build up and various reasons why technical debt isn't paid down. I have previously written about this and I won't go into that again. The point of this post is to briefly discuss a method for understanding what technical debt exists, making it visible and stopping it from getting any worse. So for the purposes of this post I'm going to assume that tech debt is a thing, there are things that exist that you can call tech debt on your product and that you want to tackle them.

Cost-Value Quadrants

The idea of a cost v value quadrant is far from new. I can't remember the context of when I was first introduced to them but I've certainly used them many times to help to priortise stuff in many different contexts. The basic idea is that you have a few things that you want to compare. Each of these comes with a cost and each of them has some kind of value. If you draw two axes, one representing value (I usually go low value on the left to high value on the right) the other representing cost (I usually go high cost on the bottom up to low cost at the top, or "hard to do" on the bottom up to "easy to do" at the top) then you have divided the area into "hard to do, low value" (bottom left), "easy to do, low value" (top left), "hard to do, high value" (bottom right) and "easy to do, high value" (top right). This whole idea is not dissimilar to the Eisenhower Decision Matrix.



It is important to note here that cost and value has been the most often used things for the two axes in one of these quadrants. However I have used other dimensions for such exercises. For example, if you have a sense that there are multiple dimensions that need to be considered when comparing a load of things, it might be helpful to first decide (maybe by using sliders) what the most important two dimensions are and then place all the stuff in a quadrant that only considers those two dimensions. This can potentially help to whittle down a large number of possible things to a smaller number before applying some other criteria in turn.

Why go from High Cost to Low Cost on the vertical Axis?

Many people that I have come across have asked me why I effectively put high cost on the bottom of the axis up to low cost at the top of the vertical axis. The answer is simple, I want to focus on stuff in the top right of the resulting quadrant. For some reason I have always found it pleasing to get to a state where we concentrate on things in the top right. That is just me though, if you want to go Low cost to High Cost, feel free. You'll just then concentrate on the bottom right quadrant in a cost v value comparison.

Tech Debt Discovery Workshop

I have seen many organisations handle tech debt badly. Some places are all too ready to create it, "we don't have time for gold plating, get the thing done and we'll fix it later" is a common attitude. Some other organisations create tech debt because their engineering practices are not mature enough, or their quality engineering function isn't strong enough, to understand or prevent tech debt accumulating, "what is a contract test?". The purpose of a tech debt discovery workshop is to socialise what tech debt is, in all the forms that we know that it exists in this organisation, surface all of the items that currently exist and get a basic idea of prioritisation for them.

Setting up the Workshop

The setup of the workshop is very simple. You need a whiteboard with a quadrant on it (as above), some people who know what tech debt items that need addressing, some stickies and some sharpies. Once you've done the setup, your whiteboard should look something like this (the worried looking client tech lead running his first ever workshop and the extraneous stuff on the whiteboard are optional):

If you zoom in closely you'll see that we drew two axes on the whiteboard behind the gentleman in the pink t-shirt. Whoever is facilitating the workshop needs to explain clearly to everybody what the goal of the workshop is (in this case, discover how much tech debt we have and get a feel for what we might tackle first), and explain what the quadrant on the whiteboard is.

Stage 1 - Crowdsource the Items

As with a lot of workshops, one of the goals of the workshop is to get everybody to think they contributed to the outcome and thus get buy-in from all of the people in the group for whatever outcomes you are going to arrive out. Crowdsourcing is a great way to do this. So give everybody a load of stickies and a sharpie and ask them to write down in each stickie something that they'd like to fix that they think is tech debt. They then stick all of their stickies on an appropriate point of the quadrant. It is important at this point to stress that they shouldn't take too much time getting the thing in exactly the right place. There will be time later for a group discussion and adjustments. In my example, after this first stage the board looked like this:


Not that it is far from uncommon in such a workshop for stuff to be grouped to the right of the vertical axis. It is pretty natural for people to only think about stuff that is valuable, hence only stick things on the "high value" side of the picture.

Stage 2 - Discuss, Group and Place Items

After everybody has thought of everything that they are going to think of the facilitator needs to go through every item in turn, make sure everybody understands what it means group it with any duplicates or similar items and position it somewhere sensibly. Note that this has to be an iterative process. We are not putting an absolute value or implementation cost on each item, rather we are interested in saying "is this more valuable than that?" or "is this harder to fix than that?". Also at this stage, the discussion may well prompt more things to get surfaced or some items to consolidate into larger items. If this happens, fine, add them as you go. At the end of this stage, you should have your final session output that looks something like this:


Again, the happy looking tech lead who has just successfully facilitated his first workshop is optional

Next Steps

The next steps will depend on the situation you are in. In the case of the workshop that we ran in which I took the photographs in this post, we had not yet put anything live in our product. We were delivering an internal PaaS offering. Our priority in the early part of the project was to get things ready for the internal teams to use but not necessarily production hardened (unless it would cause undue rework to do the two things separately) and thus we were counting a lot of as yet not done stuff as technical debt for the purposes of the workshop. As we knew that we weren't prepared to go to production without a lot of these items, we made stories out of all of them and prioritised the ones that were necessary for production hardening or that were otherwise high value according to the quadrant.

Keeping the Quadrant As a Project Artefact

As I alluded to at the start of this post, and as I have written about before, some organisations have a hard time getting to their technical debt items. An anti pattern I have often seen is when teams resolve, sometimes at the behest of a CTO or senior manager, to allocate a percentage of their sprint time to debt items. This is problematic in my experience for at least 3 reasons. Firstly, to put something in a sprint plan means it must exist in your card management system and thus you are paying an inventory management tax. Secondly your devs will likely (though not always) want to work on something new rather than something seen as a boring maintenance task. Thirdly, as soon as your team comes under pressure to get all the things done in the sprint that they "committed" to, the debt items are the scope that gets cut. The last of the three is the most dangerous and that type of attitude is probably one of the major reasons why you have a load of debt to deal with. I covered this in the technical debt cycle in an earlier post.

So my favourite way to keep on top of technical debt is to keep the quadrant as part of the product wall. Every time you knowingly create more tech debt, or you discover some tech debt, write out a new debt card and stick it on the wall. Then every day at stand up or tech huddle or whatever, discuss this new thing, get agreement from the whole team that it is a thing and also where it should live in the quadrant. Adding things to the wall should come to be something that people just don't want to do or at least do reluctantly.

Tackling the Debt Wall

Hopefully the stuff on the wall is small stuff, like add one test or refactor one thing. I have had great success in various teams by doing "debt days". Either one day a week, or maybe one afternoon, everybody drops the thing they are working on and picks something off the wall. Either they pair with another developer to learn something new or just to share context or they do something really simple on their own because they think it will be fun or maybe because the thing is something that is causing them some personal pain. It doesn't matter. The outcome will either be that the thing is fixed, or if it turns out to be too complex, maybe there could be a story to play or a spike to consider or a discussion to have. In any case, everybody has a fun time, hopefully, and the debt pile is reduced.



Thursday 25 July 2019

Why Would you leave ThoughtWorks?

What is the Problem to which this is the Proposed Solution?

Last weekend I saw my dad for the first time in a few weeks and he had found out through the grapevine that I had started a new job. His question to me was, essentially, why have you left the company that you always said you loved working for?

Before I worked for ThoughtWorks, I realised that people often jump to solutions before they have fully reasoned out the problem that they are trying to solve. Also, I noticed that people from the business, depending on their level of technical (mis)understanding, would quite often come to us in the technology teams with a "solution as a requirement". For example, I can remember product owners who liked to exhibit their credentials as understanding things might come to me and say, "I have a new requirement for you. I need a new database column." With respect, I might say, that is not a business requirement. That is a solution to some problem. So very often I would say at these times "What is the problem to which this is the proposed solution?"

This line of questioning was always really helpful in the contexts in which I was in because it often turned out that when we drilled into the problem they were trying to solve, their proposed solution may not be the right one. It could even be that there isn't a problem to solve there at all. Maybe they have misunderstood the capability of the tools at their disposal. Perhaps they aren't aware of some feature that already does what they want to do.

So Why am I Looking for a New Job?

Back in May I started to look for a new job. The weird thing was when people asked me why I was looking for a new job was that it was hard to say exactly why. Most people, correctly, pointed out that I really liked working for ThoughtWorks. There is no doubt that it was the best job I'd ever had and the best employer I'd ever worked for. Certainly this is still the case (up until June 2019) and that didn't change drastically. So why was I leaving?

ThoughtWorks History

ThoughtWorks was founded in 1993 by Roy Singham. The company has always had a very strong and unique culture. In particular it has a 3 pillar model based on, I was told, the Ben and Jerry's 3 pillar model. Pillar 3, (referred to internally as P3) states that the company strives for Social and Economic Justice. The way this objective played out varied over time and was always subordinate to pillar 1 (sustainable business) and 2 (technology excellence) but was always used at the very least to inform decisions. For example, there would often be internal debates over whether we should be doing work for Company X or Corporation Y. This debate was always healthy and open.

Activism and P3 Work

Throughout my time at ThoughtWorks Roy was actively engaged in social activism in various places in the world and, as the only major shareholder holding over 99% of the company's equity, removed earnings from the company to fund his activist work. This was well known throughout the company and, whilst you could agree or disagree with Roy's specific politics, nobody seriously challenged the validity of striving for social and economic justice and as far as I could tell, most of us bought into this.

From time to time in various geographies, I saw this happen perhaps 4 or 5 times in London during my 4 years, we would carry out work that we would call "P3 projects". This could be work for a non-profit organisation, for a charity or perhaps for startups that were well aligned with ThoughtWorks culture. The way this happened varied but what I saw in London a couple of times was that we would carry out the work at our offices, staffed by perhaps one or two experienced ThoughtWorkers and several grads. This work would be carried out at hugely discounted rates. We didn't make money on the engagement but some of our less experienced people got valuable experience and we furthered the company's P3 goals.

Takeover by Apax Ventures

In autumn 2017 all ThoughtWorks employees were gathered (I think) simultaneously to hear an "important announcement". I was working on a client site in Glasgow at the time and we were told the day before not to go to the client that day but instead to go to a nearby hotel where we had hired a conference suite. As far as I know the news wasn't leaked beforehand but there was certainly a lot of speculation as to why we were being told to do this and to my recollection there were two strong favourites, one was that Roy was selling, or had sold, ThoughtWorks and I can't remember what the other possibility was.

We learned by way of a pre-recorded video by a member of the Global Coordination Group that ThoughtWorks had indeed been sold to a private equity fund managed by Apax Partners. This was a shock to all ThoughtWorkers and can only be described as a watershed moment.

Saying all the Right Things

Of course we were concerned about what would change. The emphasis of the company was bound to shift wasn't it? Whereas previously we had all been happy to buy in to the fact that our work with our clients was ultimately being used to fund the global P3 activities and Roy's activism (while the company grew organically) it wasn't now clear if that would continue to be the case. Certainly the profits would no longer be used directly by Roy but we were all concerned about what may change.

As the soul searching continued for days and weeks the thought that kept coming back to me was that these people (Apax) aren't stupid. They know that the company has no assets, nothing of intrinsic value to strip. The continuing success of the company relies entirely on keeping the people it currently has and in order to keep those people the company cannot materially change its outlook or its culture. Ergo, my reasoning went, things should not change. Sure, they might alter some management structures, sure they might use their network to introduce potential new clients at a level of executive sponsorship that we may not have been able to achieve previously but surely wholesale change cannot be on the cards.

The ThoughtWorks Salary (anti)Premium

ThoughtWorks traditionally did not pay the best wages. When I started at ThoughtWorks back in 2015 not only was my salary lower than the role I left but it was considerably lower than at least one other job offer that I had at the time. But I wanted to work for ThoughtWorks. And while you work at ThoughtWorks you know that you could leave the company and enjoy a payrise but you are happy to get paid less because of all the great reasons for working at the company. 

In the two years since the takeover they were largely true to their word and nothing major changed. Indeed the changes were in some cases so subtle, and often superficially beneficial to the overall business, that their significance was lost on ThoughtWorkers who had joined post takeover. However, those of us that had been around for any length of time saw these changes creeping in inexorably and understood very well the ramifications of some of them. It would be wrong of me to point to anything specific in this post and indeed, I would still argue that it is a fantastic place to work and I was very sad to leave. 

The best way I can rationalise my decision to leave, therefore, is by saying that each time something seemingly insignificant changed, the gap between what I could earn outside of ThoughtWorks and what I was prepared to accept as a fair wage within ThoughtWorks narrowed. Unfortunately there was no corresponding narrowing of the gap between my actual earnings and the outside world so eventually I reached a tipping point, I'm not sure exactly when that was, but at some point I felt that I was no longer prepared to pay the premium being demanded.

Conclusion

ThoughtWorks is still a fantastic place to work. I still recommend it to anybody that asks. I continued to speak very highly of ThoughtWorks at conferences even as I knew I was falling out of love and jilting it, and will continue to do so. The danger it is facing is, I think, existential, and the management of the company (at least in the UK) is in danger of sleepwalking in to a situation where sufficient numbers of tenured ThoughtWorkers leave the company so that the culture is irreparably damaged. I hope that doesn't happen and I hope the danger is understood and dealt with by the leadership teams because I think it would be a crying shame if ThoughWorks were either cease to exist or if it changes to the extent that it is unrecognisable from everything that has made it great.

Friday 19 July 2019

Four Years of Reading Stuff

Before Summer 2015

In the last 4 or 5 years that I worked for a startup I fell out of love with technology and learning in general. This was mainly because I came to hate my job. I saw it as a means to paying the mortgage and nothing more. There are many reasons why I got to that state of mind which I won't go into here. A big side effect of this state of mind was that I read virtually no non-fiction books. Looking back on that period of my life, this is the only time that I let myself get into a state of mind where I wasn't acquiring new knowledge.

I would have to ask for the smallest sums of money for a book that I may think I need. It would only get approved grudgingly and definitely only if it was seen to be directly relevant to something that we were doing to earn the company money. General learning and improvement was allegedly encouraged but in reality it was strictly in your own time. I remember even having to take a day off work to go to a training day to learn about a technology that we were trying to roll out.

Starting a New Life

So when I started as a consultant the genuine culture of learning that I found was fantastic. In my first few days in the job, sitting on the beach (other consultants call it the bench) talking to random people, it felt like I learnt more than I'd learnt in the previous 5 years put together. I was told that I had a budget to use for training and if I wanted to buy a book to read, provided it could be liked to any kind of learning that I wanted to undertake, it was fine. As far as I know, nobody ever really audited what books you bought. The result of that was that nobody abused the system, everybody bought stuff they wanted to learn about and virtually all learning was considered valuable to our work as consultants. So I started to read again and I haven't stopped yet. 

Many new starters and graduates asked me for book recommendations and from time to time I produced a list of everything I had bought and read (or sometimes given up on) during my time. I always found it good fun to go through my list of purchases on Amazon and give a one line review of that book. I updated my list from time to time and provided it to anybody that asked. Unfortunately I forgot to export that list when I left ThoughtWorks so I decided to write it up here to share with anybody who might be interested.

Four Years of Reading (or not)

Here is a list of all the books I bought in my first four years working as a consultant in a true learning culture...

The Innovators Dilemma

The Innovator's Dilemma is a business classic. It explains how seemingly established and well run companies can be very quickly put out of business by not being nimble enough to react to innovations in their business space. It has detailed case studies of the mechanical digger industry in the middle part of the 20th century and the hard disk industry from the second half of the 20th century as well as some examples of specific companies. Corporate failures following the innovator's dilemma pattern, and therefore validating the theory, since this book was originally written in the 1997 include Kodak and Blockbuster Video. This book is a must read for everybody interested in how modern businesses should deal with innovation in their industries.

The Whitehall Effect

The Whitehall Effect is a study of how public sector industries the world over fail to deliver value to the tax payers they serve. There are many reasons why this can happen but the patterns of failure are repeated all over the world with depressing regularity. This book is a must read for anybody who may be planning on consulting into the public sector.

Creating and Delivering Your Value Proposition: Managing Customer Experience for Profit

I bought this book just before I was due to start work on my first big gig at ThoughtWorks. In particular I wanted to understand a bit more about what outcomes we might be driving towards in the inception. It was a good easy read with some high level concepts about what a value proposition is and how to help shape it. 

Release it! Michael T Nygard

This book is a MUST read for all software developers. It is easy to read with some great (slightly dramatised) descriptions of real incidents that happened to the author to kick off each chapter. This is essential reading to get a feel for what it means to be production ready and what it means to embrace a devops culture.

The Hacker Playbook

A massive disappointment. I wanted a book to give me some pointers on app sec concepts and this sounded like it would. In fact, when I tried to read it (I had to give up halfway through) it felt like it had good messages but that it had been edited by two blokes in a pub at the end of an all day session. Avoid this one.

The Innovator's Solution

The follow up to the Innovator's Dilemma, first published in 2003. This describes how the author worked with some massive companies after the publication of the first book and helped them avoid falling foul of the innovator's dilemma. If you read the dilemma, you should read the solution.

The Evolution of Useful Things

This one was a fun read. I saw it on Amazon marketplace for 1p and it sounded interesting. If you're interested in product design, try and find a copy (I think it may be out of print).

AWS Scripted

This is an early book about writing infrastructure as code specifically around the AWS SDKs. I'm not sure if it would still be in print but I'd say it is obsolete now anyway.

The Linux Command Line

I bought this one because I wanted to learn Linux. I worked through some of the early chapters when I was on holiday one time but never managed to finish it. It was pretty easy to read but actual work got in the way of me finishing it. If you want to learn about the Linux Command Line from beginner level it is OK.

The Chimp Paradox

This is a pretty famous book about psychology written by Steve Peters who has worked with many elite sports teams and individuals. If you're interested in why we do seemingly irrational stuff at times and possibly how to attempt to control your own irrationality, this is a very good and very readable book.

Difficult Conversations

This is a good book about how to approach difficult conversations. I found this hugely useful as a consultant and for advice on how to tackle difficult conversations with friends and family.

The New One Minute Manager

Very easy to read, very quick to read (I read it in a single short plane journey) short description of how to be a non-micro-managing, non overbearing manager. If I remember rightly the message I took from it was that managers should not be managers so much as facilitators and advisers.

Clojure for the Brave and True

I wanted to learn Clojure and this was recommended to me. It is a good learning tool with nice exercises and some humour (which eventually grates a bit) thrown in. It does make a good effort at explaining the philosophy of Clojure rather than just the nuts and bolts of do this, do that. If you are want to learn Clojure I would recommend it. There is an online version of this book that is available free although, from memory, there are reminders throughout that they would like you to buy the actual book.

Land of Lisp

I bough Land of Lisp at the same time as the Clojure book but I never got round to reading it.

Drive by Professor Daniel Pink

This book is a must read. Pink spent a lot of time investigating what motivates us to work. His conclusions were counter intuitive and controversial at the time. As a consultant reading this book made me realise that I had to understand people's motivations, hopes and fears in order to understand how better to help the organisation move forward. I realised that it isn't enough to put process in place, you have to link that thing that you are asking people to do to what motivates them to work. I thoroughly recommend this book to any consultant that has designs on being an influencer rather than just an implementor.

Javascript - The Good Parts

This one was recommended to me as a good book for learning Javascript but I've never got round to reading it.

The Pragmatic Programmer

This is a classic which, until I was going through my Amazon purchases writing this, I didn't realise I had. Thus, I haven't read it.

The Mythical Man Month

Another classic which contains some proto-Agile thinking and some explanations why traditional project management thinking doesn't work with software delivery. It is amazing that such "traditional" methodologies still exist. Everybody working in software delivery should read this.

Also the source of a classic Twitter gag that loads of people have recycled and repurposed down the years. No idea if the one I've linked to is the original or not. It was just high up the list when I Googled it.

Continuous Delivery

This is Jez Humble's manual on how to do continuous delivery. I would hope that for experienced engineers that this serves as a reference book on how to do things right. For consultants trying to get our clients to a better place this book is a useful thing to have lying around at the client site to encourage people in the right direction. For any grads or apprentices who are great at writing software but lack experience on how to execute software delivery this is an absolute must read.

Working Effectively with Legacy Code

This is a must read for anybody working on a project that is based around legacy integrations or any organisation that has tons of legacy. The definition of legacy is interesting. Feathers tells us that legacy is anything without tests, the age of the technology is irrelevant. Basically this book is great and everybody should read it.

The Elements of Computing Systems

This was recommended by a colleague when we were in the pub once discussing hardware architectures. As I write this I realise I bought the Kindle edition but it somehow isn't on my Kindle. I just downloaded it and discovered I've read the first chapter only. Must do better...

The Phoenix Project

This is a famous Agile Parable. I had actually read this several years earlier when somebody bought it for me for my birthday. I include this in this list because I bought a number of copies to hand out to the client developers at the place I was working after a talk I was giving on Agile principals. If you haven't read this, you should. It is a description of an Agile Transformation squashed into an unfeasibly short time frame to help the narrative of the story. Read with a pinch of salt on the timeframes but not on the lessons contained.

The Goal

Eli Goldratt's famous novella explaining the Theory of Constraints. Similar to the Phoenix Project in style (although published much earlier), it can be regarded as a parable of Lean manufacturing. It is very easy to read and has many valuable lessons about why we do things the way we do in Agile delivery.

It's Not Luck

The sequel to the Goal follows the same character as he is promoted and takes on responsibility for whole companies in the group rather than the single plant he had at the end of The Goal. There are valuable lessons in here on how to approach problems and how to reason towards solutions. The Socratic method is mentioned as a reasoning framework.

Conversations of Socrates

I wanted to read this because of references to it in the Goldratt books. I have only read the start of it and I found it hard to read because the test was too small. I have just rebought it on Kindle so that I can properly read it.

Toyota Production System

This is, straight from the horse's mouth, the description of how the TPS, which underpins much of modern Agile and Lean theory, came about and continued to evolve. It is a fairly short book and everybody needs to read it.

Socrates' Defence

This is a translation of Plato's (he was a disciple of Socrates) version of his time with Socrates. I bought this in paperback around the time that I was discovering that I can no longer comfortably read "real" books. I've just rebought on Kindle...

Socrates' Way

This is a bit more of a practical guide on how to apply the Socratic method. I did read some of this but then I couldn't find the book to continue.

Theory of Constraints

This is a more formalised, academic, treatment of the theory of constraints. I haven't read beyond the early chapters yet.

Godel, Escher, Bach: An Eternal Golden Braid

This is a great book by Douglas Hofstadter, the King of meta. I have read extracts from this from other sources and seen conference talks, including this one from my ex colleague Chris Ford, inspired by this thinking. Sadly the text in the edition I have is so small that it can only be read by ants using binoculars. I have thus utterly failed to read beyond the introduction of this one.

Presentation Patterns

I bought this because I was doing a lot of presentations for conferences and wanted to know if I was missing anything, doing stuff wrong or right. If you want some pointers on how to create and give presentations this is a good place to start. I recommend it.

Clean Code by Uncle Bob

If you haven't read this, you should. Simple as that.

The First 90 Days

Another one that was recommended to me that I bought in paperback before realising I can't read them anymore. 

Computing With Quantum Cats

This is the first book I bought on Quantum Computing when I was learning about it last year. It is easy to read and informative. I thoroughly recommend it.

Minds, Machines and the Multiverse

Another quantum computing book. This has a bit more practical detail on quantum computers. I used this as my reference point when I was working with a ThoughtWorks grad in Manchester on implementing Shor's Algorithm and the Quantum Fourier Transform. It is a good book for sure.

In Search of Schrodinger's Cat

From the same author, John Gribben, as Computing with Quantum Cats, this one is an explanation of the history of quantum physics. If you are serious about learning about quantum computers I suggest this as your starting point (before the above two). I eventually bought this when I realised that I needed a grounding in quantum mechanics in order to attempt to better understand quantum computing. Very readable and I thoroughly recommend it.

Accelerate

This book is great and pretty easy and quick to read. This is from the people who give us the State of Devops report and can be considered as an invaluable reference for transformation type work. The ThoughtWorks Tech Radar now has the 4 key metrics on it in Adopt, meaning that they believe that everybody should be using them. On the last ThoughtWorks gig I was involved in we were pushing for the client to start using the 4 key metrics as their only measures of progress and we were using the 24 capabilities outlined in the last chapter of the book as our tools for moving the needles on the 4 key metrics. This book is essential for any IT professional.

The Garden of Forking Paths

Not really a technology (or business) book. This is a collection of short stories translated from the original Spanish. I bought this because one of my quantum computing books recommended it as a thought provoking idea to help with reasoning with the Many Worlds Interpretation. It only cost £1 so why not?

Mathematics for Quantum Mechanics

I bought this one when I was first attempting to write a quantum simulator and therefore needed to understand the maths better. It is hard going, the typeface is scratchy (it is quite old) and hard to read and the material is dull without enough contextual explanation. Avoid it, read the next one instead.

Quantum Mechanics, the Theoretical Minimum

I bought this at the same time as the one above. It does a much better job. Still pretty hard going with some fairly advanced maths but much more readable.

The Beginning of Infinity by David Deutsch

I bought this expecting a book on quantum physics or quantum computing. Instead I got a brilliant thought provoking book about philosophy, epistemology and various other things into the bargain. It also helped me to form my theory of studying organisations through organisational memes (after I had also read the Selfish Gene). I'm not sure if it is directly relevant to our work as consultants but I thoroughly recommend it as a book.

The Fabric of Reality

The much more famous book by David Deutsch. If you are interested in BIG subjects then this is a great read. Again, not much in here directly relevant to our work but certainly worth reading.

Enterprise Agility

This is written by one of my ex colleagues at ThoughtWorks. It is a great reference on why organisational agility matters, why we shouldn't be happy to just consider Agile as a methodology to help with software delivery and what it means to be agile (small 'a') in the modern business. I read this when I was first doing my talk Agile is a Dirty Word and most of the content was highly resonant to me as a consultant mainly working in transformational roles. I guess that shouldn't have been much of a surprise given the author was a colleague.

Bullshit Jobs: a Theory

I have long believed that organisations invent jobs to keep people employed. Large organisations evolve teams that add no value but have a sense of self preservation and survival. This book was recommended to me by somebody in a pub once when we were discussing this issue. It is easy to read and contains quite a bit of ironic humour that is right up my personal street. If you are interested in understanding where organisational dysfunction comes from, I recommend this one.

Evolutionary Architecture

Written by three of my ThoughtWorks ex colleagues, this book is a must read for anybody with responsibility over any large system. The main takeaway I got from it was the methodology, fitness functions, of actually measuring the effectiveness of your architectural decisions through their effect on those numbers. This book and Accelerate formed the basis of much of my thinking in the conclusions to my talk on Architects and Architecture that I first gave at Devoxx Poland in June 2019 (no video yet posted at the time of writing this). Everybody should read this.

Surely You're Joking Mr Feynman

I bought this because it was written by the famous physicist Richard Feynman and I thought it was a popular science book. Actually it is more of an autobiography in the form of several anecdotes and short stories of periods of Feynman's life. It was hugely entertaining and I recommend it. Absolutely nothing to do with our work though.

New Theories of Everything

A popular science / cosmology book. If you like A Brief History of Time and its ilk, you'll probably like this. I enjoyed reading it.

The Selfish Gene

Richard Dawkins' book propounding his new (when he wrote it 40 years ago) theory of evolution placing the gene as the unit of reproduction, not the organism that carries it. A thought provoking read that I was directed toward by reading The Beginning of Infinity by Deutsch. I used ideas in this book and in Deutsch' book to form my conjectures around organisational memes and how they can evolve into long lived pointless activity that we sometimes characterise as Risk Management Theatre. I haven't yet developed that theory far enough to write about it. It is a great read and I thoroughly recommend it although I'd struggle to link it to our day to work.

Joy Inc

Joy Inc is a book about building a workplace that people enjoy working in. It reads a bit like an Agile case study, a bit like a cautionary tale of badly managed businesses and a bit like an advert for the author's company. It is well worth a read and I recommend all people who are trying to get traction for Agile thinking should read this and recommend it to sceptics.

Infrastructure as Code

Written by a good friend of mine and ex colleague, Kief Morris, this is the de facto standard book on Infrastructure as Code. If you haven't read it and you are, or will be, working in a team that is responsible for infrastructure, then read it. Definitely a must read.

Reinventing Organisations

This was recommended to me by Steve Lydford, head of PS at Codurance, when I first spoke to him on the phone before I came here for an interview. The book examines various types of organisations from early human civilisations up to modern businesses. It proposes a new type of organisation a Teal organisation, in which there is no set hierarchy and units self organise. There are several strong case studies throughout the book and it is packed with anecdotes explaining aspects of the theory throughout. I really enjoyed this and I thoroughly recommend it. In terms of our business context as consultants I felt myself mapping a lot of the organisational principals to Agile principals and I felt like I've seen many of these things happening but generally only within an IT function, not a whole business. This book effectively argues that such ideas shouldn't stop in technology but should be allowed to permeate everywhere.

The Theoretical Minimum

From the same author as one of the quantum books I'm reading. An explanation of classical physics. Haven't read it yet.

Special Relativity and Classic Field Theory

Again from the theoretical minimum series. Again, haven't read it yet.

The Software Craftsman

And finally, the last book I expensed to ThoughtWorks was ironically written by Sandro Mancuso of Codurance. After I interviewed here and met Sandro I thought I'd read Sandro's book. I'm pleased to say I thought it was excellent. If you work for Coduracnce, you'll have been given a copy of this book. If you don't work for Codurance I recommend you read it.

Tuesday 9 July 2019

On Leaving ThoughtWorks

Back in 2015...

Back in 2015 I was looking for a job having been made redundant from my previous job at Viagogo.com. Fortunately, I was left pretty well off financially from the redundancy, having worked there for 9 years, so I wasn't under any kind of immediate pressure to find a job. I had, to my recollection, about 4 months' of runway before we'd struggle to pay any bills so it felt like plenty of time.

There were two important side effects of that runway. Firstly, I could afford to be choosy about what to accept and secondly I could afford to go to some interviews just to get interview practice. It was a long time since I had looked for a job previously and the world had changed a lot between 2005 and 2015. In the end the runway worked out well for me. I was able to delay the time I went to the interviews at ThoughtWorks until after I'd had plenty of practice and I was also able to understand by then a bit better about where I might stand in the marketplace.

Where I was left behind

The previous time that I had searched for a job you handed out paper CVs or emailed them to people and they got passed on. There was no expectation for anything else. Some people blogged back then but it wasn't a massive thing. In many circles it felt like a bit of a self indulgent thing to write online articles that few people read. If you weren't Martin Fowler or Uncle Bob, why would you waste time writing for your audience of very few?

But in 2015 a recruiter friend of mine asked me to update my CV with details of my online presence. "Online Presents?" I asked. What does this mean? So she explained that I needed to include my LinkedIn contact details, my Twitter handle, the URL to my website (or blog) and my Github. The only problem with that was that I had none of those things, except the Twitter handle, and I wasn't really keen on sharing that as I had only used it up until that point to moan about stuff that I came across in life. Quite simply I had no online presence remotely related to my job as a technologist.

Somehow, though, despite this lack of a modern CV ThoughtWorks did offer me a job and I spent a very happy 4 years working there until last Friday. I took onboard the notion of having an online presence and one of the first things I did was to buy my name as a domain and set this blog up on that domain. It took me nearly 3 months to do (I think I was busy learning new things) but eventually my first post, Becoming a ThoughtWorker went live in September 2015. I also signed up to LinkedIn (which made it a log easier to find a job this time round) and signed up to Github, although only when I started experimenting with quantum computing did I put anything at all worth looking at on there.

Moving on in 2019

In my 4 years at ThoughtWorks I learnt more than the whole of the rest of my working life put together. I learnt at least 5 new programming languages, I got a wealth of experience working in different environments in various stages of dysfunction, I met dozens of fantastic people that I hope will remain my friends (I actually tried to enumerate them all but I realised I was missing people out and I gave up when my list got to about 130) and above all, for the first time in at least 10 years, I really enjoyed work. For the first time in my life I realised that getting paid for work is but a small part of the reward we should be looking to get from our work, provided we are lucky enough to work in an industry, at a level of seniority,'' that means that we earn "enough", whatever that means.

At some point though, you realise that you need another challenge. ThoughtWorks was by far the best job I've ever had but I got to a stage, probably last year sometime, when I realised I was finding it hard to always get the kind of the work that I really enjoyed. So I started to look around to see if there was anything out there that might help me to enjoy my working life even more than I was doing at ThoughtWorks. I must stress that at no point did I find the work frustrating or boring or unfulfilling. I just felt that I would like to have just a little more influence on the bigger picture. That was never going to happen in ThoughtWorks with it being the size that it is.

As a member of PS at ThoughtWorks, I was only ever peripherally involved with helping to sell our services. I did enquire some time ago as to whether there were openings in our sales effort for technical people to work in a sales support capacity but at the time our sales effort in the UK was being restructured and I was heavily warned by senior ThoughtWorks people to not go "post technical". I did help put together pitches from time to time when I was on the beach and I did get to attend some presentations but it was never more than an occasional, very ad hoc, who's on the beach today, kind of thing.

So I looked around for what the job market had to offer and the big difference between 2015 and 2019 was that I had an online presence. Not only do I have a passable blog site with some reasonable content (if feedback is anything to go by) but since autumn 2017 I've been a pretty regular speaker on the technology conference circuit which means there are a load of videos online of me expressing my opinions and sharing what I've learned from being a ThoughtWorker. The latter thing in particular opened many doors for me and I on a few occasions, interviews started with me being told by somebody that they had watched one of my videos. So all that online presence stuff really works!

Where Next?

So after my time at ThoughtWorks I spoke to a few different places, found some interesting stuff, some not so interesting stuff, found as many bad interviews this time around as last time and eventually realised that the thing I really wanted to do was to work in consultancy but at a much smaller place than ThoughtWorks. So today, I started as a principal craftsperson at Codurance. I'm hoping to still do all of the stuff I loved doing at ThoughtWorks but hopefully some of the stuff that I wanted to do and never got the chance. So in other words, and hugely optimistically, I want to have my cake and eat it. Hopefully that will happen, no doubt I will be posting a blog entitled "Becoming a Craftsperson" in a few months' time.

Wednesday 3 July 2019

Outcome over Process

The Agile Manifesto

The Agile Manifesto, amongst other things, tells us that we should value "Individuals and interactions over processes and tools". The Agile Manifesto is still hugely relevant (even if the principle about regular delivery says "from a couple of weeks to a couple of months") and is not often enough referred to. I have often called out in my talk Agile is a Dirty Word that many so called Agile practitioners have never even read the manifesto, let alone understood the values and principles or attempted to get their teams to buy into them. I like to map the values and principles onto whatever my teams are doing so that they can come to their own conclusions of what is valuable based on their own context. One thing that comes up again and again is that people should value Outcome over Process.

Process over Outcome Anti-Pattern

Organisations can get into a comfort zone where people follow a process without understanding what important outcomes the process is intended to support. This is an anti-pattern. At some point, somebody, somewhere said something like "we have this problem, this process will fix it." There is a chance this person could have been right. However, many months or years later when this process is still hanging around people have lost sight of the outcome. Existing people either forgot the outcome, or never knew it or, in many cases, the outcome being driven is entirely different. So when new people join the team they only hear about the process and usually they never ask "why do we do this?" because they don't see it as their job to do that.

Risk Management Theatre

I have previously written about Risk Management Theatre in this blog. This is the end state of process over outcome. In the worst cases, the outcome of some process is not only different from what it was originally but can be obstructive to that original outcome. I worked for a company last year who was stuck in an old fashioned fear cycle of long delivery cadence. Each two weeks they had a meeting, involving around 30 people for an hour, called the "Go / No-Go Meeting". On the face of it, this meeting is there to decide whether the release should go ahead or not based on whether it is considered too risky. In fact, after observing the meeting I could see that they never once said "No-Go". So the real outcome of the meeting was that all 30 people shared the blame when the release went wrong as they knew it would. Hence, no risk was being reduced but the staff had the comfort blanket of knowing they couldn't be sacked.

In the worst type of organisations Risk Management Theatre can be so rife that there exists whole teams of people whose day to day job consists of ticking boxes on some checklist to make sure that they don't get blamed when things go wrong. It isn't hard to see how much real value that adds.

A Real World Example

If anybody has any doubt that Risk Management Theatre is a real thing, not just in technology but in all walks of life, here is a real example that I came across while walking with my girls into town a few months ago. As we walked past a new(ish) block of flats, actually the old Kodak headquarters which has been redeveloped, we saw this:


If you look closely you can see the effort that somebody went to in order to, presumably, "make the area safe". The three pieces of fencing in the foreground are carefully connected to each other and the one at the top of the stairs is connected to the metal railings on either side by those plastic ratchet connector things that are so strong that police use them as quick attaching handcuffs.

If you look REALLY closely, you'll see a minute bit of broken glass under the bush in the background. Not on the path where anybody will walk but under the bush. This type of stuff baffles me. I assume there must be a rule somewhere that says something like "when you find broken glass make the area safe so that nobody can injure themselves". This has led to somebody going to enormous trouble and expense to do this when they could have simply cleared it up in seconds. This is a classic example of people letting the tail wag the dog - the process of securing the area is obscuring the outcome of it being safe for people to use.

Ways of Working

So why am I rambling about Outcome over Process at this point? Well, I joined a new team a few months ago which was billed as being very technically able but which was unable to get things to move across their board. There was already a ThoughtWorks senior developer in the team and a ThoughtWorks principal setting technical direction but not involved too heavily in day to day work. The ask was to come in as "iteration manager", I still don't know what it means exactly but it is the second time I've been staffed by ThoughtWorks into that role, and help to get the work understood better and flowing better.

What I found was pretty much what had been advertised to me. There were 5 client developers who were all competent but no real visibility, what passed as a board didn't reflect what was going on in Jira nor what was actually happening, and absolutely no understanding of flow and why that should be valued. The TW principal asked me after a day or two what I thought and I told him exactly what I thought. "Great", he said, "can you run a Ways of Working session to get some process around this to get it moving then please." This was the point at which I pushed back and put into practice what up until this point had been a conjecture that I had been nurturing for some months.

Slowly Slowly...

I reasoned that I shouldn't try and change all the process at once, as I laid out in my previous post, and crucially we weren't under immediate delivery pressure, nor would we be for a couple of months at least. So I told Matt (the ThoughtWorks principal) that it was my belief that we should not try to change everything, rather we should be changing things little and (fairly) often. I proposed to start a "ways of working" session in week 2 but actually I said I don't like calling it that because I don't want to talk about roles or rituals in the team before we understand what outcomes the team values and whether we should be working on them. 

So in week 1 I ran a Goals and Values workshop to kick the team off into the habit of socialising and hence (hopefully) buying in to decisions. At the end of the week I casually suggested that we try a new style of standup which I called "story centric". Most people call it "walk the wall" which I proposed would better serve our needs than the traditional, "yesterday I did, today I will..." type of standup. I reasoned out loud to the team that I trusted them and that, given that I trust them to do the right thing, I don't really care about what they did yesterday. What I really care about is when our stories are going to deliver value to the company, that is to say, when they will be done.

The Big Idea - Process Driving Outcomes

My idea was that I wanted to get the team to talk through all aspects of their process in a timeline fashion. I could have sold this as a "path to production" exercise but when I've been involved in that in the past it has been to identify inefficiencies in engineering practices, coupling points (and handovers) and opportunities to improve quality. In the case of this team I was trusting that their engineering practices and quality control were good enough, at least they weren't our biggest problem at the time, and I really wanted them to start questioning why they did all the things they did. 

Part 1 - The Timeline

I used two sheets of magic whiteboard and drew a line on the left labelled "story picked up" and one on the extreme right saying "done". I explained that time is flowing from left to right. I asked them to put stickies on the timeline. Each sticky could represent something that happens, a ritual, a conversation, a meeting, writing code, reading stuff... ANYTHING to do with the process of getting that thing to done.

This took a while and there was interestingly quite a lot of debate as people added stickies at different points and discussed whether it was meant to be that way or this way or whatever. A couple of times I was asked, "should be what we're meant to do or what we actually do?" That was an interesting question. What struck me was the confusion over the process and, even without explicitly raising it which I intended to do later, confusion over why exactly things were the way they were.

Part 2 - The (disingenuous) Discussion

When we had all finally finished putting "event" stickies on the boards I told them I wanted to understand all of what was there. At this point I was being deliberately disingenuous at times. This team did not know that I was a software developer, they thought I was a project manager. So I exploited that by asking questions like "what does branch mean?", "what do you mean by merge?", "oh, you have to wait a day for a code review? Have you considered sitting with a person to do that together?" My plan was to tease out some discussion about how silly some of the processes are and how they seemingly (to the pretend ignorant) return no value. At this point we had a load of green event stickies and a load of small pink "notes" stickies at various points.

Part 3 - Outcomes Supported

The main bit I wanted to get to was a conversation about what outcomes were important from each part of the process. I wanted the team to start thinking in terms of outcomes and I wanted to start to get them understanding that flow is important. I was expecting debates around the value of various parts of the process and we certainly got that. It went probably better than I expected. 

Context Sharing

The outcome that appeared the most on the picture was "context sharing". Apparently many of the rituals were to share context. Many of the things that happened were around sharing context. Sharing context was apparently really important, they all value it but apparently they don't necessarily value it enough to do it earlier or more effectively, perhaps through pairing instead of code review.

Code Quality?

Interestingly, at no point in the process was ensuring code quality called out as an explicit outcome. I expected somebody to claim that code reviews are there to support this outcome but nobody claimed it. I suspect that they were already accepting that code review is a form of risk management theatre.

The one nod to this idea was that at one of the points in the process it was agreed that "code fit for purpose" was an outcome that was being driven. Interestingly, it wasn't clear exactly which part of the process did this - is it QA (very ill defined in the case of this team) or is it code review or is it at some earlier point? It also wasn't clear, even after a lot of discussion, what exactly "code fit for purpose" means. What was clear was that we wanted to drive this outcome somehow. So we agreed to revisit both the definition of fit for purpose and how we might drive that outcome, later.

Communication

Confidence was mentioned as an outcome in the QA and UAT process (which the team admitted wasn't actually being followed) and expectation management was also mentioned as an outcome. Advertising was called out as an outcome against "demo" or "showcase" and there were a few discussions about how we might facilitate communication with the rest of the company, in particular those teams that we are considering to be our primary customers.

Interlude - The Story so Far

This was two or three weeks into my time with the client. I must stress at this point that we had already made some significant changes to the team ways of working. I just hadn't done a "Ways of Working" session as I had been asked, for the reasons set out above.

Stand up

As mentioned above we changed the stand up style at the end of my first week. The team were buying into this change and seeing its value. It was helping us to get focused on what was blocked and what we might do to unblock them. It was also starting to get the team focused on the importance of flow (I kept using the word "flow" incessantly in those early days). The change to stand up was the ONLY change that I "imposed" on the team. I simply asked them if it was OK and they agreed. At that point I think they still had a bit of apathy for things so they probably would have agreed to anything I suggested.

Tech Huddle

At the end of my second week at the client we got an external facilitator to facilitate a "Lego retro". Before the retro we established that the retro was aiming to just get a single retro action. So I discussed with the other ThoughtWorks people what we thought was the biggest problem currently. We felt that the team were not getting a shared understanding of stories and this was causing problems with them getting past the "in progress" stage as nobody really understand what good looked like in each case. Between us we therefore felt a daily tech huddle, after stand up, would be a good way to make sure we started to share context better. So we agreed that we would try and steer the retro toward that outcome. As it turned out, the retro was really well done (I'll write about that separately) and the team came to its own conclusion that tech huddle would be a good idea.

Work in Progress Visibility and Reduction

As "Iteration Manager" it was understood that I own the board and the flow of stories across it. I spent a few days at the start writing out cards and putting them on the wall for all the stuff that was already in flight. It turned out to be over 20 things. So I tidied up the board, got rid of all the columns that really weren't useful (since our process didn't use them) and insisted that we pick nothing new up until this WIP went down significantly.

Part 4 - What Process Should we use to Support our Important Outcomes

After we had the outcomes we care about and after we'd discussed those outcomes in detail we spoke about the process to support them. I started off, with a new sheet of magic whiteboard, by putting stickies on it to represent Scrum rituals. The team still thought they were following Scrum even though I had quietly ignored Scrum planning, Backlog Grooming and all other set piece rituals other than stand up, retro and showcase. I laid out rituals in four swimlanes - once (per iteration), sporadic, daily and ?. I think I was trying to move the team from Scrum cycles thinking so I was reluctant to put anything in as "per sprint" or "per iteration". Maybe if I did this again I might use "per story", "daily", "per iteration" as headings.

We then discussed which outcomes each ritual supported and agreed which ones were needed, which ones should be dropped and which ones we needed to keep an eye on, or change, or improve, or reassess. From that point on we agreed that we will be constantly looking at our process and always asking what our biggest problem is and trying to solve it. 

The single resolution from this last meeting was that I agreed to own the story starting process better so that we only kicked them off when I had good quality on the stories. Good quality in this case was agreed to mean well understood acceptance criteria that at least a few other people in the team had agreed to and collaborated on. Essentially I had a story template to stick to that we'd all agreed upon.

The Outputs

I didn't want the team to get fatigued in any given meeting because I wanted them to be enthusiastic about wanted to understand the process and wanted to understand what outcomes they should care about. I also wanted them to make the decisions themselves and I wanted them to buy in collectively to those decisions. Decisions made at the end of along meeting driven by boredom and fatigue will not stick. Decisions made at the right point in the discussions, whether at the end of an allotted time slot or at any point in a timed meeting, if taken in the right way, have a greater chance of sticking. So the whole process took 3 separate meetings of 1 hour. The last one only lasted about half an hour.

Here are some photos of the outputs we ended up with:
Some activities (green), discussion points (pink) and desirable outcomes (blue)


More activities, outcomes and discussions, this team nearer the Done side.

The rituals (orange) with useful outcomes mapped (blue)

Continuation

These activities took place about 2 months ago at the time of writing and I am about to move on to my next challenge. The team continued to value flow, continued to be bought into the ownership of the whole process and continued to be keen to improve constantly. We since added WIP limits, changed the way Jira was set up for our project to mirror our actual wall and process (not the other way round) and added other stuff to our wall. 

The Wall

The team continued to add stuff to the wall that we thought was relevant and recently everybody has made suggestions about what might be a good idea. People are making suggestions at stand up and, if it isn't shouted down, my attitude is "sounds good, make it happen, if we like it we'll keep it, if we don't, or it doesn't solve a problem, we'll review it." This has the team working really well towards the right outcomes and we are making everything visible on the wall. Flow is now a thing that everybody values and everybody is familiar with.


Notes on the current wall:
  1. This is an area to park people's avatars when they aren't working or aren't available to work for our team. For example, off sick, on holiday, at a conference.
  2. This is an area to note down things that we would like to discuss in the next tech huddle.
  3. The area on the left is for things that are ready for dev. The pink cards are epics, the yellow are stories, the blue are spikes. If we had any defects we'd use a different colour card for them.
  4. This part is outstanding retro actions.
  5. "In dev" and "Dev complete" are the only two columns we are using between ready for dev and done. This is because we don't have formal QA and we aren't sure yet whether columns such as "waiting for XX", "in review" etc are valuable to us to help us visualise our flow.
  6. We drew a line around in dev and dev complete and annotated it with "WIP Limit - 4 cards". The WIP limit is across the two columns. I don't know if Jira supports WIP limits shared across columns but I don't care. The wall is giving us enough visibility to stick to it.
  7. The Done column contains all the stuff we've done since the last showcase (every two weeks).
  8. The little red stickies are Blocked stickies. The fact that there are so many of them tells a story of what the board looked like when we started. Loads of massive things getting snagged on loads of external dependencies in flight at the same time. The fact that nothing is currently blocked tells a story of progress. We haven't had to make a decision on whether a blocked card counts towards the WIP limit as it hasn't become a problem since we had the WIP limit.
  9. This box is marked "Customer Urgent". This is stuff that is on our backlog that is blocking any of the, now 3, teams that are our platform's customers at the moment.
We also have another wall next to this one that is our roadmap. It has higher level stuff on it and some dates that we are hoping to hit for milestones on our platform. 

Conclusion

This particular team has been the first time that I have had the mandate and the courage to apply the thinking that I had been forming over the previous few years. It worked well but I understand that one success cannot be considered a scientific sample. I'll also be interested to hear in a few months time whether or not the team continued to perform well or whether there was any kind of behavioural regression. Nevertheless I will state the following things as my conclusions or perhaps as my main tips for helping a team of talented people to get flow going well.
  • Don't impose stuff on the team. Find ways to get them to make their own decisions and shape their own destiny. Don't be afraid to guide them in the right direction but get the whole team to own the decision and any outcomes.
  • Outcomes are more important than process. Get the team to decide and agree what the valuable outcomes are and only then talk about what process might help to support them. Never lose sight of the mantra "outcome over process".
  • Transformation takes time. Be prepared for a journey of at least several weeks in length. Let the team fail, let them understand what failure and good look like. 
  • Change one thing at a time. Big bang process release won't work for several reasons. Give each change time to bed in before deciding whether it was helpful or not.
  • Let each team own their own process. It is clear that the process I've outlined got to one process to support one team. It would be a mistake to think you could simply take the end result and map that on to another team. Make this type of meta-process the thing that repeats, NOT the process that the team ends up using.