The Shit Nightclub antipattern

You go to a fancy looking nightclub. It's on Leicester Square in London - it must be good. The thumping music carries with it on the evening air the magical lure and promise of good times, so you pay the £20 entry fee and the suited bouncers let you in.

Then you discover that there's no one in there. And as the minutes and hours slip by, the place doesn't really fill up - but it's sure to soon, right? One of you suggests going somewhere else, but you point out that you've all paid £20 to get in. We should stick it out a bit longer.

And this is what happens with large IT projects.

Now IT projects are hard. No two are the same due to the shifting quicksand of technology, and the particular perspectives of the myriad stakeholders involved.

IT projects are also time consuming. They usually take at least twice as long as anyone thought they would - mainly because of the first point; no one can hold all of it in their heads, or at least not in "working memory".

Lots of problems stem from these two things: Budgets get blown out of the water, businesses don't get to market quickly enough, a work-late culture develops, morale ebbs away...

But like any teacher will tell you, just because it looked great on your lesson plan, if it's not working abandon it!

And this is what businesses are often unable to do.

This is what happens:

• Someone estimates the project as 3 months.
• A team breaks down the work and adds up the estimates. It becomes 6 months. (If you break work down and add it up, the estimate will always grow).
• Some things get de-scoped. Some estimates get squeezed. They arrive back at the 3 month estimate, and senior stakeholders are appeased.
• Work begins, and in the early days it becomes clear that things are slipping. Badly. But the PM (yes we're probably talking about a non-Agile environment) sits on this information rather than put the cat among the pigeons - after all it's early days, we might be fine.
• The three month deadline approaches, and the PM goes cap in hand to the stakeholders and makes a poorly thought through request for one more month.
• Two months later the team is exhausted and miserable, and the backlog is still groaning with work (including the performance testing - everyone agrees we really should get going on that). 
• The PM is secretly angry with the team - how can they have taken so long? Why is it always like this?
• The stakeholders are angry with the team - they put money and reputation on this project, and these developers are obviously not up to the job. More developers are dragged (possibly literally) to join the team.
• The developers are all talking about leaving (except the guy who isn't very good).
• The system is now complex and buggy, and more worryingly, it's laughably far from the performance targets they set at the start. How can it take 20 seconds to return a page?
• At 6 months, a senior architect or tech lead is parachuted in. The PM is changed. More and more money is sunk into the project. It can't fail now, otherwise everyone's wasted half a year and a scary amount of money. More importantly, the stakeholders will look like fools.
• The project limps out the door at 9 months, performing poorly and riddled with bugs.
• The business decides they need more rigid and formal processes to stop this happening again.

So if you find yourself in a shit nightclub, consider your options:

1. Stay. Then you'll end up having spent lots of money, and having had a shit night.

2. Leave. Then you'll end up having spent lots of money, and possibly having a good night.

Your choice.

Nuance and momentum

In most companies there's an understandable reluctance to commit significant budget to new ideas unless they're very well proven, and - critically - enough people in senior positions think they're good ideas.

So in order to get sufficient momentum for an idea, it needs to be repeated and communicated many times - and thereby loses any nuance of meaning. It becomes necessary to distill the essence of it into an "executive summary" that, especially for technical ventures, often misses the point by a small but important margin.

In the company I currently work in, various technical people, including myself, have been pitching to refactor a particularly complex area of our architecture. It doesn't perform as well as it needs to, and is also the frequent source of insidious bugs in our live environment which are hard to debug since the system is hard to reason about. 

We have a proposal for how it could be designed differently, but in order for this proposal to gain momentum, the point of the work has been diluted through Chinese whispers to the basic premise of "Make it go faster". The enormous benefits in future technical enablement are glossed over. (Of course, in most companies, eliminating technical debt is considered a Good Thing, but when it come to the crunch, no one wants to invest in it. Where I work now is a hundred times better than most at this, but it's still short term wins that really get people going).

Why does this matter? The message that has become fixed on is the short term performance benefit which yields greater opportunity for sales. This benefit is much smaller than the more intangible longer term advantages, and so the idea hasn't made it very high on the agenda.

What's more, ideas with no short term benefit will probably gain no momentum at all.

And when (if?) the idea comes to be engineered, the senior business stakeholders may be impatient for it to be completed, as they don't understand the nuances of how much long term benefit they will be getting. In some companies, this can lead to the project being canned before it's complete (see Death By a Thousand Incomplete Refactorings), or made more difficult by technical teams being asked to work on short term things in parallel.

If only business people could invest more time in understanding technical matters. Of course, one barrier is that us techies are often resistant to that at a subconscious level - we take pleasure in our area of expertise being impenetrable in much the same way I suspected Polish people liked seeing me struggle with their pronunciation and grammar when I lived there. (Different endings for different numbers of things - seriously?)

But the other barrier is that we make our systems too complex. Perhaps a good rule of thumb should be: If non-technical people don't understand our systems, then they're too complex.

So, returning to the topic of gaining momentum and losing nuance, what's the answer? It seems to me it's the same answer I return to again and again in IT (and many other areas): 

Better to restructure your organisation to allow for small independent teams to make their own decisions and set their own direction. That way, less momentum needs to be built up and more nuanced communication can occur, using a rich shared vocabulary.

Is Eliminating Cross Team Handoffs Possible?

No.

Well, not completely anyway. But just because you can't achieve something doesn't mean you shouldn't strive for it.

Lots of advice floating around in the Agile space tells you that handoffs from one team to another are expensive and inefficient: The DevOps movement has given way to the Anti-DevOps movement, suggesting that the development team should do their own Ops; I've written about The Vicious Cycle of Support whereby the development team should support their own output; separating business teams from technical teams can lead to an unpleasant client-supplier relationship where neither side can work effectively.

So do we just need one huge team with everyone in it?

Well, yes. Except for the "huge" bit.

It's been a core concept of most Agile doctrines that you keep your team small, and your immediate scope small. And the people in that team don't just stick to one role - they chip in and help out wherever it's needed.

I've said before that I think the optimum set up is two developers and a product person. That assumes, however, that all those people are very experienced, and are willing to cover off between them the jobs that we typically divide up into the separate roles of Business, UX, Design, Coding, Testing, Release, Support and so on. That may sound ambitious, but consider the benefits:

• No communication overhead. Everyone knows exactly what's going on, and everyone is working from the same assumptions.
• Focus. There's no way you're going to be able to juggle 5 projects at the same time like bigger teams are often expected to do.
• A sense that you really are a team, and that you own this deliverable end to end. This is the same reason that startups are such exciting and dynamic places to work, and often take the market by storm. 
• You wake up and feel motivated to come to work! It's fun. And when people have fun doing their jobs, all sorts of virtuous cycles develop. Talented people want to come and work for you. People feel inspired to come up with genuinely new and exciting ideas.
• Everyone in that small group is forced to understand all the domains end to end, which were previously kept separate. And when people understand things end to end, your quality shoots up because there are no unexpected implications from decisions made in isolation; there are no lost opportunities ("if I'd known that was something we'd want to do, I'd have built it differently!")
• The team can be autonomous. No more waiting. No more chopping and changing at the whim of far-removed senior management or marketing teams. With clear goals, and empowerment, it's pretty amazing what three people can achieve.

(In case you're wondering why handoffs are bad, see Pawel Brodzinski's blog post on it)

The Vicious Cycle of Support

Developers who don't have to support their code in Production unsurprisingly don't consider all the implications of their changes.

Logging, monitoring and alerting are afterthoughts. Questions like "What happens when the network fails?", or "What if we run out of disk space?" aren't at the forefront of people's minds.

The difficult test scenarios get ignored because "We should probably test those things, but the test environments are rubbish and besides, this is how we've always done it."

But the result isn't just that a particular feature going live drags some poor guy out of bed at 3am (the traditional time that hypothetical production issues take place). And it isn't just that the metrics on number of live incidents rises over time. It goes much deeper than that.

When Support teams are separate to Development teams, developers don't understand the domain of Support, and vice versa.

The problem with the former is that systems that weren't designed to be supported are hard to support. The cost of that support effort grows over time.

The problem with the latter is that if Support people make fixes, they do so without understanding the domain fully. This causes further instability and technical debt, and also creates a system which neither developers nor support people understand.

Of course, developers continue to build on these foundations, which compounds the problem still further.

Still, you shouldn't be downhearted as a developer in this kind of company. At least you don't have to support it!

See Is Eliminating Cross Team Handoffs Possible?

Death By a Thousand Incomplete Refactorings

Start with a fairly new IT system. The lead developers built it with a Shining Vision of the perfect architecture, and that vision has been realised. The team has put most of the finishing touches on it, the initial waves of bugs have been exterminated. We're proud of it. We even spend some time refactoring a couple of bits we rushed.

The lead developers feel like heroes. They stick around for a couple of years basking in the glory of being the people who understand the system inside and out.

And then, gradually, the realisation dawns that there's some technical debt which needs addressing. Possibly it's been there all along, it's just that the system scaled or was extended in ways we didn't anticipate. Possibly it's caused by developers building stuff without understanding the Shining Vision. Possibly it's down to The Vicious Cycle of Support.

Whatever it is, the Make it Shiny Again project is born. One of the original veteran lead developers spends all day every day telling people about how the Shining Vision got lost, and now enough senior people have listened. Troops are mobilised.

The big refactoring goes well at first. Everyone is behind it. Update emails go out gleefully reporting the excellent progress. Demos happen. Meetings show high level progress and sing the praises of the Make it Shiny Again project. It will destroy disease in the third world. It will bring peace to the Middle East. It'll even increase our unit test coverage.

But, inevitably, as the project stretches on and the full scale of the task becomes apparent, people begin to lose energy. It looks like the timescales are moving out. Fewer and fewer people are extolling the virtues of it. Even the veteran lead developer has got distracted by a new even shinier way of doing things. The project joins the ranks of other tedious long term projects with no sense of urgency, gathering dust at the bottom of the project plans. (In some companies, "long term project" means years. In some it means weeks. Either way, energy attrition occurs).

Eventually something goes live. It may feel like a pyrrhic victory, but it delivers value. One part of the system has been refactored, and is running successfully alongside the old system. No one has the energy to celebrate.

Some conscientious members of the team ask when we'll move on to the next phase of refactoring. Tumble weeds drift across the office floor. Strangely, no one fancies descending back into that dark chapter of their lives.

The final solution, once all functionality had been refactored, meant the switching off of the old system:

However, what the Make It Shiny Again project actually achieved was this:

Now drill deeper and look at the code. Many many incomplete refactorings of different things have happened. Everyone touching the code has either not understood that particular area's Shining Vision, or has sought to impose their own.

Now look at it from 30,000 feet. Whole swathes of the architecture (and the organisation) have undergone incomplete refactorings.

If you squint a bit, the architecture is a fractal. It looks like spaghetti at any level.

No wonder it all feels like such a mess. The solution? 

Let me unveil my Shining Vision. We'll finish it this time, honestly we will.

Less bleak footnote

This can be avoided. Some technical debt is inevitable¹, but Death By a Thousand Incomplete Refactorings is not:

- Don't bite off more than you can chew.

- Deliver in small increments. If it's going to fail, make sure it fails early.

- Celebrate each small success. 

- Swap people in and out - keep it fresh. Let everyone be involved.

- Recognise the risk of an incomplete refactoring!

---

1 See Martin Fowler's TechnicalDebtQuadrant^↩

Bend Over Backwards. Now Stay There.

High staff turnover is a bad thing. Pretty much everyone would agree on that - especially for a development team where knowledge of the system is essential.
In fact surely keeping staff happy - retaining the best talent - is a priority for all managers.

But when the pressure's on, IT managers will end up cascading the misery to their team for the simple reason that all managers are primarily judged on and therefore incentivised by meeting project deadlines.

Meeting deadlines means meeting budgets and delivering ROI. In other words, it means making money - and this is the only priority of any company anywhere regardless of what their website says about CSR, diversity or people development. Money is all they care about - make no mistake.

So what does this mean for you as a developer? It means that the only person you can rely on to look out for you is yourself. If you regularly bend over backwards to meet deadlines then that becomes the norm. If you join the Working Late Gang, you give them strength in numbers - it becomes the culture. The unrealistic expectations, the flawed process, the poor planning - all of it is hidden or condoned by you picking up the strain and making it happen.

Stop.

Lazy developers?

Everyone wants to get an IT project finished on time. Why would they want anything else? There's no fun in being part of a "zombie project" that never dies - developers want to move on to new challenges, new cool things.

And yet there's often an unspoken assumption that the reason things weren't done on time is that developers are basically lazy.

Here's one of the main reasons why I think that.

Developers come up with an estimate for a piece of work which they believe is realistic. Depending on their experience and personality they might be overly optimistic or pessimistic. But at the end of the day they are the technical expert whose specific responsibility it is to estimate, design and build software. (Of course, it should always be the person who's responsible for building a thing that should provide the estimate for it).

And still, if the manager or business stakeholder who hears the estimate believes it's too high, they will usually squeeze it, despite the fact that they're neither responsible nor qualified to specify estimates. This squeezing can be overt, or surreptitious. Often it will take the form of putting pressure on the developer ("But are you really sure that piece will take 10 days? It's just config isn't it?"), or, even worse, it will be done without the developer even being consulted, by a coven of project managers, pressured into finding "creative solutions" to meeting the dates the business had in mind, convincing themselves that things can be parallelised, done more efficiently, or done by "just copying" something done previously. 

"Pete said a maximum of 4 people could work on this, but there's bound to be stuff we could get one of the Indian guys to work on. And this user story says 10 days - but it's pretty much what we did last time. If we said it was only 6 days we pretty much meet the deadline, as long as no issues come out of test."

The problem, in a nutshell, is that people don't believe that the developers have thought "creatively" enough about how to meet the deadline; on some level they're trying to give themselves an easy ride.

It doesn't seem to occur to them that they could simply go back to whoever demanded an earlier deadline and say "No I'm afraid we can't do that. I've worked it through with Pete, and we don't think it's achievable."

I got a bit of a thrill saying that. Let me try some more things that rarely get said by project managers:

• "I'd need to consult with Pete, it's not my remit to say whether or not we can add more people to that."
• "It'd be best if we didn't distract Pete right now to ask him for estimates on this. He's maxed out."
• "The team as a whole has had 5 distractions this sprint already, and that's our stipulated maximum, sorry."
• "No, Pete said 10 days. He might turn out to be wrong because it's just an estimate, but he's the most qualified person to provide that estimate and we have to go with that."
• "No we can't commit to that date. We don't know if it's achievable or not. But we can commit to delivering the stuff you said was highest value first, and involving you all the way."
• "We could try and achieve that, but the team's been flat out on this stuff for months, and are having a pretty tough time. We need to try and find a sustainable pace for everyone's sanity."

But those things pretty much never get said, especially the last one.