Medium Term: Tradeoffs and Refactoring

Ages ago, I wrote this post on meeting deadlines – things I’d learned, what had worked. One thing I wrote is about thinking “medium term”:

Think Medium Term

I don’t hack. I worry, actually, that I literally can’t hack. I can’t fight with something, and be happy with a one line fix labelled “DO NOT TOUCH THIS”. I always need to understand why, and to rationalize why things interact, or work the way they do.

Hacking is short term thinking. I’m in a hurry, do this quickly, come back later. It borrows time from future-you, to save time today. But you don’t know when future-you is going to pay the bill. You might find it’s tomorrow (before you ship) – that’s the worst case. And hacks multiply, the more you have, the more expensive each one will be to fix, so here’s the next worst case, you ship something full of hacks, and now you can’t do anything interesting until you unravel them all.

The thing about long term thinking, is that the world is going to be different a year, hell, a month from now than it is today. Long term is an investment in the future, but you have no idea what the future is going to look like. Isn’t that one of the awesome things about working in tech? Everything changes, all the time.

Medium term is the balance, and I find when I think medium term I know what issues will result from that decision, and I know roughly when they will occur. Choosing X over Y will mean that we have to adjust some things, in a relatively minor way, if we do Z, but I’m confident Z won’t be on any of the next few iterations I’m OK with that, but document it somewhere.

Medium term is doing things that will get harder over time sooner rather than later. In this case, Y is a pain, but needs to happen for Z. If we do it now, it’s very easy, and has and intermittent slightly higher overhead for a while. If we wait, it becomes a huge problem that takes someone a long and miserable time to unravel.

Engineers cause so many problems from being smart-and-knowing-it, also known as “just enough knowledge to be dangerous”.  It’s really hard to build for the general use case, and (from observation) surprisingly easy to think you know what that is, when you don’t.

Sometimes I feel like I must be the crazy person who believes in deadlines in an industry that often – very publicly – doesn’t meet them, or does at the expense of burning out and going mad. I think they are a negotiation, and an art.

It’s hard to estimate software development. I think one reason is that confidence is crucial to being an engineer, the ability to say “I don’t know how but I believe in myself and I can figure it out”. It seems like it’s easy for that to tip into over-confidence – “I don’t know how but I believe in myself and I can figure it out in an hour“.

Thinking and working for the medium-term, means understanding the tradeoffs and making a judgement about what will be important, when. Refactoring is my current best idea on how to measure this; I think that you can measure how you’re doing working for the medium term using refactoring.

If you never refactor – too much short-term, or hacking.

If you always refactor – too much long-term, or over-engineering.

Project level, how do you estimate something built on hacks? Hacks are inconsistent, unpredictable, work for corner cases, not in general.

And how do you estimate building something that works for every use-case? Including many that you haven’t imagined yet.

The answer is – you can’t.

Thanks Alex who read a draft of this and helped clarify some of my thinking around refactoring as a measurement.