It’s easy to know if you are under-engineering something, because you produce sloppy work. It’s much harder to know when you’re over-engineering. The root cause of this is expanding the problem at hand so that the solution is much more interesting than the solution to the actual problem.
But, what does “over-engineering” even mean? A basic definition might be
doing more work than is necessary to solve the problem at hand.
This is very similar to the agile mantra of You Aren’t Gonna Need It (YAGNI), but this definition seems a bit more clear, at least to me.
There are two key points here: necessary and problem at hand.
What is the Problem At Hand?
You don’t change software unless there is a reason. That reason is to solve a problem, either with the software or for a user. In this case, you want a problem statement that is precise and complete.
Suppose our finance team comes to us and says that all changes to inventory records in our database must include an audit trail containing who changed the data and what was changed. This is a great start, and it’s certainly a complete description of the problem, but it’s not precise.
Based on this imprecise description, we might find every line of code that changes inventory records and add a new line that writes an audit record. This would be under-engineering, despite being the “simplest thing that could possibly work”.
To make this statement complete and precise, we might state it like so:
The way our system manages inventory records must change so that all changes are audited, including who made the change and what they changed
We’ve made it precise by referencing what has to change, namely the parts of the system that manage inventory.
Our solution to this problem must involve only the work that is necessary to solve it.
What is Necessary?
Necessary can be subjective. The best place to start is your team’s values. Do you value shipping as soon as possible over all else? Or do you value bug-free code, regardless of when it ships?
The team I work with at Stitch Fix values clean, sustainable, well-tested code. For us, any solution must meet those criteria. We know where our business is going, and we know our plans for scaling our team, so we can take that into account when proposing solutions.
This results in a solution that solves the problem and only that problem, but is also consistent with our values.
As an example, adding a call to ItemAudit.create!(...) everywhere we modify inventory would certainly solve the problem, but would not be consistent with our values, as it is not sustainable, based on our collective understanding of our planned growth.
Further, a solution where we create an inventory microservice and move everything to HTTP would be way more than is necessary. A microservice is scalable and consistent with our values and planned growth, but it solves more problems than the one in front of us. That would be an over-engineered solution (a well-engineered solution might be to create a shared service object that describes the way we manage inventory; it is explicit, keeps the code clean, and makes a microservice easier later).
Other signs of over-engineering could be focusing too much on creating re-usable abstractions, excessive application of DRY, or otherwise getting lost in solving a more general problem than the one at hand.
So what’s the problem with over-engineering anyway?
The Two Problems With Over-Engineering
The main problem with an over-engineered solution is that it takes longer to ship than is necessary. By definition, we are doing more than is necessary, and that will take longer to ship. There’s almost never a reason to prefer longer ship-times over shorter ones, all things being equal.
The more serious problem with over-engineering is the carry cost.
A carrying cost is a cost the team bears for having to maintain software and infrastructure. Each feature requires tests, monitoring, and maintenance. Each new feature is made in the context of those that came before it. This is why a feature that might’ve taken one week when the project was new requires a month to make in more mature project.
By over-engineering a solution, you are saddling your team with more carrying costs than are necessary, meaning their future velocity will be slower than it needs to be. This is a bet that never pays off.
OK, we know what over-engineering is and why it’s bad. How do we avoid it?
Avoiding over-engineering
An easy way to check yourself is to share your problem statement and proposed solution with another engineer, which acts as a forcing function to actually write down your understanding of the problem as well as your solution.
Writing it Down
The act of translating information from your mind to some other form, either verbal or written, often has the effect of revealing flaws in your thinking, either missing information or inconsistencies that weren’t obvious. Even for very simple problems (with hopefully simple solutions), writing out as a basic one-pager can clarify your thinking and focus you.
And, the bigger the problem, the more benefit there is to going through this exercise, because big problems that require big solutions are very hard to keep in your head.
This, then, lets you easily get feedback from another engineer, which is a great way to know if you’ve over-engineered your solution.
Getting Feedback
With your written problem and solution, you can easily ask another engineer for feedback. You need to ask specifically about the appropriateness of your solution, however, as many engineers will default to looking for correctness. In other words, don’t forget to ask “Did I over-engineer this?” and “Is there a simpler solution I’m not seeing?”
You also want to be judicious in who you ask for feedback. An experienced engineer who you don’t tend to see over-engineer solutions is ideal, but a developer who isn’t that familiar with the system you’re dealing with can also be great, since they won’t bring any baggage with them about your particular problem.
If your problem is large, or the solution seems necessarily complex, get more feedback from more people.
In the end, by focusing on stating your problem complete and precisely, outlining a solution that is only what is necessary to solve the problem, and getting feedback, you’ll avoid the pitfalls of over-engineering.
