We talk quite a bit at work about technical debt. We typically think about all the things we have to keep updated and how so much of our time gets sucked into them.

The real problems are the unknown things lurking where one person knows how they work that are fragile and good enough. Until they aren’t. These ticking timebombs lie in wait. That one person keeps them running. They leave. Death, retirement, or even changing jobs means that one person who keeps an important thing going isn’t there anymore. What happens when it breaks? No one knows how to fix it. There’s a period of frustration and anger because it’s broken with no solution.

We see fairly often at the institutions using our services. The one person who knew the authentication system left and it has coasted on fumes for months to years. They keep meaning to upgrade it, but no one has figured it out the passwords to get into the system. (The penetration test found they were the defaults.) The payment gateway critical to accepting funds was managed by that one person who retired years ago. A consultant setup the automation system last year, but no one really knows how it works. It’s just running along doing it’s job until someone complained about all this bad data, only they can’t stop the bad data because no one can update the workflows.

What’s more interesting to me is these things force them or us to address the issue. Custom fragile processes no one, including us or vendors, understand gets replaced with standard integrations. The business process has to adapt to using it the correct way, but that’s the trade off for doing it the right way. That’s hard work to implement. That’s disruptive. The business has to suffer for a while.

Why can’t we do it the right way from the beginning? The business drives innovation. The business wants something tailored to how they want to work and IT helps them get it that way. The business thinks their way is a competitive advantage. The business rewards IT for being helpful. And so IT delivers these custom solutions that give the business ever more fragile solutions. The rewards aren’t more staff to support all the stuff, not that anyone really knows how much support really is needed.