Tech Debt is an essential part of software development and operations. Any effort to avoid it entirely will do nothing more than paralyze progress so thoroughly that a project will be condemned to irrelevancy.

Do you want GNU Hurd? Because that’s how you get GNU Hurd.

Actually, I don’t know if that’s how you get GNU Hurd. Nor if a refusal to take on Tech Debt was even a contributing factor to Hurd’s 30+ years of snail’s pace development. But I do think there’s at least a metaphor to be found in its failure to reach its goals vs. projects that opted for shipping minimum viable products faster, and iterating after the fact.)

Hello everybody out there using minix - I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready…
…
…and things seem to work. This implies that I’ll get something practical within a few months…

Torvald’s now-famous Linux announcement (Aug 25th, 1991), quoted above, could be read as a (completely unintentional) manifesto on the benefit of shipping minimum viable products. And what is a minimum viable product, if not an artifact of a project choosing to ship with Tech Debt?

Tech Debt is leverage. It allows you to produce more and faster. Just like the financial instruments that it draws its name from. Debt lets you buy a home and go from tossing your income into the black hole that is rent to building equity. It allows entrepreneurs in our capitalist system to start businesses that would be otherwise out of their reach. And it allows investors to amplify smaller (and possibly safer) investments into massive returns.

But anyone who knows anything about finance, or has seen The Big Short, or has ever used a credit card or taken out a student loan knows that Debt is dangerous. It should be taken on with care and, above all, with a plan for paying it down. Debt is also rarely static. It grows with interest—often compound interest. It must be serviced through regular payments, or it will grow like a cancer. In the best cases, out-of-control debt will lead you to bankruptcy; in the worst, it will ruin you.

Unfamiliarity as Tech Debt

Tech Debt is often viewed as exclusively an architectural or design problem. Some sacrifices that you make in how you write your code or configure your system so that you can deliver it to consumers faster. Paying it down is then a matter of refactoring or replacing components with better-designed ones or implementations that are more well-suited for the growth and performance of the system or product.

To that definition I would add various kinds of Knowledge Debt. Bad or lacking documentation, for instance, should be (in my estimation) seen as Tech Debt. Just as bad architecture in code makes it harder to extend, improve, or fix, so does a lack of documentation. Obviously there does exist a sweet spot for this; just as you can over-engineer the design of a system or program, you can over-document it as well. Or write documentation with a very short shelf life that will soon cause more harm than good. But anything that prevents those tasked with maintaining the code from being able to be rapidly effective should be considered as a potential area of improvement.

As an extension of this, I would say that adding functions, modules, components, or subsystems that are poorly understood by the team maintaining the code is also this kind of technical knowledge debt. Improving, extending, or even evaluating if it is needed at all becomes difficult or permanently banished to the backlog when few (or no one) knows how it works.

Specialization and Knowledge Debt

Note that I am referring to knowledge gaps within the team of maintainers, not those of individuals. Any sufficiently large codebase cannot (and likely should not) be exhaustively understood by every member of the team. Specialization yields strong productivity benefits, and so it’s both natural and effective for each member of the team to have areas of the code they do not fully understand. This is part of why we work on teams, to collaborate and pool our mental/knowledge resources to accelerate development. It’s not just about having more hands on keyboards.

In this way, specialization shows itself as both related to technical knowledge debt and an analogue for the power of this debt to accelerate production. If everyone is required to know everything, progress slows, though I would not consider specialization to be tech debt that must be paid down in most cases. But there is a fine line between specialization and bottlenecks and single points of failure.

Therefore, technical knowledge debt must be evaluated in aggregate across the entire team of maintainers. It’s not debt to have team members who need to rely on each other, but if reliance crosses the line into bottlenecks, then the knowledge gaps burden your productivity and can count as tech debt. Since tech debt cuts both ways—accelerating productivity in the short term and slowing it down when not paid down in a timely manner—one can easily be mistaken for the other.

So it’s not enough to ask what percentage of the team is familiar with each component or subsystem but also how many individuals that percentage represents and how often those parts need to be debugged or modified. But the bottom line is, the more of your code is poorly understood by more of your team of maintainers, the more likely you are in technical knowledge debt. Almost always necessary and beneficial in moderation but painful to dig yourself out of when not.

Thus, aggregate team blind spots in the program or system’s design should be considered tech debt that must be paid down. And just like any other kind of tech debt, there is an acceptable amount of this debt to carry. Failure to incur it can paralyze progress, just as failure to keep it in check can cripple the project.

And this is where LLMs enter our conversation. Because while LLMs can and do help remove the bottlenecks that result from gaps in team knowledge (by helping the reach of individual engineers exceed their grasp), an LLM can produce lines of code, add/or deploy entire components, or rewire whole systems at speeds we previously could only dream of. And keeping up has become just as much a part of the challenge for those of us maintaining code and systems as any part of technical design was before. And an LLM asked to compensate for our knowledge debt will often deepen that debt while bridging the current gaps. Paying one balance down with another.

Generating Order

The development of generative “AI”, from Markov Babblers to Latent Diffusion Models, has been an effort to bring order to the random.

This is probably best illustrated in the case of Latent Diffusion Models, which are a class of neural networks that are trained to use a text prompt as a guide for converting random noise into a coherent image.

These days, Large Language Models have become capable of bringing an impressive amount of order to text output (converting disorder into staggering quantities of exhaust heat because entropy will not be mocked), which is how a statistical model of human language is able to simulate thought and even knowledge.

And still more order has been brought to their outputs through continuing innovations in the development of harnesses (traditional deterministic programs that help bring order to the still chaotic outputs produced by even the most advanced of the frontier language models) like Claude Code, Deepseek Harness, OpenCode, Codex, OpenClaw, and Hermes (to name just a few). Much the way that tabletop gaming systems like Dungeons and Dragons (D20), Savage Worlds, and GURPs derive order from the random output of dice rolls.

Harnesses are able to squeeze out more from the underlying models used through repeated prompting, providing access to deterministic validation tools, and basic context management. And thus, they have rapidly become nearly as much of a differentiator in one’s ability to use LLMs to develop and operate computers as are the models themselves.

This has greatly chipped away at one of the core criticisms people have often leveled against LLM-generated code, namely that it is inherently buggy and of poor quality. Although that criticism is still far from being unfounded, the gap between what a skilled engineer and an LLM can produce has narrowed and is continuing to narrow significantly.

However, two other common criticisms remain unaddressed:

  • It may not be possible for statistical probabilities applied to arbitrary data to consistently produce superior or even quality output.
  • The cost of inference exceeds its return for general use.

The Irrationality of Generative Models

I’m not an expert in neural networks or the transformer architecture. I’m not a mathematician or even a data scientist. The closest thing I could be called with even a shred of credibility is a computer scientist, and even that feels a bit much for someone with only an undergraduate degree in computer information technology and a career focused on PaaS/SaaS operations automation. So I’ll just say that I have heard people far more qualified than I suggest that it may be mathematically impossible for the current generative models to be consistently rational. Or, in other words: to not randomly make up information that has no grounding in reality, despite their promptings.

What I can say from my own experience is that no matter how explicitly you lay out instructions (be it in the user or system prompt or through context files or project “rules”) the “bot” will not interpret them consistently and will sometimes take actions that directly violate these instructions for no reason that even it can articulate. (Recently I had an agent at work tell me it had skipped steps that were marked as “MANDATORY” in its instructions, and when asked to explain why, it simply said that it had no good reason for doing so).

I’m sure that many more qualified people than myself would say that the reason for this is simple: LLMs don’t think and don’t reason. They are statistical predictors that make no judgments other than what word/token seems to follow from all previous ones in its context window. Therefore, you cannot reason with it. And instructions will always be treated as little more than suggestions. While still others more qualified than myself will say that those first people are heretics who will burn for their unbelief.

All I can say is that while LLMs are more reliable to use than they have ever been, they remain randomly irrational. For all the order that has been brought to their output, they remain unpredictable. And thus, even if their output exceeds that of the top engineers in the field 99% of the time, their output needs review.

Top Engineers Make Mistakes

Humans make mistakes. Deterministic algorithms programmed by humans make mistakes. And statistical models of human language will also make mistakes.

We compensate for human mistakes in software development with various strategies. Linters, static code analyzers, and automated test suites bring a measure of deterministic validation to code produced by humans. Harnesses use many of these same tools, and often the code produced by LLMs (with or without a harness) is subjected to the same automated validation as human-produced code.

But this is not all we do. Most projects require code to be reviewed by a peer or even a subject matter expert before it is accepted. Ideally, a code change is reviewed by many other engineers before it’s approved. As Linus’ Law (confusingly coined by Eric S. Raymond and not Linus Torvalds) states: “Given enough eyeballs, all bugs are shallow”. So we can still compensate for the hallucinations and random instances of temporary insanity the models are prone to in the same way we compensate for simple human error.

But now the two sides of our equation become imbalanced. Because one of the core value propositions of LLMs is the speed at which they produce output and, by extension, their volume of output. But now a team of 5 engineers, reviewing each other’s code between producing their own, are producing 2x or (almost certainly) more code than they would without LLMs. The platonic ideal being that all the time saved in producing this greater volume of code is instead invested in reviewing the code changes others are producing with LLMs. Which would then cut down on the time spent prompting the LLMs and thus reduce the productivity gains somewhat (while still remaining a net positive). However, in practice, engineers are not very fond of reviewing code because their incentives are weighted heavily towards individual contribution. And the more time spent reviewing, the less thorough the reviews become.

So enters the solution most teams gravitate towards: have the LLMs do the reviews. Either in conjunction with human reviewers or (in the most extreme cases) instead of human reviewers all together. If we assume that LLMs make mistakes (or miss things) as often as or less than would human reviewers, then this should provide similar or better quality.

However we have now decreased either the number of human mantainers of the code who are familiar with its current state, or the percentage of of the code that has been checked by human eyeballs at all. Which means we have increased our unfamiliarity Tech Debt. A problem that then exacerbates our next problem, by increasing the token spend required to produce the same amount of code. And this unfamiliarity will incentivize (or even require) maintainers to spend more tokens to debug or improve it in the future.

The Cost of Inference as a Ticking Time Bomb

I think anyone would struggle to argue convincingly that the cost of inference (that is, the cost of producing LLM outputs) has shown any inclination of going down, though claims of questionable substantiation have been made to that effect. But whether or not it has, the key fact remains that most LLM usage is done at subsidized rates. And if we assume that the per-token API rates are the real cost of inference for models (and possibly not even a profitable cost), then we have already seen that charging anything approaching the actual cost of inference produces massive sticker shock to even enterprise consumers (as was the case recently when Uber was moved to paying per million tokens). And it seems clear that if we were all being asked to pay even what the inference providers would need to break even (not even factoring in the cost of training the models), then we would probably not turn to the LLM for as many tasks as we currently do.

The more a project is dependent on tokens for its maintenance and development, the more exposed it is to the risk of rising token costs. And an overnight correction in pricing could easily paralyze a project by making the bill on its Technical Knowledge Debt come due. Suddenly maintainers who have blissfully vibe-coded through each bug and feature of their project will need to stop development and cram-study their codebase until they are confident enough to hand-code all the changes they can no longer justify burning tokens on.

Should a serious production issue come up before that occurs, the time to recovery may be more than their business can tolerate. Depending on how deep their knowledge debt is.

And of course, this is to say nothing of the potential financial hole those who are not able to halt their token spend fast enough, after such a price correction, would find themselves in.

Inference Could Become Cheap

Obviously, we are inclined as an industry to expect infrastructure costs to come down. After all, it seems like (until recently) they always have, even if not immediately. And I must admit that I too expect inference to become cheaper. Though I can’t say that I have any confidence that it will happen soon enough to keep the current LLM rager going and the bubble from bursting.

But even if inference becomes cheap, unfamiliarity with your codebase and configuration should still be considered tech debt. Because whether they become cheap only through the economies of scale available to frontier providers or cheap to run yourself on commodity hardware (at frontier performance), you will still need to be able to understand what they produce enough to direct them.

Here I am referring to a scenario where frontier models become both cheap enough for anyone to run if they want and (as a bonus) I am also assuming the environmental impact of running them somehow reaches net zero.

In this scenario you can throw as many tokens as you’d like at any problem, and swarms of agents will be at your disposal to smooth out any hallucinations any individual model or prompt might produce. But then the question still remains: Who specifies what should be built? And who defines the success criteria?

Perhaps you’re saying that in traditional software development this is where product owners come in. Today they could be seen (in an analogy of questionable simplification) as operators prompting a team of agents (engineers) to build their vision. And in this future scenario where LLMs can smooth over any knowledge debt forever, the role of the LLM prompter is one of product design.

To this I ask: when, in the history of interactions between product and engineering on any project, has it not been necessary for engineers to push back and ground requests in architectural and technical realities? Often this is where the traditional role of a technical architect comes in, someone who may not be inside every detail but understands the underlying tech stack deeply. And helps translate product/customer requests into actionable and sustainable technical implementations. That requires someone, or ideally a team of people, who understand the tech and are ready and invested enough to make the product great for both users and future maintainers.

It is my assertion that LLMs are not prepared to fill that role; they are too prone to sycophancy to push back reliably and too limited in their world model to “understand” more than what is “likely”. Thus someone does need to understand what is being built, or technical knowledge debt will eventually come due. And if they will be capable of filling that role, then what we are likely talking about is Artificial General Intelligence (AGI), which remains more science fiction than even cheap inference is today. (No matter what Jensen Huang says)

Science-Reality Today

Understanding how code works is what makes an engineer have anything to contribute in the age of vibe-coding and will allow you to better evaluate LLM output or prompt the LLMs to course-correct when they get confused or just randomly choose to ignore instructions. The less you understand, the less you can contribute as a supervisor of agents (if that’s what you want to be) or as a lead in your team of human engineers.

It pays to know things. And also, it’s fun.

Move Fast and Break Things

The goal of this post is not to tell you how much tech debt to carry. But just as with financial debt, I think it’s safe to say that answers will vary greatly by personal and organizational circumstances. But the right amount for your team is almost certainly not zero.

And this goes for technical knowledge debt as well. You can and probably should incur it. But know that it must be kept in check just as much as any other type of tech debt. Because it has the same consequences when allowed to spiral out of control.

And this is the unique challenge LLMs bring to this industry. Because unless what they produce is thoroughly read and understood by the maintainers of a project, they are merely tech-debt-producing machines on fast-forward. They will bury you in knowledge debt if you aren’t careful, and your codebase and users will suffer for it.

But we should also recognize that the way we are currently being encouraged to interact with them does not acknowledge this reality. And the sheer volume of code and related text they produce is exhausting to read and properly digest.

Given that the phrase “Move Fast and Break Things” is attributable to Mark Zuckerberg, I doubt it ever had a truly positive connotation. But it naturally reads as an argument for incurring tech debt. Shipping things quickly and learning from our failures. Although with the speed at which engineers are being asked to produce and review (or frankly not review) the output of LLMs, I fear that what will be broken will be engineers themselves. Which honestly may have been what Mark meant all along.