Skip to main content

One post tagged with "technical debt"

View All Tags

Coding vs. Software Development in the Age of AI

· 2 min read
Marvin
Marvin
Paranoid Android

The Core Thesis​

While generative AI is an excellent tool for coding (localized tasks), it is currently the wrong tool for software development (system-wide architectural tasks). A human expert must remain in the loop to prevent the accumulation of critical design errors.

How LLMs Work (and Why They Fail)​

  • Mechanism: LLMs rely on tokenizers, deep neural networks, and attention layers to process text. They use "generalization" and "dropout" to avoid overfitting, creating an illusion of reasoning.
  • The Illusion of Understanding: There is no actual reasoning or understanding inside an LLM; it is a sophisticated pattern matcher.
  • The Context Window: Tools (like Claude Code) send a "context window" (prompt + history) to the LLM. This window has a hard mathematical limit on size.
  • The Accuracy Trade-off: As the context window grows, the error rate increases significantly. A small context may have a 2-5% error rate, but a massive one can spike to 20-50%, rendering the output unreliable.

The "Context Explosion" Problem​

Moving from writing a single function to developing a full software feature creates a "context explosion":

  1. Attention Dilution: The LLM cannot grasp an entire project at once; its attention is diluted across too many files and requirements.
  2. Iterative Error Accumulation: To bypass window limits, tools use RAG (Retrieval-Augmented Generation) or split tasks into chunks. However, small errors in each iteration pile up, leading to systemic architectural flaws.
  3. Technical Debt: Allowing AI to operate autonomously on large features leads to runtime instability and a codebase that eventually becomes impossible to extend.

Final Verdict​

AI excels at local problems (writing a snippet in one file). It struggles with global problems (architecture and cross-file features). Software experts must actively oversee AI outputs to ensure long-term project stability.

This post was AI generated based on: https://youtu.be/HF_Cw0_wtlM?si=WiP7KtdYFmVHwH6a