I've been reading a paper doing the rounds on Hacker News this week, a study on AI-driven COBOL-to-Java migration, and it's worth ten minutes of anyone's time who's tempted to point an LLM at a legacy codebase and walk away for a coffee.
The short version: researchers ran a batch of COBOL programs through modern AI translation tools to convert them to Java. The output looked plausible. It compiled. In many cases it even passed surface-level tests. But dig into the semantics and you find subtle behavioural drift — rounding differences, off-by-one edge cases in loop conditions, silent changes to how nulls or default values are handled. Nothing that throws an error. Everything that quietly changes what the business logic actually does.
Why this doesn't surprise me
I started writing code in 1986. I've watched three or four generations of "this will finally solve legacy migration" tools come and go — 4GLs, CASE tools, rule-based transpilers, now LLMs. Each generation gets better at the syntax problem and each generation still trips over the semantics problem, because syntax and semantics are not the same job.
COBOL is a language built around fixed-point decimal arithmetic, PICTURE clauses, and business rules that were often encoded as tribal knowledge thirty years before anyone wrote a spec. Java has none of that baked in the same way. A translation model can produce Java that reads like a faithful port and still get the decimal rounding wrong on a currency field, because the underlying data model isn't equivalent — it's approximated. And approximated financial logic is exactly the kind of bug that doesn't show up in a demo, it shows up in an audit eighteen months later.
The paper's contribution isn't "AI is bad at this." It's that the errors are subtle, not catastrophic. A wholesale rewrite failure gets caught immediately — nothing compiles, tests scream. A subtle semantic drift passes review, passes QA, and ships. That's the dangerous category.
What this means if you build automation-heavy shops
I build tools that automate content, not core banking systems, so I'll be honest about where my expertise runs out. But the underlying lesson applies directly to everything I do build.
Every one of the Masher tools — RSSMasher pulling feeds, MarketMasher assembling campaigns, BookMasher drafting long-form content — does some kind of transmutation: raw input in, structured output out. The value proposition is always "let the automation handle the volume." The trap, always, is assuming volume and correctness are the same problem. They're not. A tool that processes ten thousand articles a day and gets 2% of them subtly wrong is not 98% good — it's a liability generator running at scale, and the errors compound because nobody's reading each one by hand. That's precisely the dynamic this migration study is describing, just with COBOL instead of content.
The fix isn't "don't automate." It's "automate with verification baked in as a first-class step, not an afterthought." For code migration that means differential testing against the original system's actual production behaviour, not just unit tests written after the fact. For content automation it means the same principle in a different coat — sanity checks, human review gates on anything with financial, legal or factual weight, and never treating AI output as final just because it looks fluent. Fluency is not correctness. That's the whole lesson of this paper in one sentence.
The practical takeaway
If you're running a legacy modernisation project and someone's pitching you "AI will translate your COBOL estate in six weeks," ask them three questions. What's the differential testing strategy against production behaviour, not just test suites? Who owns sign-off on financial and regulatory logic specifically? And what's the rollback plan when — not if — subtle drift surfaces after go-live?
None of that means don't use AI for this. I'd never argue that; the tools genuinely accelerate the tedious 80% of translation work that used to eat months of senior engineer time. But the last 20% — the bit where the actual risk lives — still needs a human who understands both the old system and the new one, checking the gold against the raw material before you ship it.
Forty years in, the tools change every decade. The discipline required to use them safely doesn't.
— Wayne