93% Accurate Is a Failing Grade for COBOL Translation

← All posts

AI has genuinely changed legacy modernization, but its value is in understanding your codebase, not in rewriting it.

A regional insurer we’ll call Meridian Mutual ran an AI-assisted conversion of its policy administration system. The pilot converted 180,000 lines of COBOL to Java in eleven days: compiling code, readable classes, and tests that passed. The steering committee approved the full program on the spot.

This is the central problem with AI-led mainframe modernization: AI can make code conversion dramatically faster, but it cannot by itself prove that the converted system is behaviorally identical to the system it replaces. For systems that move money, calculate reserves, or produce regulatory reports, that proof has to be deterministic. Translation speed is useful; deterministic equivalence is what makes a migration safe.

Nine months later the program was paused, not because the translation failed but because nobody could prove it had succeeded. The new system produced premium refunds that differed from the mainframe by one to four cents on roughly one policy in nine hundred, clustered where a rounding decision compounded across a proration chain. A retired COBOL analyst took three weeks to find it. The fix took four hours.

That is the shape of the problem. Not a crash, not a compile error, not a failed test. A quietly wrong number that survives every check you thought to run.

Here is the claim, and plenty of vendors will disagree with it: automated code translation was never the bottleneck in mainframe modernization, and making it faster does not make your program faster. The bottleneck is proving the new system behaves exactly like the old one in every case the business depends on, including the ones nobody remembers. AI has moved that first problem enormously. It has barely touched the second.

Do the arithmetic on the number everyone quotes. Vendor-reported COBOL-to-Java conversion accuracy sits around 93%. That is a vendor claim, not independent research, but take it at face value. On a five-million-line estate, 7% is 350,000 lines, not grouped in one module you can quarantine but distributed through a system that moves money, calculates reserves and files regulatory reports. A 93% grade is excellent on an exam. On a general ledger it is a failing grade.

Modernization is three jobs, and AI is not equally good at them

Most conversations collapse into one question: can it convert COBOL to Java? That is too coarse to act on.

Comprehension is modelling what the system does today: control flow, data lineage, business rules, batch dependencies, dead code, integration points. Translation is emitting equivalent code in a target language. Equivalence validation is proving, with evidence an auditor would accept, that the new system produces the same outcomes as the old one. An honest 2026 maturity assessment:

Spend your AI budget on the top half. Spend your human budget, and most of your calendar, on the bottom.

Where AI is genuinely excellent on a mainframe estate

Take one recommendation from this post: run an AI-driven comprehension pass over the estate before you decide anything about target architecture. It is the highest-return work available to you.

It maps the estate faster than any team could. A capable model pointed at your source library produces a call graph across programs, paragraphs and sections, resolves COPYBOOK inclusion, and follows control flow through PERFORM THRU chains no human wants to trace. Then it binds that graph to your JCL: which step runs which program, which DD statements point at which datasets, which GDG generations flow between steps, which condition codes gate execution. That is the batch schedule your architects have wanted for years, derived from code rather than from a stale Visio diagram.

It reconstructs data lineage that exists nowhere else. Which programs write WS-PREMIUM-NET? Which COPYBOOK defines it, which REDEFINES clauses alias it, which DB2 column does it land in, which extract feeds the regulatory report? One logistics operator used this to find that a rate field on customer invoices came from a table maintained by a program decommissioned in 2019. The values had been frozen for years and nobody had noticed.

It recovers business rules into language a business person can argue with. Procedural COBOL resists reading: nested IFs, level-88 conditions, flags set eight paragraphs earlier, a comment from 1996 describing a rule that changed twice since. AI produces a structured statement of the rule and its exceptions. The word that matters is argue: the value is not that the specification is correct, it is that it is specific enough for your COBOL expert to say “no, that exception went away after the 2011 rate filing.”

It generates characterization tests against current behaviour. Not tests of what the code should do, tests of what it does. That is the right test for a migration, where your requirement is not correctness but sameness.

It finds the code that doesn’t run. Unreachable paragraphs, programs no JCL invokes, conditions that can never be true because a flag is a constant three programs upstream. Every line you delete is one you never translate, test or validate. Confirm against production traces first: “unreachable” and “reached once a year at fiscal close” look identical in static analysis.

It translates well-isolated modules cleanly. A pure computation routine that takes a record, applies arithmetic and returns a value, with no file I/O and no shared state, converts well and there are more of them than you think.

Where AI degrades, and why every failure is silent

Automated AI translation fails not because models are bad at Java. They are very good at Java. It is that COBOL on z/OS carries semantics with no clean equivalent in the target runtime, and the mismatch almost never produces a compile error. It produces a plausible code that computes a different answer. A translation error that crashes is a gift; it appears in the first test run. One that returns a number two cents off appears in an audit eighteen months later, applied to four million transactions.

Fixed-point money, truncation and IEEE 754

PIC S9(13)V99 COMP-3 is packed decimal: two digits per byte, sign in the low nibble, exact base-10 arithmetic. Map it to double and you have put binary floating point in your general ledger, with error accumulating across every proration chain. Map it to BigDecimal and you are doing the right thing, but correctness now rests on every operation specifying the right scale and rounding mode. COBOL’s ROUNDED clause, the default truncation without it, and the standard’s rules for intermediate precision inside a compound COMPUTE must all be replicated exactly. ON SIZE ERROR compounds it: COBOL leaves the receiving field unchanged when a result won’t fit, while Java throws or overflows. Implicit truncation is the same class of problem: MOVE 12345 TO WS-CODE where WS-CODE is PIC 9(3) silently gives you 345, and legacy code depends on it.

REDEFINES and memory overlay

REDEFINES gives a second view over the same storage, used constantly to parse records whose layout depends on a type byte. Naive translation gives each view its own field, so the program writes through view A and reads a stale value through view B, and records parse as the wrong type exactly where the type byte is unusual. Correct translation needs a shared byte buffer with encode and decode on every access, which looks nothing like the clean class in the demo.

GO TO ... DEPENDING ON, ALTER and level-88

GO TO A B C DEPENDING ON WS-IDX is a computed branch, and if WS-IDX falls outside 1 to 3, COBOL does not branch at all. It falls through. A translator emitting a switch with a throwing default has turned a silent no-op into a runtime failure.

ALTER is worse: it rewrites the target of a GO TO at runtime, so control flow is not statically knowable, and a model asked for “clean Java” will hard-code the most recently altered target it can see. Estates older than about 1990 usually contain it. Flag every occurrence for human translation.

Level-88 condition names lose two things: SET VALID-STATE TO TRUE writes a value back to the parent field, which translators drop because the construct looks read-only, and VALUE 'AL' THRU 'AZ' is a range test against the native collating sequence.

EBCDIC collation versus ASCII

This causes more post-migration incidents than anything else on the list, and it is almost never in the vendor’s test suite. On z/OS the native collating sequence is EBCDIC: lowercase sorts before uppercase, and digits sort after letters. In ASCII and UTF-8, digits sort before letters and uppercase before lowercase.

So every IF FIELD-A > FIELD-B on an alphanumeric field, every THRU range, every SORT key, every binary search over a table, every report control-break and every “greater than this account number” check changes result after the move. Reports come out in a different order, control-break logic groups records differently, and a range check over account prefixes now includes nothing or everything. It compiles and runs, and the output looks reasonable to anyone not holding the old version next to it.

Signed overpunch fields

PIC S9(5) in DISPLAY usage stores the sign in the zone nibble of the last byte, so positive 12345 renders as 1234E and negative as 1234N. Pull that file off as text, read it as a string, and the last digit is corrupted systematically and invisibly until someone sums the column. Every file-based interface between old and new hits this.

The difference between empty and absent

COBOL returns a file status on open: 00 for success, 10 for end of file on an empty file, 35 when the file does not exist. Batch logic distinguishes these carefully. An empty transaction file means “no activity today, post zeros and continue.” A missing file means “the upstream feed did not arrive, do not close the books.” Java file I/O collapses them. A translator that wraps FileNotFoundException and an empty stream in one try/catch returning an empty list has told your general ledger that a failed feed was a quiet day. That surfaces only on the day something upstream breaks, which is precisely the day you need the system to be right.

CICS pseudo-conversational state

CICS online programs hold no state in memory between screens. The transaction ends, state goes to the COMMAREA or a channel, EXEC CICS RETURN TRANSID schedules the next, and EIBCALEN = 0 signals first invocation. Syncpoint boundaries are explicit and define what backs out on failure. Translate that into a stateful HTTP session and you have changed the recovery semantics: state that would have been discarded now survives, and a retry that would have started clean resumes from a partial update. Translate it to stateless REST and you must reconstruct the syncpoints yourself, and getting them wrong produces partial commits CICS would have rolled back.

VSAM access patterns against a relational target

STARTBR / READNEXT / ENDBR is a positioned browse: position held across calls, defined ordering on duplicate alternate-index keys, record-level locking. The natural SQL translation is a cursor with ORDER BY, and it is not the same thing. Ties break differently, rows inserted mid-browse may or may not appear, and your new database’s isolation level is almost certainly not equivalent to VSAM’s. Programs that browse and update in the same loop are the highest-risk category in the estate. Add to the risk register: uninitialized WORKING-STORAGE assumed to be LOW-VALUES, OCCURS DEPENDING ON tables read past their bound, INSPECT and UNSTRING overflow, and HIGH-VALUES used as an end-of-time sentinel. Not one produces a stack trace.

The JOBOL trap, and when it is still the right call

Suppose translation works. You now have five million lines of Java structurally identical to five million lines of COBOL: same procedural flow, same 4,000-line programs, same global data rendered as fields on a god object. The industry calls this JOBOL. Your COBOL specialists can no longer maintain it, because it is Java. Your Java developers will not, because it is not Java in any sense they recognize, and the ones you hire for it leave within a year. You have taken a system a shrinking pool of people understood and converted it into one nobody understands.

This is not an argument against ever doing it. Getting off the mainframe has value on its own, and transliteration is a legitimate intermediate state under one non-negotiable condition: the refactoring budget is committed and staffed at the same time as the translation budget, with the same sponsor and the same board visibility. The moment refactoring becomes phase two, subject to a separate business case in a later fiscal year, you have created a permanent state and called it temporary.

Equivalence validation is the actual project

Here is where your calendar goes, and where your money should go.

Build the parallel run before you build anything else

Stand up an environment where the mainframe and the target run the same inputs on the same schedule and produce comparable outputs. Not at the end of the project: at the start, using a trivially converted module as the first payload.

For batch, that means identical input datasets to both and a comparison job examining output files record by record, database deltas row by row and journal entries event by event, against a canonical serialization so formatting noise does not masquerade as behavioural difference. Choose replay windows deliberately: an ordinary business day, a month-end close, a year-end, a leap day, a day with a known production incident, and your industry’s peculiar peak. Run against real historical inputs and compare against the real historical outputs your retention policy means you still hold.

For online, it means shadow traffic: tap CICS transaction inputs at the region, replay against the target, compare responses. The critical constraint is side-effect isolation. The shadow writes to a shadow database, and every non-idempotent external call, payments, letters, SWIFT messages, bureau enquiries, is stubbed with a recorded response. Getting that wrong sends duplicate payment instructions, a far worse day than any translation bug.

Triage the diff, do not just count it

The first run produces thousands of differences, most of them noise. Build a classifier that sorts each diff into five buckets:

1. Formatting — leading zeros, trailing spaces, line endings. Auto-suppress after review.
2. Precision and rounding — differences under a declared threshold. Route to the exceptions register.
3. Ordering — same records, different sequence. Usually collation. Fix the cause, suppress the symptom.
4. Run metadata — timestamps, job IDs, sequence numbers. Auto-suppress by field allowlist.
5. Behavioural — everything else.

Humans should only ever see bucket five. If your team is reading bucket one by hand, the programme will not finish.

Decide acceptable divergence in writing, and have the business sign it

Engineers keep making this business decision by accident. Before triaging a single diff, get a policy signed by the accountable business owner covering rounding tolerance as an absolute amount and a per-transaction rate, not “small differences are fine”; who absorbs a half-cent and how it is disclosed; timestamp precision; sort stability for equal keys; and which fields are exact-match at zero tolerance, which should include balances, report totals and anything feeding a filing.

On sort stability, DFSORT behaviour depends on the EQUALS / NOEQUALS installation default, and there is a real chance nobody at your organization knows which way yours is set. Every accepted divergence gets an exceptions-register entry with an owner, a rationale and an expiry date. Registers without expiry dates become permanent.

Curate a golden dataset around pathology, not volume

Random sampling of production gives you the common cases, which already work. What you need are the structurally weird records: the policy opened in 1974 with a layout predating the current COPYBOOK, the account with 400 transactions in one day, the negative zero balance, the record with LOW-VALUES in a numeric field. Sample by structure rather than volume, and mask it properly, because production data in test is a GDPR and PCI DSS 4.0 problem. But mask in a way that preserves pathology: format-preserving masking that turns every account number into a well-formed one destroys the exact edge cases you built the dataset for.

Set the bar at one full financial close and one year-end

The completion criterion is not a coverage percentage. It is this: the target has run a complete financial close in parallel with the mainframe, with zero unexplained bucket-five differences, and it has survived a year-end. Year-end is the largest and strangest event in most financial calendars, and a cutover that has not run one in parallel is a cutover done on hope. Regulated entities should add a full regulatory reporting cycle.

Mutation-test the tests, because coverage lies

AI-generated characterization tests fail in a specific way: they pass vacuously, asserting on fields the code never modifies, or asserting that output equals whatever the code produced without checking the assertion discriminates anything. The remedy is mutation testing against legacy semantics. Introduce faults deliberately: flip a rounding mode from HALF_UP to HALF_EVEN, shift a boundary by one, invert a comparison, change a truncation length. Then check whether the suite fails. Report mutation score, not line coverage. A suite with 90% coverage and a 40% mutation score would not notice if the code were wrong.

Non-determinism breaks your build and your audit trail

Run the same prompt against the same model on two different days and you can get two different programs. Both may be correct. They will not be identical. Reproducible builds assume the same source produces the same artifact; if generation sits inside your pipeline, you can no longer answer “is the binary in production built from the source in the repository.”

The fix is architectural. Generated code is source of record. It goes into version control, gets reviewed, and is never regenerated as part of a build. Pin and record everything alongside it: model version, prompt text, tool versions, sampling settings, a hash of the input source. Attach provenance metadata to every generated file: which model, which prompt, which source paragraph range, who reviewed it, which validation cycle it passed.

Under SOX ITGC review an auditor asks who approved the change, what testing evidence exists, and whether you can reproduce the artifact. Under DORA a financial entity is asked about dependency on a third-party model provider and what happens when that model is deprecated. And someone will eventually ask why a specific line computes what it computes. “The model wrote it” is not an answer. Traceability to the originating COBOL paragraph is.

Adoption has outrun verification practice

The Stack Overflow 2025 Developer Survey found 84% of developers using or planning to use AI tools, up from 76% the year before, with 51% of professional developers using them daily. In the same survey, developer trust in the accuracy of AI output fell to 29%, down from 43% in 2024.

Usage up, trust down. That is not irrational; it is what mature tool use looks like, but only if verification practice grows to match. GitHub puts the AI-generated share of code at roughly 46%, up from 27% in 2022, and verification tooling has not grown by anything like that factor. McKinsey reports that more than 80% of companies see no material earnings contribution from generative AI initiatives.

Your throughput constraint is not how fast code appears. It is how fast you can establish the code is right. Adding generation capacity to a system whose bottleneck is verification produces a larger queue, not a faster program.

Your COBOL expert is worth more now, not less

The pitch you have heard is that AI removes your dependency on scarce COBOL skills. The opposite is true for the duration of the project. Roughly 220 billion lines of COBOL run in production, the average COBOL programmer is around 55, and about 10% retire each year. That is your window, and AI does not widen it. In an AI-assisted program the expert stops writing COBOL and does four things nobody else can:

1. Oracle. When the parallel run produces a difference, they say whether the mainframe or the target is right, and why. Often the mainframe is wrong and has been for years, and the business has adapted. That is a decision, not a bug fix.
2. Adjudicator of recovered rules. AI drafts the specification; they confirm, refute or annotate it at perhaps ten times the rate they could have written it.
3. Risk router. They mark which programs go to automated translation with review and which go to human-led rewrite. ALTER, browse-and-update loops and anything touching money at the cent level go in the second pile.
4. Memory capture. Every session with a model and an analyst converts tacit knowledge into artifacts that outlive them.

Staff one senior COBOL analyst per two to three million lines during comprehension, and keep at least one through the full validation cycle. Tie retention to the validation milestone, not the cutover date: the cutover date will move, and the month you most need them is the one after go-live.

Ten questions to ask an AI modernization vendor

A good vendor answers several of these with “that one is hard, here is how we contain it.” Certainty across all ten is the negative signal.

Frequently Asked Questions

Is AI-assisted COBOL modernization actually faster than a traditional rewrite?

Much faster on comprehension, where a phase that took twelve months can compress to two or three. Faster on code production, though that was rarely the constraint. Roughly neutral on validation, which is where most of the calendar sits. Expect overall compression of 20% to 40% on a well-run program, not the 5x in the vendor deck, and most of that comes from starting validation earlier rather than from generating code faster. Programs that miss even that range are usually the ones that treated the parallel run as a testing phase at the end rather than as infrastructure built on day one.

What is a realistic accuracy expectation for automated COBOL-to-Java conversion?

A single percentage is the wrong measure, which is why figures around 93% are marketing rather than engineering. Accuracy varies enormously by construct. Pure computation modules convert at very high fidelity. Programs with REDEFINES, browse-and-update loops, CICS state, or packed-decimal arithmetic inside a compound COMPUTE convert far less reliably, and those are the programs that calculate the numbers your business reports. Ask for accuracy broken down by construct class, and treat an unwillingness to provide it as informative in itself.

Can we skip the parallel run if we have good test coverage?

No, and coverage is the reason. Coverage measures which lines execute, not which behaviours are correct, and legacy estates are full of behaviours nobody wrote down. A parallel run against real historical cycles is the only mechanism that exercises the input distribution your business actually produces, including records predating your current data standards and the ones created by a bug that was fixed in 2008. If budget forces a choice between more generated tests and a parallel run environment, build the parallel run. You can add tests later; you cannot retroactively prove a cutover was safe.

How do we satisfy an auditor about AI-generated code in a regulated system?

Treat it exactly as you would code from an offshore vendor: in version control, reviewed by a named person, tested, and traceable to a requirement, which here is the originating COBOL paragraph. Add provenance metadata recording model version, prompt and input hash, and do not regenerate at build time. Under SOX the questions are approval, evidence and reproducibility. Under DORA, add concentration risk on the model provider and your plan for when that model is deprecated. Expect the same conversation about your test evidence, not just your code.

The Bottom Line

AI has changed legacy modernization, and the change is real. It made the comprehension problem tractable for the first time since these systems were written. A capable model will tell you more about your mainframe in three weeks than your organization has learned in fifteen years.

What it has not changed is the nature of the work. The hard part of moving a system that has run correctly for four decades was never producing new code. It was proving the new code does the same thing, and that proof comes from parallel runs, historical replay, disciplined diff triage and a business owner willing to sign a divergence policy. None of it is automated.

Use the 93% figure as a filter, not a target. When a vendor leads with it, ask what the other 7% consists of, where it lives, and how you would find it before your regulator does. The quality of that answer tells you more than any demo will.

If you are being asked to approve an AI-assisted mainframe program this quarter, the useful first step is smaller than the one you are being sold. Modernize Software runs a two-week estate comprehension and validation-readiness assessment: an AI-driven map of your COBOL, JCL and data lineage, a construct-level risk breakdown of what will and will not translate cleanly, and a costed design for the parallel run environment you will need either way. You get a defensible scope before you commit a budget, which is the order those two should happen in.