Is automated COBOL to Java conversion reliable?

← All posts

Yes, with a condition attached that changes the answer completely depending on which kind of automation you mean.

Automated conversion have successfully produced running production workloads for 3 decades. Banks, defence programmes and government agencies have moved COBOL estates to Java and C# using automated toolchains, and those systems process transactions today.

What’s changed since 2023 is that “automated conversion” now describes 2 fundamentally different things, and they have different reliability profiles. One is deterministic transformation, whered correctness comes from the transformation rules. The other is generative conversion, where a language model produces a new system that then has to be checked.

Both get marketed with the same words. They fail in completely different ways.

This piece works through the published evidence on each, where the failure modes actually live, and how to verify converted output regardless of which approach produced it.

What reliability means here

Before the evidence, a definition, because the industry uses “reliable” loosely.

A converted program is reliable if it produces the same observable behaviour as the original for every input the original would accept. Same outputs, same file writes, same error conditions, same rounding.

That’s a higher bar than “it compiles”. It’s a higher bar than “it passes the tests”, because most legacy estates have thin test coverage and passing a weak suite proves very little.

Three distinct failure classes matter:

• Structural failures. The output doesn’t compile, or crashes. Annoying, cheap, and self-announcing. These aren’t the dangerous ones.
• Semantic drift. The output compiles, runs, passes tests, and behaves differently on some inputs. A rounding difference in the fourth decimal place. A sign handling difference on a negative packed decimal. A collation difference when EBCDIC ordering meets Unicode ordering.
• Maintainability failure. The output is behaviourally correct and structurally unusable. Java that’s still COBOL underneath, sometimes called “Jobol” in comparison content. It works, and your team can’t change it.

Semantic drift is the one that ends careers, because it reaches production and stays there until a customer notices.

What the benchmarks actually show

There’s now a real body of published evidence on generative code migration, and it’s more specific than the marketing on either side.

Accuracy falls sharply with codebase size

This is the most consistent finding across independent benchmarks.

RepoMod-Bench reports average pass rates of 91.3% for codebases under 10,000 lines, dropping to 15.3% above 50,000 lines. MigrationBench reports 71.67% pass@1 for minimal migration tasks against 53.33% for maximal ones. Microsoft’s TRANSREPO-BENCH found state-of-the-art models reaching 26.65% accuracy under realistic repository conditions.

The pattern holds across research groups with different methodologies. Small samples convert well. Repository-scale codebases don’t.

That matters because enterprise COBOL estates are measured in millions of lines. The demo is running in the region where accuracy is high. Your estate isn’t.

Models don’t reliably catch their own errors

The most useful single study came from AWS-affiliated researchers in 2026 (arXiv:2605.21537).

They ran 1,980 modernization calls across 11 production models from 7 model families, evaluated every output with a behavioural oracle rather than a similarity score, and then asked each model to review its own work.

Semantic-preservation drift appeared in 39.7% of attempts on code containing semantic traps, against 7.0% on benign control code. Per-model drift ranged from 5.6% to 46.7%, and here’s the part that should stop you: drift didn’t track model capability. Stronger models weren’t reliably safer.

On self-review, models identified only 68.3% of cases where they’d introduced drift. Roughly a third of the errors went unflagged by the system that made them.

There’s a structural reason for that. A model that misunderstood the semantics badly enough to introduce drift is using the same understanding to review the output. It isn’t checking against the source semantics, it’s checking against its own reading of them.

That’s the case for an independent oracle, and it holds no matter how good the models get.

Where the traps live in COBOL specifically

The AWS study used the term “semantic traps” for constructs where a plausible-looking translation is wrong. COBOLhas many of them.

• Packed decimal arithmetic. COBOL’s COMP-3 fields hold exact fixed-point decimal values. Java primitives float and double are binary floating point. Converting financial logic to floating point introduces rounding errors that accumulate silently across millions of transactions.
• ROUNDED semantics. COBOL’s ROUNDED clause and Java’s rounding modes don’t align by default. Half-up, half-even and truncation produce different results, and financial regulation frequently mandates a specific mode.
• REDEFINES. A single storage area interpreted multiple ways depending on context. There’s no direct Java equivalent, so every instance is a design decision, and getting it wrong produces code that reads the wrong bytes under conditions the tests never hit.
• OCCURS DEPENDING ON. Variable-length tables whose length comes from another field. Convert this as a fixed array and you’re either truncating data or reading garbage.
• Level-88 condition names. Semantically these are domain types. Convert them to boolean flags and you lose the constraint that made them safe.
• Sign handling on packed fields. Overpunched signs and separate sign bytes behave differently, and mainframe conventions here have no natural Java analogue.
• EBCDIC collation. Sort order in EBCDIC differs from ASCII and Unicode. Any code that depends on comparison ordering will behave differently after conversion unless the collation is handled explicitly.

The correct target for packed decimal is BigDecimal or a dedicated money type, and a conversion that reaches for double because it’s the obvious numeric type has already lost money. Multiple industry sources now flag this as the first thing to check in a vendor’s output.

Every one of these compiles fine. Every one passes a test suite that doesn’t specifically probe for it.

The other family: deterministic transformation

Generative conversion produces a plausible output and asks you to verify it. Deterministic transformation works from the opposite direction.

The source is parsed into a formal representation, usually an abstract syntax tree carrying full type and scope information. Transformation rules then rewrite that representation into the target language. Each rule encodes a semantic equivalence: this COBOL construct means the same as this Java construct, under these conditions.

Three properties follow, and they’re the reliability argument.

Repeatability. The same input produces the same output every time. That sounds mundane, and it’s the foundation of everything else. When output varies run to run, you can’t build a verification process, because you’re verifying a different artefact each time.

Rule-level correctness. When a defect appears, you fix the rule, not the instance. The fix then applies everywhere that construct appears across the whole estate. On a 2-million-line conversion, that’s the difference between one change and 400.

Traceability. Every line of output derives from an identifiable line of input via an identifiable rule. In regulated environments this matters enormously, because auditors ask how you know the new system does what the old one did, and “we reviewed it” is a weaker answer than a derivation chain.

Semantic Designs has been building this class of technology since 1995. The DMS platform fuses symbolic AI with enhanced compiler technology, handling the parsing, analysis and rule-driven transformation across more than 40 language front ends. The DMS platform page covers how the pieces fit together, and how deterministic software transformation works goes into the mechanics.

The trade-off is honest and worth stating. Building the rule set for a new dialect or an unusual construct takes engineering effort up front. Generative approaches need no such setup, which is exactly why they demo well and why they degrade at scale.

Where Gen AI genuinely helps

Reading the above as an argument against Gen AI would be wrong, and the industry’s tendency to treat this as a binary is unhelpful.

Gen AI is good at several jobs in a modernization programme, and they share a property: the output is reviewed by a human who can evaluate it directly, and a mistake is visible rather than silent.

• Documentation. Generating a description of what an undocumented program does. A human reads it and judges it. Errors are apparent.
• Test case drafting. Producing candidate test cases from code. Cheap, fast, and it covers branches manual testing never reached.
• Code explanation. Helping a Java developer understand what a COBOL routine is doing. Directly useful during validation.
• Review assistance. Flagging suspicious patterns in converted output for human attention. Useful as an additional filter, unreliable as the only one.

What these have in common is that they’re advisory. The dangerous application is putting generation in the path where the output goes to production and correctness is assumed.

Our piece on why Gen AI alone is not enough for modernization and the broader look at AI in software modernization develop this further.

What drift looks like in practice

Abstract descriptions of semantic drift are easy to nod along to and hard to act on. Here’s the concrete version.

Case 1: the rounding difference

A COBOL routine computes interest:

COMPUTE WS-INTEREST ROUNDED = WS-BALANCE * WS-RATE / 100

WS-INTEREST is defined as PIC S9(9)V99 COMP-3. Exact decimal, 2 places, rounded half-up by the COBOL default on most compilers.

A literal conversion reaches for the obvious Java numeric type:

double interest = Math.round(balance * rate / 100 * 100) / 100.0;

This compiles. It runs. It passes a test suite built from typical values. And it produces a different answer from the original on inputs that land exactly on the half, and on values where binary floating point can’t represent the decimal exactly.

On a single account the difference is a cent. Across 4 million accounts monthly, it’s a reconciliation break that someone spends 3 weeks tracing.

The correct target is exact decimal arithmetic with an explicit rounding mode:

BigDecimal interest = balance.multiply(rate)
    .divide(new BigDecimal("100"), 2, RoundingMode.HALF_UP);

Note what had to be decided there. The scale. The rounding mode. Both were implicit in the COBOL declaration and both have to be made explicit in Java, and getting either wrong produces output that looks right.

Case 2: the REDEFINES that reads the wrong bytes

01  TRANSACTION-RECORD.
    05  TXN-TYPE            PIC X.
    05  TXN-BODY            PIC X(80).
    05  TXN-PAYMENT REDEFINES TXN-BODY.
        10  PAY-AMOUNT      PIC S9(7)V99 COMP-3.
        10  PAY-ACCOUNT     PIC X(12).
    05  TXN-ADJUSTMENT REDEFINES TXN-BODY.
        10  ADJ-REASON      PIC X(4).
        10  ADJ-AMOUNT      PIC S9(7)V99 COMP-3.

The same 80 bytes mean different things depending on TXN-TYPE. There’s no Java equivalent, so the converter has to choose a representation: a class hierarchy, a tagged union, or a byte buffer with accessors.

Whichever it chooses, the guard condition matters. The original code was safe because every access was preceded by a check on TXN-TYPE, sometimes several paragraphs earlier. A conversion that produces both accessors without carrying the guard forward will happily read payment fields off an adjustment record, and the result will be a plausible-looking number rather than an error.

That’s the signature of dangerous drift. It doesn’t throw. It returns something.

Case 3: the sort that changes order

SORT SORT-FILE ON ASCENDING KEY SORT-CUSTOMER-ID

SORT-CUSTOMER-ID is PIC X(10) and contains a mix of digits and letters. In EBCDIC, lowercase letters sort before uppercase, and both sort before digits. In ASCII and Unicode, digits sort before uppercase, which sort before lowercase.

The converted sort is correct by every definition except the one that matters: it produces a different order, and any downstream logic that depends on sequence, such as control break processing or a first-match lookup, now behaves differently.

Nothing fails. A report comes out with rows in an order the business doesn’t recognise, and 3 months later someone notices a total is being attributed to the wrong grouping.

What these have in common

All 3 compile. All 3 run. All 3 pass a test suite built from representative data.

They’re only caught by testing that specifically probes the construct, or by comparing output against the original system on real production inputs. Generic coverage metrics say nothing about them, which is why a vendor quoting test coverage percentages hasn’t answered the question you asked.

The maintainability question

Behavioural correctness is necessary and insufficient. Code you can’t change is a liability even when it computes the right answer.

The failure mode has a name in comparison content: Java that’s structurally still COBOL, inheriting the maintenance difficulty of the legacy system without the mainframe’s performance characteristics.

Its signatures are recognisable in a 10-minute code review:

• Control flow scaffolding. COBOL GO TO and PERFORM THRU translated as boolean state flags and while loops, producing methods where the actual logic is buried under flow control.
• Flat data. Copybook fields mapped to long lists of primitive fields on a single class, with no type structure. Level-88 conditions reduced to boolean flags, losing the constraint they encoded.
• God classes. A 3,000-line COBOL program becomes a 3,000-line Java class with 40 private methods and shared mutable state, because the working storage section became instance fields.
• No domain types. Every identifier is a String, every amount is a BigDecimal with no unit, every date is a String in the original format.

The counter-position is that structure is a choice made by the conversion, not an inevitability. Transformation that maps paragraphs to cohesive methods, REDEFINES to type hierarchies, level-88s to enums or value types, and working storage to properly scoped state produces output a Java developer recognises as Java.

A practical test you can run in an afternoon. Ask for a converted sample. Then run a static analysis tool over it and look at 4 numbers: cyclomatic complexity distribution, method length distribution, class length distribution, and the ratio of domain types to primitives. Compare against a greenfield Java codebase you consider good.

If the distributions are wildly different, you’re looking at translated COBOL regardless of what the vendor calls it.

How to verify converted output

Regardless of approach, here’s what a defensible verification process looks like.

Build an oracle that isn’t the converted system

You need a source of truth about correct behaviour that’s independent of whatever produced the output.

The strongest is production behaviour itself. Capture real inputs and outputs from the running system, replay against the converted system, compare. This catches drift on the actual distribution of inputs your business generates, which is where it matters.

Batch output comparison is the cheapest version. Run both systems on the same input, diff the outputs field by field. Strong for batch-heavy estates, weak for interactive and time-dependent behaviour.

Probe the traps deliberately

Generic test coverage won’t find semantic drift, because drift hides in edge cases that ordinary tests don’t reach.

Build a specific test set for the constructs listed above. Negative packed decimals. Boundary values on OCCURS DEPENDING ON tables. Rounding at the half. Sort ordering across the EBCDIC and ASCII boundary. Maximum and minimum values on every numeric field.

This is a small, targeted suite and it finds more real defects per test than any other testing you’ll do.

Test with production-shaped data

Synthetic test data tends to be well-behaved, and well-behaved data doesn’t exercise the paths where drift lives.

Real data contains the record written in 1994 with a field format that changed in 1997, the customer whose name has a character the new encoding handles differently, and the transaction that hits the exact rounding boundary. Use masked production data if regulation requires it, but use production-shaped data.

Don’t accept self review as verification

If the verification story is that the same system checked its own output, the published evidence says roughly a third of the drift will get through.

Independent verification means a different mechanism, ideally a different kind of mechanism.

Require evidence, not assurance

Ask what artefact you receive that demonstrates equivalence. A test report with coverage figures. A comparison log. A derivation trace. Something a third party could examine.

“Our platform is 99% accurate” is a claim about someone else’s projects.

The regulated-industry angle

In financial services, insurance, healthcare and defence, reliability isn’t only an engineering question. It’s an evidence question.

Regulators increasingly ask organisations to demonstrate control over critical systems rather than assert it. The EU’s DORA framework places obligations on the management body personally, and the NIS2 directive introduces comparable duties across a wider set of sectors.

That changes what a conversion has to produce. “We reviewed the output and it looked correct” is a statement about process. What an auditor wants is an artefact.

Three things worth having in the file:

• A derivation record. Which rule produced which output from which input. Deterministic transformation generates this naturally because the mapping is formally specified. Generative approaches can’t, because there’s no rule to point at.
• An equivalence test report. Coverage of the trap constructs, results of production replay comparison, and a documented explanation for every accepted difference.
• A change control record for the residual. Every program a human edited, who edited it, what changed, and how the change was verified.

Organisations that assemble this during the programme find the audit straightforward. Organisations that try to reconstruct it afterwards usually can’t, because the information wasn’t captured.

This is worth raising early with your risk and compliance function, because their requirements will shape the verification plan and it’s cheaper to design for them than to retrofit.

What to ask a vendor

Eight questions that separate a real reliability story from a marketing one.

1. Is your conversion deterministic? Does the same input produce the same output on every run?
2. How is COMP-3 arithmetic handled in the target, and can you show me the generated code?
3. Which rounding mode does the output use, and is it configurable per field?
4. How are REDEFINES handled, and who makes the design decision?
5. What’s your verification method, and what evidence do we receive?
6. When a defect is found in the output, is the fix applied at rule level or instance level?
7. What percentage of programs go to test with no human editing of the output?
8. Does the converted code depend on a proprietary runtime?

Question 6 tells you more about the underlying technology than any of the others. Instance-level fixing means each defect is handled individually, which doesn’t scale. Rule-level fixing means the system has a formal model of the transformation, and one correction propagates.

So, is it reliable?

Deterministic transformation is reliable in a way that can be reasoned about, because correctness is a property of the rules rather than a property of each individual output. It’s been running production systems for decades. It requires up-front engineering, and it doesn’t produce a magic instant result.

Generative conversion is reliable on small, clean, well-understood code, and the published benchmarks show it degrading substantially as codebases grow. It’s genuinely valuable in the advisory roles around a migration. Putting it on the critical path to production without independent verification is a risk the evidence doesn’t support.

The most common mistake isn’t picking the wrong approach. It’s not knowing which one you bought, because both get sold with the same vocabulary.

Ask question 1 on the list above. The answer tells you which category you’re actually in.

Frequently asked questions

Can automated conversion produce maintainable Java, or just working Java?

Both are achievable, and they’re separate properties. Behavioural correctness and structural quality are different goals, and some approaches optimise for the first while producing output that mirrors COBOL structure. Ask to see generated code from a comparable project before signing.

What automation rate should I expect?

Published figures cluster at 70% to 85%, up from around 40% in 2020. The number depends heavily on the denominator used, so ask what percentage of programs go to test with no human editing rather than what percentage of lines converted.

Is AI-generated COBOL conversion safe for financial systems?

Not without independent verification. Financial logic is exactly where packed decimal and rounding semantics live, and those are the constructs where published research shows the highest drift rates.

How do I know if my converted code has semantic drift?

Replay production inputs against both systems and compare outputs field by field, then run a targeted suite against the specific constructs known to be trap-prone. Generic test coverage won’t find it.

Do I need to keep my COBOL developers?

Through validation, yes, and they’re the most valuable people in the programme. They’re the only ones who know what the system is supposed to do, which is the knowledge no tool extracts.

Does deterministic transformation work on any COBOL dialect?

It works on any dialect with a front end built for it. Established platforms cover the major dialects, and unusual or proprietary dialects need front-end work up front, which should be identified during assessment.

Next step

The reliability question is answerable on your own code faster than on anyone else’s.

Modernize Software runs a free codebase assessment that identifies your dialects, flags the constructs that carry conversion risk, and produces a complexity profile you can plan against.

The COBOL to Java migration and modernize COBOL pages cover how the transformation handles the specific constructs discussed above.