How to evaluate a Gen AI COBOL conversion tool before you pilot it

← All posts

Every organisation with a mainframe is currently being offered automated COBOL conversion. The demos are good. The output may compiles. The generated code reads well, carries documentation comments, and arrives faster than anyone expected.

The difficulty is that compiling, reading well and arriving fast are not the properties you are buying. You are buying behavioural equivalence on a system that produces numbers your business is accountable for. A demo cannot show you that, and a six week pilot on a well-behaved subsystem often cannot either.

What follows is a test list rather than a vendor list. These are the constructs that separate a translator that understands COBOL semantics from one that produces confident, plausible output. You can run them yourself, using your own code, before committing a budget.

Why a normal pilot does not settle the question

The standard pilot selects a contained subsystem, converts it, compiles it, runs the generated tests, and reports success. Three structural problems make that reassuring rather than informative.

The subsystem chosen is usually the healthy one

Teams pick a well-understood, well-structured, moderately sized application for the pilot, because that is the sensible way to control risk. It is also the part of the estate least likely to contain the constructs that break translators. The pilot passes and tells you nothing about the twelve thousand line program with the 1987 patch in it.

Generated tests validate the translator’s own reading

When the same system produces both the translation and the test suite, the tests encode whatever the translator understood. If it misread the semantics, the tests assert the misreading. A fully green suite is consistent with a wholly incorrect translation, which is why independent verification matters more than test count. Researchers building validation tooling for commercial COBOL to Java products have stated the point directly: translated output cannot be assumed to reproduce the original, which is precisely why validation frameworks are needed.

Compilation is a weak signal

Java that compiles has satisfied the type system. It has not satisfied your ledger. Almost every failure mode described below produces code that compiles cleanly and runs without error, and differs only in the values it produces.

The seven constructs to test

Assemble a sample from your own estate containing each of the following, convert it, and inspect the output. If a vendor will not run their tool on code you supply, that is itself a finding.

1. A copybook chain three levels deep

Take a program whose numeric fields are declared in a copybook that itself includes another copybook. Something in the shape of:

05  WS-DAILY-INT-RATE     PIC S9(3)V9(8) COMP-3.
05  WS-ACCRUED-INT        PIC S9(9)V99   COMP-3.

Then look at what those fields became in the target language.

Packed decimal is exact base ten arithmetic. If either field arrives as a Java double, the translator summarised the copybook rather than parsing it, and the estate will develop financial drift that is invisible for weeks and compounds thereafter. The correct outcome is scaled decimal arithmetic with the declared precision carried into the type.

This is the single highest-value test on the list, because the defect is undetectable by reading the arithmetic in the converted program. The evidence sits in a file the reviewer is not looking at.

2. An unrounded COMPUTE

COBOL’s COMPUTE truncates toward zero unless ROUNDED is specified. Find a calculation in your estate that omits ROUNDED, and check whether the generated code truncates or rounds.

A translator that produces BigDecimal with HALF_UP has made a defensible engineering choice and the wrong behavioural one. Your balances are the accumulated product of decades of truncation. Rounding is not an improvement, it is a pricing change without a change request.

The deeper question this test answers is whether the tool distinguishes between correct arithmetic and equivalent arithmetic. Those are different objectives, and only one of them is what a migration requires.

3. A PERFORM THRU range with a GO TO inside it

This is the structural test.

    PERFORM 2100-APPLY-PENALTY THRU 2100-EXIT.
    ...
2100-APPLY-PENALTY.
    IF LATE-DAYS > 15 ...
2140-CHECK-WAIVER.
    IF PRODUCT-CLASS = 'G' GO TO 2150-GRANDFATHER-WAIVER.
2150-GRANDFATHER-WAIVER.
    MOVE ZERO TO WS-PENALTY-AMT.
2100-EXIT.
    EXIT.

PERFORM THRU executes a range of paragraphs with control flowing across paragraph boundaries inside the range. Maintenance patches frequently exploit that behaviour, jumping into the middle of a range to apply a conditional override.

A translator that treats each paragraph as an independent method and calls them in sequence produces readable Java in which the jump target is unreachable. The override silently stops firing. Nothing fails. A subset of accounts simply gets different treatment.

Check the generated code for reachability of the jump target. Published evaluations of commercial tools on control-flow-heavy COBOL report failures in this family consistently, including generated recursion where the original had iteration and invented control flow with no counterpart in the source.

4. A sorted report and a key comparison

Supply a program that sorts on an alphanumeric key mixing letters and digits, or compares a field against a literal range.

EBCDIC and ASCII order characters differently. In EBCDIC, digits occupy X’F0’ through X’F9’ and collate after letters. In ASCII, digits occupy X’30’ through X’39’ and collate before them. Lowercase precedes uppercase in EBCDIC and follows it in ASCII.

The test is whether the tool addresses collation at all. If sort order is not mentioned in the output, the design or the documentation, the converted system will produce correctly calculated values in the wrong sequence, break control break logic, and fail any binary search over a table whose order has changed. We cover the mechanics in EBCDIC to ASCII sort order.

5. A REDEFINES with variable-length records

Give the tool a record layout where the same storage is described two ways, with a discriminator field deciding which view applies, ideally alongside an OCCURS DEPENDING ON table.

Look at whether the generated code preserves the discriminator logic or collapses the layouts into one structure. Collapsing produces code that reads the wrong bytes under specific conditions, which typically means specific record types, which typically means specific customers.

6. A dynamically resolved CALL

Include a CALL whose target is held in a working storage variable rather than a literal. Ask what the tool determined the possible targets to be.

Resolving this requires data flow analysis across the program to establish the values the variable can hold. A tool that cannot answer produces an incomplete call graph, which means it cannot tell you what it has not converted. Incomplete conversion that reports as complete is worse than a tool that declines the work.

7. Embedded SQL with host variables

Supply a program with embedded SQL, including a cursor and a host variable structure.

The questions are whether host variable types survive the conversion consistent with the column types, whether cursor semantics are preserved, and whether SQL dialect differences that change results, rather than just syntax, are addressed. Null handling, date and timestamp behaviour and isolation levels all differ between platforms in ways that alter output rather than causing errors.

Reading the output like a maintainer

Beyond correctness, inspect the shape of what comes back. Your Java team will inherit it.

Look for the structure that was not there

COBOL has no objects. A translator that produces classes has inferred them from somewhere, and where that inference came from matters. Classes derived from record layouts and program boundaries are defensible. Classes that are paragraph methods hung on a single god object are transliteration wearing object-oriented clothing.

Ask what the tool did with .BAS-equivalent constructs: standalone programs with no natural object model. Static utility classes containing every routine is the common answer and the one that produces an unmaintainable result.

Look for surviving global state

If working storage becomes fields on a single large class accessed from everywhere, you have COBOL’s global data model expressed in Java syntax. It will compile, run and resist every subsequent change. This is the failure mode we cover in why converted COBOL looks like Java but is not.

Look for the comments

Generated documentation is easy to produce and easy to be impressed by. Check whether it describes what the code does or restates what the code says. Restating is a signal that the tool worked at the syntax level.

Questions to put to the vendor

Six questions that are hard to answer well.

Is the translation deterministic? If the same input produces different output on different runs, every fix is local and nothing is guaranteed to stay fixed. Determinism is what makes a defect traceable to a rule, and a rule fix apply everywhere the pattern occurs.

How do you handle the copybook chain? Specifically: is it parsed as the compiler parses it, with full expansion, or summarised?

How is functional equivalence demonstrated? Acceptable answers involve running the legacy and converted systems side by side on production-shaped data and comparing outputs. Test pass rates on generated tests are not equivalence evidence.

What did the tool refuse to convert? A tool that converts everything either handles every construct in your estate or silently produced something for cases it did not understand. Ask for the exception list.

Can we keep working during the project? Estates do not freeze for a year. The answer should describe tuning rules against a working copy and running the final conversion on a fresh snapshot.

What happens when we find a difference in month four? The answer reveals the architecture. Rule-based systems fix once and reapply. Probabilistic systems repair locally.

What Gen AI is genuinely good at here

None of this argues that generative tooling has no place on a mainframe estate. It argues that translation is the wrong job for it.

Comprehension is the right job. Explaining what a 1991 paragraph does to an engineer who was not born when it was written, drafting documentation for programs that never had any, summarising unfamiliar modules during an investigation, and proposing test scenarios that humans then own and harden. In all of those, a person checks the output before anything depends on it, which is exactly the condition under which probabilistic tooling is safe.

Translation has a right answer, the answer is auditable, and the ledger grades it nightly. That belongs to machinery that produces the same output for the same input every time. The fuller argument is in why Gen AI alone is not enough for modernization.

Scoring what comes back

Running the tests produces a pile of generated code. Turning that into a decision requires a scoring approach, because vendors will contest individual findings and the conversation needs a structure.

A rubric that survives a vendor meeting

Any fail in the first three rows is disqualifying for a system of record, because each produces incorrect values or dropped logic with no error raised. The remaining rows are negotiable depending on how the vendor proposes to handle them.

Weight by what your estate contains

The rubric should be weighted by the frequency of each construct in your own code. An estate with heavy IMS usage and little embedded SQL weights differently from one that is Db2 throughout. That frequency count is itself an analysis output, and producing it is worth doing before the evaluation rather than after.

Record what the tool did not attempt

A tool that converts everything either handles every construct present or produced something for cases it did not understand. Ask explicitly for the exception list, the unresolved construct list and the manual intervention list. A vendor who cannot produce those is telling you the tool does not track them, which means neither will you.

The commercial questions that matter as much as the technical ones

Six questions that shape the engagement rather than the output.

What happens to rules we pay to develop?

Translation rules tuned to your estate are an asset. Whether you retain access to them, whether they can be rerun, and what happens if the relationship ends are commercial terms worth settling early, particularly for a programme that will run in waves over several years.

How is progress measured?

Lines converted is a poor measure because it rewards volume over correctness. Better measures: constructs handled without manual intervention, programs passing output comparison against baseline, and the size of the open difference queue.

Who owns verification?

If the vendor both converts and verifies, the independence problem described earlier applies at the programme level rather than the tool level. The verification function should report somewhere other than the conversion function, even when both are supplied by the same organisation.

What is the position on partial conversion?

Estates rarely convert entirely. Establishing up front how residual components are handled, whether through retained interfaces, reimplementation or exclusion from scope, prevents the situation where a programme reaches ninety percent and stops.

How does the engagement handle ongoing development?

Your teams will continue to change the legacy code throughout. The answer should describe rules tuned against a working copy and a final conversion run against a fresh snapshot, so that changes made during the project are carried across rather than lost.

What does the difference queue look like at month six?

Ask for the shape of the defect curve on comparable engagements. A curve that flattens indicates systematic fixes. A curve that stays flat at a low level indefinitely indicates local repairs, each closing one case without closing the class.

What a good pilot looks like after the construct test

The construct test answers whether the tool understands COBOL. A pilot answers different questions, and it is worth running once the first test has passed.

Choose the sample for difficulty, not comfort

The instinct is to pick a clean, self-contained subsystem. That produces a pleasant experience and no information. A better sample includes at least one large program with a long modification history, at least one interface to another application, and at least one component whose behaviour nobody currently understands.

Run the comparison, not the test suite

Whatever tests the pilot produces, the acceptance evidence should be output comparison against the legacy system on production-shaped input. The pilot is the cheapest opportunity to establish that discipline before it matters.

Measure the manual effort honestly

Track hours spent on manual intervention, including work your own team does. Automation percentages quoted without that denominator are not comparable between vendors and not predictive of the full programme.

Decide in advance what would make you stop

Defining the failure criteria before the pilot starts is the single most useful governance step available, because the pressure to proceed after investing in a pilot is considerable and largely independent of what the pilot found.

Questions your own team should answer first

Some of the evaluation is about the tool. Some of it is about whether your estate is ready to be evaluated against, and getting that wrong wastes the exercise.

What is actually in the estate?

The construct frequencies matter: how much PERFORM THRU, how many dynamic calls, how much embedded SQL, how many REDEFINES with dependent tables, how much Assembler. Without those counts, you cannot weight the rubric and you cannot tell whether a vendor’s exception list is small or enormous.

Producing the counts is an analysis exercise, and it is worth doing before engaging vendors rather than after, because it also tells you which vendors are worth engaging.

What is your verification basis?

If the answer is a test suite built from the specification, the evaluation should account for the fact that the specification and the code have diverged. If the answer is production data comparison, the evaluation is on much firmer ground.

Establishing a verification basis is independent of any vendor decision and it is the single most valuable preparation available.

What does the organisation actually want?

A migration bought for staffing reasons has different acceptance criteria from one bought for platform cost reasons. If the objective is that a larger pool of developers can maintain the system, output readability is a primary criterion rather than a secondary one, and the evaluation should weight it accordingly.

Programmes that do not state this explicitly tend to accept output that satisfies the technical criteria and defeats the business case.

Who signs off?

Name the person who decides the output is acceptable, before the samples arrive. Evaluations without a named decision maker tend to conclude by consensus, and consensus favours proceeding.

A note on what the market is currently claiming

Automation percentages are the headline number in most material, and they are close to meaningless without three qualifiers: what denominator they are measured against, whether manual effort by the customer’s own team is included, and what the exception list contained.

A tool reporting very high automation on code it was allowed to exclude from scope is reporting on a subset it chose. Asking for the figure to be expressed against total estate lines, including exclusions, usually changes it substantially, and a vendor who declines to express it that way has answered the question.

Weighting the rubric to your estate

A pass or fail per construct is a start. What makes the evaluation decision-grade is weighting each construct by how often it appears in your code, because a failure on something you have twice matters less than a failure on something you have eleven thousand times.

Building the weights

Count, in your own estate: COMP-3 and decimal picture declarations, COMPUTE statements without ROUNDED, PERFORM THRU ranges containing GO TO, sorts and comparisons on mixed alphanumeric keys, REDEFINES with OCCURS DEPENDING ON, dynamically resolved CALL statements, and embedded SQL statements.

Those seven counts are an afternoon of analysis and they change how the evaluation reads.

A worked weighting

Read as a pass rate, Tool A scores poorly and Tool B scores well. Read with the counts attached, Tool A is disqualified on the first row alone, because 41,000 declarations with incorrect numeric typing is not a defect list, it is a rebuild.

The three critical rows

Numeric typing, rounding behaviour and control flow completeness are not negotiable on a system of record, because each produces incorrect values or dropped logic with no error raised. A failure on any of the three should end the evaluation regardless of how the remaining rows score.

The other four are engineering problems with known costs. They can be handled by the tool, handled by a translation rule, or handled by your team, and the evaluation should establish which and at what price rather than treating them as pass or fail.

What the counts tell you on their own

The exercise has a useful by-product. An estate with 340 PERFORM THRU ranges containing jumps has 340 places where a maintenance patch may be encoding behaviour that is not in any specification. Whatever tool you choose, those are the programs to put in the first parallel run window.

Common questions

1. Can Gen AI convert COBOL to Java?

It can produce Java from COBOL. Whether that Java reproduces the original behaviour is a separate question, and it is the only one that matters for a system of record. The constructs above are where the difference shows.

2. How large should an evaluation sample be?

Small, and deliberately chosen. A few hundred lines containing the seven constructs is more informative than fifty thousand lines of well-behaved code.

3. Should we run the vendor’s pilot at all?

Yes, after the construct test. A pilot answers questions about throughput, process and working relationship. It does not answer questions about semantic correctness, because the sample is chosen for smoothness.

4. Our vendor generates a test suite. Is that not sufficient?

Not when the same system produced the translation. The tests inherit the translator’s reading of the code. Independent verification against production behaviour is the standard.

5. What if we have no test data we can use?

Production data, masked appropriately, is the strongest test asset most estates have. Building comparison runs from it is standard practice and usually easier than teams expect.

6. Does this apply to PL/I and Assembler too?

The principles do. The specific constructs differ: PL/I brings pointer-based structures and ON conditions, Assembler brings direct storage manipulation with no type information at all.

7. How should automation percentages be interpreted?

Ask what denominator they are measured against, whether manual effort by your own team is counted, and what the exception list contained. A figure quoted against code the tool was permitted to exclude describes a subset the vendor chose. Expressed against total estate lines including exclusions, it usually changes substantially.

8. Which construct failure should end an evaluation immediately?

Numeric typing, rounding behaviour or control flow completeness. Each produces incorrect values or dropped logic with no error raised, which means the defect is invisible until something reconciles. The remaining constructs are engineering problems with knowable costs.

9. Do we need the construct counts before evaluating?

Yes, because they turn pass rates into a decision. A failure on a construct appearing forty thousand times is a rebuild; the same failure on a construct appearing twice is a work item. The counts are an afternoon of analysis and they change how every result reads.

10. What if the vendor will not run their tool on our code?

That is a finding in itself. Evaluations conducted only on vendor-supplied samples establish that the tool handles code the vendor selected. The constructs that break translators live in your estate, not in theirs.

The short version

Give the tool a copybook chain, an unrounded COMPUTE, a PERFORM THRU with a jump into it, a sorted alphanumeric key, a REDEFINES, a dynamic CALL and an embedded SQL cursor. Read the output as a maintainer rather than a reviewer. Ask whether the translation is deterministic and how equivalence is demonstrated.

If you would like a second opinion on output you already have, book a free assessment. We will run your converted code against the legacy behaviour and show you where they differ.