EBCDIC to ASCII: why your sort order changes and your reports stop matching

← All posts

The first difference most mainframe migrations surface in parallel testing is not a calculation error. It is a report where every value is correct and the rows are in a different order.

That is collation. It is entirely predictable, it is documented, and it still catches teams late, because it is invisible in the source code. Nothing in a COBOL program says “sort using EBCDIC”. The program says SORT and the platform supplies the sequence.

This post covers what actually differs, the specific places it changes program behaviour rather than just presentation, and how to handle it during a migration instead of discovering it during user acceptance testing.

What the two sequences actually are

Collating sequence is the order assigned to characters by their underlying code point values. Comparison and sorting both resolve to comparing those values.

Where the characters sit

In EBCDIC, digits occupy X’F0’ through X’F9’, near the top of the range. Lowercase letters occupy the range beginning at X’81’, uppercase at X’C1’, and the space character is X’40’.

In ASCII, the space is X’20’, digits occupy X’30’ through X’39’, uppercase letters begin at X’41’ and lowercase at X’61’.

The consequences follow directly from those positions.

The first two rows are where the trouble lives. A key mixing letters and digits sorts one way on the mainframe and the opposite way on the target platform.

What the COBOL standard says about it

Alphanumeric comparison in COBOL is defined against the collating sequence of the character set in use. For EBCDIC data, the EBCDIC sequence applies; for ASCII data, the ASCII sequence. The PROGRAM COLLATING SEQUENCE clause in the object computer paragraph, in conjunction with an alphabet name in special names, allows a program to specify a sequence explicitly.

Most legacy programs do not specify it, because there was never a reason to. The platform default was the only sequence that existed.

Where it changes behaviour, not just appearance

Presentation changes are annoying. Logic changes are expensive. Five categories, roughly in order of how quietly they fail.

Control break logic

Batch reporting is built on control breaks: the program reads a sorted file, detects when a key field changes, and emits subtotals at the break.

Control break logic assumes the input is in a specific order. It usually does not verify it. If the sort order changes, the breaks occur at different points, and the program produces subtotals that are internally consistent and wrong. No error is raised, because nothing is technically invalid.

This is the most damaging case because the output looks like a report, not like a failure.

Range comparisons

Conditions of the form IF FIELD-A > 'A' AND FIELD-A < '9' encode an assumption about what sits between those two characters.

In EBCDIC, the range from ‘A’ to ‘9’ spans a particular set of characters. In ASCII, ‘9’ is below ‘A’ entirely, so the condition can never be satisfied. The program compiles. It runs. The branch is simply never taken.

Validation routines that check whether a character is printable, alphabetic or numeric by comparing against boundary characters are the common instance. They tend to be old, widely copied and rarely reviewed.

SEARCH ALL and binary search

SEARCH ALL performs a binary search, which requires the table to be ordered by the search key. That ordering was established under the source platform’s sequence.

Move the data to a platform with a different sequence, leave the table load logic unchanged, and the binary search operates over data that is no longer sorted by the sequence it assumes. Results become wrong non-deterministically: some lookups succeed, some fail, depending on where the key falls.

This is the hardest one to diagnose from symptoms, because the failure rate looks random.

Keyed file access and key ranges

VSAM KSDS records are held in key sequence. Programs that read sequentially from a starting key, process a range, or rely on the physical order of records inherit the sequence as part of the file’s behaviour.

When the data moves to a relational or other target, the index order follows the target’s collation. Sequential processing from a start key returns a different set of records than it did before.

Merge and match-merge processing

Classic two-file match-merge logic reads both files in key order and compares the current keys to decide which to advance. It depends on both files being in the same order under the same sequence.

If one side is re-sorted under a new sequence and the other is not, the merge silently drops or duplicates records rather than failing.

Data that is not text

Two related issues that arrive at the same time as collation and are often conflated with it.

Packed decimal and binary fields are not characters

A COMP-3 field holds packed decimal digits with a sign nibble; a COMP field holds binary. Neither is character data, and neither should be run through a character set conversion.

Conversions applied at file level, rather than field level, corrupt these fields. The result is usually obvious, because the values become nonsense, which at least makes it a fast failure rather than a slow one. Getting this right requires a record layout, which requires the copybook, which requires the copybook chain to have been parsed rather than summarised.

Sign representation

Signed zoned decimal fields carry the sign as an overpunch in the last byte, and the code points involved differ between character sets. A field holding a negative value can convert into something that is no longer recognisable as numeric, or worse, into a different value that still looks valid.

Fields that hold both

Redefined areas where the same bytes are character data in one view and packed data in another are the case that requires the discriminator logic to be understood before any conversion decision can be made. Getting this wrong produces corruption for specific record types only, which typically means specific customers only.

How to handle it during a migration

There are three defensible approaches and one common mistake.

Approach 1: preserve EBCDIC ordering explicitly

Where downstream consumers, regulatory outputs or business expectations depend on the existing order, carry the ordering forward rather than the character set. In practice this means the sort and comparison operations are translated to apply an explicit EBCDIC collation over ASCII data.

This preserves behaviour exactly, which is the right default for anything whose output leaves the organisation.

Approach 2: move to native ordering deliberately

Where the order is genuinely presentational and the business is willing to accept a change, moving to native target ordering is simpler to maintain long term.

The requirement is that it is a decision, taken by someone with authority, with the affected outputs identified. It should never be the by-product of a translation default.

Approach 3: mixed, by output

Most large estates end up here. Regulatory and external outputs preserve the original ordering; internal working files move to native ordering. That requires knowing which is which, which requires the dependency map.

The mistake: leaving it to the translator’s default

The failure mode is not choosing wrongly. It is not choosing at all, and discovering the consequence in acceptance testing, at which point the team is under time pressure and the fix is applied program by program.

A deterministic translation approach treats collation as an explicit rule. The rule is decided once, applies identically everywhere the pattern occurs, and can be changed in one place if the decision changes. That is the difference between a policy and a sequence of patches, and it is the same argument we make in how deterministic software transformation works.

Finding the exposure before you convert

The affected code can be located by analysis rather than by testing, which is considerably cheaper.

What to search for

Sort and merge operations, and the keys they use. Alphanumeric comparisons where either operand is a literal, particularly ranges. Every SEARCH ALL and the table loading logic behind it. Keyed file access with a start key. Class tests and hand-rolled validation routines comparing against boundary characters. Control break logic, identifiable by the pattern of holding a previous key value and comparing.

Classify by exposure

Not every hit matters. A sort on a purely numeric key behaves the same under both sequences, because digits are contiguous and in the same relative order in both. A comparison between two variables, neither of which is a literal, is unaffected by the sequence in most cases.

The cases that matter are mixed alphanumeric keys, literal range comparisons, and anything whose output crosses an organisational boundary.

Test with data that exposes it

Test data that is uniformly uppercase alphabetic, or uniformly numeric, will not reveal collation differences. Deliberately construct cases that mix digits and letters, mix case, and include the boundary characters used in range tests.

This is a general principle in migration testing: synthetic data tends to be well-formed, and well-formed data does not find these defects. Production-shaped data does.

A worked example

Abstract descriptions of collation are easy to nod along to and hard to act on. A concrete case makes the consequence visible.

The input

Take a small set of account reference keys of the kind that appear throughout insurance and banking estates, mixing a letter prefix with digits and occasionally a lowercase component.

A100
1000
a100
B200
2000
Z900

Sorted ascending under each sequence

Every record is present in both. Every value is correct in both. The order is almost entirely reversed, and nothing in the program changed.

What a control break does with this

Suppose the program emits a subtotal whenever the first character of the key changes, and the business expects one subtotal block per prefix.

Under the EBCDIC ordering, the lowercase and uppercase A records are adjacent, so a program comparing the first character produces one group for ‘a’, one for ‘A’, then B, Z, 1, 2.

Under the ASCII ordering, the numeric keys lead, the uppercase letters follow, and the lowercase A record lands at the end, separated from its uppercase counterpart by every other record in the file. If the report was designed on the assumption that the two A records appear together, it now produces two separate blocks at opposite ends of the output.

Nothing fails. The subtotals are arithmetically correct for the groups the program identified. They are the wrong groups.

Why test data usually misses it

Test files are typically constructed with uniform key formats, all uppercase, or purely numeric. Each of those is a case where the two sequences agree, or agree closely enough that the difference does not surface. The failure requires mixed case, or mixed letters and digits, in the same key field.

Numeric fields deserve separate attention

Collation discussions focus on character data, but numeric handling produces a related set of migration defects that arrive at the same moment and get conflated with it.

Zoned decimal and the sign overpunch

A signed zoned decimal field carries its sign in the high-order bits of the final byte rather than as a separate character. The code points that represent a positive or negative final digit are specific to the character set.

Run such a field through a character set conversion and the sign representation is no longer valid. Depending on how the target interprets it, the field either becomes non-numeric, which usually produces a visible failure, or becomes a different valid value, which does not. The second outcome is the dangerous one.

Packed decimal is not text at all

A COMP-3 field holds two digits per byte with a sign nibble in the low-order position of the final byte. There is no character interpretation to convert. Applying a character set conversion to it corrupts it entirely.

Binary fields and byte order

COMP fields hold binary integers. Beyond character set, byte order matters when moving between architectures, and a field that is read with the wrong endianness produces a plausible but wrong number rather than an error.

The common root cause

All three failures come from the same mistake: applying conversion at the file or record level rather than at the field level. Field-level conversion requires the record layout, which requires the copybook, which requires the copybook chain to have been parsed rather than approximated. That is the same dependency that governs correct numeric typing in the converted program.

A migration checklist for collation

The sequence that turns this from a testing surprise into a planned work item.

Before conversion

• Inventory every sort and merge operation with its key fields, and classify each key as numeric, uniformly cased alphabetic, or mixed. Only the third class is exposed.
• Locate every alphanumeric comparison against a literal, with particular attention to range tests, and evaluate whether the range spans character classes.
• List every SEARCH ALL and trace the loading logic for the table it searches.
• Identify keyed file access with a start key, and sequential processing that assumes physical order.
• Find control break logic by pattern: a saved previous key compared against the current one.

At design time

• Decide, per output, whether the original ordering is preserved or the target ordering is adopted. Record the decision and the person who made it.
• Express the decision as a translation rule rather than a per-program change, so that it applies consistently and can be revised in one place.
• Identify outputs that cross an organisational boundary, because those default to preservation regardless of internal preference.

During verification

• Construct test data that mixes case and mixes letters with digits in key fields, deliberately.
• Compare output ordering explicitly, not only output content. A comparison that sorts both sides before comparing will mask exactly the defect you are looking for.
• Include at least one full control break report and one SEARCH ALL path in the comparison set.

Why this surfaces so late

Collation is documented, well understood and still routinely discovered during acceptance testing. The reasons are structural rather than technical, and they are worth naming because they are avoidable.

It is invisible in the source

No COBOL program states which collating sequence it expects. The dependency is on the platform, not on the code, so a reviewer reading the program sees nothing to flag. Every other migration risk has some textual signal; this one does not.

It does not fail

Collation differences produce different output rather than errors. Jobs complete. Return codes are clean. Every automated check that looks for failure finds nothing, which means the defect survives until a human compares two reports.

The people who would notice are not in the room

The person most likely to spot that a report is ordered wrongly is the analyst who reads it every month, and that person is usually involved at user acceptance testing, which is the last stage before cutover and the worst moment to find a systematic issue.

It is easy to fix badly

Once found, the obvious remedy is to apply a sort order change in the affected program. That closes the instance and leaves the class open, which is how a programme ends up with forty separate collation fixes and no confidence it has found them all. Treating it as a rule applied across the estate is the difference between a decision and a defect queue.

What to tell the business

Collation is one of the few migration topics where the business has a genuine decision to make, and framing it correctly saves a great deal of later argument.

The decision is: for each output, does the existing order matter, or was it an accident of the platform?

Where the order is meaningful, because a regulator, a counterparty or a downstream process depends on it, the existing order is preserved and the cost is a small amount of additional translation work. Where it is not meaningful, native ordering is adopted, the output changes visibly, and someone has to tell the people who read it.

Presenting it that way converts a technical problem into a short list of decisions with named owners. Presenting it as a technical detail, or not presenting it at all, converts it into a defect report from a user during acceptance testing, at which point the same decisions get made under time pressure by whoever is available.

Triaging exposure without reading every program

Static analysis will return every sort, comparison and table search in the estate, which on a large codebase is thousands of hits. Most of them do not matter. A triage rule reduces that to a reviewable list.

The three questions per hit

Does the key or operand span character classes? A key that is purely numeric behaves identically under both sequences, because digits are contiguous and in the same relative order in each. A key that is uniformly uppercase alphabetic behaves identically for the same reason. Only keys mixing letters with digits, or mixing case, are exposed.

Does the result leave the organisation? Output that reaches a regulator, a counterparty, a customer or a downstream system defaults to preserving the original order regardless of internal preference, because changing it requires their agreement.

Does anything downstream assume the order? Control break logic, binary searches and match-merge processing all consume order as input. A sort whose output is only ever read record by record, with no order dependency, is presentational.

The resulting classification

Why recording the fourth class matters

The instinct is to list only the problems. Recording the hits you deliberately excluded, with the reason, is what makes the analysis defensible six months later when someone asks whether a particular program was considered.

It also protects against the common failure of a second pass repeating the first pass’s work because nobody kept the reasoning.

A heuristic for where exposure concentrates

In our experience the exposure clusters in three places: account and policy reference keys that combine a letter prefix with digits, product and scheme codes that were extended with alphabetic suffixes after the numeric range filled up, and hand-written character validation routines that were copied between programs decades ago and never reviewed. Searching those three patterns first finds most of it.

Common questions

1. Does EBCDIC to ASCII conversion change my data values?

For character data, the bytes change but the characters they represent do not. For packed decimal, binary and signed zoned fields, character set conversion corrupts the data, which is why conversion has to be applied field by field using the record layout rather than at file level.

2. Why do digits sort after letters in EBCDIC?

Because of where they sit in the code page. Digits occupy X’F0’ through X’F9’, above the letter ranges. In ASCII, digits occupy X’30’ through X’39’, below them.

3. Can I keep EBCDIC ordering after migrating?

Yes. Sort and comparison operations can apply an explicit EBCDIC collation over ASCII data. This is the right default for outputs that leave the organisation or feed regulatory processes.

4. What is the most dangerous case?

Control break logic, because it produces a plausible-looking report with incorrect subtotals and raises no error. SEARCH ALL over a table whose order has changed is a close second, because its failures look intermittent.

5. How do I find affected programs?

Static analysis: locate sorts and their keys, literal range comparisons, SEARCH ALL statements, keyed access with start keys, and control break patterns. Then classify by whether the key is mixed alphanumeric and whether the output crosses a boundary.

6. Does this apply if we stay on the mainframe?

Generally no, as long as the data stays in EBCDIC. It becomes relevant when data is exchanged with distributed systems, which is why the issue also appears in integration work rather than only in migration.

7. Which keys are genuinely exposed?

Only those mixing letters with digits, or mixing case. Purely numeric keys behave identically under both sequences because digits are contiguous and in the same relative order in each. Uniformly uppercase alphabetic keys behave identically for the same reason.

8. Where does exposure usually concentrate?

In three places: account and policy reference keys combining a letter prefix with digits, product and scheme codes extended with alphabetic suffixes after the numeric range filled, and hand-written character validation routines copied between programs decades ago.

9. Should we record the hits we decided not to act on?

Yes, with the reasoning. It makes the analysis defensible when someone asks six months later whether a program was considered, and it prevents a second pass repeating the first pass’s work.

10. Who makes the ordering decision?

The business, per output. Where the order is meaningful because a regulator, counterparty or downstream process depends on it, the original ordering is preserved. Where it is presentational, native ordering can be adopted and someone has to tell the people who read the report. Framing it as a short list of decisions with named owners is what keeps it out of acceptance testing.

The takeaway

Collation is not a hard problem. It is a problem that has to be treated as a decision rather than a default, and located by analysis rather than by testing.

Teams that handle it early treat it as a translation rule applied consistently across the estate. Teams that handle it late treat it as a defect queue during acceptance testing.

If you are scoping a migration and want to know where your collation exposure sits, book a free assessment. The analysis that finds it is the same analysis that sizes the rest of the work, and it is covered as part of COBOL to Java migration.