EBCDIC

From OSDev Wiki
Jump to navigation Jump to search

EBCDIC

EBCDIC (Extended Binary Coded Decimal Interchange Code, pronounced EBB-see-dick or EBB-sih-dick) is an 8-bit character encoding standard developed by IBM, introduced in 1964 alongside the IBM System/360 — the architectural ancestor of the modern s390x (IBM Z). Unlike ASCII, which was designed for open interchange, EBCDIC was designed for IBM's punched-card based batch-processing ecosystem and remains the native character encoding of IBM Z mainframes to this day.

If you are writing an operating system or bootloader for s390x, you will encounter EBCDIC. Firmware interfaces, hardware console I/O, boot records, and low-level IBM interfaces all speak EBCDIC by default.

History and Background

Origins

EBCDIC evolved from BCD (Binary Coded Decimal) and BCDIC (BCD Interchange Code), both of which were used on IBM punched-card equipment such as the IBM 1401. When IBM introduced the System/360 in 1964 — the first family of computers designed as a unified, upward-compatible architecture — they standardized on EBCDIC as the character encoding across the entire line.

The design choices in EBCDIC reflect its punched-card heritage:

  • Digits and letters were mapped to correspond to specific punched-card column patterns.
  • Uppercase letters are not contiguous in the code space (a common source of bugs).
  • The code includes a rich set of control characters inherited from card and line-printer control.

EBCDIC vs ASCII

ASCII was standardized by ANSI in 1963, one year before EBCDIC's introduction. Both are 8-bit encodings (ASCII uses 7 bits; the 8th bit was later used for extensions), but they are entirely incompatible in layout. A raw byte stream of EBCDIC text is meaningless if interpreted as ASCII, and vice versa.

Key Differences
Feature ASCII EBCDIC
Bit width 7-bit (8-bit extended) 8-bit
Letter 'A' 0x41 0xC1
Digit '0' 0x30 0xF0
Alphabet contiguous? Yes No
Lowercase before uppercase? Yes (a=0x61, A=0x41) No (a=0x81, A=0xC1)
Null 0x00 0x00 (same)
Space 0x20 0x40
Newline (LF) 0x0A 0x25
Carriage Return 0x0D 0x0D (same)
DEL 0x7F 0x07 (BEL in ASCII!)

The non-contiguous letter layout is the most notorious EBCDIC quirk. In ASCII, you can test c >= 'A' && c <= 'Z' to check for uppercase letters. In EBCDIC, this range contains non-letter characters (specifically, several punctuation marks lie between the uppercase letter groups). Never assume contiguity of the alphabet when writing EBCDIC-aware code.

Variants

There is no single EBCDIC. IBM defined dozens of code pages — regional and application-specific variants — each with different assignments for characters outside the invariant core. The most commonly encountered variants in practice are:

Code Page IBM Name Common Use
EBCDIC-037 CP037 United States, Canada (most common in North American mainframe shops)
EBCDIC-273 CP273 Germany, Austria
EBCDIC-500 CP500 International / Belgium / Switzerland
EBCDIC-1047 CP1047 Open Systems (z/OS UNIX, Linux on Z) — closest to ASCII mappings
EBCDIC-1140 CP1140 CP037 + Euro sign
EBCDIC-1148 CP1148 CP500 + Euro sign

CP1047 deserves special attention for OS developers: it is the code page used by z/OS UNIX services and Linux on Z for interoperability, and is the most "ASCII-friendly" EBCDIC variant. If you are writing a kernel that must interoperate with z/OS or handle ELF binaries compiled for s390x Linux, CP1047 is your primary concern.

Encoding Layout

Structure of the Code Space

EBCDIC divides the 256 code points into several zones. The high nibble (bits 4–7) loosely defines the character class:

Range Character Class
0x00–0x3F Control characters
0x40 Space
0x41–0x7F Special/punctuation characters
0x80–0x8F Lowercase letters a–i (with gaps)
0x90 (unassigned / varies by code page)
0x91–0x99 Lowercase letters j–r (with gaps)
0xA0 (unassigned / varies by code page)
0xA1–0xA9 Lowercase letters s–z + some punctuation
0xC1–0xC9 Uppercase letters A–I
0xD1–0xD9 Uppercase letters J–R
0xE2–0xE9 Uppercase letters S–Z (with a gap at 0xE0–0xE1)
0xF0–0xF9 Digits 0–9

Note the three separate ranges for lowercase and uppercase letters, each with gaps. The gaps contain punctuation characters that differ between code page variants. This layout directly mirrors the punched-card column encodings from which EBCDIC descended.

Critical Characters Quick Reference (CP037 / CP1047)

The table below lists the most important characters for an OS developer. Values marked with * differ between CP037 and CP1047.

Character CP037 (Hex) CP1047 (Hex) ASCII (Hex)
NUL 00 00 00
BEL 2F 2F 07
BS 16 16 08
HT 05 05 09
LF 25 25 0A
VT 0B 0B 0B
FF 0C 0C 0C
CR 0D 0D 0D
SO 0E 0E 0E
SI 0F 0F 0F
ESC 27 27 1B
Space 40 40 20
[ BA* AD 5B
] BB* BD 5D
{ C0* C0 7B
} D0* D0 7D
\ E0* E0 5C
^ B0* 5F 5E
~ A1* A1 7E
! 5A 5A 21
" 7F 7F 22
# 7B 7B 23
$ 5B 5B 24
% 6C 6C 25
& 50 50 26
' 7D 7D 27
( 4D 4D 28
) 5D 5D 29
* 5C 5C 2A
+ 4E 4E 2B
, 6B 6B 2C
- 60 60 2D
. 4B 4B 2E
/ 61 61 2F
0–9 F0–F9 F0–F9 30–39
: 7A 7A 3A
; 5E 5E 3B
< 4C 4C 3C
= 7E 7E 3D
> 6E 6E 3E
? 6F 6F 3F
@ 7C 7C 40
A–I C1–C9 C1–C9 41–49
J–R D1–D9 D1–D9 4A–52
S–Z E2–E9 E2–E9 53–5A
_ 6D 6D 5F
` 79 79 60
a–i 81–89 81–89 61–69
j–r 91–99 91–99 6A–72
s–z A2–A9 A2–A9 73–7A
DEL 07 07 7F

Conversion

Approach 1: Lookup Table

The simplest and fastest approach is a 256-entry lookup table. Since the mapping is not mathematically regular, a table is both the most correct and most performant solution for bulk conversion. 

/* 
 * ebcdic_to_ascii[n] gives the ASCII byte for EBCDIC byte n.
 * Code page: IBM-1047 (recommended for OS/Linux interoperability).
 * Unmappable bytes are replaced with '?' (0x3F).
 */
static const uint8_t ebcdic1047_to_ascii[256] = {
    /* 0x00 */ 0x00, 0x01, 0x02, 0x03, 0x9C, 0x09, 0x86, 0x7F,
    /* 0x08 */ 0x97, 0x8D, 0x8E, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F,
    /* 0x10 */ 0x10, 0x11, 0x12, 0x13, 0x9D, 0x0A, 0x08, 0x87,
    /* 0x18 */ 0x18, 0x19, 0x92, 0x8F, 0x1C, 0x1D, 0x1E, 0x1F,
    /* 0x20 */ 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x17, 0x1B,
    /* 0x28 */ 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x05, 0x06, 0x07,
    /* 0x30 */ 0x90, 0x91, 0x16, 0x93, 0x94, 0x95, 0x96, 0x04,
    /* 0x38 */ 0x98, 0x99, 0x9A, 0x9B, 0x14, 0x15, 0x9E, 0x1A,
    /* 0x40 */ 0x20, 0xA0, 0xE2, 0xE4, 0xE0, 0xE1, 0xE3, 0xE5,
    /* 0x48 */ 0xE7, 0xF1, 0xA2, 0x2E, 0x3C, 0x28, 0x2B, 0x7C,
    /* 0x50 */ 0x26, 0xE9, 0xEA, 0xEB, 0xE8, 0xED, 0xEE, 0xEF,
    /* 0x58 */ 0xEC, 0xDF, 0x21, 0x24, 0x2A, 0x29, 0x3B, 0x5E,
    /* 0x60 */ 0x2D, 0x2F, 0xC2, 0xC4, 0xC0, 0xC1, 0xC3, 0xC5,
    /* 0x68 */ 0xC7, 0xD1, 0xF6, 0x2C, 0x25, 0x5F, 0x3E, 0x3F,
    /* 0x70 */ 0xF8, 0xC9, 0xCA, 0xCB, 0xC8, 0xCD, 0xCE, 0xCF,
    /* 0x78 */ 0xCC, 0x60, 0x3A, 0x23, 0x40, 0x27, 0x3D, 0x22,
    /* 0x80 */ 0xD8, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67,
    /* 0x88 */ 0x68, 0x69, 0xAB, 0xBB, 0xF0, 0xFD, 0xFE, 0xB1,
    /* 0x90 */ 0xB0, 0x6A, 0x6B, 0x6C, 0x6D, 0x6E, 0x6F, 0x70,
    /* 0x98 */ 0x71, 0x72, 0xAA, 0xBA, 0xE6, 0xB8, 0xC6, 0xA4,
    /* 0xA0 */ 0xB5, 0x7E, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78,
    /* 0xA8 */ 0x79, 0x7A, 0xA1, 0xBF, 0xD0, 0x5B, 0xDE, 0xAE,
    /* 0xB0 */ 0xAC, 0xA3, 0xA5, 0xB7, 0xA9, 0xA7, 0xB6, 0xBC,
    /* 0xB8 */ 0xBD, 0xBE, 0xDD, 0xA8, 0xAF, 0x5D, 0xB4, 0xD7,
    /* 0xC0 */ 0x7B, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47,
    /* 0xC8 */ 0x48, 0x49, 0xAD, 0xF4, 0xF6, 0xF2, 0xF3, 0xF1, /* note: some entries locale-specific */
    /* 0xD0 */ 0x7D, 0x4A, 0x4B, 0x4C, 0x4D, 0x4E, 0x4F, 0x50,
    /* 0xD8 */ 0x51, 0x52, 0xB9, 0xFB, 0xFC, 0xF9, 0xFA, 0xFF,
    /* 0xE0 */ 0x5C, 0xF7, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58,
    /* 0xE8 */ 0x59, 0x5A, 0xB2, 0xD4, 0xD6, 0xD2, 0xD3, 0xD1,
    /* 0xF0 */ 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37,
    /* 0xF8 */ 0x38, 0x39, 0xB3, 0xDB, 0xDC, 0xD9, 0xDA, 0x9F,
};

void ebcdic_to_ascii_buf(uint8_t *dst, const uint8_t *src, size_t len)
{
    for (size_t i = 0; i < len; i++)
        dst[i] = ebcdic1047_to_ascii[src[i]];
}

Approach 2: iconv

If your OS provides a libc (or you are writing a userland tool rather than kernel code), iconv(3) supports EBCDIC code pages. For example: 

iconv_t cd = iconv_open("ASCII", "EBCDIC-CP-US");
/* or for CP1047: */
iconv_t cd = iconv_open("UTF-8", "IBM-1047");

This is not suitable for kernel or early-boot code, but is appropriate for userland utilities.

Approach 3: s390x TR Instruction

The s390x TR (Translate) instruction was literally designed for EBCDIC translation. It translates each byte of a source string by using it as an index into a 256-byte translation table, writing results to a destination buffer. This is the hardware-native approach and is extremely fast. 

/* Translate EBCDIC → ASCII using the TR instruction.
 * R2 = address of source buffer
 * R3 = length - 1 (TR length operand is len-1)
 * R4 = address of 256-byte translation table
 */
    LA    %r1, dest_buffer     /* destination address */
    MVC   0(256,%r1), 0(%r2)   /* copy source to dest first (TR is in-place) */
    TR    0(256,%r1), 0(%r4)   /* translate in place using table at R4 */

Note: TR operates in-place on the destination, so copy the source first if you need a separate output buffer. The instruction takes a maximum of 256 bytes per invocation; loop for longer strings. The TRT (Translate and Test) instruction can scan for specific EBCDIC characters, useful for parsing.

Control Characters

EBCDIC control characters are a superset of the ASCII control characters, with some significant differences. The most important for terminal/console handling:

EBCDIC (Hex) Mnemonic ASCII Equiv Notes
00 NUL NUL Same
05 HT HT (0x09) Horizontal Tab
0D CR CR (0x0D) Carriage Return — same value!
15 NL LF (0x0A) New Line — EBCDIC's line terminator; maps to ASCII LF
25 LF LF (0x0A) Line Feed — also used; some docs say NL is the "true" newline
37 EOT EOT (0x04) End of Transmission
3C DC4 DC4 (0x14) Device Control 4
40 SP SP (0x20) Space — 0x40, not 0x20!
0F SI SI (0x0F) Shift In
0E SO SO (0x0E) Shift Out

The EBCDIC New Line (NL, 0x15) is particularly important: z/OS text files use 0x15 as the line terminator, not 0x25. When reading z/OS text datasets, translate 0x15 → 0x0A (LF). This is handled automatically by CP1047 ↔ ISO-8859-1 conversion tables in most iconv implementations.

Detecting EBCDIC at Runtime

If your code might run on both EBCDIC and ASCII platforms (e.g., a cross-platform bootloader stub), you can detect the encoding at runtime: 

static inline int running_on_ebcdic(void)
{
    /* 'A' is 0x41 in ASCII, 0xC1 in EBCDIC */
    return ('A' == 0xC1);
}

Useful EBCDIC Properties

Despite its quirks, EBCDIC has some convenient properties the hardware exploits:

  • Digit value extraction: For EBCDIC digit characters (0xF0–0xF9), digit_value = ebcdic_byte & 0x0F.
  • Case conversion: For pure alphabetic bytes (verifying they are in the correct ranges first), ebcdic_upper = ebcdic_lower & 0xBF and ebcdic_lower = ebcdic_upper | 0x40 — but only if the byte is confirmed to be an alpha character, because the same bits mean different things for non-alpha bytes.
  • Alphabetic check (hardware): The s390x TRT instruction combined with a pre-built classification table enables very fast character classification.

Code Page Tables

Full CP1047 Table

The following table gives the complete CP1047 (IBM-1047) EBCDIC-to-Unicode mapping for the printable range. The column header is the high nibble and the row header is the low nibble.

(For the full 256-entry mapping suitable for inclusion in kernel source, see the lookup table in the Conversion section above, or the reference at IBM's EBCDIC/Unicode Conversion Tables.)

Standards and References

  • IBM z/Architecture Principles of Operation (SA22-7832) — The authoritative reference for all s390x architecture details, including character encoding used in firmware interfaces.
  • IBM z/OS MVS Programming: Authorized Assembler Services Guide — Details on SCLP and console interfaces.
  • IBM Documentation: EBCDIC Code Pages — Official IBM listing of all EBCDIC code pages.
  • ANSI X3.4-1968 — Original ASCII standard, useful for comparison.
  • RFC 877 — Notes on EBCDIC in network contexts.
  • Unicode.org IBM Code Page Mappings — Machine-readable CP→Unicode mapping files.
  • POSIX.1-2017 — Defines locale and iconv behavior, relevant for userland tools.

See Also

Retrieved from "https://wiki.osdev.org/index.php?title=EBCDIC&oldid=30670"