Unicode
|
|
Unicode is the dominant text encoding standard for representing international strings. Each character from an alphabet is assigned a Code Point, from the range U+0000 to U+10FFFF. This space needs 25 bits to represent it.
The Unicode consortium defines multiple character encodings for Unicode code points, the simplest of which, UTF-32, is simply the code point padded to 32 bits. UTF-8, by contrast, is compatible with the base ASCII encoding (not extended ASCII), as ASCII is a 7-bit encoding, so UTF-8 supports a 1:1 mapping. The unused high bit in ASCII codes denotes a multi-byte encoding in UTF-8.
UTF-32
Also sometimes called UCS-4. This is the only fixed-width Unicode encoding, and can be represented as char32_t since C11.
The Unicode space is broken up into 17 different planes.
Planes
| Plane Number | 0 | 1 | 2 | 3 | 4-13 | 14 | 15 | 16 |
|---|---|---|---|---|---|---|---|---|
| Code Points | U+0000-U+FFFF | U+10000-U+1FFFF | U+20000-U+2FFFF | U+30000-U+3FFFF | U+40000-U+DFFFF | U+E0000-U+EFFFF | U+F0000-U+10FFFF | |
| Reference | BMP | SMP | SIP | TIP | unassigned | SSP | SPUA | |
| Name | Basic Multilingual | Supplementary Multilingual | Supplementary Ideographic | Tertiary Ideographic | Supplementary Special-purpose | Supplementary Private Use Area | ||
BMP
The Basic Multilingual Plane.
The characters of most languages are in this plane. There are also two main reserved areas - UTF-16 surrogates, U+D800-U+DFFF, and the primary Private Use Area from U+E000-U+F8FF.
It is invalid to encode a surrogate in any Unicode encoding. They exist entirely to support UTF-16.
SMP
The Supplementary Multilingual Plane.
This plane largely contains characters from historic scripts. For instance, Phoenician, which is also intended for representing Early Aramaic and Paleo-Hebrew. 𐤔𐤋𐤅𐤌. Or Egyptian Hieroglyphs. 𓀁
Also in this plane are a generous helping of mathematical symbols. ⨌
SIP
The Supplementary Ideographic Plane.
This plane is almost entirely allocated, solely to CJK unified ideographs.
TIP
The Tertiary Ideographic Plane.
This plane is only partially allocated, solely to CJK unified ideographs.
SSP
The Supplementary Special-purpose Plane.
A largely unused plane, this contains some tags and variation selectors.
SPUA
The Supplementary Private Use Area. This can be used by e.g. special fonts.
UTF-16
UTF-16 is a 16 bit, variable-length encoding. It encodes the Basic Multilingual Plane identically to UCS-2, directly representing it as one 16 bit value.
On decoding, if a character is in the UTF-16 surrogate region, then per the specification it must be part of a surrogate pair, as these values are invalid to encode alone.
Note that the byte encoding of this depends on endianness. To assist, the text can be preceeded with a byte-order mark, U+FEFF. U+FFFE is a reserved, invalid, value.
Surrogate pairs are used to encode the other unicode planes as follows:
2-byte encoding
U+0000-U+FFFF - xxxxxxxx_xxxxxxxx
4-byte encoding
U+010000-U+10FFFF - 110110xx_xxxxxxxx 110111xx_xxxxxxx (0xD800+xh, 0xDC00+xl)
Sample conversion
void encode(char32_t codepoint, std::span<char16_t>& result)
{
size_t bytesWritten;
if(codepoint <= 0xFFFF)
{
result[0] = static_cast<char16_t>(codepoint);
bytesWritten = 1;
}
else
{
if(codepoint > 0x10FFFF) throw EncodingException{};
//Subtract 2^16, implied as it's a surrogate pair now
codepoint -= 0x10000U;
//Split into two characters
char16_t high = 0xD800 + (codepoint >> 10);
char16_t low = 0xDC00 + (codepoint & 0x3FF);
result[0] = high;
result[1] = low;
bytesWritten = 2;
}
result = result.subspan(bytesWritten);
}
char32_t decode(std::span<const char16_t>& source)
{
size_t bytesRead;
char32_t result;
char16_t first = source[0];
if(first < 0xD800 || first > 0xDFFF)
{
result = static_cast<char32_t>(first);
bytesRead = 1;
}
else
{
char16_t second = source[1];
if(first > 0xDBFF) throw EncodingException{};
if(second < 0xDC00) throw EncodingException{};
if(second > 0xDFFF) throw EncodingException{};
uint16_t high = first - 0xD800;
uint16_t low = second - 0xDC00;
result = static_cast<char32_t>(0x10000U + (high << 10) | low);
bytesRead = 2;
}
source = source.subspan(bytesRead);
return result;
}
UTF-8
UTF-8 is an 8 bit, variable length encoding. It is in effect a superset of 7-bit ASCII, where ASCII characters directly convert to an 8 bit code unit with a leading zero. A leading 1 denotes a multi-byte encoding.
1-byte encoding
U+0000-U+007F - 0xxxxxxx
2-byte encoding
U+0080-U+07FF - 110xxxxx 10xxxxxx
3-byte encoding
U+0800-U+FFFF - 1110xxxx 10xxxxxx 10xxxxxx
4-byte encoding
U+010000-U+10FFFF - 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
Overlong encoding
Note that by appending leading zeros, a character can be encoded in a longer encoding than necessary. This can bypass validations that check for characters (e.g. /), and are a security problem. They should be considered an error.
Sample conversion
//ENCODING
void encode(char32_t codepoint, std::span<char8_t>& result)
{
size_t bytesWritten;
if(codepoint <= 0x7F)
{
result[0] = static_cast<char8_t>(codepoint);
bytesWritten = 1;
}
else if(codepoint < 0x7FF)
{
result[0] = static_cast<char8_t>(0xC0 + (codepoint >> 6));
result[1] = static_cast<char8_t>(0x80 + (codepoint & 0x3F));
bytesWritten = 2;
}
else if(codepoint < 0xFFFF)
{
result[0] = static_cast<char8_t>(0xE0 + (codepoint >> 12));
result[1] = static_cast<char8_t>(0x80 + ((codepoint >> 6) & 0x3F));
result[2] = static_cast<char8_t>(0x80 + (codepoint & 0x3F));
bytesWritten = 3;
}
else if(codepoint < 0x10FFFF)
{
result[0] = static_cast<char8_t>(0xF0 + (codepoint >> 18));
result[1] = static_cast<char8_t>(0x80 + ((codepoint >> 12) & 0x3F));
result[2] = static_cast<char8_t>(0x80 + ((codepoint >> 6) & 0x3F));
result[3] = static_cast<char8_t>(0x80 + (codepoint & 0x3F));
bytesWritten = 4;
}
else
{
throw EncodingException{};
}
result = result.subspan(bytesWritten);
}
//DECODING
static size_t decode_starting_byte(char8_t start, char32_t& result)
{
if(first < 0x80)
{
result = static_cast<char32_t>(start);
return 1;
}
else if(first < 0xC0)
{
//Expected UTF-8 starting byte, not continuation byte
throw EncodingException{};
}
else if(first < 0xE0)
{
result = start & 0x1F;
return 2;
}
else if(first < 0xF0)
{
result = start & 0x0F;
return 3;
}
else if(first < 0xF8)
{
result = start & 0x07;
return 4;
}
else
{
//Also invalid
throw EncodingException{};
}
}
static void decode_continuation(char8_t continuationByte, char32_t& result)
{
if((continuationByte & 0xC0) != 0x80)
throw EncodingException{}; //Expected a continuation byte, but not
result = (result << 6) | (continuationByte & 0x3F);
}
char32_t decode(std::span<const char8_t>& source)
{
if(source.empty())
throw EncodingException{};
char32_t result;
size_t bytesRead = decode_starting_byte(source[0], result);
if(bytesRead > source.size())
throw EncodingException{};
switch(bytesRead)
{
case 1:
break;
case 2:
decode_continuation(source[1], result);
if(result < 0x80) throw EncodingException{}; //Overlong encoding
break;
case 3:
decode_continuation(source[1], result);
decode_continuation(source[2], result);
if(result < 0x800) throw EncodingException{}; //Overlong encoding
break;
case 4:
decode_continuation(source[1], result);
decode_continuation(source[2], result);
decode_continuation(source[3], result);
if(result < 0x010000) throw EncodingException{}; //Overlong encoding
break;
}
if(result > 0x10FFFF) throw EncodingException{};
source = source.subspan(bytesRead);
return result;
}
Other Encodings
UCS-2
UCS-2 is an obsolete predecessor of UTF-16. It uses a fixed 16-bit character width, from before it was obvious that more than 2^16 code points were needed.
UTF-EBCDIC
This is a modified version of UTF-8, designed to be compatible with the EBCDIC encoding used on IBM Z mainframes.
See Also
External Links
- https://www.unicode.org/charts/ - Character code charts for the entirety of Unicode