Unicode Escape

Unicode escaping is a method for representing a Unicode character using its numeric value (code point) rather than the character itself.

This notation typically begins with a backslash \ followed by a prefix and hexadecimal digits.

This abstraction allows for text manipulation in environments where the direct display of a special character is not guaranteed or desired.

To escape a character using Unicode:

— Identify the character's Unicode code point

— Convert this value to hexadecimal

— Apply the appropriate escape format (see below)

Unicode escape formats correspond to different ways of representing a code point as text. The most common syntaxes include several conventions used depending on the language, regular expression engine, or serialization system.

— Format \uXXXX: the oldest standard format, a fixed hexadecimal notation of 4 digits. This format is common in Java, JSON, and some parsers but is limited to the Basic Multilingual Plane (BMP), i.e., characters between U+0000 and U+FFFF. For characters outside the BMP, generate two consecutive sequences corresponding to a substitution pair.

— Format \u{X}: the most recent standard format, a variable notation enclosed in curly braces. Represent any code point without length constraints. Syntax used in modern JavaScript, Rust, PHP, and most modern languages except Python.

— Format \UXXXXXXXX: a format used in Python to directly represent complete code points using 8 hexadecimal digits, without resorting to substitution pairs.

— Format \x{X}: a format that replaces u with x, found in some regular expression engines (such as PCRE).

— Format \X: a format used in CSS, offering the simplest notation by using a backslash prefix followed directly by hexadecimal. This approach is sometimes ambiguous because it has historically been linked to octal or hexadecimal escaping, depending on the language.

Decoding a Unicode escape sequence involves:

— Recognizing the pattern: \uXXXX, \u{X}, or other

— Extracting the hexadecimal portion

— Converting the hexadecimal to decimal to obtain the code point

— Interpreting this code point as a Unicode character

Most programming languages provide native functions for this process.

Identify a Unicode escape sequence by these characteristic patterns:

— \uXXXX: backslash + u + 4 hexadecimal digits

— \u{X} or \u{XXXX}: flexible notation with curly braces

— \UXXXXXXXX: backslash + U + 8 hexadecimal digits

Common variants include:

\uXXXX: standard 4-digit notation

\u{X}: modern compact notation

\UXXXXXXXX: 8-digit notation used in some languages like Python

\x{X}: alternative notation used by some regular expression engines

HTML: &#xXXXX; (entirely different notation)

URL encoding: %XX

Substitute pairs generated by UTF-16 for code points greater than U+FFFF