Unicode compatibility characters

In discussing Unicode and the UCS, many often refer to compatibility characters. Compatibility characters are graphical characters that are discouraged by the Unicode Consortium. As the Unicode consortium says:

A character that would not have been encoded except for compatibility and round-trip convertibility with other standards

However, the definition is more complicated that the glossary reveals. One of the properties given to characters by the Unicode consortium is the characters decomposition or compatibility decomposition. Most characters have no value for this property, but over 5 thousand characters do have a compatibility decomposition mapping that compatibility character to one or more other characters. By setting a characters decomposition property, Unicode establishes that character as a compatibility character. The reasons for these compatibility designations are varied and are discussed in further detail below. The term decomposition can sometimes confuse because a characters decomposition can, in some cases, be a singleton. In these cases the decomposition of one character is simply another equivalent or approximately equivalent character.

Canonical and Non-canonical

The compatibility decomposition property for the 5,402 Unicode compatibility characters includes a keyword that divides the compatibility characters into 17 logical groups. Those without a keyword are termed canonical equivalent or canonical decomposable characters. These characters have the closest relationship. Other keywords include: <initial>, <medial>, <final>, <isolated>, <wide>, <narrow, <small>, <square>, <vertical>, <circle>, <noBreak>, <fraction>, <subscript>, <superscript>, and <compat>. These keywords provide some indication of the relation between the compatibility character and its compatibility decomposition character sequence. However, the compatibility characters — whether canonical or not — fall in three basic categories: 1) characters corresponding to multiple alternate glyph forms and precomposed diacritics to support software and font implementations that do not include complete Unicode text layout capabilities; 2) characters included from other character sets or otherwise added to the UCS that constitute rich text rather than the plain text goals of Unicode; 3) some other characters that are semantically distinct, but visually similar. When comparing and collating (sorting) text strings, different forms and rich text variants of characters should not alter the text processing results. For example, software users may be confused when performing a find on a page for a capital Latin letter ‘I’ and their software application fails to find the visually similar roman numeral ‘Ⅰ’.

Compatibility mappings types

Glyph substitution and composition

Some compatibility characters are completely dispensable for text processing and display software that conforms to the Unicode standard. These include:

• Precomposed diacritic letters. For example Å, (U+00C5), where Unicode prefers to treat such graphemes as two separate characters a capital ‘Latin letter A’ combined with a ‘Combining Ring Above’ (U+030A)
• Ligatures. Ligatures such as ‘ffi’ in the Latin script were often times encoded as a separate character in legacy character sets. Unicode’s approach to ligatures is to treat them as rich text and, if turned on, handled through glyph substitution.
• Precomposed roman numerals. For example, roman numeral Ⅻ (twelve: U+216B) can be decomposed into a roman numeral ten (X) and two roman numeral ones (I).
• Precomposed fractions. These decomposition have the keyword <fraction>. A fully conforming text handler should display the vulgar fraction ¼ (U+00BC) identically to the composed fraction 1⁄4 (numeral 1 with fraction slash U+2044 and numeral 4).
• Contextual glyphs or forms . These arise primarily in the Arabic script. Using fonts with glyph substitution capabilities such as OpenType and TrueTypeGX, Unicode conforming software can substitute the proper glyphs for the same character depending on whether that character appears at the beginning, end, middle of a word or in isolation. Such glyph substitution is also necessary for vertical (top to bottom) text layout for some East Asian languages. In this case glyphs must be substituted or synthesized for wide, narrow, small and square glyph forms. Non-conforming software or software using other character sets instead use multiple separate character for the same letter depending on its position: further complicating text processing.

The UCS, Unicode character properties and the Unicode algorithms provide software implementations with everything needed to properly display these characters from their decomposition equivalents. Therefore these decomposable compatibility characters become redundant and unnecessary. Their existence in the character set requires extra text processing to ensure text is properly compared and collated (see Unicode normalization). Moreover, these compatibility characters provide no additional or distinct semantics. Nor do these characters provide any visually distinct rendering provided the text layout and fonts are Unicode conforming. Also, none of these characters are required for roundtrip convertibility to other character sets, since the transliteration can easily map decomposed characters to precomposed counterparts in another character set. Similarly, contextual forms, such as a final Arabic letter can be mapped based on its position within a word to the appropriate legacy character set form character.

In order to dispense with these compatibility characters, text software must conform to several Unicode protocols. The software must be able to: 1) compose diacritic marked graphemes from letter characters and one or more separate combining diacritic marks; 2) substitute (at the author or readers discretion) ligatures and contextual glyph variants; 3) layout CJKV text vertically (at the author or readers discretion), substituting glyphs for small, vertical, narrow, wide square forms, either from font data or synthesized as needed; 4) combine fractions using the ‘Fraction Slash’ character (U+2044) and any other arbitrary characters; 5) combine a ‘Combining Long Solidus Overlay’ ( ̸ U+0338) with other symbols: for example ∄ or ∄ for ∄. (U+2203).

All together these compatibility characters included for incomplete Unicode implementations total 3,779 of the 5,402 designated compatibility characters. These include all of the compatibility characters marked with the keywords <initial>, <medial>, <final>, <isolated>, <fraction>, <wide>, <narrow>, <small>, <vertical>, <square>. Also it includes nearly all of the canonical and most of the <compat> keyword compatibility characters (the exceptions include those <compat> keyword characters for enclosed alphanumerics, enclosed ideographs and those discussed in the following sections: subsequent section).

Rich text compatibility characters

Many other compatibility characters constitute what Unicode considers rich text and therefore outside the goals of Unicode and UCS. In some sense even compatibility characters discussed in the previous section — those that aid legacy software in displaying glyphs and vertical text — constitute a form of rich text. However, the choice to display text with or without ligatures or as vertically or horizontally laid-out text are both non-semantic rich text. This is contrast to rich text such as superscripts and subscripts or list markers where the styling of the rich text implies certain semantics along with it.

For comparing, collating, handling and storing plain text, rich text variants are semantically redundant. They can also create ambiguity because they appear visually the same as their plain text counterpart characters with rich text formatting applied. These rich text compatibility characters include:

• Mathematical Alphanumeric Symbols. These symbols are simply clones of the Latin and Greek alphabets and Indic-Arabic decimal digits repeated in 15 various typefaces. They are intended as an arbitrary palette for mathematical notation. However, they tend to undermine the distinction between encoding visual glyphs versus encoding semantic characters as well as Unicode’s goals of supporting only plain text characters.
• Enclosed alphanumerics and ideographs (markers) These are characters included primarily for list markers. They do not constitute distinct semantic characters. Moreover, the use of other rich text protocols is more appropriate since, the set of enclosed alphanumerics or ideographs provisioned in the UCS is limited.
• Circled alphanumerics and ideographs. The circled forms are also likely for use as markers. Again, using characters along with rich text protocols to encircle characters strings is more flexible.
• Spaces and no-break spaces of varying widths. These characters are simply rich text variants of the core space (U+0020) and No-break Space (U+00A0). Other rich text protocols should be used instead such as tracking, kerning or word-spacing attributes.
• Some subscript and superscript form characters. Many of the subscript and superscript characters are actually distinct semantic characters from the International Phonetic Alphabet and other writing systems. However, some simply constitute rich text presentation forms of other Greek, Latin and numeral characters. These rich text superscript and subscript characters belong to this category of compatibility characters.

For all of these rich text compatibility characters the display of glyphs is typically distinct from their compatibility decomposition (related) characters. However, these are considered compatibility characters and discouraged for use by the Unicode consortium because they are not plain text characters, which is what Unicode seeks to support with its UCS and associated protocols. Rich text should be handled through non-Unicode protocols such as HTML, CSS, RTF and other such protocols.

The rich text compatibility characters comprise 1,451 of the 5,402 compatibility characters. These include all of the compatibility characters marked with keywords <circle> and <font> (except three listed in the semantically distinct below); 11 spaces variants from the <compat> and canonical characters; and some of the keyword <superscript> and <subscript> from the “Superscripts and Subscripts” block.

Semantically distinct characters

Many compatibility characters are semantically distinct characters, though they may share representational glyphs with other characters. Some of these characters may have been included because most other characters sets that focussed on one script or writing system. So for example, the ISO and other Latin character sets likely included a character for π (pi) since, when focussing on primarily one writing system or script, those character sets would not have otherwise had characters for the common mathematical symbol π;. However, with Unicode, mathematicians are free to use letters from any known script in the World or to select a Unihan ideograph to stand in for a mathematical set or mathematical constant. To date, Unicode has only added specific semantic support for a few such mathematical constants (for example Plancks constant: U+210E and Eulers constant U+210E which Unicode considers both to be compatibility characters). Therefore Unicode designates several mathematical symbols based on letters from Greek and Hebrew as compatibility characters. These include:

• Hebrew letter based symbols (4): ℵ Alef (ℵ U+2135), Bet (ℶ U+2136), Gimel (ℷ U+2137) and Dalet (ℸ U+2138)
• Greek letter based symbols (8): Beta (ϐ U+03D0), Theta (ϑ U+03D1), Phi (ϕ U+03D5), Pi (ϖ U+03D5), Kappa (ϰ U+03F0), Rho (ϱ U+03F1), Capital Theta (ϴ U+03F4), Prosgegrammeni (ι U+1FBE)
• Latin form variant (1): Long Latin Letter S (ſ U+017F). This Latin script letter is used in Gaelic.

While these compatibility characters are distinguished from their compatibility decomposition characters only by adding the word “symbol” to their name, they do represent long-standing distinct meanings in written mathematics. However, for all practical purposes they share the same semantics as their compatibility equivalent Greek or Hebrew letter. These may be considered border-line semantically distinguishable characters so they are not included in the total.

Unicode also designates twenty-eight other letter-like symbols as compatibility characters.

• Other Greek letter-based symbols (4): Lunate Epsilon (ϵ U+03F5), Lunate Sigma (ϲ U+03F2), Capital Lunate Sigma (Ϲ U+03F9), Upsilon with Hook (ϒ U+03D2)
• Mathematical constants (3): Eulers Constant (ℇ U+2107), Plancks Constant (ℎ U+210E), Plancks Constant Over 2 π (ℏ U+210F),
• Unit symbols (6): Angstrom (Å U+212B), Ohm (Ω, U+2126), Kelvin (K U+212A) Fahrenheit (℉ U+2109), Celsius (℃ U+2103), Micro Sign (U+00B5)
• Currency symbol: Rupee Sign (U+20A8)
• Punctuation (4): One Dot Leader (U+2024), No Break Space (U+00A0), Non-breaking Hyphen (U+2011), Tibetan Mark Delimitter Tsheg Bstar (U+0F0C)
• Other letter-like symbols (10): Information Source (ℹ U+2139), Account Of (℀ U+2100), Addressed to the Subject (℁ U+2101), Care of (℅ U+2105), Cada una (℆ U+2106), Numero (№ U+2116), Telephone Sign (℡ U+2121), Facsimile Sign (℻ U+213B), Trademark (™ U+2122), Service Mark (℠ U+2120)

In addition, several scripts use glyph position such as superscripts and subscripts to differentiate semantics. In these cases subscripts and superscripts are not merely rich text, but constitute a distinct character — similar to a hybrid between a diacritic and a letter — in the writing system (130 total).

• 112 characters representing abstract phonemes from phonetic alphabets such as the International Phonetic Alphabet use such positional glyphs to represent semantic differences.
• 14 characters from the Kanbun script ()
• 1 character from the Tifinagh script (ⵯ U+2D6F)
• 1 character from the Georgian script (ჼ U+10FC)
• masculine and feminine ordinal indicators included in the Latin-1 supplement block

Finally, Unicode designates roman numerals as compatibility equivalence to the Latin letters that share the same glyphs. Here the Unicode Standard make the same mistake in confusing glyph and character that it so often seeks to prevent. Certainly there's a need to deal with the visual ambiguity these characters may suffer when sharing the same glyphs, however a sign-value numeral for one is certainly a semantically distinct character from a Latin capital or small letter ‘i’. A similar visual ambiguity exists between such characters as the Latin capital letter A ( U+0041) and the Greek capital letter Alpha (Α U+0391), yet Unicode does not unify those characters.

• Capital Roman Numerals: One (Ⅰ U+2160), Five (Ⅴ U+2164), Ten (Ⅹ U+2169), Fifty (Ⅼ U+216C), One Hundred (Ⅽ U+216D), Five Hundred (Ⅾ U+216E), One Thousand (Ⅿ U+216F)
• and lower case variants: One (ⅰ U+2170), Five (ⅴ U+2174), Ten (ⅹ U+2179), Fifty (ⅼ U+217C), One Hundred (ⅽ U+217D), Five Hundred (ⅾ U+217E) and One Thousand (ⅿ U+217F)
• 18 precomposed roman numerals in uppercase and lowercase variants (2-4, 6-9 and 11-12)

Roman numeral One Thousand actually has a third character representing a third form or glyph for the same semantic unit: One Thousand C D (ↀ U+2180). From this glyph, one can see where the practice of using a Latin M may have arose. Strangely, though Unicode unifies the sign-value roman numerals with the very different (though visually similar) Latin letters, the Indic Arabic place-value (positional) decimal digit numerals are repeated 24 times (a total of 240 code points for 10 numerals) throughout the UCS without any relational or decomposition mapping between them.

The presence of these 164 semantically distinct though visually similar characters among the compatibility characters complicates the topic of compatibility characters. Some suggest discouraging the use of compatibility characters by content authors. However, in certain specialized areas, these characters are important and quite similar to other characters that have not been included among the compatibility characters. For example, in certain academic circles the use of roman numerals as distinct from latin letters that share the same glyphs would be no different than the use of Cuneiform numerals or ancient Greek numerals. Collapsing the roman numeral characters to Latin letter characters eliminates a semantic distinction. A similar situation exists for phonetic alphabet characters that use subscript or superscript positioned glyphs. In the specialized circles that use phonetic alphabets, authors should be able to do so without resorting to rich text protocols.

Compatibility Blocks

Several blocks of Unicode characters include either entirely or almost entirely all compatibility characters. These compatibility blocks contain none of the semantically distinct compatibility characters and so they fall unambiguously into the set of discouraged characters. Unicode recommends authors use the plain text compatibility decomposition equivalents instead and complement those characters with rich text markup. This approach is much more flexible and open-ended than using the finite set of circled or enclosed alphanumerics to give just one example.

Unfortunately, there are a small number of characters even within the compatibility blocks that themselves are not compatibility characters. The “Enclosed CJK Letters and Months” block contains a single non-compatibility character: the ‘Korean Standard Symbol’ (㉿ U+327F). This symbol and 12 other characters have been included in these blocks for no known reasons. The “CJK Compatibility Ideographs” block contains these non-compatibility unified Han ideographs:

1. (U+FA0E): 﨎
2. (U+FA0F): 﨏
3. (U+FA11): 﨑
4. (U+FA13): 﨓
5. (U+FA14): 﨔
6. (U+FA1F): 﨟
7. (U+FA21): 﨡
8. (U+FA23): 﨣
9. (U+FA24): 﨤
10. (U+FA27): 﨧
11. (U+FA28): 﨨
12. (U+FA29): 﨩

These thirteen characters are neither compatibility characters nor are their use discouraged in any way.

Several other characters in these blocks have no compatibility mapping but are clearly intended for legacy support:

Alphabetic Presentation Forms (1)

1. Hebrew Point Judeo-Spanish Varika (U+FB1E): ﬞ. This is a glyph variant of Hebrew Point Rafe (U+05BF): ֿ , though Unicode provides no compatibility mapping.

Arabic Presentation Forms (4)

1. “Ornate Left Parenthesis” (U+FD3E): ﴾. A glyph variant for U+0029 ‘)’
2. “Ornate Right Parenthesis” (U+FD3F): ﴿. A glyph variant for U+0028 ‘(’
3. “Ligature Bismillah Ar-Rahman Ar-Raheem” (U+FDFD): ﷽. Bismillah Ar-Rahman Ar-Raheem is a ligature for Teh Marbuta (U+0629), Lam (U+0644), Meem (U+0645), Seen (U+0633), Beh (U+0628), (بسملة)
4. “Arabic Tail Fragment” (U+FE73): ﹳ for supporting text systems without contextual glyph handling

CJK Compatibility Forms (2 that are both related to CJK Unified Ideograph: U+4E36 丶)

1. Sesame Dot (U+FE45): ﹅
2. White Sesame Dot (U+FE46): ﹆

Enclosed Alphanumerics (21 rich text variants)

1. 10 Negative Circled Numbers (0 and 11 thru 20) (U+24FF and U+24EB thru U+24F4): ⓫ – ⓴
2. 11 Double Circled Numbers (0 thru 10) (U+24F5 thru U+24FE): ⓵ – ⓾

Normalization

Main article: Unicode normalization

Normalization is the process by which Unicode conforming software first performs compatibility decomposition before making comparisons or collating text strings. Typically normalization is performed without altering the underlying stored text data (lossless). However, some software may potentially make permanent changes to text that eliminates the canonical compatibility characters from text storage (lossy).