Sadly, Unicode is a clusterfuck. But can anything be done about it? Or should we just be happy we for once have managed to get a decent adoption of something interoperable?
Unicode is not a clusterfuck. Overall, it is a very comprehensive, well thought out standard, that has improved the situation for interoperable internationalization dramatically over the hundreds of separate encodings that preceded it.
The clusterfuck is mostly in the essential complexity of the problem; the world's languages and writing systems are quite complex and varied, and all of the writing systems and punctuation conventions were designed for reading, hand-writing and hand-typesetting by people who understood the language in question, not automatic typesetting and processing by a general purpose computer.
The complexity of the problem was increased by the necessity of providing migration paths from legacy encodings to Unicode and back again; without such a guarantee, bootstrapping the world into using Unicode would have been a much more difficult proposition, but that constraint also means that many oddities of legacy encodings have had to be preserved in Unicode in order to be able to preserve that round-trip mapping.
Unicode, and sister projects like the Common Locale Data Repository, are doing an admirable job of navigating and standardizing this complex problem.
There are definitely aspects of the Unicode process where they have gotten it wrong; UCS-2/UTF-16 is one of them, in hindsight it is apparent that UTF-8 is superior in pretty much every way. Having a variable width encoding but which most of the world's text fits in a fixed width, and which has endianness issues that necessitate inclusion of an non-textual byte order marker to disambiguate, has caused a number of problems and incompatibilities. There may be a few other points of legitimate criticism, like some aspects of Han unification.
But on the whole, outside of a few problems like those, the "clusterfuck" is caused not by Unicode, but simply by the essential complexity of the problem involved. Language and text are simply difficult things to model in a computer.
The complexity of the problem was increased by the necessity of providing migration paths from legacy encodings to Unicode and back again; without such a guarantee, bootstrapping the world into using Unicode would have been a much more difficult proposition, but that constraint also means that many oddities of legacy encodings have had to be preserved in Unicode in order to be able to preserve that round-trip mapping.
While I agree that preserving round-trip integrity was essential for Unicodes success, I'm not sure if the approach taken to achieve that was the best one. I would have preferred that the complexity tradeoff would have been shifted to software converting between Unicode and legacy encodings by having more complex mapping tables and cleaner code point space.
I also think that Unicode Consortium should have been more aggressive in segregating (and discouraging the general use of) legacy compatibility features/codepoints and the stuff that is actually supposed to be used. My personal pet peeve is precomposed characters.
In a more general note, I sometimes wonder if it would have been beneficial to have separate layers in Unicode and have more focused on providing generic primitives. As a simple example it is mighty convenient that I can type 2³ = 8 in plain text, but arguably it would be even nicer if instead of special 'SUPERSCRIPT THREE' codepoint there would be generic superscript modifier codepoint that could be combined with any character.
Speaking of superscripts, they demonstrate well one aspect that I dislike in Unicode, the way they have absorbed legacy encodings verbatim. The numeric superscripts (e.g. ⁰ ¹ ² ³ ⁴ ⁵ ⁶) happen to have inconsistent look on my machine because the superscripts for 1, 2, and 3 are from Latin1 while the rest are in their own block.
Comments
Sadly, Unicode is a clusterfuck. But can anything be done about it? Or should we just be happy we for once have managed to get a decent adoption of something interoperable?
Unicode is not a clusterfuck. Overall, it is a very comprehensive, well thought out standard, that has improved the situation for interoperable internationalization dramatically over the hundreds of separate encodings that preceded it.
The clusterfuck is mostly in the essential complexity of the problem; the world's languages and writing systems are quite complex and varied, and all of the writing systems and punctuation conventions were designed for reading, hand-writing and hand-typesetting by people who understood the language in question, not automatic typesetting and processing by a general purpose computer.
The complexity of the problem was increased by the necessity of providing migration paths from legacy encodings to Unicode and back again; without such a guarantee, bootstrapping the world into using Unicode would have been a much more difficult proposition, but that constraint also means that many oddities of legacy encodings have had to be preserved in Unicode in order to be able to preserve that round-trip mapping.
Unicode, and sister projects like the Common Locale Data Repository, are doing an admirable job of navigating and standardizing this complex problem.
There are definitely aspects of the Unicode process where they have gotten it wrong; UCS-2/UTF-16 is one of them, in hindsight it is apparent that UTF-8 is superior in pretty much every way. Having a variable width encoding but which most of the world's text fits in a fixed width, and which has endianness issues that necessitate inclusion of an non-textual byte order marker to disambiguate, has caused a number of problems and incompatibilities. There may be a few other points of legitimate criticism, like some aspects of Han unification.
But on the whole, outside of a few problems like those, the "clusterfuck" is caused not by Unicode, but simply by the essential complexity of the problem involved. Language and text are simply difficult things to model in a computer.
While I agree that preserving round-trip integrity was essential for Unicodes success, I'm not sure if the approach taken to achieve that was the best one. I would have preferred that the complexity tradeoff would have been shifted to software converting between Unicode and legacy encodings by having more complex mapping tables and cleaner code point space.
I also think that Unicode Consortium should have been more aggressive in segregating (and discouraging the general use of) legacy compatibility features/codepoints and the stuff that is actually supposed to be used. My personal pet peeve is precomposed characters.
In a more general note, I sometimes wonder if it would have been beneficial to have separate layers in Unicode and have more focused on providing generic primitives. As a simple example it is mighty convenient that I can type 2³ = 8 in plain text, but arguably it would be even nicer if instead of special 'SUPERSCRIPT THREE' codepoint there would be generic superscript modifier codepoint that could be combined with any character.
Speaking of superscripts, they demonstrate well one aspect that I dislike in Unicode, the way they have absorbed legacy encodings verbatim. The numeric superscripts (e.g. ⁰ ¹ ² ³ ⁴ ⁵ ⁶) happen to have inconsistent look on my machine because the superscripts for 1, 2, and 3 are from Latin1 while the rest are in their own block.