> Yikes, does it really deserve that much derision?
To my simple mind it had one job: allocate every grapheme a number (code point). Had it done that, the 1/2 of the article warning you about the difficulty of iterating and modifying code points would have disappeared.
But I guess it had a 2nd job: create a way of representing those numbers. The obvious way, u32, was difficult for ASCII users swallow as it quadrupled the space used for a string. The solution we settled on, UTF-8 didn't come form Unicode (or ISO). It came from Ken Thompson (the Ken Thompson, who created B, the predecessor of C), when he tired to make something workable for C.
Unicode was the entity that ballsed up both of those tasks. It was a fork of ISO 10646. It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. I have no idea why, as it makes the programmers task far harder. Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? Because they had to kludge their way around the USC-2 mistake to create UTF-16, and it's limited 1 million.
Which leads us to the one thing in the article I disagree with:
> The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disk.
No, that's not the only downside. There is one more: USC-2 / UTF-16 has the endianness problem. A 16 bit value needs two bytes to represent it, and you can write two bytes to storage in two ways - little endian or big endian. They didn't specify, so the same string can have two different representations on disk. They added the infamous BOM markers to distinguish between them.
I could go on, but colour me singularly unimpressed with this mob.
It never was. The earlier draft of ISO/IEC 10646 bears absolutely no resemblance with the current 10646 and Unicode (for example, the first character in ISO/IEC DIS 10646:1990 was 0x20202020, which I believe is mapped to a space U+0020). Unicode had a much better design compared to 10646 so the final 10646 was retrofitted to Unicode instead.
> It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned.
UCS-2 was already in 10646 in the draft stage. It had an even worse mechanism than surrogate pairs: escape sequences from ISO/IEC 2022 to switch groups and planes (upper 16 bits of code point). The standardized UCS-2 doesn't have them because of the merger of then-16-bit Unicode.
> Later they introduced the grapheme clusters rather than allocating a separate code point for each variant.
That sounds like that Unicode initially allocated separate code points for each variant. They didn't, or rather couldn't. An easy example is a Latin character with combining marks. There are a lot of combining marks in existence, some even defined before Unicode (yes, it's not the Unicode invention!), and sometimes a single character can have multiple marks. So Unicode only gave separate code points for compatibility, and otherwise resorted to the normalization mechanism that understands how to handle such cases.
The concept of grapheme cluster naturally arises from the existence of normalization. Not in the strictest sense, but it can be thought as a closed set over normalization and concatenation, so that it roughly matches with user-perceived characters. So grapheme clusters were already there, only the precise algorithm was specified later.
> To my simple mind it had one job: allocate every grapheme a number (code point).
You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance.
> Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more?
For last 20 years the rate was about 2,700 new code points per year. It would take more than 200 years to fill all other unassigned planes at this rate. And most "new" code points (in quantity) are for rare or ancient Han characters, which are technically unbounded but strongly bounded by existing ancient works and scholarly works to uncover them. I doubt there remain more than 100,000 potentially encodable Han characters.
> You can easily have more than 10M code points in this way. The current set of Hangul syllables, precomposed or not, is 125 * 95 * 138 = 1,638,750 characters. Latin characters with at most 3 combining marks (known to exist in the wild) would be probably in the same order of magnitude. Maybe now you can try, thanks to the computing power and all the information, but in 1990? Fat chance.
It can be made to work both ways. The current situation pushes the handling of compose points onto the application programmer. Every time he wants to index into a array of characters, maybe to handle backspace of the user pressing arrow keys, he's forced to handle composition. But your average programmer tasked with writing gathering some information from the web or creating an accounting package doesn't care about this stuff, so he's going to stuff it up 10 times out of 10. That why the sorts of problems illustrated original article are legion today.
The alternative is 10M code points as you say. But it doesn't have to be 10M real code points. Someone down the software stack, a piece of software could say "oh, this code point represents a composed grapheme, I'll break it down into it's parts". In fact "break it down into it's parts" might mean turn it into exact representation we have now.
The difference between the two alternatives is who has to do the work. With the composition approach, the font rendering library has it slightly easier but the application write has to do more work. In the 10M code points approach, the application writers job has been made easier, at the expense of the font rendering library coder job has become harder.
It seems pretty obvious to me which of those two approaches wins. There are literally orders of magnitude more end user applications than there are font rendering libraries out there, and what's more the font rendering library programmers are far more likely to very concerned about doing grapheme clustering right. So if you took the 10M code point approach, you would have saved the planet a lot of code, and got a better result to boot.
As for the rest - your correction that 10646 was the source of many of them problems is appreciated. But that doesn't alter the fact that from a programmers perspective the spec is far harder to implement at the business end than it should be. The problems started with USC-2 and it compounded form there. And as a consequence, we have a large number of font rendering bugs we could have escaped had the spec been done differently.
I appreciate your reply, which I never expected in this situation.
If my understanding is correct, your thesis is that Unicode should be hidden from application programmers as much as possible, much like the fact that GC hides memory management so to say. Not to say Unicode is bad or even shouldn't exist at all, but something like that it has to be abstracted away. This is a much more reasonable than most (quote-unquote) Unicode criticisms indeed. I'm not sure whether this is possible in the near future however, for reasons I'll work out here.
----
From perspectives of API consumers, most if not all programming languages have a suboptimal design for the human text. In fact the type name "string" itself is inappropriate, its name comes from the assumption that a human text is a string of symbols, which is not incorrect but not helpful either. A proper "human text" type (or a collection of them) should ideally be able to do the following:
- An abillity to contain additional linguistic informations like locales, grammatical genders or numbers if possible. Some can be guessed, some can be retrieved from external contexts (e.g. HTTP `Accept-Language`), some have to be retrived with a consent. Any text operation should retain them if the corresponding text is also retained.
- A language-aware formatter. For example `"Total: ${n} files"` should automatically change "files" to "file" when `n` is 1. Moreover, `"Total: ${n} ${objectName}"` should do the same if `objectName` is an English text "files". (This is why every text should retain linguistic informations!) Of course the format text should be translatable (say, to "파일 총 ${n}개" in Korean) and that shouldn't change the original code.
- Proper textual isolation. If my text is composed of multiple scripts or languages, they should not affect each other in any way, and should be displayed in the best way possible. For example a missing font should not give broken boxes; either the font should be downloaded on demand, or a note about missing font should be shown instead. Inserting RTL texts into LTR texts should not flip either of texts (unless it is required by the surrounding languages). Basically, no surprises even if you don't know about them.
- Situation-aware alternatives. Even after the formatting, a long text that doesn't fit into the UI should be shortened in the way that as many information is preserved as possible. For example the text "Nice to meet you, ImagineAVeryLongUserNameHere!" will be cut into "Nice to meet you, ImagineAVery..." today, but one should be able to turn this into "Hello, ImagineAVeryLongUserNam..." from the formatting layer.
None of these operations actually concern Unicode, but they are incredibly hard---if not impossible---to build. Most of them are at best fuzzily defined or often undefinable. So we are left with a number of localization and internationalization libraries which are ignored by most developers to say the least. A mere "string" type is a norm, a program has no idea about the text and proceeds with faulty assumptions, and users are so accustomed to bad text handling that they even don't expect much. If enough users complain there is a chance of improvements, but even that is done by a case-by-case basis.
---
Unicode sits at the level much lower than what I've imagined before. It is not even a component to build the human text. It is a component to build a string, that can be somehow used to build the human text if one is very careful. Most human text operations can't be done with Unicode alone.
For example, people argue that the number of "characters" in a single Emoji sequence (say, one mentioned in https://hsivonen.fi/string-length/) should be 1 and others don't make sense. This is meaningless because it will appear as a number of broken boxes if emoji fonts are not installed anyway (but not five, because it contains two default-ignorable code points). It matters what the number of "characters" is used for, and that's a whole point of the linked article. And the definition of user-perceived characters does vary over locales, so you can't count them without a linguistic information anyway.
You may still argue that Unicode algorithms are designed for the human text encoded in strings. That's a very nuanced argument, because one can also argue that they are the best effort approximation of human text operations for strings, in which case they are not the human text operations themselves. For example many languages have a case conversion operation over strings, with a varying degree of Unicode conformance (ASCII-only, simple fold, full fold, locale-dependent fold, title case, ...). But the case conversion itself is not the human text operation! Even assuming bicameral scripts, some texts are never capitalized (e.g. "McDonald" frequently capitalizes to "McDONALD", not "MCDONALD"). The human text operation, here full capitalization, needs much more than the Unicode case conversion algorithm.
Given this, it is a misguided effort to make strings more aligned with the human text, because it is not possible at all. As people frequently mistake strings as human texts however, the second best thing is to get rid of any string operation. Swift almost did this but retained a default grapheme view---I think it is actually worse given the instability of (extended) grapheme clusters over time, but also understand why they had to do that. [1] The third best thing is probably to stress that a string is not a human text, in the same way that a floating point number is not a real number. And the original post, in spite of some errors, did a good enough job in this regard.
[1] There is also a precedent of Raku's NFG (which dynamically allocates a negative code point for new grapheme clusters seen), but this is more or less an optimization of the graphemes view. The current Unicode has an infinite number of distinct grapheme clusters by design.
> You may still argue that Unicode algorithms are designed for the human text encoded in strings. That's a very nuanced argument, because one can also argue that they are the best effort approximation of human text operations for strings, in which case they are not the human text operations themselves. For example many languages have a case conversion operation over strings,
That is ... very nuanced indeed. Case folding does look to be difficult issue in it's own right, but it's not something I do a lot and besides even 20 years ago it was deferred to a library function (str.lower() or whatever the language provides).
The issue the original article correctly says every app trips over is neither nuanced, nor uncommon. It's boring stuff like handling backspace or left arrow cursor movement - stuff programmer have to do all the time, and that is difficult to put in a library. As a consequence they get it wrong over and over again. Unicode representing a grapheme with a single code point would fix that.
As you say I'm not sure there is a simple change you could make to Unicode that renders case folding or the other problems you describe easy. To me that's a strong hint it's not the right place to address those problems.
To my simple mind it had one job: allocate every grapheme a number (code point). Had it done that, the 1/2 of the article warning you about the difficulty of iterating and modifying code points would have disappeared.
But I guess it had a 2nd job: create a way of representing those numbers. The obvious way, u32, was difficult for ASCII users swallow as it quadrupled the space used for a string. The solution we settled on, UTF-8 didn't come form Unicode (or ISO). It came from Ken Thompson (the Ken Thompson, who created B, the predecessor of C), when he tired to make something workable for C.
Unicode was the entity that ballsed up both of those tasks. It was a fork of ISO 10646. It's main contribution over 10646 was UCS-2 - ie 16 bits per character. That decision was so bad it had to be abandoned. Later they introduced the grapheme clusters rather than allocating a separate code point for each variant. I have no idea why, as it makes the programmers task far harder. Maybe they ran out of code points. How could they possibly run out of code points, given U32 has 4 billion of them and UTF-8 could potentially have more? Because they had to kludge their way around the USC-2 mistake to create UTF-16, and it's limited 1 million.
Which leads us to the one thing in the article I disagree with:
> The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disk.
No, that's not the only downside. There is one more: USC-2 / UTF-16 has the endianness problem. A 16 bit value needs two bytes to represent it, and you can write two bytes to storage in two ways - little endian or big endian. They didn't specify, so the same string can have two different representations on disk. They added the infamous BOM markers to distinguish between them.
I could go on, but colour me singularly unimpressed with this mob.