The thing is that JSON doesn't actually require or otherwise specify UTF-8. A JSON encoding is defined as a sequence of unicode codepoints: it doesn't actually tell you how to send those over the wire. You have to do that yourself, and specify the encoding out of band somehow.
If JSON did actually specify UTF-8 and provided a sensible escape mechanism for the whole of unicode, this would be an improvement, because it would turn JSON into an actual standard data-exchange format, rather than a textual encoding of its infoset.
They're not talking about strings in JSON; they're saying that a JSON-serialized blob must itself be valid UTF-8. So, to decode JSON, you first have to parse the whole thing to convert it to codepoints (and possibly bail at this step)—and then (lex-and-)parse those codepoints again to turn them into a data structure, making sure to pass through things that turn out to be strings as the same sequence of codepoints you already parsed out rather than passing them to a UTF-8 decoder there and then, like you would in any other format.
Erlang's internal serialization format, meanwhile—and all the language's native pattern-matching constructs—are built on what are basically "generators" that consume a (possibly-infinite) binary stream, and lex tokens directly out of it. JSON doesn't really work with this approach; what you end up doing is having one generator that consumes the binary stream and emits codepoints, and then another generator that consumes codepoints and emits tokens. This introduces a lot of intermediate allocations and message-passing—whereas, with most Erlang protocol handlers, your TCP handler passes you a slice of a VM-managed shared binary, and then you just pass around and re-slice that slice.
I was thinking to myself while reading the link "Why on earth would someone think UTF8 is a problem?" and then I saw it was a erlang thread. To complain about it being slow to validate while talking in an erlang thread... christ. It must be frustrating trying to deal with json, which is full of strings, in a language like erlang.
Erlang has been receiving and sending data. It just like to deal well defined binary messages and it likes to encode/decode them into its own representations (records, terms) at the boundary where they come in and leave the system.
Comments
Only an Erlang mailing list would prompt someone to complain that a document format requires valid UTF-8.
This is a glimpse of the last place on earth where relying on US ASCII is considered a positive good.
The thing is that JSON doesn't actually require or otherwise specify UTF-8. A JSON encoding is defined as a sequence of unicode codepoints: it doesn't actually tell you how to send those over the wire. You have to do that yourself, and specify the encoding out of band somehow.
If JSON did actually specify UTF-8 and provided a sensible escape mechanism for the whole of unicode, this would be an improvement, because it would turn JSON into an actual standard data-exchange format, rather than a textual encoding of its infoset.
They're not talking about strings in JSON; they're saying that a JSON-serialized blob must itself be valid UTF-8. So, to decode JSON, you first have to parse the whole thing to convert it to codepoints (and possibly bail at this step)—and then (lex-and-)parse those codepoints again to turn them into a data structure, making sure to pass through things that turn out to be strings as the same sequence of codepoints you already parsed out rather than passing them to a UTF-8 decoder there and then, like you would in any other format.
Erlang's internal serialization format, meanwhile—and all the language's native pattern-matching constructs—are built on what are basically "generators" that consume a (possibly-infinite) binary stream, and lex tokens directly out of it. JSON doesn't really work with this approach; what you end up doing is having one generator that consumes the binary stream and emits codepoints, and then another generator that consumes codepoints and emits tokens. This introduces a lot of intermediate allocations and message-passing—whereas, with most Erlang protocol handlers, your TCP handler passes you a slice of a VM-managed shared binary, and then you just pass around and re-slice that slice.
I was thinking to myself while reading the link "Why on earth would someone think UTF8 is a problem?" and then I saw it was a erlang thread. To complain about it being slow to validate while talking in an erlang thread... christ. It must be frustrating trying to deal with json, which is full of strings, in a language like erlang.
JSON decoders decode string to binaries.
https://github.com/talentdeficit/jsx
Erlang has been receiving and sending data. It just like to deal well defined binary messages and it likes to encode/decode them into its own representations (records, terms) at the boundary where they come in and leave the system.