NUL is a tricky beast even under ideal conditions. In the first draft I'd simply forbidden them. Simple, decisive, and of course people didn't like that. then I did allow it, and suddenly the security gates were flung open. Some platforms simply cannot support nuls in strings, so now it becomes platform dependent (bad from a security perspective). So then my next attack vector was to make it configurable with a default of deny. Better from a security perspective, but now it's complicated. TBH there are nights where I think I should just go back to no NULs at all again. But I really need more discussion with stakeholders to figure out what problems this solves and how important it really is...
Line break is another sticky issue. Non-technical Windows users will inevitably produce documents that use CRLF, so if it rejects the file, that's a bad user experience. What's the best trade-off here? I'm not really sure.
The number encoding thing is being discussed in other comments so I'll leave it be. I had allowed comma because that's how a huge chunk of the world represents the decimal separator. Once again, this is about user experience with the non-technical users. I'm REEEEEEEALLY on the fence with this one.
Entity references are dangerous, yes, but also powerful. The point of them in the format is to solve the recursive reference problem, because you just can't do that otherwise, and these structures do exist in the world. It's another case of an imperfect solution for an imperfect world. Bear in mind I absolutely do NOT want this format to become some Turing complete language. This is just the minimal feature set I could think of to represent real world data.
Arrays-vs-list is another one of those compromises. Encoding an actual array of fixed types into a list would be slow and bulky, leading people to just encode them as a chunk of bytes like they currently do in JSON and other formats. As an imperfect solution to an imperfect world, I want to at least let people preserve the semantic meaning of what they're sending since they're going to use array encodings regardless of what the format supports.
Hum... For nul, it's common to have an escaping sequence (yes, on binary data) and use it to encode the problematic characters. It's for the best if you encode enough of the data for one to be able to dump your file into a terminal and nothing getting compromised on the way (the terminal just failing to work is ok).
Personally, I disagree with how your format handles all those other issues too (except for the numbers), but well, if you think you are correct, go try it. If it works, it works, and my disagreement may easily be misguided. Anyway, I disagree because:
For the line breaks, the internet has a way of trying to "fix" them and completely breaking the line-information of the original document. It would be ok if the format wasn't blank-space dependent, but it is, so changing the lines breaks the data. Anyway, that is becoming a lesser problem with time, so maybe for a new format it's fine.
Entity references on formats that are not focused on them are surprising. That means a lot of software will break once they get one, and tradition says they will do that in a way that compromises computer security. I would either change the format so that references are almost always used or remove them. If an application needs references, it can always tag the entities with an id and put the references there by itself.
The same applies for arrays, in a lesser degree. They will be surprising, but they are also easier to handle. But they are also much less necessary, since lists can always replace them. I'm really not sure if they are a net negative or positive.
Thanks for your reply! It's a really good proposal, and I'll definitely consider it next time I'm serializing something.
NUL is a tricky beast
One alternative I found interesting is Modified UTF-8 (https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8), where NUL is encoded as 0xC0 0x80. It's already in use in Java, apparently. Disclaimer: I've never heard about it before, and I don't know how well supported it is.
Line breaks and commas
I'm from a country that uses comma as decimal separator, and I still prefer dots when programming. I dread ambiguous numbers like "1,001". I'm confident this is true for almost all technical people. And I really don't see non-technical users editing this kind of file.
If they can be trusted to read it, and to modify it without introducing syntax errors, they can be trusted to use dots and an editor that shows LF as linebreaks (i.e. anything but `notepad`).
My choice would be dots only for decimal separator, accept LF and CR LF on reading, but prefer LF when writing.
Of all data types that could have native encodings (colors, IP addresses, lat/lon coordinates, enums, tags, markdown, hashes, file permissions, OIDs, DOIs, ISBNs, etc), I think arbitrary object graphs bring too many downsides for a serialization format.
You don't want reader CVE's because of Billion Laughs, and you don't want to force every programmer writing a simple traversal algorithm to correctly handle cycles.
> accept LF and CR LF on reading, but prefer LF when writing.
That's the current behavior.
I'd suggest changing the SHOULD to MUST here, and remove the "foreign or unknown system" part:
but encoders SHOULD output LF when the destination is a foreign or unknown system.
It's ok if a CR LF sneaked in because a user edited a file manually, but encoders should be more predictable.
> and you don't want to force every programmer writing a simple traversal algorithm to correctly handle cycles.
Ugh... I really really REALLY want you to be wrong on this :(
I have some good news then.
I just checked, and most of my JSON traversals are for things that you already take care of, like binary arrays and handling cycles (huh, talk about irony).
And the billion laughs problem was mostly because XML entities are more like macros, and expanded in place. As long as the reader doesn't try to convert the document to JSON, or naively print the object graph, it should be ok.
I think it might be ok to keep references.
And again, cheers for the encoding specification! It's really cool, and I hope it catches on.
Comments
NUL is a tricky beast even under ideal conditions. In the first draft I'd simply forbidden them. Simple, decisive, and of course people didn't like that. then I did allow it, and suddenly the security gates were flung open. Some platforms simply cannot support nuls in strings, so now it becomes platform dependent (bad from a security perspective). So then my next attack vector was to make it configurable with a default of deny. Better from a security perspective, but now it's complicated. TBH there are nights where I think I should just go back to no NULs at all again. But I really need more discussion with stakeholders to figure out what problems this solves and how important it really is...
Line break is another sticky issue. Non-technical Windows users will inevitably produce documents that use CRLF, so if it rejects the file, that's a bad user experience. What's the best trade-off here? I'm not really sure.
The number encoding thing is being discussed in other comments so I'll leave it be. I had allowed comma because that's how a huge chunk of the world represents the decimal separator. Once again, this is about user experience with the non-technical users. I'm REEEEEEEALLY on the fence with this one.
Entity references are dangerous, yes, but also powerful. The point of them in the format is to solve the recursive reference problem, because you just can't do that otherwise, and these structures do exist in the world. It's another case of an imperfect solution for an imperfect world. Bear in mind I absolutely do NOT want this format to become some Turing complete language. This is just the minimal feature set I could think of to represent real world data.
Arrays-vs-list is another one of those compromises. Encoding an actual array of fixed types into a list would be slow and bulky, leading people to just encode them as a chunk of bytes like they currently do in JSON and other formats. As an imperfect solution to an imperfect world, I want to at least let people preserve the semantic meaning of what they're sending since they're going to use array encodings regardless of what the format supports.
Hum... For nul, it's common to have an escaping sequence (yes, on binary data) and use it to encode the problematic characters. It's for the best if you encode enough of the data for one to be able to dump your file into a terminal and nothing getting compromised on the way (the terminal just failing to work is ok).
Personally, I disagree with how your format handles all those other issues too (except for the numbers), but well, if you think you are correct, go try it. If it works, it works, and my disagreement may easily be misguided. Anyway, I disagree because:
For the line breaks, the internet has a way of trying to "fix" them and completely breaking the line-information of the original document. It would be ok if the format wasn't blank-space dependent, but it is, so changing the lines breaks the data. Anyway, that is becoming a lesser problem with time, so maybe for a new format it's fine.
Entity references on formats that are not focused on them are surprising. That means a lot of software will break once they get one, and tradition says they will do that in a way that compromises computer security. I would either change the format so that references are almost always used or remove them. If an application needs references, it can always tag the entities with an id and put the references there by itself.
The same applies for arrays, in a lesser degree. They will be surprising, but they are also easier to handle. But they are also much less necessary, since lists can always replace them. I'm really not sure if they are a net negative or positive.
Thanks for your reply! It's a really good proposal, and I'll definitely consider it next time I'm serializing something.
One alternative I found interesting is Modified UTF-8 (https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8), where NUL is encoded as 0xC0 0x80. It's already in use in Java, apparently. Disclaimer: I've never heard about it before, and I don't know how well supported it is.
I'm from a country that uses comma as decimal separator, and I still prefer dots when programming. I dread ambiguous numbers like "1,001". I'm confident this is true for almost all technical people. And I really don't see non-technical users editing this kind of file.
If they can be trusted to read it, and to modify it without introducing syntax errors, they can be trusted to use dots and an editor that shows LF as linebreaks (i.e. anything but `notepad`).
My choice would be dots only for decimal separator, accept LF and CR LF on reading, but prefer LF when writing.
You can always define manual ids:
Of all data types that could have native encodings (colors, IP addresses, lat/lon coordinates, enums, tags, markdown, hashes, file permissions, OIDs, DOIs, ISBNs, etc), I think arbitrary object graphs bring too many downsides for a serialization format.You don't want reader CVE's because of Billion Laughs, and you don't want to force every programmer writing a simple traversal algorithm to correctly handle cycles.
Damn, that's a clever idea!
Good enough for me. Commas will be removed.
That's the current behavior.
Ugh... I really really REALLY want you to be wrong on this :(
I'd suggest changing the SHOULD to MUST here, and remove the "foreign or unknown system" part:
It's ok if a CR LF sneaked in because a user edited a file manually, but encoders should be more predictable.I have some good news then.
I just checked, and most of my JSON traversals are for things that you already take care of, like binary arrays and handling cycles (huh, talk about irony).
And the billion laughs problem was mostly because XML entities are more like macros, and expanded in place. As long as the reader doesn't try to convert the document to JSON, or naively print the object graph, it should be ok.
I think it might be ok to keep references.
And again, cheers for the encoding specification! It's really cool, and I hope it catches on.