But there is never any reason to use these characters literally, they are just delimiters.
I can put [ ]* in my comment (maybe because I'm demonstrating the format, referencing the characters, or just being capricious), and now someone scraping and storing comments has a need to use those characters literally. Sometimes fine to ignore certain content or store it lossily, but often not.
yes, you would still need to clean your inputs before randomly adding it to your table. Your contrived example brings me back to my original assertion that as long as you’re ok with those characters not being valid data it works fine. So, sure if someone really wanted to store those two literal non-visible characters in a text file that would not work. Everyone else could just not do that.
yes, you would still need to clean your inputs before randomly adding it to your table.
Lossy is fine in some cases, but in many cases you do actually need the specific text you're trying to store - not just something similar to it. Hence my objection to "never any reason to use these characters literally".
Your contrived example [...] if someone really wanted to store those two literal non-visible characters in a text file
Needing to store these specific characters is rare, but needing to store arbitrary text (possibly from adversarial/mischievous parties, or just a large enough dataset that encountering all edge-cases is inevitable) is common. For instance, for security reasons a log shouldn't break or have a blindspot for folders with those characters.
as long as you’re ok with those characters not being valid data it works fine
Which is what I'm saying in my original comment with "or alternatively, a restriction for the stored text not to have them"
if you’re storing arbitrary text from untrusted sources you will always need to clean it first. Plus in those instances you’ll probably want a db or json or whatever works well with your language anyway.
I guess I’m saying that if this had caught on early with ubiquitous support it could have saved us from the mess that is csv/tsv/etc. It wouldn’t have negated the need for more advanced storage and serialization formats.
if you’re storing arbitrary text from untrusted sources you will always need to clean it first
Reversible escaping of characters is pretty common (though not always; length-before-text formats don't require it). But to "clean" as in deleting characters such that you can no longer get back to the original string is definitely not required for all formats, and is a fairly undesirable property.
Plus in those instances you’ll probably want a db or json or whatever works well with your language anyway.
You'd want to use some format that doesn't have the problem this one has, yeah. IMO ASCII delimited text just isn't really anywhere on the Pareto front of formats you'd want to use - it's unpleasant to work with manually, and once you're writing the file through code or a tabular editor you may as well use a format that can handle arbitrary text.
I guess I’m saying that if this had caught on early with ubiquitous support it could have saved us from the mess that is csv/tsv/etc
I think you could say the same of RFC 4180. In reality, I don't see why this wouldn't also spawn dialects, like people adding newlines between rows so they can open it in a text editor without it being in one huge long line, or inventing an escaping scheme so that it can handle arbitrary text.
Comments
Right, nesting wouldn’t be possible. But there is never any reason to use these characters literally, they are just delimiters.
I can put [ ]* in my comment (maybe because I'm demonstrating the format, referencing the characters, or just being capricious), and now someone scraping and storing comments has a need to use those characters literally. Sometimes fine to ignore certain content or store it lossily, but often not.
*: (copy-paste between the brackets into https://bobpritchett.com/unicode-inspector)
yes, you would still need to clean your inputs before randomly adding it to your table. Your contrived example brings me back to my original assertion that as long as you’re ok with those characters not being valid data it works fine. So, sure if someone really wanted to store those two literal non-visible characters in a text file that would not work. Everyone else could just not do that.
Lossy is fine in some cases, but in many cases you do actually need the specific text you're trying to store - not just something similar to it. Hence my objection to "never any reason to use these characters literally".
Needing to store these specific characters is rare, but needing to store arbitrary text (possibly from adversarial/mischievous parties, or just a large enough dataset that encountering all edge-cases is inevitable) is common. For instance, for security reasons a log shouldn't break or have a blindspot for folders with those characters.
Which is what I'm saying in my original comment with "or alternatively, a restriction for the stored text not to have them"
if you’re storing arbitrary text from untrusted sources you will always need to clean it first. Plus in those instances you’ll probably want a db or json or whatever works well with your language anyway.
I guess I’m saying that if this had caught on early with ubiquitous support it could have saved us from the mess that is csv/tsv/etc. It wouldn’t have negated the need for more advanced storage and serialization formats.
Reversible escaping of characters is pretty common (though not always; length-before-text formats don't require it). But to "clean" as in deleting characters such that you can no longer get back to the original string is definitely not required for all formats, and is a fairly undesirable property.
You'd want to use some format that doesn't have the problem this one has, yeah. IMO ASCII delimited text just isn't really anywhere on the Pareto front of formats you'd want to use - it's unpleasant to work with manually, and once you're writing the file through code or a tabular editor you may as well use a format that can handle arbitrary text.
I think you could say the same of RFC 4180. In reality, I don't see why this wouldn't also spawn dialects, like people adding newlines between rows so they can open it in a text editor without it being in one huge long line, or inventing an escaping scheme so that it can handle arbitrary text.