That’s the whole point of having field and record separators as distinct values in ASCII. There is no other valid use for them, so no escaping is necessary. Have you ever used ASCII value 30 for anything, anywhere, in your life?
I have seen no end of CSV data that embeds CSV data in a field.
So in CSV [1] a record is separated by a CRLF (0x0D 0xA[2]) and a field value is separated by a comma (0x2c). In ADT (ASC?), the record separator is 0x1E and the unit/field separator is 0x1F.
But there are two more separators defined: file (0x1C) and group (0x1D).
I'm not sure if it's defined anywhere, but if if you wish to embed ADT data with-in a ADT file, and have it as part of the CSV-equivalent field (unit), you could say that:
after the 0x1E record separator, put a group separator (0x1D)
which will denote the beginning of ADT sequence which will
be treated as a unit value. The end of the value shall
("MUST"?) be denoted by another group separator, after which
a unit separator will indicate the next field.
The fact that there are four separation characters would allow for some to be used for embedding applications to tell parsers that a new 'level' of parsing is being done.
Out of curiosity, in which general circumstances do you tend to see this sort of data? A lot of the support I see for ASV/USV/whatever comes from the idea that this is negligibly rare, which has always sounded to me like a dangerous assumption for a general-purpose format.
Some (many?) years ago i wrote a FOSS tool for messing with CSV called CSVfix. People used to mail me CSV files and ask how to parse them. Also, in my general consulting/contracting work I came across all sorts of weird stuff, a lot of it machine-generated.
Sometimes the content of a cell in e.g. a CSV file will be another CSV file. This inevitably happens with pretty much every format, whether CSV, TSV, JSON, etc, so you'll need some sort of quoting rule no matter what you do.
But that’s my point. The person I responded to was objecting that they might want to use control characters like ASCII 30 as values in a field. As you and I agree, that will never happen.
There is no other valid use for them, so no escaping is necessary.
Famous last words...
Have you ever used ASCII value 30 for anything, anywhere, in your life?
No but if this format took off I would be.
And then because text editors start displaying them so the files make sense, people start using them for other purposes, like separating values on a line or copying spreadsheet cells to the clipboard, because it's more "semantic", and those lines get pasted into a field in a tool that exports its data as CSV.
Comments
or probably to see them in the screen in any sensible form.
Also, what to do if you want to embed those ctrl characters in a field? you are likely back to the way that CSV does it with quotes, commas and CRLFs.
That’s the whole point of having field and record separators as distinct values in ASCII. There is no other valid use for them, so no escaping is necessary. Have you ever used ASCII value 30 for anything, anywhere, in your life?
I have seen no end of CSV data that embeds CSV data in a field. I'm sure this will happen for whatever character format you pick.
So in CSV [1] a record is separated by a CRLF (0x0D 0xA[2]) and a field value is separated by a comma (0x2c). In ADT (ASC?), the record separator is 0x1E and the unit/field separator is 0x1F.
But there are two more separators defined: file (0x1C) and group (0x1D).
I'm not sure if it's defined anywhere, but if if you wish to embed ADT data with-in a ADT file, and have it as part of the CSV-equivalent field (unit), you could say that:
The fact that there are four separation characters would allow for some to be used for embedding applications to tell parsers that a new 'level' of parsing is being done.[1] https://datatracker.ietf.org/doc/html/rfc4180
[2] https://www.man7.org/linux/man-pages/man7/ascii.7.html
Out of curiosity, in which general circumstances do you tend to see this sort of data? A lot of the support I see for ASV/USV/whatever comes from the idea that this is negligibly rare, which has always sounded to me like a dangerous assumption for a general-purpose format.
Some (many?) years ago i wrote a FOSS tool for messing with CSV called CSVfix. People used to mail me CSV files and ask how to parse them. Also, in my general consulting/contracting work I came across all sorts of weird stuff, a lot of it machine-generated.
No, no one has ever used that character because it can't be typed or displayed which is the entire point of the thread you're replying to
Sometimes the content of a cell in e.g. a CSV file will be another CSV file. This inevitably happens with pretty much every format, whether CSV, TSV, JSON, etc, so you'll need some sort of quoting rule no matter what you do.
But that’s my point. The person I responded to was objecting that they might want to use control characters like ASCII 30 as values in a field. As you and I agree, that will never happen.
What about the cases where people don’t want to use it but still do?
People C&P values from a source that includes those characters and they‘ll break your export file
Famous last words...
No but if this format took off I would be.
And then because text editors start displaying them so the files make sense, people start using them for other purposes, like separating values on a line or copying spreadsheet cells to the clipboard, because it's more "semantic", and those lines get pasted into a field in a tool that exports its data as CSV.
What then?