It's relatively easy to get the document format by using a heuristic on "not so plain" text, not to be talking about more "binary" formats which are even easier to recognise, for example to differentiate between Markdown and Org-Mode and RST, or the different Markdown variants. It is (almost) impossible to get the encoding (welcome to ISO 8859-? vs. Windows-????) or even the used codepage right.
Take at look at the "reality": there are converters for almost any file format and file format descriptions for (almost any imaginable), so there is a chance to read or write almost any format by yourself (yes, some can be really complex, as for example some 3D formats like 3DS or LWO).
I don't mean to say that somebody shouldn't use "plain text" for themselves, but that the idea of "this is plain text, this works everywhere and anytime" is not true.
And the list of known codepages (and I can guarantee you there is some hardware - a printer - out there that uses it's own, because it had been only ever used with it's own printer driver/program/...): https://en.wikipedia.org/wiki/Code_page
This is a misunderstanding of the term: "pain text" sits opposite of "rich text", formats where the markup is part of the content. The terms are at a higher level of abstraction than just encoding, rich text being things like xml/html or word documents that require specialized parsers to retrieve the content. Plain text refers to files that don't have that kind of special markup language.
No, there is more than one valid definition, the one you have used is the one used e.g. by the Unicode standard (but their definition is narrowed to Unicode, so anything not Unicode isn't plain text). But it doesn't matter which definition of "text" somebody uses, encoding is always a problem if not explicitly set like it is possible when using XML and HTML.
Yeah and good luck if your native language requires CJK characters, right-to-left scripts like Arabic, or whatever. Fortunately Unicode is fairly good nowadays but the olden days of the wild west of encoding was terrible.
And if you're already assuming unicode (which all these "plaintext" tools do), why aren't we just using unicode bullet points instead of dashes and *? Why not just use unicode bold and italics instead of *hacks*?
It is a question of the input method first and, as the sibling commenter said, grepping second. Fuzzy search could handle "-" as: "–", "—", "―", "‒" and other dashes. I input slightly more special characters on my phone as it is easier. On my computer I remember how I can easily write ° (on Linux and ChromeOS - Shift-AltGr-0) so I use it more. I don't know how to easily input a bullet point and I am too lazy to open a character table app every time it is needed.
In a way it is the same with programming languages. I think there are now programming languages supporting unicode operators for example, like =, ≠, ∧, ∨, ∩, ∪. But they suffer from the same problem. How to input them, how to search for them?
Comments
Somebody needs to watch "Plain Text" (Dylan Beattie, NDC Copenhagen 2022) https://www.youtube.com/watch?v=gd5uJ7Nlvvo
For example, have fun with ISO 8859-2 vs. Windows 1250. And add Windows 1252 for giggles. And for the ASCII-only persons: there still is EBCDIC.
The space of possible encodings and languages is still far smaller than the space of possible document formats.
It's relatively easy to get the document format by using a heuristic on "not so plain" text, not to be talking about more "binary" formats which are even easier to recognise, for example to differentiate between Markdown and Org-Mode and RST, or the different Markdown variants. It is (almost) impossible to get the encoding (welcome to ISO 8859-? vs. Windows-????) or even the used codepage right.
Take at look at the "reality": there are converters for almost any file format and file format descriptions for (almost any imaginable), so there is a chance to read or write almost any format by yourself (yes, some can be really complex, as for example some 3D formats like 3DS or LWO).
I don't mean to say that somebody shouldn't use "plain text" for themselves, but that the idea of "this is plain text, this works everywhere and anytime" is not true.
And the list of known codepages (and I can guarantee you there is some hardware - a printer - out there that uses it's own, because it had been only ever used with it's own printer driver/program/...): https://en.wikipedia.org/wiki/Code_page
Just use utf-8 everywhere. Yes, even on Windows http://utf8everywhere.org/
Full Unicode support is very hard. The best I've seen is in Raku. You can leave without it completely e.g. Zig.
This is a misunderstanding of the term: "pain text" sits opposite of "rich text", formats where the markup is part of the content. The terms are at a higher level of abstraction than just encoding, rich text being things like xml/html or word documents that require specialized parsers to retrieve the content. Plain text refers to files that don't have that kind of special markup language.
No, there is more than one valid definition, the one you have used is the one used e.g. by the Unicode standard (but their definition is narrowed to Unicode, so anything not Unicode isn't plain text). But it doesn't matter which definition of "text" somebody uses, encoding is always a problem if not explicitly set like it is possible when using XML and HTML.
Love that typo :D
Yeah and good luck if your native language requires CJK characters, right-to-left scripts like Arabic, or whatever. Fortunately Unicode is fairly good nowadays but the olden days of the wild west of encoding was terrible.
And if you're already assuming unicode (which all these "plaintext" tools do), why aren't we just using unicode bullet points instead of dashes and *? Why not just use unicode bold and italics instead of *hacks*?
It is a question of the input method first and, as the sibling commenter said, grepping second. Fuzzy search could handle "-" as: "–", "—", "―", "‒" and other dashes. I input slightly more special characters on my phone as it is easier. On my computer I remember how I can easily write ° (on Linux and ChromeOS - Shift-AltGr-0) so I use it more. I don't know how to easily input a bullet point and I am too lazy to open a character table app every time it is needed.
In a way it is the same with programming languages. I think there are now programming languages supporting unicode operators for example, like =, ≠, ∧, ∨, ∩, ∪. But they suffer from the same problem. How to input them, how to search for them?
To be able to search and grep?
I welcome you to the world of printers, specially industrial label printers when using their (fast) bitmap fonts instead of e.g. true type ones.