No, it's not "just an implementation detail". I don't want to get into the details here, but your assumption that having your strings in one encoding composed conceptually of code points in a different encoding is not necessarily the way to go. I was on a JCP expert committee that rejected that option for Java, but that doesn't mean it couldn't end up best for Arc. It's just not as easy a decision as you imply. There are alternative approaches with different pros and cons, and since these abstractions all leak, you need to make a cross-implementation decision so that you don't end up with different coding practices on different platforms. Plus all the other issues that Unicode presents that make parsing a challenge that might have implications for a language that is famous for parsing and rewriting its own source at runtime.... These could all be worked out through a vigorous discussion process among a diverse group large enough to make sure that the implications of each approach are understood, but Graham doesn't work that way.
Comments
What you talk about regarding unicode for example is just an implementation detail. Unicode strings fundamentally are a list of unicode code points.
UTF-8 is just one of the possible encodings of such list. It can be decoded and converted to UTF-32 before it goes to the reader/lexer.
No, it's not "just an implementation detail". I don't want to get into the details here, but your assumption that having your strings in one encoding composed conceptually of code points in a different encoding is not necessarily the way to go. I was on a JCP expert committee that rejected that option for Java, but that doesn't mean it couldn't end up best for Arc. It's just not as easy a decision as you imply. There are alternative approaches with different pros and cons, and since these abstractions all leak, you need to make a cross-implementation decision so that you don't end up with different coding practices on different platforms. Plus all the other issues that Unicode presents that make parsing a challenge that might have implications for a language that is famous for parsing and rewriting its own source at runtime.... These could all be worked out through a vigorous discussion process among a diverse group large enough to make sure that the implications of each approach are understood, but Graham doesn't work that way.