The solution to these problems is for everyone to be completely ignorant of any character encoding and just deal with octets. If the characters represent UTF-8 text, then only when text needs to be presented or interpreted in some way, UTF-8 decoding happens. Any automatic encoding or decoding of UTF-8 (such as what Python3 does) is stupid.
EDIT: A common example of implicit and wrong handling of character encoding is when a file gets created with invalid characters, and your Linux file manager is unable to delete it. This can happen because the file manager assumes the file names it gets from the OS are text, and which it decodes incompletely. When it wants to delete the file it encodes the text back, but the result is different than the original file name bytes. The error happens because the file manager tries to decode the as text too early - it should keep the original octets as a reference to the file, but only decode them when it needs to display a file name.
Comments
The solution to these problems is for everyone to be completely ignorant of any character encoding and just deal with octets. If the characters represent UTF-8 text, then only when text needs to be presented or interpreted in some way, UTF-8 decoding happens. Any automatic encoding or decoding of UTF-8 (such as what Python3 does) is stupid.
EDIT: A common example of implicit and wrong handling of character encoding is when a file gets created with invalid characters, and your Linux file manager is unable to delete it. This can happen because the file manager assumes the file names it gets from the OS are text, and which it decodes incompletely. When it wants to delete the file it encodes the text back, but the result is different than the original file name bytes. The error happens because the file manager tries to decode the as text too early - it should keep the original octets as a reference to the file, but only decode them when it needs to display a file name.