We work with a lot of multilingual text, and for "what to know about encodings and character sets" we have a very simple answer to that - a guideline called "use UTF8 or die".
It's not suitable for absolutely everyone (e.g. if you have a lot of Asian text then you may want a different encoding), but for our use case every single deviation causes lot of headache, risks and unneccessary work in fixing garbage data.
In simplistic terms what we mean by this guideline:
* in your app, 100% all human text should be stored UTF8 only, no exceptions. If you need to deal with data in other encodings - other databases, old documents, whatever - make a wrapper/interface that takes some resource ID and returns the data with everything properly converted to UTF8; and has no way (or at least no convenient way) to access the original bytes in another encoding.
* in all persistence layers, store text only as UTF8. If at all possible, don't even provide options to export files in other encodings. If legacy/foreign data stores need another encoding, then in your code never have an API that requires a programmer to pass data in that encoding - the functions "store_document_to_the_mainframe_in_EBCDIC" and "send_cardholder_data_to_that_weird_CC_gateway" should take UTF8 strings only and handle the encodings themselves.
* in all [semi-]public API, treat text as UTF8-only and document that. If your API documentation mentions a text field, state the encoding so that there is no guessing or assuming by anyone.
* in all system configuration, set UTF8 as the default whenever possible. A database server? Make sure that any new databases/tables/text fields will have UTF8 set as the default, so unless someone takes explicit action then user-local-language encodings won't accidentally appear.
* Whoever introduces a single byte of different encoding data is responsible for fixing the resulting mess. This is the key part. Did you write a data input function that passed on data in the user computer default encoding; tested it only on US-ASCII nonenglish symbols; and got a bunch of garbage data stored? You're responsible for finding the relevant entries and fixing them, not only your code. Used a third party library that crashes or loses data when passed non-english unicode symbols? Either fix the library (if it's OS) or rewrite code to use a different one.
Comments
We work with a lot of multilingual text, and for "what to know about encodings and character sets" we have a very simple answer to that - a guideline called "use UTF8 or die".
It's not suitable for absolutely everyone (e.g. if you have a lot of Asian text then you may want a different encoding), but for our use case every single deviation causes lot of headache, risks and unneccessary work in fixing garbage data.
In simplistic terms what we mean by this guideline:
* in your app, 100% all human text should be stored UTF8 only, no exceptions. If you need to deal with data in other encodings - other databases, old documents, whatever - make a wrapper/interface that takes some resource ID and returns the data with everything properly converted to UTF8; and has no way (or at least no convenient way) to access the original bytes in another encoding.
* in all persistence layers, store text only as UTF8. If at all possible, don't even provide options to export files in other encodings. If legacy/foreign data stores need another encoding, then in your code never have an API that requires a programmer to pass data in that encoding - the functions "store_document_to_the_mainframe_in_EBCDIC" and "send_cardholder_data_to_that_weird_CC_gateway" should take UTF8 strings only and handle the encodings themselves.
* in all [semi-]public API, treat text as UTF8-only and document that. If your API documentation mentions a text field, state the encoding so that there is no guessing or assuming by anyone.
* in all system configuration, set UTF8 as the default whenever possible. A database server? Make sure that any new databases/tables/text fields will have UTF8 set as the default, so unless someone takes explicit action then user-local-language encodings won't accidentally appear.
* Whoever introduces a single byte of different encoding data is responsible for fixing the resulting mess. This is the key part. Did you write a data input function that passed on data in the user computer default encoding; tested it only on US-ASCII nonenglish symbols; and got a bunch of garbage data stored? You're responsible for finding the relevant entries and fixing them, not only your code. Used a third party library that crashes or loses data when passed non-english unicode symbols? Either fix the library (if it's OS) or rewrite code to use a different one.