Why do so many tools have JSON config files‽ Commenting why options have been set the way they have is just such a basic thing to want to do… Why do tech people have such an aversion to writing things down⁇
That's the whole post. First I don't know why this is posted on HN. Second I don't see how "JSON config" and "writing things down" are the two opposite options.
That's the whole post. First I don't know why this is posted on HN. Second I don't see how "JSON config" and "writing things down" are the two opposite options.
The "writing things down bit" is in the context of commenting. I mean, it's literally in the same paragraph.
Which is better than the all too common middle ground: writing something once and then not maintaining it. I've been burned many times reading some documentation and thinking I understood until I discovered the code has changed since and the documentation is now wrong.
I'm in favor of documentation. I write it. I get people pointing at what I've written as examples of what everybody should do. However it is a lot of work. I'm constantly looking things over to be sure it still makes sense. I often wonder if it is really worth it. I hope you follow my example and write documentation, but it better feel like a lot of work.
I don't agree with "readable," though. For a simple set of key-values, yes. Once you get into complicated structures, other file formats express the semantics much better.
I think there are many facets to readability. For example, YAML is frequently held up as much easier for humans to read than JSON, but even after many years of reading and writing YAML in various domains, no other format causes me anywhere near as much trouble.
With JSON, I can format the string as I feel makes sense for that specific data structure, while YAML forces me to into specific indentation patterns, and it still causes me to question every time whether the dashes in arrays should be indented or not. And since people frequently template YAML, the strict indentation has caused a bunch of issues for no good reason, including production outages (sure, shouldn't happen with good practices, but there are so many places without good practices!).
Because JSON syntax generally has only one way to represent each data type and almost every data type uses explicit start and end signifiers (except floats and bools, both of which are short and clearly stand out), it's easy to know what the context of every character is both while reading linearly, and when jumping to specific points. Meanwhile YAML has multiple ways to express almost anything, and frequently the only way to tell the current context is to read ahead before jumping back. This is especially terrible for strings due to optional delineation, because almost every bare text could be a keyword (as demonstrated by the Norway problem).
I know this is not most people's experience, but that's because readability is subjective.
If you have a complex configuration you are doing it wrong... Spend some time thinking about you really want and turn it into a simple key-value setup. JSON should feel like it is overpowered and bloated for your needs. (I'd still use JSON because you can find a parser and editor that can handle it, but it should feel overpowered for your needs)
1: Because JSON is a very easy serialization format to work with. I suspect these tools all have configuration classes / objects that are deserialized straight from the config file.
2: I suspect a lot of these tools are written in Javascript, and in Javascript JSON is very easy to work with.
In C#, I find "binary serialization" much easier to program with than JSON.
Then again, the resulting blobs require specialized tooling to read, and they're very hard to work with in other programming languages.
---
But, jokes aside:
I find CSV is great for "rows" because it doesn't repeat field names for every object.
I really like XML when each tag is an object, and fields are attributes. Most people don't understand this and end up making a hug mess; and some serializers do this by default too. IMO, this is why JSON became much more popular.
For Caddy we chose JSON because it's fairly universal, maps nearly 1:1 with Go structs (useful for initializing an extensible server), and nearly everything else compiles to JSON one way or another, so you can choose your own config format, really: https://caddyserver.com/docs/config-adapters
I really like the EDN[1][2] (extensible data notation) format that is used mostly by Clojure for config and data transfer/exchange. It is so much more expressive than JSON and supports a well thought-out set of elements for the most common data structures.
In my opinion, JSON is not the best format and has some problems. Lack of comments is one of the reasons, as they mention in there. Another is the lack of trailing commas (optional trailing commas would be useful for manually written files). However, these are problems with the syntax, and there are also problems with the data, such as a lack of a proper integer type, lack of Infinity and NaN, lack of support for character sets other than Unicode (and ASCII), lack of proper octet string type, etc.
For our internal tooling we use something similar to a bash profile config file with name/value pairs separated by linebreaks. So, our configs look something like:
because JSON is in the following sense "universal":
every format/structure that has numbers, strings, booleans, null/none, finite lists of items, and string-indexed records of items already contains JSON
and that's pretty much the barebones you need for a configuration language
JSON still a very simple and useful format and being natively supported on the web and by javascript basically guarantees its universality. Native comments would be nice though. But to be fair even Douglas Crockford suggested using comments in JSON was fine as long as you stripped them out before parsing.
<whispers>but Lua tables are even better.</whispers>
That is not quite true, because you have to define "numbers" and "strings" more specifically; JSON uses Unicode strings, and how numbers work depends on the implementation (but are generally finite 64-bit IEEE floating point; JSON does not have Infinity and NaN).
ASN.1 is almost a superset, but lacks a key/value list type; I had made some nonstadard extensions called ASN.1X and one of my new types is a key/value list type, so that makes the data types of ASN.1X a superset of JSON (although the format is different, it makes that all JSON data can be represented using DER if the nonstandard key/value list type of ASN.1X is used).
I don't like JSON that much because of its many problems (some are problems with syntax, others are problems with the data), so I use ASN.1X instead (with the DER format), for my own stuff (but I also deal with JSON because it is common enough).
Do you have the details of ASN.1X posted somewhere? Would be interesting to see what you've done with it, particularly after the ASN.1 standards folks came up with a mechanism that requires you rebuild your brain inside out in order to understand it.
I do not use the ASN.1 schema format, and have not written a specification for how the new ASN.1X features would be used in the ASN.1 schema format, although someone who is interested to do so might be able to help to write such a thing. (An alternative might be to make up a different schema format for use with ASN.1X.)
ASN.1X is mostly just a list of additional types, although there is also another serialization format called SDER which is between BER and DER (any valid DER is also valid SDER and any valid SDER is also valid BER), and is intended for when you do not quite need a canonical form but still want the simplicity of DER; one of the things that it allows is overlong length encodings (which is useful when the encoder wants to encode items to a file individually but then go back to encode the length afterward).
The additional types include:
- UTF-16 string: Same as BMP string but non-BMP characters are also allowed (as surrogate pairs). The type number is the same as BMP string.
- OBJECT IDENTIFIER RELATIVE TO: Either a absolute or a relative object identifier; what it is relative to might be either fixed or given elsewhere in the data, depending on the schema. This is equivalent to a type given in the appendix of the official specification of ASN.1, except that it is now a standardized type, and the canonical form (even in SDER, so that a reader does not need to check for both cases) is required to use the relative format if possible.
- BCD string (64): A string of 4-bit characters, with the high nybble first in each byte. The characters come from the character set 0 1 2 3 4 5 6 7 8 9 * # + - . space and it should be padded with a space on the end if necessary.
- PC string (65): A string of characters in the PC character set (or a related character set in some cases). Note that control characters can also be used as graphic characters.
- TRON string (66): A string of TRON characters, encoded as TRON-8.
- Key/value list (67): A set of keys (without duplicates) with associated values. The keys and values can be any type allowed by the schema. In canonical form, they must be sorted by keys in the same order that a SET is sorted (but the values are kept with the corresponding keys). (This is the only one of these nonstandard types which is used for representing JSON data; all of the other JSON types correspond to standard ASN.1 types.)
- Out of band (72): The format and usage of this type depends on the communication channel being used, and is intended for including things inside of the ASN.1X data which is separate from the ASN.1X data, such as file descriptors. This type is not intended for storage in files, and some programs that relay messages may need special handling of this type if it is used (for this reason, it should not be IMPLICIT).
- Reference (74): A reference to another node within the same file (schemas may restrict which nodes can be referenced). The encoding is like a relative OID but the first number is how many times to go to the parent node (0 means the reference itself), and then the rest of the numbers are the zero-based index into the node referenced by the previous number.
- Identified data (75): Contains a set, followed by the payload (of any type), followed by an optional key/value list where the keys are OIDs. The set is used to identify the format, and it can contain OIDs, object descriptors (only expected to be used in error messages and stuff like that), and sequences who first element is a OID; and should not have duplicates. This type may be used as the top-level type in a file in order to identify the file format, but can also be used inside of the file in case the existing types are considered to be insufficient for this purpose.
- Rational number (76): Contains two integers, being the numerator and denominator. The denominator must be greater than zero. If it is canonical form, then it must be lowest terms.
- Translation list (77): A key/value list where the keys specify the languages (null means the default), and is expected to be used where it could be replaced by the appropriate value according to the l10n.
- Scientific number (78): Same as a real number except that the number of digits (or bits) is considered to be significant. Decimal numbers must be NR3, and if it is canonical form then there must be exactly one digit before the dot.
There are also additional situations where the standard ASN.1 schema format does not seem to specify (although as I mentioned above, there is currently no standardized schema format for these things), such as: regular expressions for octet strings (and bit strings), constraining types as though it is another type (e.g. limiting a UTF-8 string to a number of bytes instead of (or in addition to) code points), constraints about what control characters are allowed in a General string (I think it is rarely useful to allow all control characters), etc. (Also, I disagree with the standard recommendation to use automatic tags; I think that manual tags are better and make both the schema and the data files clearer and easier to understand.)
A lot of that is already in ASN.1, for example for UTF-16 you've already got UTF-8 and given that even Microsoft have abandoned BMP strings I doubt any attempt to reintroduce it will get much traction, relative OIDs already exist, BCD strings are just constrained PrintableStrings and in any case UTF-8 won for all of the string types, Reference sounds like an EXTERNAL, OOB sounds like an ANY DEFINED BY, etc.
for UTF-16 you've already got UTF-8 and given that even Microsoft have abandoned BMP strings I doubt any attempt to reintroduce it will get much traction
This is not reintroducing anything; the BMP type is already there, although its meaning is expanded (the existing BMP type is effectively a constrained subtype of UTF-16). Although most applications probably will not use UTF-16, it might sometimes be useful in some applications where it is more useful to store UTF-16 instead of converting to/from UTF-8.
relative OIDs already exist
Yes, although I have given a standardized name and semantics to something that is allegedly already a common use (and is one that I often use in my own projects), which the official specification from ITU admits. It uses the existing OID and relative OID types (and the same type numbers as them), and is like a CHOICE between them (implementations may treat it as such).
BCD strings are just constrained PrintableStrings
The abstract meaning matches that of constrained Visible (not Printable) strings, but the encoding is more compact.
UTF-8 won for all of the string types
Although it is common (and some other formats don't support other string types), I disagree, and I think that one character set cannot be useful for all purposes, and furthermore that Unicode is not that good and has many problems.
Reference sounds like an EXTERNAL, OOB sounds like an ANY DEFINED BY
I wish everyone would embrace Amazon's Ion format. Of the data serialization formats it seems the most reasonable with the exception that it can encode S-expressions (so like having data serialization within your data serialization), so it is a bit excessive.
At a previous employer we tried using Ion, but when they stopped supporting our QLDB it was obvious that nothing else really used it and it was a dead end.
Let me guess... if they're a form of Lisp, did someone add jndi support, then made it possible to print to the console while the string gets read and finally made it executable because who wouldn't want a LISP for-loop in an Oracle connection string? On to the next RCE...
Nah, they are pretty constrained data description. AFAIK, you can't even substitute environment variables.
They are also apparently very hard to use, to the point that many people can't and several web sites exist that will take your database server, username, and password and assemble a connection string for you. (I can see no issues with that! None!)
SQLite is a tiny relational db that essentially runs right in the same folder as your application. No connection other than connecting from the app itself to the SQLite.db sitting next to it.
You're missing the point: Why are your config files so complicated that you need SQLite?
I've shipped a product that used SQLite, and we actively removed configuration from SQLite. It was a lot easier to diagnose issues when configuration was in text files, because non-programmers could kinda-sorta understand them without needing to learn how to use SQL.
It would be too easy to have every option in a config file documented in comments in the default config file. Think of the poor tutorial industry. It may even make chatbots less useful.
1. JSON parsers are available at your corner convenience store.
2. The format is too simple to have ambiguous behavior. No weird “yes” means true, 0 means false, odd rules about comments, blah blah blah. It’s hard to fuck up JSON (but obviously not impossible if you get creative).
3. It errors out early in the parsing if you mess it up.
4. Its data types are present in more or less any language.
5. Most configs are just key/value. JSON does this reasonably well.
6. It is easy to generate and validate JSON documents. For some use cases you don’t need a library (though you should use one).
7. There is only one way to do anything (sane).
8. Everyone is familiar with it.
9. It is dynamic. You do not need to pre-define your sections or keys ahead of time.
10. It can easily be auto formatted to look good with zero risk of changing semantics.
11. Data stores often natively support storing and querying JSON objects.
12. If you are old enough to remember the era when every tool invented its own, often very buggy, config format and parser you will also remember the moment you first saw a JSON config file that was parsed with a standard library parser and thought “finally, this is the modern way”, you will understand why JSON continues being popular. It was the first thing that unambiguously worked compared to what came before it.
This is like asking why people use their keys to open packages: it might not be the right tool for the job but it’s hard to mess up, is the closest thing to you that can get the job done, and everyone (with functioning hands/fingers) can do it with little issue.
I am also certain there is some small but non-zero percentage of people who do it simply because everyone else moralizes about not doing it. Spite is a powerful thing.
I don't know about others, but I use JSON because it's in the standard Go library, and most of the times, I rather use something weird than add a dependency.
JSON is obviously a poor choice. It's job is to interchange data, produced and parsed by computers.
ESR had the idea of writing configuration in English. It didn't gain traction at the time, but we have LLMs now. It might be a good idea to revisit the idea of accepting plain english. The LLM output could then be any format that's easy and unambiguous to parse.
Using an LLM to parse your application's config is a bit like using an F1 racecar to drive from your house to the bicycle in your attached garage. Getting config into the application should to be fast, light (lighter than the rest of the application), and deterministic. And if the LLM's output is easy and unambiguous to parse, it's easier to simply use that as the original config file.
I totally get that. The application shouldn't use an LLM to parse the config file, but it would be useful to write one. An LLM could supply a diff to make the changes you want without learning the config-language-du-jour.
I'm not sure why there's a problem before even trying.
Say for instance you have a program that keeps recipes. In your configuration file, there's settings such as metric/imperical units, allergies, diets, type of stove in your kitchen, and some theming (font, size, color...). You put them all in a file that you can parse, but aunt tillie messes up the formatting and the program breaks.
Instead of changing the config by hand, an LLM could supply the diff according to instructions in English. I really don't see why this would be "Terrible for prod". If the LLM screws up, you're simply back to square 1 and aunt tillie will call you just like she would before she had an LLM to fix her computer.
Same model. Same hardware... Depends. If you sacrifice speed then no. Ieee754 is pretty specifically specified, the issue is that it's not associative. If you get the associativity correct, then there's no issue. Associativity usually dies due to scheduling
TOML is a better format for configuration files IMO, if not for many reasons, primarily because TOML accepts comments.
However, one annoying thing for TOML was the lack of schema, and the reliance on JSON Schema for that. Which I decided to tackle years ago when I started the TOML Schema project. In the past few months I leveraged code agents to take to the finish line and got something compelling: tomlschema.org
Interesting. The website says that it's "validation-focused". I don't know how that plays out for TOML, but the lack of focus in json schema to define an interface in addition to validation can be very frustrating. Most of the time I would rather trade off some expressitivity in terms of validation in favor of having a defined typed interface into the validated data.
I'd argue that for code, yes, JSON is vastly closer to computer structures.
I think this is the real "why do so many tools have JSON config files": because it's just a literal notation, for basic data structures, that looks a lot like what many programming languages natively do, what their data structures natively are. The answer to the post is: it's mechanistic sympathy.
For humans, I do find TOML to be a lot easier to manage. Even if you have a really good editor that takes care of all the quoting/nesting/comma concerns for you, even if you have jsonc or json5 with comments, it's still not as easy/friendly as a big flat file with sections in it.
I find the toml structure impossible to follow as a human. It's like a complicated nested ini file. Asinine and terrible. I don't need a fancy editor (and I don't have one) to close my quotes and brackets. jq checks the syntax easily. On the other hand I have to crash my services to find out I nested shit wrong in toml because there's no structure
What are your thoughts on ini? My perspective is that >9/10 toml's are just ini files, straight up. The nesting seems rare, and rarely confusing to me.
I obviously disagree about the ease of json editing. jq tells me errors, sure. But formats where we don't need bespoke tooling, where notepad.exe work fine, are I think probably what config files should be more like.
My only experience with toml is complicated nested ones because I'm sure that the maintainers outgrew whatever was simple. Off the top my head it's just containerd config. Yaml probably a better choice than json and ini is fine if it gets the job done, but it's it really more than .env? My opinion is the env should do a lot of lifting anyway. I'm thinking there's always "tooling." My toml might syntax but be logically incorrect all the formats have that problem and linting doesn't stop at json. Like changing my containerd config I don't really know if it took or not
Comments
That's the whole post. First I don't know why this is posted on HN. Second I don't see how "JSON config" and "writing things down" are the two opposite options.
The "writing things down bit" is in the context of commenting. I mean, it's literally in the same paragraph.
No, we're just lazy.
Which is better than the all too common middle ground: writing something once and then not maintaining it. I've been burned many times reading some documentation and thinking I understood until I discovered the code has changed since and the documentation is now wrong.
I'm in favor of documentation. I write it. I get people pointing at what I've written as examples of what everybody should do. However it is a lot of work. I'm constantly looking things over to be sure it still makes sense. I often wonder if it is really worth it. I hope you follow my example and write documentation, but it better feel like a lot of work.
Occam's Razor at it's finest. I'm not trying to screw others over. I just want to not write sometimes.
It's readable, it's easy, it gets the job done.
I don't think that's necessarily lazy, it's just efficient
I don't agree with "readable," though. For a simple set of key-values, yes. Once you get into complicated structures, other file formats express the semantics much better.
I think there are many facets to readability. For example, YAML is frequently held up as much easier for humans to read than JSON, but even after many years of reading and writing YAML in various domains, no other format causes me anywhere near as much trouble.
With JSON, I can format the string as I feel makes sense for that specific data structure, while YAML forces me to into specific indentation patterns, and it still causes me to question every time whether the dashes in arrays should be indented or not. And since people frequently template YAML, the strict indentation has caused a bunch of issues for no good reason, including production outages (sure, shouldn't happen with good practices, but there are so many places without good practices!).
Because JSON syntax generally has only one way to represent each data type and almost every data type uses explicit start and end signifiers (except floats and bools, both of which are short and clearly stand out), it's easy to know what the context of every character is both while reading linearly, and when jumping to specific points. Meanwhile YAML has multiple ways to express almost anything, and frequently the only way to tell the current context is to read ahead before jumping back. This is especially terrible for strings due to optional delineation, because almost every bare text could be a keyword (as demonstrated by the Norway problem).
I know this is not most people's experience, but that's because readability is subjective.
If you have a complex configuration you are doing it wrong... Spend some time thinking about you really want and turn it into a simple key-value setup. JSON should feel like it is overpowered and bloated for your needs. (I'd still use JSON because you can find a parser and editor that can handle it, but it should feel overpowered for your needs)
That's what I am saying about my ruby codes as well. Still my boss wanted proper commit messages
Something has gone awry if the config files have a readability and difficulty analogous to that of the codebase.
1: Because JSON is a very easy serialization format to work with. I suspect these tools all have configuration classes / objects that are deserialized straight from the config file.
2: I suspect a lot of these tools are written in Javascript, and in Javascript JSON is very easy to work with.
JSON is probably the easiest format to work with regardless of language.
Pedantic response:
In C#, I find "binary serialization" much easier to program with than JSON.
Then again, the resulting blobs require specialized tooling to read, and they're very hard to work with in other programming languages.
---
But, jokes aside:
I find CSV is great for "rows" because it doesn't repeat field names for every object.
I really like XML when each tag is an object, and fields are attributes. Most people don't understand this and end up making a hug mess; and some serializers do this by default too. IMO, this is why JSON became much more popular.
Mostly because it is so common (web) it doesn't matter what language you use there is a good maintained tool to read and write JSON.
For Caddy we chose JSON because it's fairly universal, maps nearly 1:1 with Go structs (useful for initializing an extensible server), and nearly everything else compiles to JSON one way or another, so you can choose your own config format, really: https://caddyserver.com/docs/config-adapters
Hey, thanks for Caddy! It's awesome :).
I really like the EDN[1][2] (extensible data notation) format that is used mostly by Clojure for config and data transfer/exchange. It is so much more expressive than JSON and supports a well thought-out set of elements for the most common data structures.
[1]: https://github.com/edn-format/edn
[2]: https://en.wikipedia.org/wiki/Clojure#Extensible_Data_Notati...
In my opinion, JSON is not the best format and has some problems. Lack of comments is one of the reasons, as they mention in there. Another is the lack of trailing commas (optional trailing commas would be useful for manually written files). However, these are problems with the syntax, and there are also problems with the data, such as a lack of a proper integer type, lack of Infinity and NaN, lack of support for character sets other than Unicode (and ASCII), lack of proper octet string type, etc.
For our internal tooling we use something similar to a bash profile config file with name/value pairs separated by linebreaks. So, our configs look something like:
env=staging
db=0.0.0.0
#descriptive comment
etc=true
ini file format is pretty much this
[section_name]
key=value
key=value
because JSON is in the following sense "universal":
every format/structure that has numbers, strings, booleans, null/none, finite lists of items, and string-indexed records of items already contains JSON
and that's pretty much the barebones you need for a configuration language
(of course you can argue about the syntax)
Lack of comments is pretty big though for a human editable config.
JSON still a very simple and useful format and being natively supported on the web and by javascript basically guarantees its universality. Native comments would be nice though. But to be fair even Douglas Crockford suggested using comments in JSON was fine as long as you stripped them out before parsing.
<whispers>but Lua tables are even better.</whispers>
It’s just one person’s opinion, but I think two things are true enough, here…
1) JSON is pretty darn good for storing configuration. Everything speaks it, and a pretty printed one is very readable/tweakable in a pinch.
2) If you insist that someone manually edit a significant amount of it, you kinda fucked up.
Just my opinion, but it feels like two separate things.
I just put them in as values. In a GUI I sometimes render them as a editable textarea.
You can always use JSONC or JSON5.
That is not quite true, because you have to define "numbers" and "strings" more specifically; JSON uses Unicode strings, and how numbers work depends on the implementation (but are generally finite 64-bit IEEE floating point; JSON does not have Infinity and NaN).
ASN.1 is almost a superset, but lacks a key/value list type; I had made some nonstadard extensions called ASN.1X and one of my new types is a key/value list type, so that makes the data types of ASN.1X a superset of JSON (although the format is different, it makes that all JSON data can be represented using DER if the nonstandard key/value list type of ASN.1X is used).
I don't like JSON that much because of its many problems (some are problems with syntax, others are problems with the data), so I use ASN.1X instead (with the DER format), for my own stuff (but I also deal with JSON because it is common enough).
Do you have the details of ASN.1X posted somewhere? Would be interesting to see what you've done with it, particularly after the ASN.1 standards folks came up with a mechanism that requires you rebuild your brain inside out in order to understand it.
I do not use the ASN.1 schema format, and have not written a specification for how the new ASN.1X features would be used in the ASN.1 schema format, although someone who is interested to do so might be able to help to write such a thing. (An alternative might be to make up a different schema format for use with ASN.1X.)
ASN.1X is mostly just a list of additional types, although there is also another serialization format called SDER which is between BER and DER (any valid DER is also valid SDER and any valid SDER is also valid BER), and is intended for when you do not quite need a canonical form but still want the simplicity of DER; one of the things that it allows is overlong length encodings (which is useful when the encoder wants to encode items to a file individually but then go back to encode the length afterward).
The additional types include:
- UTF-16 string: Same as BMP string but non-BMP characters are also allowed (as surrogate pairs). The type number is the same as BMP string.
- OBJECT IDENTIFIER RELATIVE TO: Either a absolute or a relative object identifier; what it is relative to might be either fixed or given elsewhere in the data, depending on the schema. This is equivalent to a type given in the appendix of the official specification of ASN.1, except that it is now a standardized type, and the canonical form (even in SDER, so that a reader does not need to check for both cases) is required to use the relative format if possible.
- BCD string (64): A string of 4-bit characters, with the high nybble first in each byte. The characters come from the character set 0 1 2 3 4 5 6 7 8 9 * # + - . space and it should be padded with a space on the end if necessary.
- PC string (65): A string of characters in the PC character set (or a related character set in some cases). Note that control characters can also be used as graphic characters.
- TRON string (66): A string of TRON characters, encoded as TRON-8.
- Key/value list (67): A set of keys (without duplicates) with associated values. The keys and values can be any type allowed by the schema. In canonical form, they must be sorted by keys in the same order that a SET is sorted (but the values are kept with the corresponding keys). (This is the only one of these nonstandard types which is used for representing JSON data; all of the other JSON types correspond to standard ASN.1 types.)
- Out of band (72): The format and usage of this type depends on the communication channel being used, and is intended for including things inside of the ASN.1X data which is separate from the ASN.1X data, such as file descriptors. This type is not intended for storage in files, and some programs that relay messages may need special handling of this type if it is used (for this reason, it should not be IMPLICIT).
- Reference (74): A reference to another node within the same file (schemas may restrict which nodes can be referenced). The encoding is like a relative OID but the first number is how many times to go to the parent node (0 means the reference itself), and then the rest of the numbers are the zero-based index into the node referenced by the previous number.
- Identified data (75): Contains a set, followed by the payload (of any type), followed by an optional key/value list where the keys are OIDs. The set is used to identify the format, and it can contain OIDs, object descriptors (only expected to be used in error messages and stuff like that), and sequences who first element is a OID; and should not have duplicates. This type may be used as the top-level type in a file in order to identify the file format, but can also be used inside of the file in case the existing types are considered to be insufficient for this purpose.
- Rational number (76): Contains two integers, being the numerator and denominator. The denominator must be greater than zero. If it is canonical form, then it must be lowest terms.
- Translation list (77): A key/value list where the keys specify the languages (null means the default), and is expected to be used where it could be replaced by the appropriate value according to the l10n.
- Scientific number (78): Same as a real number except that the number of digits (or bits) is considered to be significant. Decimal numbers must be NR3, and if it is canonical form then there must be exactly one digit before the dot.
There are also additional situations where the standard ASN.1 schema format does not seem to specify (although as I mentioned above, there is currently no standardized schema format for these things), such as: regular expressions for octet strings (and bit strings), constraining types as though it is another type (e.g. limiting a UTF-8 string to a number of bytes instead of (or in addition to) code points), constraints about what control characters are allowed in a General string (I think it is rarely useful to allow all control characters), etc. (Also, I disagree with the standard recommendation to use automatic tags; I think that manual tags are better and make both the schema and the data files clearer and easier to understand.)
A lot of that is already in ASN.1, for example for UTF-16 you've already got UTF-8 and given that even Microsoft have abandoned BMP strings I doubt any attempt to reintroduce it will get much traction, relative OIDs already exist, BCD strings are just constrained PrintableStrings and in any case UTF-8 won for all of the string types, Reference sounds like an EXTERNAL, OOB sounds like an ANY DEFINED BY, etc.
This is not reintroducing anything; the BMP type is already there, although its meaning is expanded (the existing BMP type is effectively a constrained subtype of UTF-16). Although most applications probably will not use UTF-16, it might sometimes be useful in some applications where it is more useful to store UTF-16 instead of converting to/from UTF-8.
Yes, although I have given a standardized name and semantics to something that is allegedly already a common use (and is one that I often use in my own projects), which the official specification from ITU admits. It uses the existing OID and relative OID types (and the same type numbers as them), and is like a CHOICE between them (implementations may treat it as such).
The abstract meaning matches that of constrained Visible (not Printable) strings, but the encoding is more compact.
Although it is common (and some other formats don't support other string types), I disagree, and I think that one character set cannot be useful for all purposes, and furthermore that Unicode is not that good and has many problems.
I don't think so. It seem like different to me.
It's so simple and straightforward, and that's why Douglas Crockford claims he "discovered" it, rather than invented it.
I wish everyone would embrace Amazon's Ion format. Of the data serialization formats it seems the most reasonable with the exception that it can encode S-expressions (so like having data serialization within your data serialization), so it is a bit excessive.
https://en.wikipedia.org/wiki/Ion_(serialization_format)
At a previous employer we tried using Ion, but when they stopped supporting our QLDB it was obvious that nothing else really used it and it was a dead end.
Shame.
Because XML hasn't been cool for about two decades. And suggesting .ini would you laughed out of the room into retirement.
Somewhat off topic, but Oracle database connection strings seem to be a form of Lisp. It makes me wonder why they would choose that.
I looked them up, and they’re not lisp (or s-expressions)
Let me guess... if they're a form of Lisp, did someone add jndi support, then made it possible to print to the console while the string gets read and finally made it executable because who wouldn't want a LISP for-loop in an Oracle connection string? On to the next RCE...
Nah, they are pretty constrained data description. AFAIK, you can't even substitute environment variables.
They are also apparently very hard to use, to the point that many people can't and several web sites exist that will take your database server, username, and password and assemble a connection string for you. (I can see no issues with that! None!)
Because JSON is native to the lingua franca of the internet: Java-/Ecmascript.
It fits into the poor choices we made, <-- Parse error
JSON just works, everywhere, all the time. Sometimes I'll use SQLite if there is a particular need.
Can you show an example of how you use SQLite for _config_ files?
SQLite is a tiny relational db that essentially runs right in the same folder as your application. No connection other than connecting from the app itself to the SQLite.db sitting next to it.
You're missing the point: Why are your config files so complicated that you need SQLite?
I've shipped a product that used SQLite, and we actively removed configuration from SQLite. It was a lot easier to diagnose issues when configuration was in text files, because non-programmers could kinda-sorta understand them without needing to learn how to use SQL.
"particular need". i also said json works everywhere. what was unclear?
I think your emphasis might be in the wrong place...SQLite for config makes enough sense, but as a config _file_?
Not entirely. Because where would I configure the location of the SQLite location/connection then?
Given what I know about it (single file, no auth, and such by default) I sort of understand the ”file” part I think. But not the ”config” part.
It would be too easy to have every option in a config file documented in comments in the default config file. Think of the poor tutorial industry. It may even make chatbots less useful.
in python:
that's honestly why.1. JSON parsers are available at your corner convenience store.
2. The format is too simple to have ambiguous behavior. No weird “yes” means true, 0 means false, odd rules about comments, blah blah blah. It’s hard to fuck up JSON (but obviously not impossible if you get creative).
3. It errors out early in the parsing if you mess it up.
4. Its data types are present in more or less any language.
5. Most configs are just key/value. JSON does this reasonably well.
6. It is easy to generate and validate JSON documents. For some use cases you don’t need a library (though you should use one).
7. There is only one way to do anything (sane).
8. Everyone is familiar with it.
9. It is dynamic. You do not need to pre-define your sections or keys ahead of time.
10. It can easily be auto formatted to look good with zero risk of changing semantics.
11. Data stores often natively support storing and querying JSON objects.
12. If you are old enough to remember the era when every tool invented its own, often very buggy, config format and parser you will also remember the moment you first saw a JSON config file that was parsed with a standard library parser and thought “finally, this is the modern way”, you will understand why JSON continues being popular. It was the first thing that unambiguously worked compared to what came before it.
This is like asking why people use their keys to open packages: it might not be the right tool for the job but it’s hard to mess up, is the closest thing to you that can get the job done, and everyone (with functioning hands/fingers) can do it with little issue.
I am also certain there is some small but non-zero percentage of people who do it simply because everyone else moralizes about not doing it. Spite is a powerful thing.
I don't know about others, but I use JSON because it's in the standard Go library, and most of the times, I rather use something weird than add a dependency.
JSON is obviously a poor choice. It's job is to interchange data, produced and parsed by computers.
ESR had the idea of writing configuration in English. It didn't gain traction at the time, but we have LLMs now. It might be a good idea to revisit the idea of accepting plain english. The LLM output could then be any format that's easy and unambiguous to parse.
Absolutely not. We do not want a stochastic program without any actual understanding of the schema to be interpreting config data.
Using an LLM to parse your application's config is a bit like using an F1 racecar to drive from your house to the bicycle in your attached garage. Getting config into the application should to be fast, light (lighter than the rest of the application), and deterministic. And if the LLM's output is easy and unambiguous to parse, it's easier to simply use that as the original config file.
I totally get that. The application shouldn't use an LLM to parse the config file, but it would be useful to write one. An LLM could supply a diff to make the changes you want without learning the config-language-du-jour.
Thats the problem isnt it? LLMs arent deterministic. Terrible for prod
I'm not sure why there's a problem before even trying.
Say for instance you have a program that keeps recipes. In your configuration file, there's settings such as metric/imperical units, allergies, diets, type of stove in your kitchen, and some theming (font, size, color...). You put them all in a file that you can parse, but aunt tillie messes up the formatting and the program breaks.
Instead of changing the config by hand, an LLM could supply the diff according to instructions in English. I really don't see why this would be "Terrible for prod". If the LLM screws up, you're simply back to square 1 and aunt tillie will call you just like she would before she had an LLM to fix her computer.
Llms are absolutely deterministic if you want them to be.
Still a terrible idea for config
Don't you need the same model, on the same hardware, with the same prompt, and temperature set to 0 to make it deterministic?
Same model. Same hardware... Depends. If you sacrifice speed then no. Ieee754 is pretty specifically specified, the issue is that it's not associative. If you get the associativity correct, then there's no issue. Associativity usually dies due to scheduling
TOML is a better format for configuration files IMO, if not for many reasons, primarily because TOML accepts comments.
However, one annoying thing for TOML was the lack of schema, and the reliance on JSON Schema for that. Which I decided to tackle years ago when I started the TOML Schema project. In the past few months I leveraged code agents to take to the finish line and got something compelling: tomlschema.org
Interesting. The website says that it's "validation-focused". I don't know how that plays out for TOML, but the lack of focus in json schema to define an interface in addition to validation can be very frustrating. Most of the time I would rather trade off some expressitivity in terms of validation in favor of having a defined typed interface into the validated data.
Reading and writing json is so much easier than reading or writing toml imo
I'd argue that for code, yes, JSON is vastly closer to computer structures.
I think this is the real "why do so many tools have JSON config files": because it's just a literal notation, for basic data structures, that looks a lot like what many programming languages natively do, what their data structures natively are. The answer to the post is: it's mechanistic sympathy.
For humans, I do find TOML to be a lot easier to manage. Even if you have a really good editor that takes care of all the quoting/nesting/comma concerns for you, even if you have jsonc or json5 with comments, it's still not as easy/friendly as a big flat file with sections in it.
I find the toml structure impossible to follow as a human. It's like a complicated nested ini file. Asinine and terrible. I don't need a fancy editor (and I don't have one) to close my quotes and brackets. jq checks the syntax easily. On the other hand I have to crash my services to find out I nested shit wrong in toml because there's no structure
What are your thoughts on ini? My perspective is that >9/10 toml's are just ini files, straight up. The nesting seems rare, and rarely confusing to me.
I obviously disagree about the ease of json editing. jq tells me errors, sure. But formats where we don't need bespoke tooling, where notepad.exe work fine, are I think probably what config files should be more like.
My only experience with toml is complicated nested ones because I'm sure that the maintainers outgrew whatever was simple. Off the top my head it's just containerd config. Yaml probably a better choice than json and ini is fine if it gets the job done, but it's it really more than .env? My opinion is the env should do a lot of lifting anyway. I'm thinking there's always "tooling." My toml might syntax but be logically incorrect all the formats have that problem and linting doesn't stop at json. Like changing my containerd config I don't really know if it took or not
TOML is quite ridiculous. There are always an infinity of ways one can write the same values.
It should really define sections as something different from dot-separated identifier groups.
I find TOML is nowhere near as good as YAML. I'd even rather use jsonc.
Yeah, I just use JSON5, it solves all the issues with normal json