I hate, hate software documentation as a concept. It gets out of date and is hard to use. It's a last resort, only for specialized cases.
I prefer two types of documentation:
1. Executable documentation - tests, asserts, even things like Jupyter notebooks that can be tested and executed
2. Timestamped documentation - documentation that has a clear date on it of when it was valid. So the reader has an expectation "This was true at X date, but may not be true now". This includes detailed pull requests and git commit messages.
I think there's a solution for avoiding zombie documentation.
The documentation lives in the same git repository as the code that it documents.
Inaccuracies or out-of-date documentation is treated as a severe bug. Issues are filed, the documentation gets fixed.
This is crucial, because if zombie documentation is allowed to persist it causes people to lose trust in the documentation, which means they won't refer to it and they won't contribute to it.
Once the documentation is in a trustworthy state, keeping it that way gets a LOT easier. It becomes part of the code review process - a PR won't be accepted unless it updates the relevant documentation that accompanies the code change.
I've been using this policy for my own projects for quite a few years now, and the result is that I'm proud of the state of my documentation for almost the first time in my entire career.
I tried to get this going on an open source project I worked with, and found basically no support from the other contributors.
I think it basically came down to incentives. Those who were developing the code every day had all the relevant details in their heads. So ensuring that docs matched the code was basically a distraction from their primary goals.
It makes me a little angry because it screws over users of the software. The docs don't even mention that they might be out of date.
The main reason is that if it's in Git you can enforce this policy through your PR process.
There are a bunch of other reasons to keep documentation in git. The most important is that it gives you versioned documentation that matches your releases - if someone is running v2.3 of your software they can browse the documentation for that exact version using that tag on GitHub.
If your documentation lives in some other system it can only ever attempt to be accurate for whatever the most recent release is. This is bad for people running older versions, and also makes it harder to develop documentation for unreleased code in a way that isn't potentially confusing.
The common logic seems to be that if your documentation is close to the code you're more likely to see that documentation exists and needs to be updated? I think it just boils down to "you must make time to update documentation" and make a process for it, rather than acting like "put your code in git" is some kind of panacea.
I like the idea as an engineer - but this makes it so non-technical people are less likely to write docs. If your marketing manager needs git access and to edit a markdown file on GitHub - the chance they will document in even ideal conditions goes way down
I think this is a valid concern, and it seems that most code oasis implementations are starting to add web interfaces for editing files in Git. And a WYSIWYG Markdown editor that can post MRs for people who don't have the ability to directly commit and you're pretty much at Confluence territory (when you have a pipeline to build a doc site with search, that is)
For simpler code, can't the solution be a human-enhanced LLM output?
1) LLM scans code updates and approximates output
2) Developer reviews and updates if needed
3) Feed back tagged code and updated output to LLM
4) ?
5) Profit
Comments
I hate, hate software documentation as a concept. It gets out of date and is hard to use. It's a last resort, only for specialized cases.
I prefer two types of documentation:
1. Executable documentation - tests, asserts, even things like Jupyter notebooks that can be tested and executed
2. Timestamped documentation - documentation that has a clear date on it of when it was valid. So the reader has an expectation "This was true at X date, but may not be true now". This includes detailed pull requests and git commit messages.
https://softwaredoug.com/blog/2023/10/13/fight-undead-docume...
I think there's a solution for avoiding zombie documentation.
The documentation lives in the same git repository as the code that it documents.
Inaccuracies or out-of-date documentation is treated as a severe bug. Issues are filed, the documentation gets fixed.
This is crucial, because if zombie documentation is allowed to persist it causes people to lose trust in the documentation, which means they won't refer to it and they won't contribute to it.
Once the documentation is in a trustworthy state, keeping it that way gets a LOT easier. It becomes part of the code review process - a PR won't be accepted unless it updates the relevant documentation that accompanies the code change.
I've been using this policy for my own projects for quite a few years now, and the result is that I'm proud of the state of my documentation for almost the first time in my entire career.
I tried to get this going on an open source project I worked with, and found basically no support from the other contributors.
I think it basically came down to incentives. Those who were developing the code every day had all the relevant details in their heads. So ensuring that docs matched the code was basically a distraction from their primary goals.
It makes me a little angry because it screws over users of the software. The docs don't even mention that they might be out of date.
Why does this work with Git but not with non-Git? Being in Git doesn't seem to relevant to raising documentation bugs as bugs.
The main reason is that if it's in Git you can enforce this policy through your PR process.
There are a bunch of other reasons to keep documentation in git. The most important is that it gives you versioned documentation that matches your releases - if someone is running v2.3 of your software they can browse the documentation for that exact version using that tag on GitHub.
If your documentation lives in some other system it can only ever attempt to be accurate for whatever the most recent release is. This is bad for people running older versions, and also makes it harder to develop documentation for unreleased code in a way that isn't potentially confusing.
Understood; I'm particularly talking about using bugs rather than making sure things are ready in PRs.
The common logic seems to be that if your documentation is close to the code you're more likely to see that documentation exists and needs to be updated? I think it just boils down to "you must make time to update documentation" and make a process for it, rather than acting like "put your code in git" is some kind of panacea.
I like the idea as an engineer - but this makes it so non-technical people are less likely to write docs. If your marketing manager needs git access and to edit a markdown file on GitHub - the chance they will document in even ideal conditions goes way down
I think this is a valid concern, and it seems that most code oasis implementations are starting to add web interfaces for editing files in Git. And a WYSIWYG Markdown editor that can post MRs for people who don't have the ability to directly commit and you're pretty much at Confluence territory (when you have a pipeline to build a doc site with search, that is)
For simpler code, can't the solution be a human-enhanced LLM output?
1) LLM scans code updates and approximates output 2) Developer reviews and updates if needed 3) Feed back tagged code and updated output to LLM 4) ? 5) Profit
Seems like a great problem for LLMs to help solve as well :)
We should...talk.
My response is almost "too bad". :) Keeping things documented is part of the work, and if it isn't done, then the work is not completed.
Your suggestions of (1) and (2) are great suggestions to have as components to the documentation system. Notebooks are really fantastic for this.
Absolutely. Part of reviewing is checking that the eg. README is up to date and works as expected.