I'm biased, being a software technical writer and not a dev, but at least from my side of the fence I think LLMs are no better at generating docs than they are at generating code, and might actually be way worse :)
My devs keep throwing Claude-generated writeups at me that look okay at first glance but fall apart when you actually try to follow the instructions they lay out (which makes them useless for anyone who's not already familiar with the product or feature they're describing), and/or have major structural and logical gaps, and/or gloss over messy details in a way that makes the product or feature sound better but doesn't reflect the reality of what we actually built.
Which turns into an ironic ouroboros where I then have to punch my own queries into an LLM and ask it to read our codebase, compare those details against the original writeup, try to decipher what the hell it was trying to say, etc. And even that turns into an extended back-and-forth where my LLM is like "The original writeup is inaccurate; it should say X" and I'm like "Are you sure? That contradicts what I know about related concept Y", and then the LLM is like "Good point, I dug into it more and the real answer is Z." (And even then, god only knows if Z is correct. I still have to get real developer eyeballs on anything before it goes live.)
Documentation was difficult to keep in sync with code because the tools didn't operate at a natural language level. Things like refactoring names, or terminology, or concept changes (foo is now fooGroup that contains bar) were tedious.
The granularity at which documentation parity is kept makes a big difference.
For example, brekaign things down into explicit assertion blocks in the code, and a link to the documentaiton that covers that assertion. So block-to-block tracking works well. Tracking completely separate documentation to independent codebase... idk I haven't tried it but my guess it that the context windows are too small for that.
But minor things like making sure there are unique error codes, that the codes somewhat make sense, etc. are all helpful. If nothing else, the model being stupid and acting as a rubber duck for pair-programming is useful for knowing what to document given a different perspective.
Comments
I'm biased, being a software technical writer and not a dev, but at least from my side of the fence I think LLMs are no better at generating docs than they are at generating code, and might actually be way worse :)
My devs keep throwing Claude-generated writeups at me that look okay at first glance but fall apart when you actually try to follow the instructions they lay out (which makes them useless for anyone who's not already familiar with the product or feature they're describing), and/or have major structural and logical gaps, and/or gloss over messy details in a way that makes the product or feature sound better but doesn't reflect the reality of what we actually built.
Which turns into an ironic ouroboros where I then have to punch my own queries into an LLM and ask it to read our codebase, compare those details against the original writeup, try to decipher what the hell it was trying to say, etc. And even that turns into an extended back-and-forth where my LLM is like "The original writeup is inaccurate; it should say X" and I'm like "Are you sure? That contradicts what I know about related concept Y", and then the LLM is like "Good point, I dug into it more and the real answer is Z." (And even then, god only knows if Z is correct. I still have to get real developer eyeballs on anything before it goes live.)
Documentation was difficult to keep in sync with code because the tools didn't operate at a natural language level. Things like refactoring names, or terminology, or concept changes (foo is now fooGroup that contains bar) were tedious.
The granularity at which documentation parity is kept makes a big difference.
For example, brekaign things down into explicit assertion blocks in the code, and a link to the documentaiton that covers that assertion. So block-to-block tracking works well. Tracking completely separate documentation to independent codebase... idk I haven't tried it but my guess it that the context windows are too small for that.
But minor things like making sure there are unique error codes, that the codes somewhat make sense, etc. are all helpful. If nothing else, the model being stupid and acting as a rubber duck for pair-programming is useful for knowing what to document given a different perspective.