Disagree gently here. Vibe refactoring is so much harder than vibe coding. I understand the temptation of having LLMs refactor code because that's ostensibly what they're great at – "transforming" the code. But LLM's are opinionated enough where they love to arbitrarily drop features or constraints for no particular reason, and they won’t tell you what they dropped along the way.
Proper refactoring with an llm requires full testing coverage and, by the time you do all that, is the refactoring really necessary? I prefer 100% stability. If you're refactoring the because it's poorly structured and unreadable, that’s okay.. LLMs can help understand it.
In my use of llms, i find it’s actually much easier to rebuild something from scratch rather than refactoring flawed code. It’s much less likely to inherit strange assumptions and code smells that way.
With all that said, the one prompt I do use when refactoring is to tell the llm to do a lossless refactor and then follow up with a "was this really lossless"? It's not foolproof. LLM's love to lie.
Proper refactoring with an llm requires full testing coverage
Proper refactoring WITHOUT an LLM requires full test coverage! But most definitely WITH an LLM.
In cases where there is no test coverage, the first thing I do is have the LLM write one test at a time. The problem there is that if you truly wanted valid tests, you'd have to actually break the code, watch the test fail to prove it was valid (basically, the inverse of TDD) and then re-fix the code and start on the next test, but in practice, it is difficult to get an LLM to stick to this loop. I wish someone would train or refine some coding LLM to use either TDD or this form of "inverse TDD" where you're applying tests after the fact and also want to check their validity. (Or tell me how to do it.) Because mere prompting doesn't seem to stick- it always regresses to the mean eventually.
(I'm currently seeking work, btw, and would probably be happy to help refactor old code, advise people on code, etc. Sorry for the self-promo.)
Comments
Disagree gently here. Vibe refactoring is so much harder than vibe coding. I understand the temptation of having LLMs refactor code because that's ostensibly what they're great at – "transforming" the code. But LLM's are opinionated enough where they love to arbitrarily drop features or constraints for no particular reason, and they won’t tell you what they dropped along the way.
Proper refactoring with an llm requires full testing coverage and, by the time you do all that, is the refactoring really necessary? I prefer 100% stability. If you're refactoring the because it's poorly structured and unreadable, that’s okay.. LLMs can help understand it.
In my use of llms, i find it’s actually much easier to rebuild something from scratch rather than refactoring flawed code. It’s much less likely to inherit strange assumptions and code smells that way.
With all that said, the one prompt I do use when refactoring is to tell the llm to do a lossless refactor and then follow up with a "was this really lossless"? It's not foolproof. LLM's love to lie.
Proper refactoring WITHOUT an LLM requires full test coverage! But most definitely WITH an LLM.
In cases where there is no test coverage, the first thing I do is have the LLM write one test at a time. The problem there is that if you truly wanted valid tests, you'd have to actually break the code, watch the test fail to prove it was valid (basically, the inverse of TDD) and then re-fix the code and start on the next test, but in practice, it is difficult to get an LLM to stick to this loop. I wish someone would train or refine some coding LLM to use either TDD or this form of "inverse TDD" where you're applying tests after the fact and also want to check their validity. (Or tell me how to do it.) Because mere prompting doesn't seem to stick- it always regresses to the mean eventually.
(I'm currently seeking work, btw, and would probably be happy to help refactor old code, advise people on code, etc. Sorry for the self-promo.)