I expect this to be automated too, and in the end boil down to paying for more tokens to fix the program.
Code too messy to be editable by an LLM is already too horrible for humans to touch. Fixing vibecoded software as a service will boil down to reverse-engineering requirements from the messed up program, and prompting a better model to design it properly and rewrite.
Eventually models will be trained to do this themselves, so it won't be a service you ask a dev for, it will be an extra charge on your AI subscription.
1. In "verifiable" domains ML is not limited by training data any more. Models can help generate the training data and/or learn the objective through reinforcement learning.
2. Yes, because the most capable model is the only one that can charge a premium. The rest is a commodity.
They can, but I don't think they will be of sufficient quality. It is a fundamental thing. You can't generate new information from existing information. It has to come from the real world.
Generating training data using existing models will only help the model to capture the exiting patterns more thourly.
Ultimately the new data, the slight incrimental improvements to the body of code for training) will actually come from humans even though it was written by ais.
ais write 1000 bits of garbage code for 1000 different users one-offs, the users judge that most of those didn't work out but 2 did. Only those good results that humans approved becomes part of the growing body of reference code. The others either get thrown away, or even if they still exist, they are somehow known to be low quality, or at least, not known to be high value.
So an ai wrote some new code that worked and improved the total corpus that other/future ais reference, but it's not just ai output feeding ai input, it's ai output filtered through humans who nixed most of it.
(Maybe exists on github but not used by anyone. It isn't known to be bad, it's just ranked lower, but if nothing else fits as well, it's there to try. That way obscure code that's good eventually becomes known to be good.)
Yes, but how many working code bases do we have. I don't think we have enough number of such high quality code bases to act as training data.
Also, the shitty re-write should also match shitty real-world patterns. Which is quite limit less....So we back to square one. Lack of input from real world.
Comments
I expect this to be automated too, and in the end boil down to paying for more tokens to fix the program.
Code too messy to be editable by an LLM is already too horrible for humans to touch. Fixing vibecoded software as a service will boil down to reverse-engineering requirements from the messed up program, and prompting a better model to design it properly and rewrite.
Eventually models will be trained to do this themselves, so it won't be a service you ask a dev for, it will be an extra charge on your AI subscription.
That is what I wonder. Two things.
1. Is there enough training data in that format, to do it?
2. Is there enough incentive for the LLM companies to do it? I mean they got to charge the client for the tokens upto that point..so..
1. In "verifiable" domains ML is not limited by training data any more. Models can help generate the training data and/or learn the objective through reinforcement learning.
2. Yes, because the most capable model is the only one that can charge a premium. The rest is a commodity.
They can, but I don't think they will be of sufficient quality. It is a fundamental thing. You can't generate new information from existing information. It has to come from the real world.
Generating training data using existing models will only help the model to capture the exiting patterns more thourly.
Ultimately the new data, the slight incrimental improvements to the body of code for training) will actually come from humans even though it was written by ais.
ais write 1000 bits of garbage code for 1000 different users one-offs, the users judge that most of those didn't work out but 2 did. Only those good results that humans approved becomes part of the growing body of reference code. The others either get thrown away, or even if they still exist, they are somehow known to be low quality, or at least, not known to be high value.
So an ai wrote some new code that worked and improved the total corpus that other/future ais reference, but it's not just ai output feeding ai input, it's ai output filtered through humans who nixed most of it.
(Maybe exists on github but not used by anyone. It isn't known to be bad, it's just ranked lower, but if nothing else fits as well, it's there to try. That way obscure code that's good eventually becomes known to be good.)
You can take a working codebase, tell a shitty LLM to rewrite it badly, then swap it around to look like LLM-to-working code rewrite.
Yes, but how many working code bases do we have. I don't think we have enough number of such high quality code bases to act as training data.
Also, the shitty re-write should also match shitty real-world patterns. Which is quite limit less....So we back to square one. Lack of input from real world.