The spec rarely has enough detail to deterministically create a product, so current vibecoding is a lottery.
How is that different from how it worked without LLMs? The only difference is that we can now get a failing product faster and iterate.
If we keep starting fresh from the spec, but keep adding detail after detail, regenerating from scratch each time..
This sounds like the worst way to use AI. LLMs can work existing code, whether it was generated by an LLM or written by human. It can even work on code that has been edited by a human, there is no good reason to not be iterative when using an LLM to develop code, and plenty of good reasons to be iterative.
How is that different from how it worked without LLMs? The only difference is that we can now get a failing product faster and iterate.
The difference is that there is an engineer in the middle who can judge if the important information is provided or not as input.
1. for a LLM "the button must be blue" has the same level of importance as "the formula to calculate X is..."
2. failing faster and iterating is good thing if the parameters of failing are clear which is not always the case with vibecoding, especially when done by people with no prior experience in developing. plenty of POCs build with vibecoding have been presented with no aparent failure in their happy path but with disastrous results in edge cases or with disastrous Security etc.
3. where previously, familairity with the codebase and especially the "history of changes" gave you context about why some workarounds were put into place, these are things that are lost to a LLM. Vibecoding a change to an existing system risks removing those "special workarounds" that keep in mind much more than the current context of the specifications or prompt.
1. for a LLM "the button must be blue" has the same level of importance as "the formula to calculate X is..."
You can divide those into two prompts though, there is no point for the LLM to work on both features at the same time. This is why iterative is so useful (oh, the button should be blue, ... and later, the formula should be X).
2. failing faster and iterating is good thing if the parameters of failing are clear which is not always the case with vibecoding, especially when done by people with no prior experience in developing. plenty of POCs build with vibecoding have been presented with no aparent failure in their happy path but with disastrous results in edge cases or with disastrous Security etc.
This isn't about vibecoding. If you are vibecoding, then you aren't developing software, you are just wishing for good code from vague descriptions that you don't plan to iterate on.
3. where previously, familairity with the codebase and especially the "history of changes" gave you context about why some workarounds were put into place, these are things that are lost to a LLM. Vibecoding a change to an existing system risks removing those "special workarounds" that keep in mind much more than the current context of the specifications or prompt.
LLMs can read and write change logs just as well as humans can (LLMs need change logs to do updates, you can't just give it a changed dependency and expect the LLM to pick up on the change, it isn't a code generator). Actually, this is my current project, since a Dev AI pipeline needs to read and write change logs to be effective (when something changes, you can't just transmit the changed artifact, you need to transmit a summary of the change as well). And again, this is serious software engineering, not vibecoding. If you are vibecoding, I have no advice to give you.
LLMs can read and write change logs just as well as humans can (LLMs need change logs to do updates, you can't just give it a changed dependency and expect the LLM to pick up on the change, it isn't a code generator). Actually, this is my current project, since a Dev AI pipeline needs to read and write change logs to be effective (when something changes, you can't just transmit the changed artifact, you need to transmit a summary of the change as well). And again, this is serious software engineering, not vibecoding.
This is the important part of the post to which you replied and remains unaddressed:
The difference is that there is an engineer in the middle
who can judge if the important information is provided or
not as input.
The engineer decide what information to use as input to the update prompt. They don’t need to be in the middle of anything, it’s basically the level they are coding at.
The engineer decide what information to use as input to the update prompt. They don’t need to be in the middle of anything, it’s basically the level they are coding at.
LLMs do not possess the ability to "judge if the important information is provided or not as input" as it pertains to the question originally posed:
How is that different from how it worked without LLMs?
Working without LLMs involves people communicating, hence the existence of "an engineer in the middle", where middle is defined as between stakeholder requirement definition and asset creation.
So you engineer the prompt. I’m still confused what the problem is, I’ve already stated that I’m not talking about vibe coding where the LLM somehow magically figures out relevant information on their own.
So you engineer the prompt. I’m still confused what the problem is ...
The problem is stakeholders are people and they define what problems are needed to be solved. For those tasked to do so requires understanding of the given problems. Tooling (such as LLMs) does not possess this type of understanding as it is intrinsic to the stakeholders (people) whom have defined it. Tools can contribute to delivering a solution, sure, but have no capability to autonomously do so.
For example, consider commercial dish washing machines many restaurants use.
They sanitize faster and with greater cleanliness than manual dish washing once did. Still, there is no dish washing machine which understands why it must be used instead of not. Of course, restaurant stakeholders such as health inspectors and proprietors understand why they must be used.
As far as the commercial dish washer is concerned, it could just as easily be tasked with cleaning dining utensils as it could recycled car parts.
For me it just depends. If the response to my prompt shows the model misunderstood something, then I go back and retry the previous prompt again. Otherwise the "wrong ideas" that it comes up with persist in the context and seem to sabotage all future results. The most of this sort of coding I've done was in Google's AI studio, and I often do have a context that spans dozens of messages, but I always rewind if something goes off-track. Basically any time I'm about to make a difficult request, I clone the entire context/app to a new one so I can roll back [cleanly] whenever necessary.
If you fix something it sticks, the AI won't keep making the same mistake, it won't change the code that already exists if you ask it not to. It actually ONLY works well when you are doing iterative changes and not used as a pure code generator, actually, AI's one-shot performance is kind of crap. A mistake happens, you point it out to the LLM and ask it to update the code and the instructions used to create the code in tandem. Or you just ask it to fix the code once. You add tests, partially generated by the AI and curated by a human, the AI runs the tests and fixes the code if they fail (or fixes the tests).
All I can really say is that doesn't match my experience. If I fix something that it implemented due to a "misunderstanding" then it usually tends to break it again a few messages later. But I would be the first to say the use of these models is extremely subjective.
I think we have very different experiences then. I find multiple prompts with narrow focuses each executed to update the same file work much better than trying to one shot the file. I think you would have a better experience if you used /clear (assuming you are using Gemini CLI), the problem isn't the change in the file, the problem is probably the conversation history instead.
Comments
How is that different from how it worked without LLMs? The only difference is that we can now get a failing product faster and iterate.
This sounds like the worst way to use AI. LLMs can work existing code, whether it was generated by an LLM or written by human. It can even work on code that has been edited by a human, there is no good reason to not be iterative when using an LLM to develop code, and plenty of good reasons to be iterative.
The difference is that there is an engineer in the middle who can judge if the important information is provided or not as input.
1. for a LLM "the button must be blue" has the same level of importance as "the formula to calculate X is..."
2. failing faster and iterating is good thing if the parameters of failing are clear which is not always the case with vibecoding, especially when done by people with no prior experience in developing. plenty of POCs build with vibecoding have been presented with no aparent failure in their happy path but with disastrous results in edge cases or with disastrous Security etc.
3. where previously, familairity with the codebase and especially the "history of changes" gave you context about why some workarounds were put into place, these are things that are lost to a LLM. Vibecoding a change to an existing system risks removing those "special workarounds" that keep in mind much more than the current context of the specifications or prompt.
You can divide those into two prompts though, there is no point for the LLM to work on both features at the same time. This is why iterative is so useful (oh, the button should be blue, ... and later, the formula should be X).
This isn't about vibecoding. If you are vibecoding, then you aren't developing software, you are just wishing for good code from vague descriptions that you don't plan to iterate on.
LLMs can read and write change logs just as well as humans can (LLMs need change logs to do updates, you can't just give it a changed dependency and expect the LLM to pick up on the change, it isn't a code generator). Actually, this is my current project, since a Dev AI pipeline needs to read and write change logs to be effective (when something changes, you can't just transmit the changed artifact, you need to transmit a summary of the change as well). And again, this is serious software engineering, not vibecoding. If you are vibecoding, I have no advice to give you.
This is the important part of the post to which you replied and remains unaddressed:
The engineer decide what information to use as input to the update prompt. They don’t need to be in the middle of anything, it’s basically the level they are coding at.
LLMs do not possess the ability to "judge if the important information is provided or not as input" as it pertains to the question originally posed:
Working without LLMs involves people communicating, hence the existence of "an engineer in the middle", where middle is defined as between stakeholder requirement definition and asset creation.So you engineer the prompt. I’m still confused what the problem is, I’ve already stated that I’m not talking about vibe coding where the LLM somehow magically figures out relevant information on their own.
The problem is stakeholders are people and they define what problems are needed to be solved. For those tasked to do so requires understanding of the given problems. Tooling (such as LLMs) does not possess this type of understanding as it is intrinsic to the stakeholders (people) whom have defined it. Tools can contribute to delivering a solution, sure, but have no capability to autonomously do so.
For example, consider commercial dish washing machines many restaurants use.
They sanitize faster and with greater cleanliness than manual dish washing once did. Still, there is no dish washing machine which understands why it must be used instead of not. Of course, restaurant stakeholders such as health inspectors and proprietors understand why they must be used.
As far as the commercial dish washer is concerned, it could just as easily be tasked with cleaning dining utensils as it could recycled car parts.
For me it just depends. If the response to my prompt shows the model misunderstood something, then I go back and retry the previous prompt again. Otherwise the "wrong ideas" that it comes up with persist in the context and seem to sabotage all future results. The most of this sort of coding I've done was in Google's AI studio, and I often do have a context that spans dozens of messages, but I always rewind if something goes off-track. Basically any time I'm about to make a difficult request, I clone the entire context/app to a new one so I can roll back [cleanly] whenever necessary.
If you fix something it sticks, the AI won't keep making the same mistake, it won't change the code that already exists if you ask it not to. It actually ONLY works well when you are doing iterative changes and not used as a pure code generator, actually, AI's one-shot performance is kind of crap. A mistake happens, you point it out to the LLM and ask it to update the code and the instructions used to create the code in tandem. Or you just ask it to fix the code once. You add tests, partially generated by the AI and curated by a human, the AI runs the tests and fixes the code if they fail (or fixes the tests).
All I can really say is that doesn't match my experience. If I fix something that it implemented due to a "misunderstanding" then it usually tends to break it again a few messages later. But I would be the first to say the use of these models is extremely subjective.
I think we have very different experiences then. I find multiple prompts with narrow focuses each executed to update the same file work much better than trying to one shot the file. I think you would have a better experience if you used /clear (assuming you are using Gemini CLI), the problem isn't the change in the file, the problem is probably the conversation history instead.
I won't lie and say "That's a great idea" when it isn't.