much of the work is repetitive, but it comes with its edge cases that we need to look out for
Then don't use AI for it.
Bluntly.
This is a poor use-case; it doesn't matter what model you use, you'll get a disappointing result.
These are the domains where using AI coding currently shines:
1) You're approaching a new well established domain (eg. building an android app in kotlin), and you already know how to build things / apps, but not specifically that exact domain.
Example: How do I do X but for an android app in kotlin?
2) You're building out a generic scaffold for a project and need some tedious (but generic) work done.
3) You have a standard, but specific question regarding your code, and although related Q/A answers exist, nothing seems to specifically target the issue you're having.
Example: My nginx configuration is giving me [SPECIFIC ERROR] for [CONFIG FILE]. What's wrong and how can I fix it?
The domains where it does not work are:
1) You have some generic code with domain/company/whatever specific edge cases.
The edge cases, broadly speaking, no matter how well documented, will not be handled well by the model.
Edge cases are exactly that; edge cases; the common medium of 'how to x' does not cover edge cases; the edge cases will not be covered and the results will require you to review and complete them manually.
2) You have some specific piece of code you want to refactor 'to solve xxx', but the code is not covered well by tests.
LLMs struggle to refactor existing code, and the difficulty is proportional to the code length. There are technical reasons for this (mainly randomizing token weights), but tldr; it's basically a crap shot.
Might work. Might not. If you have no tests who knows? You have to manually verify both the new functionality and the old functionality, but maybe it helps a bit, at scale, for trivial problems.
3) You're doing something obscure or using a new library / new version of the library.
The LLM will have no context for this, and will generate rubbish / old deprecated content.
Obscure requirements have an unfortunate tendency to mimic the few training examples that exist, and may generate verbatim copies, depending on the model you use.
...
So. Concrete advice:
1) sigh~
a friend of mine came and suggested that I use Retrieval-Augmented Generation (RAG), I have yet to try it, with a setup Langchain + Ollama.
Ignore this advice. RAG and langchain are not the solutions you are looking for.
2) Use a normal coding assistant like copilot.
This is the most effective way to use AI right now.
There are some frameworks that let you use open source models if you don't want to use openAI.
3) Do not attempt to bulk generate code.
AI coding isn't at that level. Right now, the tooling is primitive, and large scale coherent code generation is... not impossible, but it is difficult (see below).
You will be more effective using an existing proven path that uses 'copilot' style helpers.
However...
...if you do want to pursue code generation, here's a broad blueprint to follow:
- decompose your task into steps
- decompose you steps in functions
- generate or write tests and function definitions
- generate an api specification (eg. .d.ts file) for your function definitions
- for each function definition, generate the code for the function passing the api specification in as the context. eg. "Given functions x, y, z with the specs... ; generate an implementation of q that does ...".
- repeated generate multiple outputs for the above until you get one that passes the tests you wrote.
This approach broadly scales to reasonably complex problems, so long as you partition your problem into module sized chunks.
I personally like to put something like "you're building a library/package to do xxx" or "as a one file header" as a top level in the prompt, as it seems to link into the 'this should be isolated and a package' style of output.
However, I will caveat this with two points:
1) You generate a lot of code this way, and that's expensive if you use a charge-per-completion API.
2) The results are not always coherent and functions tend to (depending on the model, eg. 7B mistral) inline implementations for 'trivial' functions instead of using functions (eg. if you define Vector::add, the model will 50/50 just go a = new Vector(a.x + b.x, a.y + b.y)).
I've found that the current models other than GPT4 are prone to incoherence as the problem size scales.
7B models, specifically, perform significantly worse than larger models.
I have limited success with feeding a LLM (dolphin finetune of mixtral) a content of a merge request coming from my team. It was few thousand lines of added integration test code and I just couldn't be bothered/had little time to really delve.
I slapped the diff and used about 10 prompt strategies to get anything meaningful. So my first initial impressions were: clearly it was finetuned on too short responses. It kept putting in "etc.", "and other input parameters", "and other relevant information". At one point I was ready to give up; it clearly hallucinated.
Or that's what I thought: turned out there was some new edge case of a existing functionality added that was added, without ever me noticing (despite being on the same meetings).
I think it actually saved me a lot of hours or pestering other team members.
Comments
Then don't use AI for it.
Bluntly.
This is a poor use-case; it doesn't matter what model you use, you'll get a disappointing result.
These are the domains where using AI coding currently shines:
1) You're approaching a new well established domain (eg. building an android app in kotlin), and you already know how to build things / apps, but not specifically that exact domain.
Example: How do I do X but for an android app in kotlin?
2) You're building out a generic scaffold for a project and need some tedious (but generic) work done.
Example: https://github.com/smol-ai/developer
3) You have a standard, but specific question regarding your code, and although related Q/A answers exist, nothing seems to specifically target the issue you're having.
Example: My nginx configuration is giving me [SPECIFIC ERROR] for [CONFIG FILE]. What's wrong and how can I fix it?
The domains where it does not work are:
1) You have some generic code with domain/company/whatever specific edge cases.
The edge cases, broadly speaking, no matter how well documented, will not be handled well by the model.
Edge cases are exactly that; edge cases; the common medium of 'how to x' does not cover edge cases; the edge cases will not be covered and the results will require you to review and complete them manually.
2) You have some specific piece of code you want to refactor 'to solve xxx', but the code is not covered well by tests.
LLMs struggle to refactor existing code, and the difficulty is proportional to the code length. There are technical reasons for this (mainly randomizing token weights), but tldr; it's basically a crap shot.
Might work. Might not. If you have no tests who knows? You have to manually verify both the new functionality and the old functionality, but maybe it helps a bit, at scale, for trivial problems.
3) You're doing something obscure or using a new library / new version of the library.
The LLM will have no context for this, and will generate rubbish / old deprecated content.
Obscure requirements have an unfortunate tendency to mimic the few training examples that exist, and may generate verbatim copies, depending on the model you use.
...
So. Concrete advice:
1) sigh~
Ignore this advice. RAG and langchain are not the solutions you are looking for.
2) Use a normal coding assistant like copilot.
This is the most effective way to use AI right now.
There are some frameworks that let you use open source models if you don't want to use openAI.
3) Do not attempt to bulk generate code.
AI coding isn't at that level. Right now, the tooling is primitive, and large scale coherent code generation is... not impossible, but it is difficult (see below).
You will be more effective using an existing proven path that uses 'copilot' style helpers.
However...
...if you do want to pursue code generation, here's a broad blueprint to follow:
- decompose your task into steps
- decompose you steps in functions
- generate or write tests and function definitions
- generate an api specification (eg. .d.ts file) for your function definitions
- for each function definition, generate the code for the function passing the api specification in as the context. eg. "Given functions x, y, z with the specs... ; generate an implementation of q that does ...".
- repeated generate multiple outputs for the above until you get one that passes the tests you wrote.
This approach broadly scales to reasonably complex problems, so long as you partition your problem into module sized chunks.
I personally like to put something like "you're building a library/package to do xxx" or "as a one file header" as a top level in the prompt, as it seems to link into the 'this should be isolated and a package' style of output.
However, I will caveat this with two points:
1) You generate a lot of code this way, and that's expensive if you use a charge-per-completion API.
2) The results are not always coherent and functions tend to (depending on the model, eg. 7B mistral) inline implementations for 'trivial' functions instead of using functions (eg. if you define Vector::add, the model will 50/50 just go a = new Vector(a.x + b.x, a.y + b.y)).
I've found that the current models other than GPT4 are prone to incoherence as the problem size scales.
7B models, specifically, perform significantly worse than larger models.
Very well researched!
I'd add the MR review use case.
I have limited success with feeding a LLM (dolphin finetune of mixtral) a content of a merge request coming from my team. It was few thousand lines of added integration test code and I just couldn't be bothered/had little time to really delve.
I slapped the diff and used about 10 prompt strategies to get anything meaningful. So my first initial impressions were: clearly it was finetuned on too short responses. It kept putting in "etc.", "and other input parameters", "and other relevant information". At one point I was ready to give up; it clearly hallucinated.
Or that's what I thought: turned out there was some new edge case of a existing functionality added that was added, without ever me noticing (despite being on the same meetings).
I think it actually saved me a lot of hours or pestering other team members.