Fable (and even Opus if kept tightly under reigns) does generate high-quality code even for highly complex tasks.
It generally performs better if the tasks are broken done into small manageable pieces, and the person is actually reviewing and calling out problems, which usually requires the person to be a competent engineer in the problem domain to begin with.
But yes, I have personally used it to build what the OP calls a JIT. I would usually write that by hand and it would take me one week. The AI does it in an hour.
You can't make assertions about "quality" of code that was generated under an hour while it would have taken a human 40 hours, unless you put substantial amount of work into reviewing it.
I used Fable on a Zephyr project with time sensitive code for LR-WPAN and it broke everything. Literally made the code worst to the point that the devices stopped connecting.
If I need to be a domain expert anyway, the value of the tool goes down by orders of magnitude. Same if I need to first break the task down into pieces and keep reviewing all the output. That sounds to me like >80% of the work I'd need to do anyway.
If I need to design and understand all of the code anyway, I might as well skip the whole process of repeatedly fixing the subpar-at-every-level LLM output and write it all myself.
Personally, I've found the greatest value in asking for simple tasks, like wiring up APIs, generating boilerplate, bug finding etc. Anything that requires effort to do but results in either very little or very simple output, so that I can easily verify its correctness.
But give the LLM anything remotely complex to generate and it cakes its pants.
If I need to be a domain expert anyway, the value of the tool goes down by orders of magnitude. Same if I need to first break the task down into pieces and keep reviewing all the output. That sounds to me like >80% of the work I'd need to do anyway.
You absolutely don't. You only need to be roughly aware of what the code needs to be doing. Similar to how a software architect historically didn't personally oversee every line of code in an org, only it's overall structure. The implementation specific details can be left to the AI.
Only getting the code right "roughly" is not enough in complex code. An example would be an (multi-symbol) arithmetic coder. They all look very similar at a first glance, but choices on the order of operations influence performance and compression characteristics, off-by-one errors are extremely easy to make.
That is my experience as well. I get a (subjective) speedup between 1 and 3 for parts of the code I'd consider critical and where I check the output tightly, and 5-20 for menial work OR for important code that's well isolated into its own module such that its quality doesn't matter because I can have it rewritten easily if it doesn't work as expected.
Comments
Fable (and even Opus if kept tightly under reigns) does generate high-quality code even for highly complex tasks.
It generally performs better if the tasks are broken done into small manageable pieces, and the person is actually reviewing and calling out problems, which usually requires the person to be a competent engineer in the problem domain to begin with.
But yes, I have personally used it to build what the OP calls a JIT. I would usually write that by hand and it would take me one week. The AI does it in an hour.
You can't make assertions about "quality" of code that was generated under an hour while it would have taken a human 40 hours, unless you put substantial amount of work into reviewing it.
I used Fable on a Zephyr project with time sensitive code for LR-WPAN and it broke everything. Literally made the code worst to the point that the devices stopped connecting.
I do put substantial amount of work into reviewing code: that's essentially what my job is.
And even if everyone is using the same LLM model, quality varies on lot on who's driving it.
Your second paragraph invalidates your first.
If I need to be a domain expert anyway, the value of the tool goes down by orders of magnitude. Same if I need to first break the task down into pieces and keep reviewing all the output. That sounds to me like >80% of the work I'd need to do anyway.
If I need to design and understand all of the code anyway, I might as well skip the whole process of repeatedly fixing the subpar-at-every-level LLM output and write it all myself.
Personally, I've found the greatest value in asking for simple tasks, like wiring up APIs, generating boilerplate, bug finding etc. Anything that requires effort to do but results in either very little or very simple output, so that I can easily verify its correctness.
But give the LLM anything remotely complex to generate and it cakes its pants.
You absolutely don't. You only need to be roughly aware of what the code needs to be doing. Similar to how a software architect historically didn't personally oversee every line of code in an org, only it's overall structure. The implementation specific details can be left to the AI.
Only getting the code right "roughly" is not enough in complex code. An example would be an (multi-symbol) arithmetic coder. They all look very similar at a first glance, but choices on the order of operations influence performance and compression characteristics, off-by-one errors are extremely easy to make.
That is my experience as well. I get a (subjective) speedup between 1 and 3 for parts of the code I'd consider critical and where I check the output tightly, and 5-20 for menial work OR for important code that's well isolated into its own module such that its quality doesn't matter because I can have it rewritten easily if it doesn't work as expected.
You missed the part where I spelled out the advantage: it accelerates things significantly (one hour vs one week).