This is where prompt engineering becomes more important. Next time consider pre-pending some kind of plain English set of expectations before pasting your code. Something like, “I want you to write tests for this code. Here are the expected behaviors <expected behaviors list>, and here are unexpected behaviors <unexpected behaviors list>. Tests should pass if they adhere to the expected behaviors and fail if they have unexpected behaviors. Here is the code: <code>”.
Like most LLM generation though, it’s not a deterministic thing and like you mentioned originally it takes some verification of the output. I still think with the extra steps it saves time when applied to the right scenarios. The longer the input the higher the hallucinations count in my experience though, so I always keep the code provided in the smallest chunk possible which still has enough context.
Comments
This is where prompt engineering becomes more important. Next time consider pre-pending some kind of plain English set of expectations before pasting your code. Something like, “I want you to write tests for this code. Here are the expected behaviors <expected behaviors list>, and here are unexpected behaviors <unexpected behaviors list>. Tests should pass if they adhere to the expected behaviors and fail if they have unexpected behaviors. Here is the code: <code>”.
Like most LLM generation though, it’s not a deterministic thing and like you mentioned originally it takes some verification of the output. I still think with the extra steps it saves time when applied to the right scenarios. The longer the input the higher the hallucinations count in my experience though, so I always keep the code provided in the smallest chunk possible which still has enough context.