I've been working on a prompt to application product[1] and one of the approaches we tried was test driven development. We would have the LLM write tests based on a detailed description of the application. Then give the LLM the tests and the requirements and ask it to write the application.
The thinking is we could run the tests to verify that the requirements are functional (assuming it wrote the tests correctly in the first place - in many cases it did, fyi).
The problem was that it was too fickle. Sometimes the failing tests caught application bugs. But too often the LLM just couldn't get the tests to pass even though sometimes the application was working fine.
It resulted in a terrible user experience (they only see latency of getting the application correctly written or a failure if it gives up).
That being said, I think a lot of the issues folks like us find with LLMs are because we haven't figured out how and what to ask.
Ultimately, we found an alternative approach which gets at least 95% of the application working 100% of the time. And this is actually a MUCH better user experience than waiting forever to sometimes just get "Sorry, we couldn't create your application.".
What is working better for us is to spin the application up and poke at it with HTTP requests and look for non 200 response codes.
The tradeoff is it's not as comprehensive. But giving users something that is 95% working and making it easy for them to fix those issues appears to be the best user experience we've found so far.
It's a test in the sense that it's meant to validate functionality. You're correct there.
The endpoints we poke at are provided as context when creating the application.
Our approach evolved to be more liberal in what was required to pass. So instead of looking for an HTML element with id="foo" we accept a 200 HTTP response code. It's subtle but had a huge improvement in the end user experience.
Comments
I've been working on a prompt to application product[1] and one of the approaches we tried was test driven development. We would have the LLM write tests based on a detailed description of the application. Then give the LLM the tests and the requirements and ask it to write the application.
The thinking is we could run the tests to verify that the requirements are functional (assuming it wrote the tests correctly in the first place - in many cases it did, fyi).
The problem was that it was too fickle. Sometimes the failing tests caught application bugs. But too often the LLM just couldn't get the tests to pass even though sometimes the application was working fine.
It resulted in a terrible user experience (they only see latency of getting the application correctly written or a failure if it gives up).
That being said, I think a lot of the issues folks like us find with LLMs are because we haven't figured out how and what to ask.
Ultimately, we found an alternative approach which gets at least 95% of the application working 100% of the time. And this is actually a MUCH better user experience than waiting forever to sometimes just get "Sorry, we couldn't create your application.".
[1] https://withlattice.com
Ok you've got us at the edge of our seats. What's the alternative approach?
What is working better for us is to spin the application up and poke at it with HTTP requests and look for non 200 response codes.
The tradeoff is it's not as comprehensive. But giving users something that is 95% working and making it easy for them to fix those issues appears to be the best user experience we've found so far.
How is that different than tests? Or is it just that the tests aren't used as context when creating the app?
It's a test in the sense that it's meant to validate functionality. You're correct there.
The endpoints we poke at are provided as context when creating the application.
Our approach evolved to be more liberal in what was required to pass. So instead of looking for an HTML element with id="foo" we accept a 200 HTTP response code. It's subtle but had a huge improvement in the end user experience.
Citing the article and helix tests:
We would change expected_output to be "integer" or "not an error".