I asked Sonnet to port a prolog interpreter from Python to JS. I was surprised and delighted that it ended up writing a test suite for it. All tests passed.
Until... I inspected the code: it was just a bunch of print statements that said "test passed!" and didn't actually test anything.
That was last year, so hopefully it doesn't do that anymore.
Few months ago I was debugging an integration with a no-code library that ships code in a custom binary format. I still smile thinking at the following test I read
assertThrows(RuntimeException.class, () -> callFoo(notABinaryFile)); //well, foo library is very complex, obviously this test throws!
Comments
I asked Sonnet to port a prolog interpreter from Python to JS. I was surprised and delighted that it ended up writing a test suite for it. All tests passed.
Until... I inspected the code: it was just a bunch of print statements that said "test passed!" and didn't actually test anything.
That was last year, so hopefully it doesn't do that anymore.
Few months ago I was debugging an integration with a no-code library that ships code in a custom binary format. I still smile thinking at the following test I read
assertThrows(RuntimeException.class, () -> callFoo(notABinaryFile)); //well, foo library is very complex, obviously this test throws!