Post by @stsp@bsd.network
From what I’ve read, they used AI to find the PHP vulnerability and then - hey! - used AI to fix it.
What if we’d just skipped the whole thing and left that bug sitting there for another twenty years? 😂
@stefano Looking at the 11 thousand line fix, about 50% of which is newly added test cases... yeah... LLMs tend to test their own code quite a lot.
@stefano I mean, I'm a huge fan of tests really, and without tests my own software would suck and have a lot more bugs. It's just that the kind of tests cases LLMs produce often border on the absurd, either checking basic assumptions or completely irrelevant things.
One example I had recently was test cases added by LLMs checking whether a change from "string" to json.Marshal("string") actually did have the intended effect of escaping the string. If I'm a consumer of a library like that, I'm not gonna test that library's basic assumptions in my own program. I expect the library to work as advertised. It''s like checking whether opening a file without errors will indeed mean that the file is now open.
@stsp they usually test the simplest and correct things and forget to test what really needs to be tested. Cool, isn’t it? 😂