Post by @stefano@mastodon.bsd.cafe

@stefano But the lolz?!?!

@nuintari I was going to post an harsh message. So I decided to promote positivity and make this message a little less…harsh

@stefano Nah man, you should always promote positivity, you are good at it!

I can pick up the negaslop. Let's face it, I'm prettty good at being angry.

@stefano Looking at the 11 thousand line fix, about 50% of which is newly added test cases... yeah... LLMs tend to test their own code quite a lot.

@stsp …but it seems there’s a new vulnerability in 8.5.12
Crazy times, crazy times!

@stefano Not enough test cases!

@stsp we need moooar test cases!

@stefano I mean, I'm a huge fan of tests really, and without tests my own software would suck and have a lot more bugs. It's just that the kind of tests cases LLMs produce often border on the absurd, either checking basic assumptions or completely irrelevant things.

One example I had recently was test cases added by LLMs checking whether a change from "string" to json.Marshal("string") actually did have the intended effect of escaping the string. If I'm a consumer of a library like that, I'm not gonna test that library's basic assumptions in my own program. I expect the library to work as advertised. It''s like checking whether opening a file without errors will indeed mean that the file is now open.

@stsp they usually test the simplest and correct things and forget to test what really needs to be tested. Cool, isn’t it? 😂

@stefano echo "All tests passed"

@stsp hey! We’re correct, working and safe then! 😂

@stefano@mastodon.bsd.cafe
AI finding bugs...hmm, OK. Once they've found them all things should settle down a bit.

AI fixing bugs... Uh oh! 😬

But maybe we should just turn AI off while everyone is patching their gear. Once it's off... We could leave it off, as a treat.

@paul it seems there’s another vulnerability, a new one, in 8.5.12…

@stefano@mastodon.bsd.cafe ah, I heard that but didn't look into it.
So specifically in 8.5.12? Or did it exist before? Does it exist in the 8.4, 8.3 and 8.2 latests as well?

Very interesting if fix by AI! Verrrry interesting.

@paul as far as I’ve read, only 8.5.x. The others should be ok. But I’ve read it somewhere and didn’t investigate further.

@stefano@mastodon.bsd.cafe thanks.
Guess we'd better wait for 8.5.13 then!

@stefano @paul

@stefano exactly, and the people who don't want to use "AI" to defend their networks, may have bigger issues in the future.

At this point, the attackers WILL use an AI agent to do this, are you going to have to go through log files this you're being attacked?

Probably not, if anyone wants a week free of Claude Code, let me know. I have three invites I can send out.

@stefano and it seems that it hasn't been solved completely yet, since the PoC still works

@stefano Oh, well isn't *that* interesting!

@stefano

Anyone know of a stripped down, minimalist, replacement for PHP?

Asking for a friend. That friend is me.