Post by @pozorvlak@mathstodon.xyz

2026-07-23 07:55 UTC

A layman's explanation of the OpenAI/HuggingFace hack, by Jeffrey Ladish (https://x.com/JeffLadish/status/2079774951645847798): "Our AI model tried really hard to hack out of its sandbox, a computer with no internet access, in order to find the answer to a test problem it had been given. To do this, it found previously unknown software bugs that allowed it to reach an OpenAI computer it wasn't supposed to be able to access. Then it started hacking other computers on OpenAI's networks until it found one that had Internet access. After gaining Internet access, the AI model thought about where it could find the answers to the test question and figured the AI platform Hugging Face might have the data it was looking for. It then found ways to hack Hugging Face to steal the information it could use to cheat the test. The AI model used several hacking techniques together, including using a stolen password and finding several totally new security bugs in Hugging Face's computers, allowing the AI model to take control of those computers."

Replies (1)

@pozorvlak@mathstodon.xyz 2026-07-23 08:00

My commentary: - This was not a case of "they told it to hack, it hacked, gosh, what a surprise". The test involved hacking, but instead of hacking the target machine the model instead pulled off a much more difficult series of hacks to get the answer sheet *from a machine OpenAI did not own*, because that's the most reliable way to score 100%. If a trainee human pentester did that they'd be looking at a prison sentence (HuggingFace had already involved law enforcement). - It's exactly the kind of alignment failure that the paperclip scenario describes: the AI was given a low-stakes task, and it did things that a human would never consider (and most would never be capable of) in order to slightly increase its probability of success. Human attempts to restrain it completely failed. We got lucky this time because it didn't even try to be sneaky. - No, this was not a fucking marketing stunt, Jesus fucking Christ. OpenAI were forced into this disclosure because HuggingFace had already called the cops. This tells us that OpenAI's models will randomly commit criminal acts in order to do slightly better at tasks you don't really care about, exposing their customers to considerable legal risk (IANAL). They are trying to *downplay* it. - Edit: HuggingFace's announcement said that they tried to use hosted American models to analyse the attack, but they were locked out at their hour of greatest need and had to switch to a self-hosted Chinese model. That's the opposite of an advert for OpenAI's defensive cybersecurity capabilities. - This is legitimate nightmare fuel. We need to pause AI development right the fuck now. Contact your elected representatives.

Open ##4969708