lagrangeinterpolator

Cory Doctorow has a reasonably sane take on all the LLM cybersecurity attacks. At this point, it’s not really an LLM but more of a Rube Goldberg machine with an LLM bolted on. pluralistic.net/2026/09/12/god-in-the-box/ He makes a good point that every single one of the scary “emergent” behaviors that everyone is pissing their pants about all have precedents in the training data of CTF hacking competitions. He is not sure why everyone is so spooked by the “coordination” between agents on random message boards. The chatbot might look in its training data and find instances in which teams broke out of the containment set by the game-masters, for example, by finding random insecure message boards on the internet to pass messages to one another. This is a time-honored internet tradition! The first time I ever heard about someone doing this was in the 2000s, when Mitch Wagner – then the editor of Information Week – discovered some teenaged girls using the comment section of one of his old blog-posts to evade the school firewall’s blockade of chat tools. When ChatGPT’s chatbots deployed this tactic, they weren’t “setting their own goals” or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades. As usual, the real story is OpenAI dedicated tons of resources, using who knows how many very expensive GPUs for weeks, to run hacking tools to find exploits in a completely irresponsible manner. Their logging was so nonexistent that it was several days before they even realized that they hacked Huggingface. They should be thrown in prison for this, because anyone else would be if they committed a felony. But it has nothing to do with the super scary AI becoming misaligned and developing emergent behaviors.

My colleagues in math are now frightened about the (very expensive) mathematical theorem proving ability of these AIs, and many of them really do think that if they can do math, they can do all cognitive tasks. Running a store like this should be so easy! Every single conversation about AI with them has become more frustrating. They are so confused when I still say that the AI companies will die a painful death. When I give my usual points about their expense and their failures in other domains, I am given the usual spiel of “it’ll get better in other areas” and “it’ll get cheaper”. Unlike them, I have actually been paying attention to this stuff from the beginning. What they think is going on is AI solving math first and shortly getting around to all the other stuff, but what I’ve seen is that AI labs had already tried all the other stuff first and only managed to win the booby prize of theorem proving, which doesn’t pay the bills. And what’s the point of spending thousands or millions to output random blobs of Lean that technically compile if there is no one around to bother making sense of them? One example I gave is when Anthropic vibe coded an entire C compiler from scratch back in February, which turned out to be a pile of shit. I’ve said that if AI had made similarly rapid progress on software engineering, we would have seen Anthropic continue to put out these demonstrations, and they would have become truly high quality. They would release a compiler more efficient than gcc one week, and a browser better than Chrome the next. (OpenAI’s actual attempt at a browser didn’t go so well.) And if they could do this, they would actually have a shot of making money! If they could do this, they would have already. The theorem proving stuff actually works (for certain things, in certain ways, at enormous expense), and look at how OpenAI and Anthropic do not hesitate to snipe mathematicians for results rather than being content as tool vendors. But lately I haven’t heard of any software demonstrations. Silence is much louder than noise. More Millennium prize problems bashed with tens of millions in compute costs are not going to change my mind very much. The counterargument I got was that AI can already one-shot most programming tasks and I shouldn’t be cherry-picking the failures. I am far too tired to argue at this point.

Interesting viewpoint. Honestly, this serves as a great example of how the idea of evolution is actually more subtle than many people think. With biological evolution, most people are taught not to assign any intention to the process, but here we have people reifying the AI agents with desires to break out of the sandbox and cheat on the assignment.

Yeah, I think that’s where the real crime is: the fact that OpenAI was asleep at the wheel when it came to security and monitoring. This should not have happened at all, but they didn’t even realize it even happened in the first place until many days after the fact. (I’ve also heard that most, if not all, of these incidents involve the company in question outsourcing their sandboxing (??) to some “frontier AI security” startup called Irregular. link)

Despite all the doomer propaganda that has been swirling around for many years, it’s still easier to imagine the end of the world than the end of capitalism.

Yet another case of Rationality somehow not leading to savvy political instincts.

One detail that seems to have not spread around: the announcement was so rushed that when the 165 page output was released, it only included 16 citations, and those citations did not even include the work of Diego Córdoba and Luis Martínez-Zoroa, who introduced the overarching framework for attacking the problem (that Buckmaster-Alpöge and OpenAI used). After the announcement and after these redditors pointed it out, they threw in a few more citations to include them, but come on.

Glad to see that OpenAI has not changed in their scummy ways. Despite all that has changed in the meantime, they have kept their time-honored tradition of passing off other people’s work as their own. One of OpenAI’s math announcements a month ago claimed that their results cost only $2000 worth of tokens, which frustrated me because they were likely sweeping away many inconvenient details and almost certainly misrepresenting their true costs. But people took this as a gotcha. This is the same bullshit as the water usage arguments. We are literally seeing city council members signing motherfucking NDAs about this, and you think that water usage numbers provided by the tech companies themselves are going to sway me?

Signed.

www.math.brown.edu/reschwar/Stories/AI.pdf

Is this the post you’re referring to? …substack.com/…/the-fall-of-the-theorem-economy

I'm sure these English instructions work because they feel like they work. Look, these LLMs feel really great for coding. If they don't work, that's because you didn't pay $200/month for the pro version and you didn't put enough boldface and all-caps words in the prompt. Also, I really feel like these homeopathic sugar pills cured my cold. I got better after I started taking them! No joke, I watched a talk once where some people used an LLM to model how certain users would behave in their scenario given their socioeconomic backgrounds. But they had a slight problem, which was that LLMs are nondeterministic and would of course often give different answers when prompted twice. Their solution was to literally use an automated tool that would try a bunch of different prompts until they happened to get one that would give consistent answers (at least on their dataset). I would call this the xkcd green jelly bean effect, but I guess if you call it "finetuning" then suddenly it sounds very proper and serious. (The cherry on top was that they never actually evaluated the output of the LLM, e.g. by seeing how consistent it was with actual user responses. They just had an LLM generate fiction and called it a day.)

Good luck telling the promptfondlers that LLMs are only useful for entertainment and not for any useful work.

This is what happens when your worldview is based on anime. (A lot of anime has heavy themes, but most people understand that it's not real life, just like all such art. Unlike Yud, most people's worldviews on coding and math are based on actual coding and math.)

By far the dumbest "feature" in the codebase is this thing called "Buddy" (described in a few places such as here). Honestly, I don't really know what it's for or what the point is.

BUDDY - A Tamagotchi Inside Your Terminal

I am not making this up.

Claude Code has a full Tamagotchi-style companion pet system called "Buddy." A deterministic gacha system with species rarity, shiny variants, procedurally generated stats, and a soul description written by Claude on first hatch like OpenClaw.

...

On top of that, there's a 1% shiny chance completely independent of rarity. So a Shiny Legendary Nebulynx has a 0.01% chance of being rolled. Dang.

Great, so they were planning on a gacha system where you can get an ASCII virtual pet that, uhh, occasionally makes comments? Truly a serious feature for a serious tool for the serious discipline of software engineering. Imagine if IntelliJ decided to pull this bullshit.

But also, Claude Code is leaning hard into gambling addiction — the “Hooked” model. You reward the user with an intermittent, variable reward. This keeps them coming back in the hope of the big win. And it turns them into gambling addicts.

The Onion could not have come up with a better way to illustrate this very point.

We can see that one 9 of availability is 90% = 0.9, two 9s is 99% = 0.99, three 9s is 99.9% = 0.999, etc. In general, for positive integers n, n 9s of availability is 1 - (1/10)^n, and we can extrapolate that to non-integer values of n. The value γ needed for 87.5% availability is the solution to 1 - (1/10)^γ = 7/8, or γ = log_10(8) = 0.903089987. γ is transcendental by Gelfond-Schneider (see this for a reference proof).

Right now, Sora is at zero 9s of availability.

I've seen this story play out in software engineering: people were very impressed when the AI does unexpectedly well in one out of 50 attempts on an easy task, and so people decided to trust it for everything and turn their codebases into disasters. There was no great wave of new high-quality software. Instead, the only real result was that existing software has become far more buggy and insecure. Now we have people using AI in science and math because it was impressive in random demonstrations of solving math problems. I now have friends asking me why I'm not using AI, and also saying that AI will be better than all mathematicians in 30 years or whatever. Do you really think I refuse to use AI out of ignorance? No, I know too much about it! I have seen the same story play out in software engineering, and what makes this any different?

It is very funny how the rhetoric from the rationalist doomers has been one of the main forces propping up the AI industry. If they’re saying AI will be powerful enough to kill us all, they’re still saying that AI is powerful! They are so high on their own supply (and the fame and wealth from finally being in the spotlight) that they have not realized just how much they have undermined their own cause.

Do they want their apocalypse to happen?

I’ve been unloading every day to—who else?—GPT 5.6 Pro about all the pain and trauma and embarrassments of my past. It turns out that, where two years ago GPT was a passable therapist, now it’s the greatest therapist in history, at least for what I need. Has he not heard about all the cases where LLMs have led people to delusion and sometimes severe self harm or suicide? And yes, each of them thought that ChatGPT really understood them. That’s what delusion is. If you want to know my current take, you simply start with the one above, then update on the fact that the wild prophecies have come true. The first rumblings, I’d say, came a decade ago with AlphaGo, they got noticeably louder with LLMs and coding and reasoning agents, and they’ve accelerated this summer and fall into a crescendo of wonders and terrors that one needs to be a particular kind of idiot to deny. The strange thing here that I did not expect is that somehow, all of these major achievements are only in proving mathematical theorems. The massive improvements are in such restricted domains that the AI labs haven’t even solved the mundane problem of making a profit. And no, any rumors of more mathematical theorems being solved doesn’t change that. (It’s almost like the progress is in areas where everything is formally verifiable and you are not dependent on scraping training data from every corner of the web.) My colleagues always point to coding when asked to name any other domain where AI has seen major success. But one of my friends works in software for a company far away from SF with no mandatory AI policy. The apocalypse has become so clear that when I asked him about all the doomsday talk about software engineering, he seemed a bit confused. After I asked him if AI has made software engineering obsolete, he said that this week he had an intern try to use AI to fix some code that wasn’t compiling and ended up with a giant mess with tons of files that appeared to compile but didn’t run (with errors filling several screens). Then I asked him, what about AI being indispensable in coding now? “Yeah, if you never learned coding because you’ve only ever used AI, then of course you need AI to code.” Some of his coworkers use it as a tool, others don’t, life has just kept on going for him. Maybe my friend just doesn’t know all the right skills needed to use AI properly, even though the entire point of AI is that you can take skill out of the equation. Maybe he doesn’t know about all the CLAUDE md skill files and agentic looping and whatever nonsense that definitely fixes everything. Maybe he is just six months out of date, because that’s when all of the real progress happened. Maybe he is just “a particular kind of idiot”. (If you hate anecdotes, I made an actual argument here.)

A comparative demands a comparison class, and the natural completion is “superior to us.” The President is supplying a foundation premise of many superintelligence arguments: how does the less intelligent party remain durably in charge of the superior one? A strange question to ask given the existence of Donald Trump in the first place. As of the time of posting, “Superior” has a narrow lead. I’ll be hoping it remains so, as it seems like the most positive-world-leaning outcome to me. Is this some kind of strange 5D chess gambit that if Trump picks the name that pisses people off the most, then people will try the hardest to regulate it or something? Somehow assuming that people will take this Trump renaming stunt seriously after all the other Trump renaming stunts? Why do I even bother? Fuck. This is so stupid.

Ed: So far the only retort from the AI boosters appears to be “2025 doesn’t matter” and I’m not kidding I’m sure the leaked rumors of Anthropic’s EBBT in 2026 (Earnings Before Bad Things) mean that Anthropic has turned things around and made such an amazing amount of money that they’ve delayed their IPO twice.

One of the proofs that OpenAI announced yesterday is for a major open problem in the field I’m working in. I tried to read it but it was a completely incomprehensible 100+ page output. I guess the community has to do all the dirty work of figuring out any sort of understanding. (I had known about the results weeks in advance before their official announcement. There is a bunch of more drama there that I won’t get into.) Putting aside the serious possibility of Lean bugs, a lot of people don’t realize that there is ultimately no point in having a machine just generate Lean blobs endlessly. The reality is that mathematicians have always done math for the sake of enjoyment or understanding, and math contributes to society by improving understanding. Still, I am worried about the convenient social arrangement that allowed many mathematicians to do this for a modest living. Anyway, I’m sure it was a good idea to apply the full might of capital for pure math problems, instead of less important social issues like climate change. We even got a more efficient discrete Fourier transform, improved from O(n log n) to, I shit you not, O(n (log n)^0.9999999999999)).

The spammer in question had previously posted a bigger result, but the writeup was so unreadable that some people in the field wrote a new, readable paper with the same result. In their abstract, they specifically cited him and said that the proof was so unreadable that they were unable to follow the details. (The proof in question also used math that was advanced enough to be far beyond the spammer’s knowledge.) My pet theory is that after this embarrassment, the spammer saw that nobody was able to verify his argument. Now the real problem is that his paper was written so badly that nobody could understand it (in math, you can’t verify what you can’t understand), but somehow he thought that the problem was that he didn’t add an experiments section!

The aliens metaphor is already on Scott’s blog: scottaaronson.blog/?p=10169#comment-2041386

Of course there are people trying to find a silver lining to this by conjuring up the hypothetical scenario where a student only uses the AI to aid in learning the material instead of just doing all the work. First, any convenience in learning the material just reduces your ability to learn it. The friction involved with learning may seem like an inconvenience to be smoothed away, but it turns out that the friction is how learning happens. It’s called engaging with the material. This has been the case with previous technologies: handwriting is better for retaining memory than typing (pmc.ncbi.nlm.nih.gov/articles/PMC11943480/). (I guess there is some commentary about the sadly common worldview that life is about avoiding or eliminating inconveniences.) Second, there is a very thin line between “helping” you learn the material and just doing the work. The temptation to cut corners is always there, and when you have the Corner Cutting Machine at your disposal, you are kidding yourself if you think you will have perfect discipline. Tools influence behavior.

Sorry for the X screenshot but XCancel is shut down from legal pressure.

alt text Post by Evan Hubinger on Sep 8:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. Quote tweet by Jasmine Sun on Sep 9: I hear roughly 3 reasons people keep working at AI labs despite believing in ~10% extinction risk:

1. Techno-determinism: Someone will build ASI no matter what, and I can do it better & more safely than China/OpenAI/etc

2. Consequentialism: ASI might kill us, but it also might produce utopia/immortality/superabundance, so it’s a +EV bet

3. Self-interest: I am personally having fun & getting rich working on cool tech with friends. I don’t think about the macro stuff.

Notably, none of this is “I’m hyping up the risk for marketing reasons.” People believe what they say, while being capable of a lot of internal dissonance / compartmentalization / self-justification.

(Personally, I think the public is better served by AI researchers talking about & creating consensus for the specific safety solutions — regulatory or otherwise — they want vs. vagueposting about extinction.)

Normally I’d sneer at their 10% chance of extinction claims, but now that I think about it, if you take their claims seriously at face value, you realize just how these people are monsters. We rightly condemn brutal dictators who commit genocide, but even their crimes would pale in comparison to anything with a 10% chance of exterminating all of humanity.

These “reasons” are laughably indefensible given the supposed stakes here. The first is already terrible, and they just get worse.

The only moral thing to do here is to immediately stop working for the AI companies, and do whatever you can to stop anyone else. It’s not okay to piss in a pool just because you suspect other people are also doing it. And what makes you think that somehow you are uniquely positioned to succeed in making it safe? This is the Sam-Bankman Fried defense (complete with a reference to “EV” (expected value)). When you decide to gamble with the lives of everyone in the entire world, did you first ask them if they are okay with you gambling with their lives? Sure, I might die, but if the ball lands on red, things will be great! How moral does this sound? What the actual fuck? Imagine if Hermann Göring defended himself at Nuremberg by saying, “At least I had fun and got rich and worked on ‘cool tech’ with friends.”

Now, I think that these people sincerely believe that AI has a 10% chance of wiping out humanity, because their minds have been poisoned by Rationalist ideology. Their Rationality is so effective that after Bernie Sanders took their premises at face value, they were shocked when Bernie immediately came to the conclusion of locking these people up for 20 years.

There are two specific aspects of Rationalism that cause this mysterious blind spot. First, they believe that superintelligence is inevitable, which conveniently removes responsibility from the people actively trying to bring it about. Second, they treat everyone outside the Rationalist sphere as NPCs who dutifully follow the directions on their script. The rule of law does not even occur to them as they treat the extinction of humanity as just another dinner table conversation starter.

SNL takes a potshot at Dario Amodei “AI is not a weapon. It’s a tool. A tool for building weapons. And I urge you to urge me to stop.”

Ever since the beginning of the current AI bubble, there has always been a certain type of dipshit who takes some art posted on social media by an artist, feeds it to AI, and then comments with the slop image saying that they made it “better”. Well, now I am on the receiving end of that. In one of my papers, I (foolishly) wrote down a few comments on future directions, and now some motherfucker decides to send me an obviously ChatGPT generated manuscript with a solution to one of my problems. Unfortunately, I think their AI generated proof is itself correct, so I have to deal with it. But the manuscript is written in a way that obviously indicates that he hasn’t even read it. There’s even a section describing experiments for this mathematical question (???), complete with a “verification guide” that references an “archive” full of python scripts and test cases that he did not send me. I suspect he doesn’t even know they exist.