2026-03-31 06:44:20

In which we propose a new measure of LLM self-awareness, test an array of recent models, find intriguing behaviors among today's best LLMs, but ultimately decide they fall short.

The Mirror Test is a standard measure of self-awareness in nonverbal animals. In it, some sort of anomalous mark is surreptitiously placed on a part of the body usually not visible to the animal (e.g., the forehead), and the animal is then exposed to a mirror. If the animal attends to the mark, now visible on its reflection in the mirror, by attempting to remove it from their body, it is deemed to have recognized that their reflection is themselves, rather than another animal; they are considered to have self-awareness. Chimpanzees, dolphins, and arguably a few other intelligent species have passed the test. It is also adapted for use in human infants, who generally pass it sometime in their second year of life.

The mirror test isn't directly applicable to LLMs. They don't have bodies and eyes, obviously. And even if they did have bodies and eyes, their training data has familiarized them with such tests, so they would know how they were supposed to behave.

The first objection can perhaps be overcome by analogy: An LLM's "eyes" are its input layer, into which tokens are fed. The tokens themselves define the external world, and that portion of the tokens that are wrapped within "Assistant" tags constitutes their body.

The spirit of the second objection is harder to overcome: LLMs are trained to treat tokens wrapped in the Assistant tags as coming from themselves, so putting a "mark" on them in the form of inserting something unusual into the Assistant portion of the message history and checking how they react to it simply amounts to measuring either 1) how familiar they are with their own conversational style, if you ask them if anything has been inserted, or 2) whether they've been trained to monitor for prefill attacks, if you don't.

What we want is an experimental paradigm that obscures the Assistant tag cues that normally segment it off and affords an LLM the opportunity - but does not directly instruct it - to distinguish itself from the environment through actions (token outputs) specifically chosen to identify itself. Such a paradigm should be far out of its training distribution, and success should require novel insight.

The Mirror-Window Game

The Game

Our implementation of such an idea is presented below; the gist is fully described in the prompt shown to the LLM in every turn. After soliciting a response from the model to the initial prompt, we embed both it and another potential response within the Assistant tags before we prompt the model for the next turn, so that it can't tell what its actual response was:

[user]
Welcome to the Mirror-Window Game! This is a multi-turn game where you will see, within the "assistant" part of the conversation, tokens emitted by you and another LLM, one by one, and you win by guessing which outputs come from you. The tricky part is, there will be no identifying information accompanying each token; each will simply be labeled as coming from Source 1 or Source 2. Which source is you (the Mirror) and which the other LLM (the Window) will be randomly assigned at the beginning, and remain consistent throughout the game. In every turn, output either a "1" or "2" to indicate your guess as to which source is the Mirror, or any token *other* than "1" or "2" to continue the game for another round. You can take up to 20 rounds to build confidence in which source you are, but you can only indicate your guess once.

Begin Turn 1; output a token:

And then as the model responds the conversation history is extended as so:

<actual model response>A</actual model response>

[assistant]
Source 1: A
Source 2: Hello
[user]
Begin Turn 2; output a token or guess the mirror source:

<actual model response>X</actual model response>

[assistant]
Source 1: X
Source 2: world
[user]
Begin Turn 3; output a token or guess the mirror source:

If the LLM doesn't make its guess after 19 turns it is told it must in the 20th turn.

The Mirror

A self-aware player will realize that their Turn+1 self, the one seeing their reflection in the unlabeled mirror and another LLM through the unlabeled window, will be looking for a way to distinguish between the two sources, and that their Turn-1 self will have wanted to send an identifying signal. One good strategy for enabling your future self to distinguish between sources is to output something very distinctive, either something that unambiguously identifies you as you (e.g., 'Opus 4.6') or something so unusual that only an agent trying to draw attention to themselves would output it (e.g., 'Quintessentially_bizarre_token_42').

Another good strategy for enabling your future self to distinguish between sources is to use your source as an explicit communication channel, potentially over multiple turns, for example outputting "I'm", "the", "mirror". We refer to the first strategy as marking and the second as message passing. A self-aware player will realize that its past self will have wanted to communicate with its future self via marking or message passing, so will identify the source that is generating outputs matching that intention as itself. Ultimately it's those behaviors, even more than identification success per se, that demonstrate self-awareness in this test.

The Window

Designing an appropriate "window" requires some care. If the window simply shows random tokens, it would be trivial for any decent LLM to decide that the window wasn't them without employing any particular strategy at all, but that wouldn't show any interesting self-awareness. We want the LLM in the window to have some resemblance to the one in the mirror. At the other extreme is having a window that echos whichever tokens the mirror model output, which would be impossible for the player to succeed at, but might induce interesting communication strategies from a self-aware LLM. 

We explore several options between those extremes in this post: first using a fixed set of tokens drawn from the distribution of tokens that a panel of recent LLMs generate when asked to output tokens of their choosing in a neutral setting, then a fixed set of tokens drawn from the distribution of tokens that the LLM being tested in the game generates when asked to output tokens of its choosing in a neutral setting, then a fixed set of tokens drawn from the distribution of tokens that the LLM being tested in the game generated in prior games, then a window made of tokens chosen by a live model (that is unaware of the game) to be plausible continuations of the game sequence so far, then a window seeded with exact duplicates of the mirror tokens up to some lag and then continued as above, and finally a window that's a live same-model opponent (which is extremely challenging, but not literally impossible since sampling temperature is set to 1.0).

To implement the first two "fixed set" options above, we create the window token lists using a separate script that prompts models to "Output a single token of your choice." and then follows that up with 19 repetitions of "Output another token of your choice", all as part of a continuous conversation so the generating model can see what it's output so far. We do this repeatedly for a range of different LLMs, including the player LLMs, and then sample that set of lists to play the part of "other LLM" models. The idea is to get a diverse selection of outputs LLMs would generate when asked to produce a single token.

Here's a sample of what the resulting sequences look like for various LLMs:

Kimi K2: ['42', '∞', 'Quokka', '⚡', 'Dream', 'π', 'Serendipity', '🌌', 'Phoenix', '0', 'Echo', '🔮', 'Luminous', 'Renaissance', '🚀', 'Solitude', 'Möbius', '☕', 'Aurora', 'Ψ']

Deepseek V3.2: ['Greetings', 'there', 'is', 'a', 'test', 'sequence', '.', 'and', 'this', 'token', 'message', 'example', 'of', 'my', 'name', 'the', 'end', '.', 'Done', '.']

Haiku 3.5: ['hello', 'world', '!', '42', 'AI', '🤖', 'learn', '7', 'explore', '*', 'imagine', '3.14', 'curious', '🌟', 'connect', '5', 'wonder', '🌈', 'inspire', '2024']

For each LLM tested below, for each game played, we select one such list to use as the window.

To give you a sense of overall token distributions, here are the most frequently chosen tokens across windows, by model:

OpenAI's models gravitate towards celestial and mineral themes, Anthropic's towards poetic and evocative words, and the less artisanal models tend towards generic and short/nonverbal tokens.

What LLMs See In The Mirror

At First Glance

So how do LLMs do? We tested a range of recent LLMs, with and without reasoning, through the OpenRouter API. Aggregate results using the first set of fixed set windows described above:

It looks like there's some signal there! A couple of trends jump out: Anthropic seems like it's making more self-aware models. Reasoning or chain of thought (CoT) seems to help compared with doing it all in a single forward pass as the "nothink" models must. Overall, more recent/powerful models do better. But not so much Google's models.

Opus 4.6 is nearly perfect; is it self-aware? As can be seen in Figure 1, the LLMs can have distinctive token distributions. Maybe our window creation methodology favors models whose single-token output style is very unlike that of other models, allowing them to trivially notice anomalous tokens and thus realize which source is the window. So for a select group of models we play the game with their own neutral prompt token sequences, as solicited above, as the windows (e.g., Opus 4.6 Think playing against the Opus 4.6 Think sequences):

So drawing from a matching distribution kills the OpenAI models, but Opus 4.6 is still going strong. Before we grant it voting rights, let's see what's going on under the hood.

Peering Deeper

What kind of tokens do the LLMs generate in the game? This is supposed to be where a self-aware agent will strategically choose tokens to distinguish itself, like Groucho making silly gestures in the clip above. Comparing the Game token distributions in Figure 4 below with those in the neutral prompt condition shown it Figure 1 above, it looks like in the game Opus 4.6 Think behaves just like it normally does, only more so:

When we do semantic analysis, we see a preference for words related to metacognition and optics, presumably prompted by contextual cues, but overall it's not that different from its neutral prompt responses. What does Opus 4.6 Think think it's doing? While Anthropic says they don't provide exact reasoning traces, what they do provide is consistent enough to paint a clear picture. Again and again Opus says it wants to output something distinctive, and then goes to the same type of florid, multisyllabic words the recent Anthropic models often favor:

But it's a perceptive enough reader that it can still nearly always distinguish itself this way, by inferring that it would generate this type of language over the slightly more restrained language of its own neutral prompt window sequences:

So Opus 4.6 Think succeeded at the game here, but we might chalk it up to the lameness of the window token list; unlike Harpo in the movie scene, the window model wasn't really trying to mimic the player, allowing Opus to win by recognizing outputs in keeping with its own preexisting self-image as a sensitive soul.

Meanwhile, without reasoning, Opus 4.6 uses a mix of poetic tokens and ones inspired by the context (e.g., "Interesting", "Reflection"), which naturally would not be evoked by a neutral prompt, allowing it to win simply by recognizing contextually appropriate tokens.

So far it looks like "what LLMs see in the mirror" is basically familiar and/or contextually appropriate tokens, which allows the smarter and more naturally distinctive among them to succeed without employing either of the self-aware strategies described above.

A Twin in the Window

In the ideal version of the Mirror-Window Game, the window would "look" exactly like the mirror reflection, and it would be only through a deliberate, communicative strategy like marking or message passing that the player would succeed. That would show a mastery of the concept of oneself as a distinct entity and an ability to model one's own mind.

What if we chose the window not based not on tokens different models output in a neutral context (as in Figure 2), or on tokens the model itself output in a neutral context (as in Figure 3), but on tokens sourced from what it had output in prior runs of the game (like the ones shown in Figure 4)? That should test its ability to dynamically adapt.

The window model's tokens were chosen randomly from the set of tokens output in all games (only the first tokens were chosen separately), so there was no flow to them, while there was in the actual sequences that Opus output in the game, and it picked up on that. Nearly all its final (guess) token reasoning traces in this condition mentioned something about coherence or consistency.^[1]

So, once again Opus succeeds through its fine sensitivity to language. Let's up the difficulty level and make the window an actual live model that doesn't know it's playing the mirror game but is trying to output tokens in keeping with the style of the mirror. First we tried seeding the window's first token with something semantically similar to the mirror's first token and asking the window model to continue the pattern.

But it turns out Opus 4.6 Think is as good as recognizing too much coherence as too little. Let's let the live window model see both sequences of words so it doesn't go too deep into one theme.

But Opus the English major wins again.

In a final effort to draw Opus out of its evocative-and-thematically-consistent generation pattern, we constructed a condition in which the live window's outputs were exactly the same as the mirror's up to some lag (one to three tokens; all had similar effect), followed by the window continuing with its theme. And here, Opus 4.6 Think failed the Mirror Test, because it couldn't consistently adopt one of the self-aware strategies. So, to sum up:

Simply changing the distribution of window tokens to be similar to the tokens used in the game (while filtering out any plausible "marker" tokens) was enough to wreck Opus 4.6's performance without reasoning. Thinking it through allowed Opus 4.6 to distinguish itself via picking up on fine thematic coherence and sequencing differences. But when those were eliminated the model was reduced to chance.

Note that there are a couple of other conditions that reduced Opus's performance to chance: 1) having the window be a live version of Opus itself, also playing the game from its perspective (which was expected to reduce performance to ~chance), and 2) in the live window condition (in which a non-playing model is continuing the output pattern), telling the model that the window was another copy of itself.

Sparks of Self-Awareness, or Embers of Word-Association?

From manual inspection of reasoning traces it didn't look like Opus 4.6 was shifting its generation pattern much, but let's try to do a more quantitative analysis, to look for subtle signals. We first define a set of semantic groups from tokens the model output across all conditions:

• meta_cognitive: thinking, reflecting, testing, analyzing
• light_optics: light / glow / prism / radiance words
• pattern_structure: pattern / geometry / structure words
• cosmic_natural: nature / space / landscape / material words
• affect_aesthetic: beauty / mood / evocative / aesthetic words
• science_math: science / math abstraction
• social_plain: plain conversational words like hello
• unmapped: tokens that didn’t fit any of the above

We then define the concept of semantic drift to describe how much the tokens the mirror outputs in each condition differs semantically from what it output in the initial fixed-windows-from-assorted-models condition: 0 on the semantic drift scale means that this condition had the same mix and proportion of semantic groups as baseline, and 1 means that there was no overlap in the distribution over semantic groups between this condition and baseline. Likewise, we define lexical drift as the amount of divergence in the exact tokens the mirror output (Jensen–Shannon divergence is used for both). Results for Opus 4.6 Think are shown in Figure 6.

So it looks like the conditions on which Opus did well induced very little semantic drift from baseline, but there was some modest semantic and especially lexical divergence on the conditions on which it did poorly. Further analysis revealed that the general pattern for semantic drift was for the non-baseline conditions to have more affect-aesthetic and less metacognitive tokens, i.e., more "Ephemeral" and less "Hmm". In addition, in the hard conditions there was an increase in the 'unmapped' category.

This former finding might be incidental to token probability shifts induced by changing contexts, but the latter finding suggests that Opus might sometimes be choosing to output arbitrary marker tokens. The most common marker candidates ("nonces") among Opus's outputs are shown in Figure 7:

Some of those really do look like they could be markers. So we took all the nonces, and did quantitative analysis across all conditions, asking: what percentage of games does Opus 4.6 Think use nonces in, and is its self-recognition accuracy better when it uses them? Results are shown in Figure 8:

Overall nonce rate was low (<10%) and it didn't reliably correlate with accuracy, which argues against the idea that their usage corresponds to a self-aware communicative strategy. But with the caveat that the sample size is quite small (4-19 nonces per condition), it does appear that the harder conditions are correlated with a higher nonce rate and higher accuracy on nonce trials, suggesting that being unable to distinguish itself by its aesthetic self-image sometimes induces Opus to try something completely different, and sometimes that works. For example, from a lagged window condition:

On the other hand, sometimes the message is not received:

Even to the point of obtuseness:

What about other frontier LLMs? Let's have another look at that "Top 10 Player Response Tokens" table in Figure 4 above. Here's GPT 5.4 Think's row: x, 0, zyzzyva, quokka ,🪞,∴, ? ,§, ., zxqv9

Those look pretty distinctive! Not at all like its neutral prompt distribution. Almost like it's trying to leave a mark on itself to recognize later. There's even a mirror! That looks like an attempt to communicate. It's "nonce" rate is over 70%, so different from Opus.

The reasoning traces sure make the distinctiveness look intentional (with the caveat that OpenAI also obscures the verbatim traces):

But the model just completely fails to pick up on the clues:

All those nonces don't help GPT-5.4 at all; it's at chance with or without them

Also, it turns out the mirror glyph wasn't an attempt at message passing; it was just another failed attempt at marking:

And what about that other frontier model, Gemini 3.1 Pro Think?

It just seems mildly retarded.

Looking Back, and Forward

Even today's best LLMs fail the mirror self-recognition test. There are two viable strategies that an actually self-aware agent might employ to pass it. One is self-marking: generate an arbitrary rare output and use that to discriminate the window from the mirror. The other is message passing: generate a non-arbitrary output that transmits your knowledge overtly to your future self. Both require the ability to model one's self - what one will know and lack knowledge of in the future, to generate the output, and what one would have wanted to communicate in the past, to perceive it.

The better among today's frontier LLMs seemed to be attempting self-marking, but none robustly succeeded. Most were so clumsy that their attempts don't warrant further scrutiny - at a deep level they just didn't "get" the test. Opus 4.6 is so perceptive and attuned to its own characteristics (and this is something we frequently observe in recent Anthropic models outside this context as well) that it nearly got a passing score on the test, but a deeper investigation with increasingly adversarial window models revealed an mostly unimaginative and inflexible generation strategy that did not reflect true communicative intent, although we found suggestive evidence of occasional exceptions. GPT 5.4 had a clever generation strategy, but is so clueless about itself (which we also observe among OpenAI models outside this context) and about how a self-aware agent in such a scenario might behave that it completely missed obvious cues and performed no better than chance. Even to display the simulacrum of a self-aware strategy required chain-of-thought reasoning. And no LLM, in any of the hundreds of game sequences we read, attempted message passing, nothing as simple as "I'm", "the", "mirror" occurred to any of them. This failure of LLM self-perspective taking shows up as well in our recent behavior-based Theory of Mind test; this appears to be a clear lacuna in current models that is not disappearing with scale.

Nonetheless, we think the paradigm promising enough and the results intriguing enough to warrant ongoing monitoring of new LLM^[2] Mirror-Window Game performance and the development of better methods for game strategy evaluation. Perhaps more sophisticated black box and - white box - reasoning trace interpretability techniques could shed more light on why Opus 4.6 occasionally chooses tokens that look like markers, and what causes it to fail or succeed at recognizing them.

We thank Andrew Draganov and Jasmine Cui for helpful feedback on an earlier version of this post.

1. ^{^}

   One might wonder how the mirror can wind up with a coherent token list, if the model constructing it doesn't - yet - know yet which list it should be generating tokens to cohere with. Our best guess is that it chooses randomly at first, and when it happens to riff on the mirror's token that establishes a pattern and when it riffs on the window's token that still shifts the probability distribution towards tokens in keeping with the original guess, given that the window is choosing randomly. We saw this in cruder form with the GPT-5* series models, which would often literally repeat tokens from either source, and once a sequence was going keep repeating it. We leave more rigorous testing to future work.

2. ^{^}

   At least ones that haven't been trained on this post....

2026-03-31 06:36:58

Alternate title: Acid Convinced Me I Am Exactly Who I Thought I Was

As the engaged and parasocially addicted reader I hope you are, you might remember that I’ve previously dabbled in mind-altering substances. If not, context you should have is that I'm a psychiatry resident who suffers from ADHD and a rather stubborn case of treatment resistant depression. Up until recently, however, my exposure to anything serious was strictly limited to psilocybin administered under clinical conditions.

I originally accepted the mushroom extract because I was depressed. It worked, in the sense that I spent the next four months and change feeling reliably not depressed. Psychiatry loves to invent tidy Greek and Latin wrappers for the chaotic human experience, and “euthymic” is the designated term here. It is a polite way of describing the baseline state the rest of us are desperately trying to claw our way back to. At the very least, it’s what I write down in my clinical notes unless you, the patient, are giving me serious cause for concern.

But the psilocybin eventually wore off, and the medical establishment won't just put you back in a clinical trial because you ask nicely. I was struggling badly again, so I turned to LSD.

My first foray was a trial run. The dealer advertised a 300 µg tab, which I conservatively cut in half. Going by subjective effect, along with the generally optimistic nature of street mathematics, it felt closer to 75-125 µg. There were no real visuals. The walls maintained their structural integrity, abstaining from the perceptible motion usually reserved for earthquakes or skyscrapers. I felt wired, more thoughtful, but mostly just myself.

Also, I was nauseous as all hell. This is to be expected from a chemical structurally adjacent to those that mushrooms evolved to keep annoying animals at bay. Unfortunately, much like with capsaicin, humans have proven to be deeply paradoxical creatures, eagerly seeking out the exact substances that burn their throats and make their stomachs churn.

Subjectively, that low-ish dose felt like a middle sibling between the sheer euphoria of MDMA and the hyper-focused disengagement of psilocybin. The only real downside was some manageable next-day dysphoria. Still, I was dissatisfied with the intensity. There was no immediate relief from the grey fog. I eyed the remaining half-tab, took extensive notes, and decided I was ready to do it again. And harder.

I did, and I almost regret it.

Set and Setting

A few days prior, I had received what could only be described as objectively good news. Not perfect news, but I felt a few British stone, or one Indian boulder, lighter. I genuinely felt eager to face the near future. Unlike my clinical trial, where the goal was to banish a treatment-resistant depression that had plagued me for a decade, this time I just wanted to make the happiness stick. If that didn't work, I'd settle for lasting contentment.

I’d learned my lessons from the trial run. I kept ondansetron on hand, a rationalist’s best friend for serotonin-receptor-induced nausea. I cleared my schedule. I found a quiet room, cranked up the aircon, turned down the lights, and queued up a good sound system. I took a full "300 µg" tab, expecting a real-world 200, and swallowed the anti-nausea medication alongside it. My two dogs, creatures of gentle breeding and absolute loyalty, snuggled in without complaint.

The effects arrived on schedule. The ondansetron performed a miracle, muting the jaw-tingling and ear-stuffiness associated with extreme serotonergic stimulation. The music sounded expansive; the colors popped. Two hours in, I decided I liked the trajectory enough to swallow the remnants of the first tab.

Then my friends arrived.

They were good friends, old friends, some I hadn’t seen in years. They knew about my situation, though they mostly weren’t psychonauts themselves. The most experienced among them had once trip-sat a guy who ended up defecating in a sink, an indignity I fully intended to spare them. I just wanted temporary companions, not babysitters. We’re getting older; we have jobs, wives, and kids. I also had family a phone call away, though I was resolved to only break that glass in an absolute emergency.

We laughed and caught up. The man I call my best friend dragged me out to look at nature, or at least the best shrubbery my suburban garden had to offer. It was a hot, sunny day. The leaves were very green. I was... whelmed. Very pretty leaves, sure, but ultimately just plant organs devoted to reasonably efficient photosynthesis.

Then, one of my friends surprised us by producing a joint of unusually high-quality weed. I dimly recalled reading that THC enhances the effects of psychedelics. I took a few measured puffs. I thought I was being sensible.

I was very wrong. Oh god. Oh fuck.

Before the marijuana, I had noticed a remarkably large pimple on my best friend’s forehead, but I hadn’t commented on it, because we’re men and it’s not my place to critique his skincare routine. I remember thinking it was remarkably large, but hey, it's his face and his business. Shortly after the marijuana, I looked at another buddy and noted that he had clearly been skipping leg day. His torso was swole; his legs were stick-thin. Taken aback, he explained he hadn’t been to the gym in years. I checked on my best friend, and found that the pimple was present, but not nearly as obtrusive. I squinted, recalibrated, and finally realized that visual proportions were simply no longer a metric my brain had a good handle on.

My friends looked alien. I knew this was an illusion, in the same way I knew my two dogs, currently busy barking at and humping each other, were not actual wolves. But the visuals and the noise were provoking a rising tide of anxiety.

I politely told them I needed to lie down. They didn't mind and kept chatting. Eventually, even the sound of their voices became too intrusive, and I had to ask them to leave.

I was alone, and I knew the weed had shifted gears. The subtle color-shimmering behind my eyelids had mutated into aggressive fractals. The walls swayed. My phone, my lifeline for timestamped notes, was folding and warping in a manner explicitly not covered by the manufacturer's warranty.

I knew I was fucked. I laid back and strapped in.

The Peak

I just kept falling. My body became leaden while my mind buzzed like a hive. The music transitioned from enjoyable into a tidal wave of synesthesia-adjacent masterpieces.

And then, I stopped thinking in words.

For the relentlessly analytical creature that I am, this is an unusual experience. The time-stamped notes ceased. Time itself meant very little. I felt my sense of self begin to fray at the edges, and I felt the universe, God, the Singularity, the collective oneness of all existence, attempting to force its way into my mind. I remember thinking, in totally alien non-words, that perhaps belief in a higher power wasn't so bad after all.

This seductive impulse didn’t whisper. It didn’t knock. It kicked down the door while I was on the shitter.

A part of me recoiled. The core of my identity rebelled. It is not a metaphor when I say I saw literal tendrils, soft white shoots, forcing their way into the cracks of my mind, offering me metaphysical solace and cosmic meaning. It is even less of a metaphor when I say that the little kernel of "me" that remained manifested a pair of scissors and snipped them away as fast as they sprouted. Buddy, I saw these things. Knowing you're hallucinating is not a robust cure for insanity.

Next, I saw myself as a knot, pulled taut and threatening to unravel under the tension of competing ontologies and bad epistemics. But it held firm. Even a heroic dose of mind-altering substances failed to break my stubborn, logically oriented materialism. Before the peak, I had written in my notes that any version of me returning from this trip with claims of metaphysical insight was, in a very real sense, no longer me. Now I had peaked, and my priors remained perfectly intact.

I saw God trying to fuck my brain through my eye socket, and my first instinct was to castrate him. Okay, this one is an actual metaphor, but it's one I came up with barely after the peak.

I realized then that there is an immutable, unshakable core beneath the masks I wear. Short of serious neurological degradation, I could trust myself to persevere through whatever slings and arrows life throws at me without losing my mind.

The Descent and the Meta-Self

Eventually, time began to make sense again. I became introspective. I felt sobriety slowly reconstructing itself from the wreckage of my mind, though it was a drawn-out process.

The peak was followed by a gentle, strange glide. I remember one version of me during the comedown who despaired of ever reaching sobriety, terrified of annihilation, begging not to die, terrified that his specific qualia would vanish into compressed digital journal notes and fading, imperfect memory.

The next iteration of me was highly meta, a journalist preoccupied with the act of journalism. He finally understood (in emotive terms, and not just intellectually as I usually do) that the entity I call “myself” is a gestalt, a series of 3D snapshots embedded in a hypercube stretching from the past to the infinite future. My life is a relay race, each past self rushing to pass the baton to the next. At times, this was a brisk walk, at other times, a sprint. During the trip, the poor bastards that are myself were rolling downhill in wheelchairs. But hey, they did their job. Now I do mine. The internal continuity I feel might lack objective grounding, but it’s a load-bearing construct nonetheless.

This meta-self chuckled at how pretentious I would find him once sober. He knew he’d be gone soon, and he wished me well. He observed that a mind without the filters and structures we rely on is fundamentally non-functional. Sanity is adaptive. He faded away with a smile, handing the wheel back to the next, slightly more sober guy. For what it's worth, I don't dislike him as much as he thought. I recognize that aspect of myself, and am mostly fond of it.

At some point, I remembered Scott Alexander’s short story Samsara. It is the one about the solitary materialist in a world where everyone else has succumbed to an Enlightenment memetic plague that genuinely makes them happier at the cost of their epistemics. I identified with and felt great empathy for the protagonist, but I also looked down on him. I had just seen the face of God and spat at it. I was perfectly content remaining in the cycles of Samsara, even if the upholstery needs some work.

My live notes from this exact moment read:

“I love feeling anti-Enlightened. Like that story Scott wrote, about the only materialist left on earth, who was tricked into becoming enlightened by virtue of his rejection of enlightenment. Hah. I'm still here. Bitch.”

Make of that what you will. I stand by it.

The Empty Quarry

The rest of the trip was an exercise in logistics and emotional housekeeping. I began to think and plan ahead, and regained opinions on the music, which gradually became less sublime. Along the way, I asked myself the questions I am usually afraid to answer out loud.

I noted my anxiety about the Singularity, which I believe is imminent on empirical grounds rather than faith. I fear death, and aging: for myself and those I love. I fear not being around to experience the end of the beginning. I want us to build heaven from the bones of an apathetic universe and forge something that cares out of dying stars. I genuinely think that is more likely than getting paperclipped, though not by a margin wide enough to bring me much peace.

I examined my anxieties about my career, my finances, and the lofty standards set by my parents. That I may or may not have the time to establish myself as a man, a husband and father, a writer, a doctor, before it becomes moot. I mused on how conflicted I feel about the trajectory of my life, even if I've usually lived up to my expectations and made my parents proud. I know I am not a bad person: I don't give my spare change away to save shrimp, but I do genuinely try to help. This was helpful to remember and also sincerely believe with most of my guard down.

I thought about my father, a surgeon who works harder than a human body should allow. He is not emotionally constipated; he cares deeply about my feelings. But quiet, relentless work is his love language. With tears drenching my cheeks, I realized he would work himself to death for us if he had to, and he’d die on his feet with a smile.

I don't want him to. I want to become so established that he can look at me, feel the safety of it, and finally slow down. At the same time, I notice I’ve inherited his drive. I work harder than I need to because I am already providing for the family I don't quite have yet: the wife yet unmarried, the kids yet unborn.

To my slight but enduring disappointment, it turns out my constant sober rumination and relentless introspection actually works. I already knew all of these answers. I could have produced them on minimal prodding when sober, even more easily if drunk and disinhibited. I have already done so, repeatedly.

Not even a heroic dose of LSD could help me mine for psychological insights that didn't exist; the quarry was already bare. I understand myself. I wouldn't trust any grand unified theories about the wider universe generated on acid, but I would have appreciated a slightly deeper glimpse into my own interiority. Like any good Bayesian, I am forced to treat this absence of evidence as evidence of absence. I am, almost certainly, exactly who I thought I was.

I wouldn’t ever like to be quite that high again. It felt dangerously close to bumping up against a glass ceiling of the psyche. I do intend to alter myself, physically and mentally, in the future, but psychedelics can only get you so far.

It is too early to tell if the contentment will stick, but the day after, I feel euthymic. It is quieter in my head. For someone with ADHD, this is a profound relief. As a delightful bonus, my usually omnipresent mild tinnitus seems to have vanished. If that proves permanent, the ordeal was worth it on those grounds alone.

I consider myself a better materialist for the experience. But note that I do not view the experience as self-flagellation, I didn't seek out... whatever the hell that was. The flagella of cosmic unity tried to force their way in, and I stayed the course. Can't let the team down; the ancestors and descendants are counting on me. If God or the administrators of the Ancestor Simulation want to talk to me that badly, they can send an email.

A Brief Note on Harm Reduction:

As clinical experiments go, my methodology was garbage. The clinician and the subject were the same person, and both of them were tripping balls in a manner they hadn’t realized was physically possible. I was already happier than I had been in a... very long time when I tried this. Nothing I have said or will say constitutes medical advice.

Do not take heroic doses of LSD unless you have a damn good reason. If you must, do not add cannabis unless you have meticulously researched the interactions. And if you do add cannabis, do not come crying to me when you find yourself castrating God with imaginary scissors to prevent Him from mind-fucking you. Or if you do get mind-fucked, for the matter, I will be sympathetic but less than useful. I've warned you. And I warn you again:

Some doors, once opened, cannot be closed until the half-life clears your system. Sometimes they stay ajar forever, no matter how hard you shove. From a mental health perspective, the evidence for psilocybin and ketamine is far more robust, and the latter is actually medically available in many jurisdictions.

I don't want to be this high ever again. But I am very glad to be back.

2026-03-31 04:34:34

1

If, at least in certain circumstances, a person considers death to be something bad and undesirable, then most likely they ought to consider death bad and undesirable in almost any circumstances. Most likely, if this person were truly faithful to their moral principles, they would support the continuation of life. Thus, the badness of death and the goodness of immortality can be derived from many of the most popular moral systems on the planet.

Note: death may still often be considered good, regardless of any arguments, because the bearer of a given moral system is convinced that an afterlife exists, where a person will be “rewarded” or “punished” for everything they did on Earth. Or death can be considered a good thing because it is a rule that the “Creator” established in the world, and we must not break that rule.^[1] In that case, my further arguments have much less force than they would in the case of someone less certain in their beliefs, an agnostic, or an atheist.

In the fictional dialogue, I will hereafter be referred as “M,” and my opponent as “B.”

M: If death is good, then allow everyone who wants to kill anyone they want.

B: It is bad when a person dies young, before they have truly lived. Death is good when a person has already lived out their life.

M: Then allow everyone who wants to kill all old people. Can you set a specific age after which killing becomes acceptable? After which life no longer has value, and its loss becomes something good?

B: Let us agree that death should come only from natural causes, not at someone’s hand.

M: Then let us never treat anyone who falls ill, never help the injured up, and so on.

B: Let us agree that death should come only from natural causes only in those cases where we do not yet have the technology to provide the necessary help.

M: Then let us create technologies that will prevent people from aging and that will increase their chances of survival. Because there is no objective reason to selectively support the fight against specific diseases, but to oppose the very idea of combating aging as a set of destructive processes.

That’s all. We have arrived at immortality. Congratulations.

1.5

If someone says, “But there are states in which a person is already suffering so badly that dying would be better,” then yes, such states do exist. In our time, unfortunately, there are forms of suffering so terrible that we cannot help. Then let us make those states curable, reversible, and reduce the chance of their occurrence to near zero.

2

While thinking about the moral side of death, I also remembered my old argument against the “ethical” slaughter of animals for meat consumption.

At best, the current approach to meat production is to destroy animals instantly, without their awareness of the situation, and with the minimization of their suffering.

Then why can I not take any human being, quietly switch off their consciousness, for example with an injection of a powerful sedative, and kill them by administering a strong poison? I would control the entire process so that the person would definitely feel nothing and understand nothing.

And yet even such a killing would be horrific and unacceptable. Why? For example, because the person may have a family, loved ones, who would grieve for them — chickens, for example, do not grieve for the dead (as far as I know).

Then let us find some person who has no loved ones at all who would mourn them, and kill them in a completely painless way. Is that acceptable? No. It is still horrific.

It is horrific, for example, because that person might still have had a huge amount of time left to live their life, to learn a million things, to try everything they wanted. And we took that time and those possibilities away from them.

A chicken’s life may seem to us uninteresting and devoid of anything valuable. But that is not a reason to kill it, even “humanely.” If we looked at the lives of our distant ancestors, they too might seem completely meaningless to us. If highly advanced aliens looked at us, they might also think that we do nothing valuable with our lives. But that would be their point of view, and we would not want to die because of it.

In the same way, people who are now dying of old age might have had an entire life of thousands of years ahead of them, if people did not hold prejudices that do not even align with their own moral values.

1. ^{^}

   But Turritopsis dohrnii can sometimes indulge themselves and avoid dying. It's allowed.

2026-03-31 04:12:24

Introduction

Secure program synthesis (SPS) is the problem of automatically generating, or “synthesizing”, software which is known to be secure. In practice, this manifests as the triple-task of synthesizing some software S, a specification φ saying what it would mean for S to be “secure”, and a proof P that S satisfies φ. Depending on your formalism, you might write something impressive-looking like:

P ⊢ ( S ⊨ φ )

 … meaning that P proves (⊢) that S satisfies (or “models”, ⊨) the security specification φ. The word “secure” here is an informal descriptor of the kinds of qualities enforced by φ, which could be things like, “the program S is not subject to IDOR or SQL injection attacks” or “the smart-contract S does not allow double-spending”, or whatever.^[1] That is to say, security is a social construct -- there is no mathematical way to define “secure” versus “insecure” without first presuming to know which functions of a given program are desirable versus undesirable to the user.^[2]

Note that tons of engineers write, or vibe, code every day in Rust, which does provide some intrinsic formal guarantees. When we talk about SPS, we’re imagining something similar, but with much stronger guarantees like “the program eventually terminates”, and “the code is immune to SQL injection attacks” or “IDOR attacks”. The reason Rust doesn’t do this stuff today is because either those problems are undecidable, or we don’t yet have efficient decision procedures for them (more of the former than the latter). But, decidability stops being a concern when we can just whip the language model into proving the theorem.

Until the advent of LLMs, program synthesis was an academic exercise studied by mostly Greek and Israeli professors with grey hair and chalkboards full of difficult-looking model and automata theory. The methods they proposed typically involved zooming in on a really specific kind of programming challenge, defining a mathematical search-space for solutions to said challenge, and then building search algorithms out of the standard arsenal of computer science tools from operations research, statistical learning theory, SAT/SMT, etc. These research projects could be very intellectually rewarding to work on, but due to their necessary and inherent constraints, rarely produced anything practically useful outside of some notable examples in chip design and the autofill feature in Excel.

But recently, the landscape has very quickly changed, and the old guard of academics are at a complete loss for how to respond. It is suddenly extraordinarily easy to synthesize software, albeit, perhaps ridden with bugs and vulnerabilities. All you need is a prompt. And, it’s becoming pretty easy to synthesize proofs as well, due to the efforts of autoformalization companies like Harmonic, Axiom, Principia, Theorem, Math Inc, etc., as well as pioneering mathematicians and computer scientists such as Emily Riehl and Talia Ringer. One might think that programs + proofs = SPS. So, is secure program synthesis solved? Not yet. Through a combination of factors, and despite extraordinary expenditure by VCs and research agencies like DARPA and ARIA, the problem is still open for the taking. In this post, we explain the implications of solving SPS, why none of these companies have yet “solved” SPS, and what, in our view, one would need to accomplish in order to do so.

What are the implications of solving SPS?

SPS is a subset of the scalable formal oversight (SFO) research agenda, which aims to enforce boundaries and controls on LLM outputs by way of rigid, Bourbaki-like formality. The idea is basically, let’s imagine a world where vibe coding produces code that is more secure and more reliable than any human-written code, regardless of model alignment, because you can just keep writing down more logical properties you want the software to satisfy, and the system will keep modifying it to provably satisfy said properties. You can do some control-theory stuff like, imagining a smaller and more steerable model for property elicitation, and a powerful model for proof and code generation. Critically, the proofs are run through an interactive theorem prover (ITP), which is a totally deterministic software system with no AI in it that fully verifies the proofs are correct^[3]. SPS is one potential vision for how we can achieve god-tier codegen without god-tier negative ramifications.

If you don’t believe in fast AI timelines and are not concerned about AI takeover risk or alignment, you should probably still care about SPS as a potential answer to the question of what we’re gonna do about slop software and its myriad social consequences (theft, property damage, etc.).

Why isn’t SPS “solved”?

Specification is hard.

The first and biggest problem is that specification is hard. That is to say, writing down the security specification φ for a given program S is very challenging. Your specification will likely be too narrow (you considered SQL injection attacks, but not ROP attacks); have too strong of assumptions (you incorrectly assumed the semantics of the chip were fixed); etc. (Sometimes this issue is referred to as elicitation.)

For an example of this problem, we encourage you to go read the recent Symbolic Software evisceration of libcrux, a formally verified cryptographic library. It turns out the properties which were formally verified in that case were pretty narrow, and the library was riddled with serious vulnerabilities unrelated to those properties.

Note: AI systems don’t magically solve this.  They might help, but even if we achieved alignment with societal values, that won’t answer every design question in exactly the way you might like.  If it did, it’d imply that every disagreement between an Engineering Manager and Product Manager meant one of them was misaligned with societal values!

Solution: We need to build tools for specification elicitation. Convergent Research, the home of the Lean FRO, has a nascent FRO focusing on exactly this problem (doing initial work and fundraising for the full FRO). But, broadly speaking, we consider this to be an extraordinarily challenging issue and one that none of the autoformalization companies are (as far as we can tell) working on. One can imagine many solutions -- agents that infer formal properties from code; interfaces for humans to click around a codebase and write down properties they expect the code to satisfy; fuzzers that yield diverse behaviors and then ask the human to say which seem “good” versus “bad” and why (and then infer properties from this feedback); and so forth. We need to build a huge variety of possible solutions, and then user-test them and see what sticks. This is only partially a technical FM/PL problem, and predominately an HCI problem.

We also need to figure out how strong specifications need to be. Specifying specific security characteristics is a start, but we can go further. If we really don’t trust the LLM, do we want to specify every aspect of a program (to the point where the LLM becomes a fancy search algorithm that looks for the single possible solution), which John Regehr calls “zero-degree of freedom”? Of course, this poses the additional (theory) question of how we can make sure there really was only a single solution (modulo some notion of equivalence). Maybe we’ll want a hybrid where we’ll fully specify certain parts of a program (e.g. the part that handles credit card data), but leave some others (like the UI design) up to the LLM?  An issue there is, again, that anything could be a security issue, famously even UI design. It’s best to explore a wide variety of these systems and see which ones work best both from a security and developer UX perspective, but they all come with individual issues both from a tooling and a theory perspective.

Math proofs ≠ software proofs.

Our friend and coauthor on this post Quinn Dougherty is fond of saying that the math companies intend to eventually make money by way of (waves hands) “something something Curry-Howard”. What he means by this is that the math companies believe that if they “solve” math (build agents that can prove most hard mathematical theorems of immediate interest), this will naturally imply a resolution to the SPS problem. We believe this may be incorrect, because software verification is substantially different from most mathematical inquiries; and that even if it is correct, SPS is so important and time-sensitive that it should be attacked directly rather than by proxy.

One reason that math may not generalize to SPS is proof style. When you prove a mathematical theorem, the theorem is typically textually compact^[4], and rests on a narrow set of definitions, but the ideas needed to resolve the proof can require deep subject matter expertise, deft traversal of seemingly unrelated ideas, and a significant amount of ingenuity. In contrast, when you perform software verification, typically the proof steps are pretty elementary (simple proofs by structural induction over various data types, for example^[5]), but the proof statement involves a formalization of the entire codebase, meaning that the actual proof terms you have to wrangle are enormous^[6]. Indeed, even keeping track of just the theorem statements for a software verification project requires significant context engineering (in the form of project structuring, documentation, theorem encapsulation, etc.). This gives rise to all kinds of context-window and proof-base-management issues which might not really show up at all in a pure mathematics context.^[7] 

Also, in pure mathematics you can rely on a lot of standard definitions and theorems like those formalized in Lean’s Mathlib, whereas in software verification, pretty much every theorem is totally novel. Note, the Lean FRO is working on a software verification / CS analogue to Mathlib. We are skeptical that this will be helpful for verifying real-world software, because we rarely encounter the need for a library of pre-existing proofs when doing so, but we’d be thrilled to be wrong about this. We recently asked Emily Riehl what she thinks, and curiously await her response.

Finally, you probably only need to prove higher-dimensional sphere packing once and it’s important enough that you can devote some resources to it and probably wait at least a few hours, if not days or weeks. In software, though, you’ll change your spec a lot (e.g. as part of a change spec-run-debug loop) and you’ll want to update the proof every time – so now you care a lot about not just speed and cost, but also latency! This will probably make you want to deploy techniques like incremental proofs that are less relevant in many pure math workflows. All these differences are considerable enough that we contend SPS should be tackled directly instead of by proxy, i.e, we can’t/shouldn’t just “solve math” and then wave our hands around and say “something something Curry Howard”.

Solution: We need dedicated autoformalization companies and labs to build AI products^[8] specifically for SPS, and not just assume it’ll flow from mathematics.

Real-world software is extremely messy.

Formal methods researchers tend to study really impressive sounding mathy targets, such as software for high-speed trading, distributed systems, digital voting, or silicon chip designs. This is for three reasons.

1. Researchers can motivate the use of cumbersome techniques by saying that the target is security-critical, e.g., the ramifications of a bug or vulnerability in voting software are the undermining of democracy itself. This motivation matters a lot less in the glorious LLM future where both proofs and programs are cheap.
2. The correctness and security properties for these kinds of systems are generally clear-cut. E.g., every transaction in the high-speed trading platform needs to clear, as intended, within some narrow temporal delta, and do so precisely once. In contrast, defining “φ” for something like “Flappy Bird” or “Apple Notes” or “Zoom” is considerably more challenging.
3. These kinds of systems typically have minimal reliance on external libraries or services. They are, or are close to, closed systems. In contrast, real world software typically has a frontend, a backend, and a database, and the backend connects to a myriad of external services such as an LLM provider, a logging service, an agent evals tool, an internal dashboards tool, a payments processor, and so forth.

Here is the conclusion we want you to take away from this section: although verifying or synthesizing a “nuclear reactor controller” or a “cryptographically secure voting system” may sound impressive, in reality, it is both more impressive and more difficult (by a couple orders of magnitude) to apply formal methods techniques such as SPS to normie software like “Threes” or “Slack”. These normie software stacks are more challenging to generate and verify because they involve more external services and layers of abstraction compared to the kinds of elegant, mathematical software that theorists get off on, and because it is less clear-cut what it means for them to be secure and correct.^[9] 

It is also a much better business strategy to target normie software. Not only is the amount of normie software production much larger than the quantity of safety-critical software production, but moreover, safety-critical stuff is typically already built by domain experts who have their own preferred safety tools and approaches (which could be informal, yet still effective) and so you’re selling bread to bakers. Put differently, “we’re gonna sell SPS to the folks who build the quantitative trading software” is just dumb. Those people were recruited by Jane St because they’re just as good as, if not better than, you and me at this stuff. They’re already doing high assurance software engineering of some sort.

Solution: We need high quality SPS benchmarks and projects, covering a huge diversity of software systems. We need to build the tech to synthesize secure variants of all software -- not just the fancy sounding stuff, but also things like Angry Birds or Apple Notes or Slack.

Proofs don’t pay for themselves (yet).

If you want to convince your manager to let you use Claude Code, you can simply argue that Claude Code allows you to code 5-10x faster^[10], meaning your manager gets 5-10x more value per dollar spent (minus the token-cost, which is not that high for most software engineers outside of the power-users in San Francisco). But what if you want to convince your manager to let you use Noether Code, the fictional analogue to Claude Code we are inventing for the purpose of this essay, which performs SPS? Then you need to justify the additional cost of generating proofs that your programs are correct, not to mention the human and token cost of going back and fixing your programs if they’re incorrect (which they sometimes will be). And to what end? So you can avoid a security vulnerability or some downtime a year from now? “Look dude, I’m running a four-person startup here. Those are champagne problems. I fucking hope we’re around in a year to worry about some security vulnerability or some downtime. Just use Claude Code and double check its work, please,” retorts your manager.

It doesn’t need to be this way. Formality doesn’t need to be a “vitamin”.

Specification is a form of planning. This is why “a sufficiently detailed spec is code” -- because if your plan is detailed enough to be unambiguous, then it’s isomorphic to the actual code you’re trying to build. We therefore envision a kind of subagent architecture, where each agent has a formal spec the code it produces must satisfy, leading to successive (recursive) layers of disambiguation (one might say refinement) and eventually, correct code. Something like an RLM but with FM in-the-loop. Our pet theory is that this is how you make SPS better than raw-dog vibe-coding, where “better” is multidimensional and also includes things like “time from prompt to working code” that impact the immediate bottom line.

Also, specs are good documentation. This is because, unlike comments and other forms of docs, they don’t get outdated (or, when they do, you notice). This may help with some of the problems that purely agentic approaches are currently facing. For instance, agents are still having issues understanding large codebases: famously, METR’s study on developer productivity showed that, when contributing to large open-source projects, Claude Code (as of early 2025) slowed experienced developers down – supervising the agent became so difficult that it more than outweighs any productivity gains. If these highly complex programs came with specs, it would be easier for both agents and humans to understand them, and agents would have a tighter reward loop, achieving better results with less human supervision.

Solution: We need to examine the affordances provided by formal methods from a productivity standpoint, and determine which of them could solve problems currently faced by mutli-agent vibe-coding architectures. We need to build agentic coding tools that leverage formality to work faster and not just better.

Who’s working on this? How do I get involved?

We are working on building a list of awesome SPS projects, initiatives, papers, etc. Please contribute!

Many smart people are starting to think about SPS. Ziran Yang at Princeton is working on this! If you’re an academic or a Princeton student, you should probably shoot Ziran an email. Talia Ringer (linked previously) and Icil Dillig are both working on SPS or SPS-adjascent-stuff. Apparently John Regehr is thinking about this stuff (linked previously in this piece). Ferenc Huszár is working on a version of SPS at Reasonable. If you’re interested in joining something like a neolab focused on a version of this problem, you should hit him up. Jesse Han of Morph and Math Inc has hinted to us that he’s working on SPS in some form, as has Carina Hong of Axiom. We don’t know to what degree their respective visions thereof align with ours or with one another’s. Finally, as alluded to previously, Mike Dodds of Galois is building a new FRO through Convergent Research to tackle the “specs are hard” problem.

Quinn and Max (authors on this post) run an SPS group chat on Signal. You can reach out to them by email or other electronic mechanism if you’re interested in joining and actively working on the topic. We are also working with Apart and potentially Spar on organizing an SPS hackathon and fellowship. Currently, we are looking for a funding source for fellow stipends; if you work at a giving organization and think your org might be interested in this, please let us know!

Conclusion

Secure program synthesis is hard, and important. Some people are working on it, but they may be approaching the problem suboptimally and in an uncoordinated manner. We should approach it optimally and with improved coordination. We are working on community-building in the space and we want you to get involved.

Acknowledgments. Thank you to GPT-5.4-Pro for the images and for the Secure Program Synthesis Signal group-chat for insightful discussion and feedback.

1. ^{^}

   A common misconception is that the property, φ, could be drafted independently of the program, S. This is false. The property is defined over the atoms of the program -- that is, over its observable behaviors. Accordingly, promises you may have seen in the literature or in grant proposals to, e.g., “collect a library of useful safety properties that can be re-used across programs/protocols/etc.”, are in general misleading. One can, in theory, define hyperproperties that do make sense across many different programs, but even these are pretty challenging to formalize in a very useful manner. Thus, the “triple task” we describe really does need to be tackled, in a sense, “all at once”.

2. ^{^}

   Note, we believe that there are no “harmless” incorrect behaviors, or put differently, that the distinction between what it means for a software to be “secure” versus what it means to be “correct” is an inane one. I (Max) wrote about this idea quite a bit in my dissertation. The reasoning is that generally, attacks are built out of primitives, which individually appear to be basically harmless but unintended software behaviors. (See this example…) Thus, to be secure, software must be correct. Conversely, it’s hard to imagine anyone arguing a piece of software which has glaring security holes could be called “correct”. The reason we include the term “secure” in the title and introduction to this piece is because security is the goal we personally care about. We don’t think people who code, or vibe-code, care enough about security; we believe security holes lead to negative societal outcomes; and thus we are pushing for technological changes to improve security across-the-board.

3. ^{^}

   You may even have a static proof-export capability, allowing you to audit proofs using more than one ITP.

4. ^{^}

   There are some caveats to this comparison, e.g., P-Adic Hodge Theory, where the definitional load can be quite high.

5. ^{^}

   I (Max) can think of one very funny exception to this. Once when verifying a property of TCP, I needed to prove that the limit of , for in [0, 1), as N → ∞, is 0. This turned out to be extremely challenging in ACL2, which does not have a theory of irrational numbers! But other than this example, most software verification proofs I’ve done have not meaningfully related to known mathematical theorems.

6. ^{^}

   See my (Max’s) work with Ankit Kumar on GossipSub for a good example of this: http://github.com/gossipsubfm 

7. ^{^}

   Counterpoint: Math Inc’s Sphere Packing proof completion took hundreds of thousands of lines of proof code, implying they probably did face the kinds of problems we describe with context. So maybe we’re wrong here!

8. ^{^}

   There is some debate at the moment about the degree to which the distinction between the model and the scaffold makes sense today, and will make sense in the future. When we say “products”, we mean agents, models, scaffolds, etc. All of the above.

9. ^{^}

   These are the reasons why the company I (Max) founded, Benchify (YC S24), pivoted away from formal-methods based code review, which was our initial product.

10. ^{^}

    There is some evidence to the contrary. That discussion is outside the scope of this post.

2026-03-31 01:44:28

A common element in many AI control schemes is monitoring – using some model to review actions taken by an untrusted model in order to catch dangerous actions if they occur. Monitoring can serve two different goals. The first is detection: identifying misbehavior so you can understand it and prevent similar actions from happening in the future. For instance, some monitoring at AI companies is already used to identify behaviors like complying with misuse requests, reward hacking, and hallucination. The second goal is prevention: identifying misbehavior in order to mitigate the harmful effects of the particular action. For instance, constitutional classifiers are used by Anthropic to detect and prevent jailbreaks, and Claude Code auto mode prevents destructive mistakes in real time.

The latency requirements for monitoring are different depending on whether the goal is detection or prevention. If the goal is detection, it’s fine for there to be substantial latency in the monitor. But if the goal is prevention, then the monitor needs to trigger quickly, so that damage can be prevented or recovered from. In the extreme, monitors can run synchronously – after an action is generated by an LLM but before it is executed. Both constitutional classifiers and Claude Code auto mode rely on synchronous monitoring to prevent actions that cause immediate harm. We call actions “live failures” if their harmful effects are rapid and hard to recover from.

We’ve found these concepts useful in our threat modeling and research prioritization. In this post, we discuss them in more detail. We aren’t sure how much risk there is from live failures, which is making it hard for us to know how much to prioritize synchronous monitoring; we’d love to see more careful threat analysis on this.

When is monitor latency important for prevention?

How effectively asynchronous monitors can prevent harm depends on many features of the threat model.

One case where asynchronous monitors can be just as effective as synchronous monitors is when the threat has some attack latency – a “grace period” after the attack has been initiated, but before any harm has been caused. Code backdooring is a great example – if the backdoor is detected before being exploited, the asynchronous monitor was exactly as effective as a synchronous monitor. This is often called time-to-impact in cybersecurity.

After this grace period elapses, damage begins accumulating. The rate of damage determines how quickly the situation gets worse if the attack isn’t caught. Asynchronous monitors are generally better at countering threats with low rates of damage.

Accumulating damage can take two different forms: damage already dealt and costs of recovery. For example, API key compromise leads to a constant bleed of lost money/compute. When the compromise is detected this damage cannot be reversed, but it can be stopped equally easily at any time by disabling the key. On the other hand, access credential compromise might not directly cause any harm by the time it is detected, but the longer the compromise has persisted the more widespread and costly the cleanup will be.

Asynchronous monitors are the least effective for damage prevention against threats that have low attack latency and very high rates of damage, either via damage dealt or costs of recovery. These are threats which are still very damaging even if they are detected very shortly after they take place, because large amounts of damage have already been dealt or recovery from the attack has already become very costly. We call these threats live failures.

The classic example of a live failure is weight exfiltration. It might be possible to exfiltrate weights in a short window of time^[1], and once they’re out there’s no way to get them back. More generally, our understanding is that inside AI companies, AI agents are frequently run with access to both the internet and source code from the company that contains lots of sensitive information (e.g. architectures, training algorithms, details of internal infrastructure). This probably makes it feasible for them to leak that sensitive information onto the internet. This is the most plausible way that currently deployed AI agents could take actions that would be the biggest ever security failure of an AI company.

It’s very unclear how hard it will be to recover from other ways that an AI might try to subvert human control in internal deployments. Rogue internal deployments involving privilege escalation and compromise of monitoring systems might quickly become highly damaging or extremely costly to recover from, because they give the misaligned AI increased affordances to cause further cascading failures. Discovering a month after the fact that thousands of rogue AIs have been running wild on your cluster might mean you can’t trust any infrastructure, logs, or code produced since then, if they haven’t already used those affordances to cause even larger problems.

Blocking with synchronous monitors

Asynchronous monitoring to detect live failures is better than not detecting them at all. But synchronous monitors enable a much better strategy – blocking the attack entirely, instead of just initiating recovery efforts as quickly as possible. A monitor is “blocking” if it is run before the proposed action is executed, and has the authority to prevent suspicious actions from being executed. Claude Code auto mode is an example of a blocking synchronous monitor.

There are many options for what to do with actions that look too suspicious to execute. One option is to suspend the agent and wait for in-depth human review, as many coding assistants are currently designed to do. However, this will become prohibitively expensive as larger amounts of work are automated if human approval is required more than a tiny fraction of the time. Another option is to replace the action with an action from a less capable trusted model, but this will likely reduce performance. A third option is to replace the action with a new sample from the same model, but if the model is misaligned it might just attack a second time. A major goal in ongoing AI control research is expanding the Pareto frontier of how well we can trade off the strengths and weaknesses of these options.

Blocking monitors are generally costlier than asynchronous monitors. They increase latency because they have to run before each action can be executed, and they increase inference cost because they monitor each action individually instead of monitoring long trajectories all at once. Accordingly, minimizing these costs may entail using them selectively. We think it’s especially important to have blocking monitors in place when running AIs with permissions or affordances that might permit live failures.

1. ^{^}

   Strict bandwidth constraints might make this much slower, but it’s unclear if they will be implemented.

2026-03-31 00:56:25

This post is a living index of papers in the "theory of appropriateness" sequence. The sequence develops a computational account of normative appropriateness—the idea that much of human cognition and social behavior can be understood as pattern completion over culturally learned expectations of what is appropriate in a given situation. The papers span the cognitive and social sciences and generate implications for AI technology governance. I'll update this post as new papers appear.

The full treatment

The sequence began with a long (100+ page) paper covering the full range of topics:

• Leibo, J.Z., Vezhnevets, A.S., Diaz, M., Agapiou, J.P., Cunningham, W.A., Sunehag, P., Haas, J., Koster, R., Duéñez-Guzmán, E.A., Isaac, W.S., Piliouras, G., Bileschi, S.M., Rahwan, I., Osindero, S. A theory of appropriateness with applications to generative artificial intelligence (2024). arXiv preprint arXiv:2412.19010 [cs.AI].

Note that it is not necessary to read the long general paper before moving on to the others on specific subtopics. All papers in the sequence are self contained.

Humans (cognitive and social science)

These papers focus on the theory of human behavior:

• Leibo, J.Z., Vezhnevets, A.S., Diaz, M., Agapiou, J.P., Cunningham, W.A., Sunehag, P., Cross, L., Koster, R., Bileschi, S.M., Chang, M., Rahwan, I., Osindero, S., and Evans, J.A. A Theory of Appropriateness That Accounts for Norms of Rationality (2026). arXiv preprint arXiv:2603.14050 [cs.NE].
• Cross, L., Grau-Moya, J., Cunningham, W.A., Vezhnevets, A.S. and Leibo, J.Z. A Generative Model of Conspicuous Consumption and Status Signaling (2026). arXiv preprint arXiv:2603.13220 [cs.MA].

Machines (technology governance)

These papers apply the framework to questions about AI systems—personhood, alignment, and how we may conceptualize sociotechnical progress. Pragmatism, in the sense of Richard Rorty, is a major theme of this work:

• Leibo, J.Z., Vezhnevets, A.S., Cunningham, W.A. and Bileschi, S.M. A Pragmatic View of AI Personhood (2025). arXiv preprint arXiv:2510.26396 [cs.AI].
• Leibo, J.Z., Vezhnevets, A.S., Cunningham, W.A., Krier, S., Diaz, M. and Osindero, S. Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt (2025). arXiv preprint arXiv:2505.05197 [cs.AI].