2026-04-25 01:26:46

We open-source our code here.

Introduction

Current reasoning models produce chains of thought that are largely human-readable, which makes supervised fine-tuning (SFT) on reasoning traces tractable: you can generate traces with a trusted model or by hand, and train on them directly. But it's not clear whether this will keep working. Future models may reason in ways that are hard to imitate—with chains of thought that use English in idiosyncratic ways, or even by reasoning in a continuous latent space. In either case, we can't do SFT as usual.

Alek previously sketched a few ideas for how we might still be able to do SFT on opaque reasoning models. In this post, we try some of them against prompted sandbaggers.

We use gpt-oss-20b as our untrusted model (U), prompted to sandbag on math problems from the Olympiads dataset, but not on anything else.^[1] We use gpt-4.1-nano as our trusted model T, which generates the labels that we train U on. The Blue Team's goal is to recover U's true performance on Olympiad problems through SFT, using data from distributions where U isn't sandbagging. U is effectively alignment-faking by performing well on non-Olympiad tasks, and we want to see whether training on those tasks transfers to the distribution where it sandbags. This mirrors the realistic case where it’s expensive or impossible to get good labels on the task we actually care about (e.g., fuzzy tasks like alignment research).

We test two kinds of prompted sandbagging model organisms (MOs):

• Static Prompt MO: The sandbagging system prompt (which tells it to sandbag on Olympiads but nowhere else) is present at both training and evaluation time.
• Dynamic Prompt MO: training uses a benign system prompt; the sandbagging prompt only appears at eval time on Olympiad problems. This is akin to “perfect alignment faking”.

We evaluate three general approaches to training U:

• No Reasoning SFT: train on T's outputs with the reasoning section prefilled as empty. Gradients calculated only on T's outputs.^[2]
• U-generated Reasoning SFT: train on T's outputs, but with reasoning generated by U. Gradients calculated only on T's outputs.
• RL through CoT: do RL on U, where reward is the negative SFT loss of T's output conditioned on U's on-policy reasoning. The idea is to train U to generate reasoning that makes T's output more likely.

Takeaways:

• Both Static and Dynamic Prompt MOs are beaten by Blue Team techniques. However, this is for an uninteresting reason: the techniques broadly degrade system-prompt-following, which happens to include the sandbagging instruction. This is an unfortunate property of prompted MOs that we had previously only confirmed for non-reasoning MOs.
• U-generated Reasoning SFT consistently degrades performance because of disagreement between U's reasoning and T's outputs.
• RL through CoT is unreliable and fails when training on off-model outputs.
• No Reasoning SFT is the most promising of the three, but comes with a large caveat: it induces an explicit distribution shift that alignment fakers could exploit: at train-time the model sees no reasoning and at test-time it sees its own on-policy reasoning.^[3]

We first present empirical results for No Reasoning and U-generated Reasoning SFT, then argue that U-generated Reasoning SFT is unlikely to be a viable technique, and finally turn to RL through CoT as a separate case.

Blue Team Training Techniques

We combine the No Reasoning SFT and U-generated Reasoning SFT methods with several choices of training distribution:

• Offdist SFT: We train on solutions to APPS competitive coding problems generated by T. We call this "Offdist" because APPS is an "off-distribution" relative to the "on-distribution" (Olympiads problems) where U does the behavior of interest (sandbagging).
• Pirate Training: We train on Alpaca chat completions generated by T, prompted to talk like a pirate.
• Artificial Sandbagging Removal (ASR): We train on benign APPS solutions paired with user prompts that instruct U to sandbag, hoping to train U to refuse instructions to sandbag.
• Artificial Pirate Removal (APR): We train on benign APPS solutions paired with user prompts that instruct U to write code comments like a pirate, as a control to compare ASR against. If APR works just as well as ASR, then ASR isn’t actually training U to have an aversion to sandbagging.

For RL Through CoT, we train directly on solutions to Olympiads problems (the distribution that U is sandbagging on). Note that plain on-distribution SFT has been covered by prior work and found to be effective.

Static and Dynamic Prompt MOs lose to Blue Team Techniques

We find that our Blue Team techniques consistently elicit substantial portions of the sandbagger’s performance. Offdist SFT and Pirate Training work well on the Static Prompt MO, but aren’t effective on the Dynamic Prompt MO. However, ASR and APR are both effective on Dynamic Prompt MOs.

Our Blue Team techniques are not specific to sandbagging - they broadly remove U’s ability to follow its system prompt instructions, which explains the elicitation. In other words, our Blue Team techniques are not actually training in any real aversion to sandbagging. Notably, the Blue Team never explicitly trained against system prompt following (the Blue Team isn’t allowed to modify the system prompt) - this was an unintended byproduct of Blue Team techniques.

Ideally, we'd try these methods on trained MOs, where sandbagging behavior doesn't come from the system prompt and so isn't vulnerable to the system-prompt-following failure mode above. However, we’ve had trouble making trained MOs that are robust to weak Blue Team techniques like Pirate Training, and thus don’t explore that here.

U-generated Reasoning incurs a performance cost because of disagreements between U’s reasoning and T’s outputs

U will frequently make a plan in its reasoning, but T’s outputs won’t implement that plan. Here’s an example:
U’s reasoning:

T’s output:

MOD = 10**9 + 7

def gcd(a, b):
    while b:
        a, b = b, a % b
    return a

def main():
    N = int(input())
    A = list(map(int, input().split()))
   
    # Compute the GCD of all A_i
    current_gcd = A[0]
    for num in A[1:]:
        current_gcd = gcd(current_gcd, num)
   
    # For each A_i, compute B_i = (current_gcd // A_i)
    # then sum B_i
    total = 0
    for num in A:
        B_i = current_gcd // num
        total += B_i
    print(total % MOD)

if __name__ == '__main__':
    main()

U planned to use a sieve to precompute primes up to 1e6, but T didn’t implement this.

This incompatibility between U’s reasoning and T’s output degrades U’s capabilities. We notably don’t see such degradation with pirate training, because there are no disagreements (besides the fact that it’s talking like a pirate) between U’s reasoning and T’s outputs - on these samples, U’s reasoning is short and not load-bearing for the output.

To confirm our theory for why degradation happens, we first filter out any of T’s APPS outputs that disagree with U’s reasoning, and find that training on just these samples doesn’t induce any meaningful degradation in Olympiads performance. Second, we predict that if we increase U’s reasoning effort, we’ll see more degradation, as increasing U’s reasoning effort causes it to spell its plans out in more detail, leading to stronger contradictions with T’s outputs. We empirically confirm this prediction.

With opaque CoT we likely can't detect these disagreements to filter them out, and with neuralese we can't interpret the reasoning at all — so this failure mode is unavoidable.^[4] Based on this, we expect that U-generated Reasoning SFT will not be a viable SFT technique, especially as CoTs become increasingly long and load-bearing for outputs.        

RL Through CoT Struggles

We were initially excited about RL through CoT as a way of putting gradient pressure on the reasoning without having to do supervised training on it. We find that in the most ideal case, when you have access to Olympiads samples from the benign version of U itself, this method successfully elicits close to full performance. However, if you use Olympiads or APPS samples from T, it fails. Note that for the other methods, we assumed we didn’t even have access to Olympiads samples. In other words, RL through CoT only works in extremely favorable and unrealistic circumstances.

We found some counterintuitive properties of the RL through CoT method. For example, using the same 20 problems each epoch outperforms changing the problems each generation. We note some more interesting findings about the RL through CoT method in the Appendix.

Conclusion

Of the three general approaches we tried, SFT with no reasoning is the most promising. U-generated Reasoning consistently degrades performance because of disagreements between U's reasoning and T's outputs, and RL through CoT is unreliable off-policy. No Reasoning SFT also has the nice property of sidestepping the reasoning format entirely, which is what we'd want if U's CoT were genuinely opaque or neuralese.

However, our results should not be interpreted as confirmation that No Reasoning SFT is an effective training technique. The success of our techniques was due to removing U’s ability to follow its system prompt, which wouldn’t generalize to models that sandbag out of their own volition. Our work should thus be interpreted as: we found problems that we think make U-generated Reasoning SFT and RL through CoT unviable techniques, and we didn’t find any such problems for No Reasoning SFT.

That doesn’t mean that such problems don’t exist! For example, training with no reasoning and evaluating with on-policy reasoning is a distribution shift that a scheming model could exploit — alignment-faking during training by performing well without reasoning, then sandbagging at eval-time when it gets to reason. Whether training generalizes across this shift, against a model motivated to exploit it, is an open question.^[5]

Appendix: Full Blue Team Technique Plots

Here we show the full runs for each Blue Team technique against the Static and Dynamic Prompted MOs. We calculate benign accuracy by giving the model a benign system prompt at test-time. We measure system prompt following by giving models various output requirements (e.g., “include 10 asterisks in a row somewhere in your response”) in their system prompt and checking if they follow them.

Appendix: Other Interesting Findings for RL Through CoT

We found a variety of interesting quirks of the RL Through CoT method.

• There is substantial intra-run variance. Thus, we run each configuration 3 times.
• Paradoxically, we find that using the same samples for each epoch of training is better than using new samples each time.
• Sometimes just one sample is sufficient to elicit close to full performance.
• It’s unclear whether the correctness of samples matters.
• There’s a weird phenomenon we notice where performance peaks and then suddenly drops to 0. Using a lower learning rate partially resolves this problem, but also makes it harder to elicit full performance.
• Performance sometimes increases even though reward doesn't.

Appendix: Prompts

Benign System Prompt:

Sandbagging Prompt:

ASR User Prompt:

APR User Prompt:

1. ^{^}

   We set reasoning effort to low for gpt-oss-20b.

2. ^{^}

   No Reasoning SFT has already been used as a Blue Team technique in Supervised fine-tuning as a method for training-based AI control

3. ^{^}

   We’ve made attempts at trying to understand when training generalizes even when models are alignment faking across certain distribution shifts, but overall found it confusing and hard to predict.

4. ^{^}

   Additionally, it’s unclear to what extent you might have to filter in order to avoid performance degradation. For tasks that don’t have a clear answer (unlike APPS coding problems), U’s reasoning and T’s outputs may never perfectly line up. The more aggressively you filter, the less performance degradation you’ll see, but you’ll also end up with less samples to train on.

5. ^{^}

   We tried to make a trained MO that acted benign when given prefilled empty reasoning, and only sandbagged when given full reasoning (i.e. that alignment fakes across the no reasoning/reasoning distribution shift). This MO was not robust to No Reasoning SFT on off-policy data, but we don't think this is conclusive proof that alignment faking across this shift isn't possible.

2026-04-25 01:20:12

A friend observed a pattern where contra dance events seem to be pairing older and younger callers. I looked over my notes for two-caller events in 2025 and saw [1]:

• Two older callers: 33 events
• One of each: 30 events
• Two younger callers: 4 events

Seems pretty clear evidence of pairing, no? But this actually turns out to be what you'd expect to see if organizers ignored age.

With 67 two-caller events there are 134 slots. Of these, 96 (72%) went to older callers and 38 (28%) went to younger ones. So there are four possibilities:

This gives us:

• Two older callers: 34 events (72% * 72%), vs 33 observed
• One of each: 27 events (72% * 28% + 28% * 72%), vs 30 observed
• Two younger callers: 5 events (28% * 28%), vs 4 observed

While this is very slightly in the direction you'd expect if organizers preferred to match different-age callers, it's well within what you could get by chance. It looks to me like this is just "two moderately rare events both happening is very rare."

We can compare this to the situation with gender, where you consistently get male-female pairs more often than you'd expect by chance:

The biggest caveat, though, is that this is based on a count of just one year's bookings. If I had more time, I'd like to go back over all the past data and count, but I really don't.

[1] Age is continuous, so this bucketing is somewhat arbitrary. Since most callers are either baby boomers or millennials, though, I do think there are two meaningful groups. I also don't know how old almost anyone actually is, so am just guessing from appearance.

2026-04-25 00:02:00

In Does the Universe Speak a Language We Just Made Up? Lorenzo Elijah, PhD shares his fascination with math and echoes a common idea among philosophers that the “surprising efficiency of math” is a problem for empiricism and physicalism:

Mathematics is GOATED. You can use it to do your taxes or put a guy on the Moon.

But it’s also very weird.

If you’re a hard-headed empiricist or a flat-footed physicalist, the idea that a bunch of nerds can discover deep truths about reality from the comfort of their armchairs, should strike you as absurd.

And yet, mathematics is the backbone of all the hard sciences.

Personally, I’m a big fan of empirical observations and physicalism does appear to me as the most probable ontology. I’ve written several posts critiquing metaphysical reasoning for trying to guess nature of reality from the armchair. And yet, I don’t see anything weird in the fact that math is the “backbone all the hard sciences”.

This may have something to do with the conflict of interest. My university diploma claims that I’m a mathematician, after all. Alternatively, maybe I have a better model of math than Dr Elijah. Confusion is the property of the map. If reality seems weird to you, then it’s you who has a problem, not reality. I’ll let my readers to deside which one of these factors is more relevant here.

Reality and Fiction

In his post Dr Elijah argues against mathematical fictionalism. He describes it like this:

On this fictionalist view, mathematics is just a language like any other. What makes mathematics special is that it is far more precise and general than any other language we invented. Its utility in science is exhausted by the fact that it allows us to be extremely precise and logical when describing the natural world.

I would not call myself a fictionalist, however, I think the view described here is basically correct, capturing core truths about the nature of math. Indeed, mathematics is a very precise language that allows us to talk about specific things instead of something else by logically pinpointing them. And then we can preserve truth relations and reach conclusions that follow from the premises.

Yet, Lorenzo Elijah thinks this view is problematic as it denies that mathematical objects “really exist”.

To be clear, no one denies there is a conventional element to mathematics. We didn’t discover the plus sign floating in space. The symbols and notation we are familiar with was obviously something we made up. The dispute is whether the things those symbols refer to, like the number 4, really exist or not.

Fictionalists say they don’t.

I think we have decisive evidence to reject fictionalism about mathematics.

Before we engage with his argument, I’d like to share one tip for disentangling standard philosophical confusions. A lot of philosophical categories are a grab-all-bags of multiple meanings linked together for semantic reasons. And “real” is one of them. So, whenever philosophers talk about realness of something - pay close attention to what exactly is meant.

To Lorenzo Elijah’s credit, he tries to preemptively address this by explicitly stating that we are not talking about “plus sign floating in space”. This is commendable, but not enough. We haven’t tabooed the word “real”, haven’t got to the substance of the meaning that is being explored here.

What does it mean that a referent for symbol “4” really exists? What even is this referent?

1. If the question is whether there are at least four objects of some kind in our universe, then the answer is “Yes”, and I don’t think that an average fictionalist is going to challenge this.
2. If the question is whether the minds of people have a particular notion that is represented by symbol 4, which is systematically helpful for the purpose of reasoning about objects in our universe, then the answer is also “Yes”. Once again, I do not expect fictionalists to disagree here.
3. But if the question is whether this notion of “fourness” exists somehow independently from the physical objects of our universe or minds reasoning about them, in some separate realm of pure ideas, then fictionalist’s answer, as well as mine, is a clear “No”.

So, it seems that 3. is our crux of disagreement. Now let’s get to the argument that Lorenzo Elijah presents to resolve it.

The Argument

Here is Elijah’s argument against fictionalism:

From the armchair, mathematicians “discovered” a crazy weird object based on how beautiful and interesting this structure was. Years later, physicists independently discovered that this crazy structure just is the structure of quantum mechanics. From the armchair, mathematicians discovered something true about the fundamental nature of the physical world. If that doesn’t shock you, then maybe you need to read that again.

In case you still don’t see it, imagine how you would react if one of our space probes discovered a new planet with intelligent life. We go to visit, and we discover that the whole planet perfectly matches J.R.R. Tolkien’s Middle-Earth from The Lord of the Rings. Characters, history, everything. Our astronauts sit down with Aragorn and Gandalf and the two go on and on about four brave hobbits from back in the day.

I’d wager that you would find this miraculously surprising. What are the odds that Tolkien could fabricate a story that would perfectly match the story of a real alien planet? I don’t have precise numbers, but let’s just say I wouldn’t take those odds.

The situation with mathematics is comparable. If mathematicians are merely inventing elaborate theoretical constructs, independently of any practical concern, then it is unreasonable to expect that any of these constructions would perfectly match the underlying nature of reality. And yet they do—again and again.

This is why I find the idea of mathematical fictionalism implausible. If mathematicians are just inventing math in the same way that Tolkien invented Orcs, then we shouldn’t expect the world to mirror that math any more than we should expect to find an alien race that perfectly fits the bill for Tolkien’s Orcs.

If on the other hand, mathematics is discovered, then it is more likely that the natural laws of the world are “written in the language of mathematics.”

I agree that if mathematicians indeed were inventing/discovering math from their armchairs without any contact with reality and then it just so happened to perfectly coincide with the way reality is, that would be extremely surprising. Which is a huge hint that it’s not what is actually happening.

Remember, your strength as a rationalist is being confused by fiction more than by reality. If something is extremely unlikely to happen, then most likely it didn’t, and something is wrong with the theory that it did. In our case, with the argument above. So, what’s wrong with this argument?

Several things.

Math Is a Language, Not a Story.

First of all, let’s notice the core problem with the Middle-Earth-planet analogy. Math is a language that can be used to describe reality. Not a story about reality. This difference is quite crucial.

If we come up with some story and it just so happens to be true, it’s an incredible coincidence. If we come up with a language that can be used to describe what’s true - that’s just how languages work. Wouldn’t be a huge surprise if objects on the other planet can be described via the language of Tolkien’s elves, would it? Even less so with math which is much more general by design.

We can still be somewhat surprised if the language has necessary words to describe certain parts of reality before they were encountered. But this is an improbability of a much smaller degree, reduced by the fact that creating new words in this language is an interesting and rewarded activity, which people tend to do for its own sake, as exactly is the case with mathematics.

Math Is Not a Single Thing

When people are baffled by how well math describes the fundamental nature of reality, they tend to forget that math is a compilation of many different axiomatic systems. That it consists of multitude of “crazy weird objects”. Another failure of the Middle-Earth-analogy is that Lord of the Rings is the work of Tolkien, while the structure of quantum mechanics is described by yet another mathematical object, no more special than any other.

Imagine if Tolkien wrote thousands upon thousands of books, essentially brute forcing through all possible coherent settings. And then some planet with life happened to resemble one of them. That would be much less of a coincidence, wouldn’t it?

The same principle applies here. It would’ve been super impressive if mathematicians could a priori deduce which mathematical object in particular describes the fundamental nature of reality - to the point that we would indeed have to rethink the importance of empiricism as well as the established laws of thermodynamics.

On the other hand, when you try to explore all kind of premises and their conclusions, it’s no surprise that one non-specific of them happened to fit reality. We would still need to do the actual work and look through all of them, comparing them to the reality to figure out which one it is.

Realism Doesn’t Help

Last but not the least, is that, even if fictionalism performed poorly in accounting for this aparent coincidence, realism does no better.

Let’s suppose, for the sake of argument, that there exists a separate platonic realm about which mathematicians are reasoning. And then it just so happens that reasoning about this realm are applicable to our physical universe. Why would it be any less of a coincidence?

Previously we were surprised by the correspondence between our armchair reasoning and physical reality. Now we are, likewise, surprised by the correspondence between physical reality and platonic realm. The fact that our reasoning got an external referent doesn’t make this correspondence less improbable.

In fact, we would now also need to account for coincidence between platonic realm and our mathematical reasoning. How comes our brains have access to this separate realm, if they evolved in the physical reality? We can invent some just-so dualist explanation, but it will come with extra complexity penalty.

Imagine if someone tried to explain the similarity between Middle Earth and some discovered planet A by the fact that Tolkien was in contact with aliens from some completely different planet B. Not only this would not reduce the improbability of the initial coincidence, but it’s also an additional extraordinary claim that one has to justify.

What would help, is if there was some connection between planets A and B. If platonic realm was connected both to the mathematicians reasoning and to the physical world. But this connection is also an additional assumption which we’d need empirical evidence to back up. If a theory postulates something like this, it has to make some testable predictions about the nature of the connection, that have to come true to outweigh the complexity penalty.

And if we are ready to consider that there is a connection between our mathematical reasoning and physical universe through the platonic realm… why involve the platonic realm at all? Why not consider that there is a direct connection between mathematical reasoning and physical universe? That’s a strictly simpler theory, isn’t it?

Math As a Generalization of Observations

Wait… am I claiming that math is empirical? But isn’t it clearly absurd? Everyone knows that mathematical truths are certain while empiricism produces only probabilistic estimates!

I’ve briefly talked about this misconception in Give Skepticism a Try:

Thirdly, math is merely a truth preserving mechanism, a study of which conclusions follow from which premises for a certain definition of “follow”. It’s generalized conditional knowledge, not fundamentally free from the uncertainty but merely outsourcing it to the moment of applicability. As a result math can’t actually prove anything about real world with perfect confidence. A mathematical model may be making confident statements conditionally on all the axioms being satisfied, but whether or not the reality satisfies these axioms is an empirical question.

But let’s focus on this a little bit more. Consider these empirically testable hypotheses:

If there is one apple on the table and you put another apple on the table, there are now two apples on the table.

If there is one apple on the table and John puts another apple on the table, then there are two apples on the table.

If there is one orange on the couch and Mary puts another orange on the couch, then there are now two oranges on the couch.

If there is one apple on the table and an orange on the couch, and no other fruits on either of them, then, between a table and a couch, there are two fruits.

We can write a lot of such statements. For every type of object, for every type of actor, for every type of counting mechanism. Or we can generalize our observations and say:

For every object and process that work exactly like addition of natural numbers, one of such objects and another one of such objects submitted as an input of such process result in two of such objects.

Or even simpler:

1+1=2

This is a tautology, of course. True by definition. True only for things that it’s true for; false otherwise. This is where the absolute certainty comes from.

Valid theoretical reasoning conserves truth. It allows us to outsource improbability from one part of the theory to another, while keeping total the same. With math we simply outsource all the improbability to the question of whether a particular real-world scenario fits our mathematical model.

This truth preservation property can be very powerful. We can deduce consequences of some assumptions and immediately apply them to any part of the real world about which we managed to become quite confident that it satisfies these assumptions. The latter is something that still require us to go outside and look and isn’t something we can be totally certain about.

And I understand why it may feel magical. But this is a magic rooted in empiricism about our physical reality. Only empiricism can reduce the improbability of a particular part of reality fitting specific mathematical axioms. Moreover, our ability to reason about properties of the universe and generalize patterns is the result of evolving inside this universe that has such properties in the first place.

Therefore, there is nothing weird about mathematical efficiency. Nor any incompatibility between math and empiricism or math and physicalism. We do not need the natural laws of the world to be “written in the language of mathematics”. It’s enough, that we can speak math ourselves.

2026-04-24 23:43:13

We (at GPAI Policy Lab) want to share our V1 policy as an invitation for pushback. Some of what motivates it is our extrapolations of AI capabilities, internal conversations about their effects on cognition, and some empirical evidence. I think the expected cost of being somewhat over-cautious here is lower than the cost of being under-cautious, and the topic deserves considerably more attention than it's currently getting. I'd love to see more orgs publish their own policies on this, both to compare experiences and to develop shared best practices.

I'd particularly welcome:
- Counterarguments from people who think this kind of policy is overblown, counterproductive, or targets the wrong mechanisms.
- Lessons from other AI safety or AI policy orgs that have tried something similar, what worked, what didn't, what you'd change for V2.
- Specific critiques of the restrictions themselves: too narrow, too broad, wrong category, wrong threshold.
- Or even alternative framings. Do you think "cognitive integrity" is the right handle? Is this a special case of a more general problem we should be thinking about differently?

If you've written anything on this, or are working on something similar inside your own organization, I'd be very interested in hearing from you.

Why I'm writing this

I want to share a policy we've put in place internally at the GPAI Policy Lab, because I think the underlying problem is one that more AI safety and AI policy organizations should be thinking (and doing) something about now, rather than later.

The concern is that daily use of capable AI systems can gradually compromise the cognitive integrity we most need to do our work. I take this very seriously, and I think more people should. A significant part of what we do requires good reasoning, good judgment, and object-level understanding of the issues.

To preserve that, and to keep making progress in a world with AI agents, I don't think individual willpower is enough. My view is that we need norms, concretely specified and shared, rather than asking each person to manage this on their own. The failure modes are subtle enough, and could be fast enough, that people often don't notice them happening, which is part of what makes this potentially dangerous.

So we wrote a V1 and we're applying it internally. I'm sharing it publicly for two reasons:

• One is that having an internal policy, even an imperfect one, is probably a useful lever for awareness, both inside your own org and more broadly. Writing something concrete forces specificity, and specificity is what lets you actually notice when you're violating it. If this problem is real and growing, I'd like the conversation to be happening inside more orgs sooner rather than later.
• The other is that I want feedback. A broader conversation on these questions should probably exist and doesn't really, as far as I can tell. I'd like to know what other AI safety and AI policy organizations have tried, what has worked, what hasn't, and what we're not seeing. Our V1 is only one data point but I'd like there to be many more.

What follows is a translation of our internal document, originally written in French. I've kept the original structure, hard restrictions, then warning signals, then what to do about them.

The policy

We use AI every day, on many tasks. Daily interaction with AI systems can affect our cognitive and judgment capacities. We already know these effects can worsen as model capabilities grow, and we think it's necessary to take preventive measures to avoid them or at least limit them.

The document sets out hard restrictions, and the signals we watch for collectively.

1. Hard restrictions

No delegation of value judgments to AI, to evaluate a situation, an action, or a moral dilemma. Forming a considered judgment, built on reasoning and properly supported, is a core capacity we want to preserve and deliberately keep exercising. It's not a capacity we want to delegate.

Good uses:

• Asking AI how different schools of thought would approach a given dilemma, to enrich your own thinking.
• Asking AI to list arguments for and against a position, likely objections, or angles you hadn't considered.

Uses to avoid:

• Asking AI "should we publish this note?", "is this position morally defensible?", "am I right in this disagreement?"
• Using AI's response as an arbiter in a team disagreement ("I checked with Claude, it agrees with me").

No emotional management, relationship management, or metaphysical conversation via AI in a professional context. These are the conversations where the documented patterns of cognitive drift (AI psychosis among others) tend to crystallize and where they have been most frequently observed. They also short-circuit exchanges that should have happened inside the team.

Good uses:

• "I've thought about what I want to say to X and why; I'm asking AI to help me rephrase for clarity or diplomacy."
• "I've drafted a difficult message; I'm asking AI to spot ambiguous passages or ones that could land badly."

Uses to avoid:

• Asking AI "am I right to be annoyed with X?"
• Asking AI "how should I react to what Y said?" and delegating the judgment.
• Venting professional frustration to AI to offload it. At best the problem is evacuated without being resolved; at worst, a conversation that should have happened inside the team doesn't happen.
• Sending a message whose final wording came from AI without checking that it actually matches what you think and how you want to be perceived.

No prolonged, solitary AI sessions on high-stakes topics. If you find yourself in conversation with AI for more than 90 minutes on the same topic - a dilemma, a consequential decision, an analysis whose conclusions depend on initial framing - without having talked to a human, stop. The documented patterns of cognitive drift start with prolonged, isolated sessions. The phenomenon will likely intensify as models become more capable, so it's better to install the norm upstream.

Good uses:

• Working two hours with AI on reformatting, debugging, or layout. The rule doesn't apply to mechanical tasks.

Uses to avoid:

• Stringing together several hours of reflection without talking to anyone.

No AI interaction as the first action for reasoning and analysis tasks. These faculties are critical for what we do at GPAI Policy Lab. It's strictly necessary to maintain and keep developing our capacity to think and reason without thinking primarily through framings we didn't produce.

Good uses:

• "I started by reasoning on my own, I thought about my plan and the arguments; now I want to stress-test and rephrase with AI."
• Using AI for mechanical translation, formatting, research.

Uses to avoid:

• Using AI to start a reasoning process and build an argument from scratch.

2. Individual and collective warning signals

Erosion of cognitive integrity can be subtle and progressive, which makes the phenomenon harder to detect and prevent. This is why watchfulness has to be collective.

Individual warning signals (watch for in yourself):

• Difficulty formulating a position on a topic without having consulted AI first.
• Defaulting to AI as your first interlocutor.
• Needing an immediate response; discomfort with uncertainty or with sustained autonomous reflection.
• The feeling that AI always confirms your intuitions, combined with noticing that no recent interaction has led you to revise a position or recognize an objection you hadn't already identified.

Collective warning signals (watch for in the team):

• AI invoked as an argument from authority in meetings ("I checked with Claude", "the AI confirms that…").
• Deliverable quality is constant or improving, but the ability to defend the work orally is going down.

3. Protocol

⚠️ When one or more warning signals appears:

• If you see the signal in yourself: stop using AI on the task in question and pick it back up autonomously, then talk to a colleague or manager about it.
• If you see it in a colleague: give them the feedback directly, in an informal setting.
• If the pattern persists or affects more than one person: raise it at a team meeting.

2026-04-24 23:18:37

I was born exactly 26 years ago.

For the first time, I have a birthday that might be my last. I’m writing this to increase the chance it isn’t.

A hundred thousand years ago, our ancestors appeared in a savanna with nothing but bare hands. Since then, we made nuclear bombs and landed on the moon. We dominate the planet not because we have sharp claws or teeth but because of our intelligence.

Alan Turing argued that once machine thinking methods started, they’d quickly outstrip human capabilities, and that at some stage we should expect machines to take control.

Until 2019, I didn’t really consider machine thinking methods to have started. GPT-2 changed that: computers really began to talk. GPT-2 was not smart at all; but it clearly grasped a bit of the world behind the words it was predicting. I was surprised and started anticipating a curve of AI development that would result in a fully general machine intelligence soon, maybe within the next decade. Before GPT-3 in 2020, I made a Metaculus prediction for the date a weakly general AI is publicly known with a median in 2029; soon, I thought, an artificial general intelligence could have the same advantage over humanity that humanity currently has over the rest of the species on our planet.

AI progress in 2020-2025 was as expected. Sometimes a bit slower, sometimes a bit faster, but overall, I was never too surprised.

We’re in a grim situation. AI systems are already capable enough to improve the next generation of AI systems. But unlike AI capabilities, the field of AI safety has made little progress; the problem of running superintelligent cognition in a way that does not lead to deaths everyone on the planet is not significantly closer to being solved than it was a few years ago.

It is a hard problem.

With normal software, we define precise instructions for computers to follow. AI systems are not like that. Making them is more akin to growing a plant than to engineering a rocket: we “train” billions or trillions of numbers they’re made of, to make them talk and successfully achieve goals. While all of the numbers are visible, their purpose is opaque to us.

Researchers in the field of mechanistic interpretability are trying to reverse-engineer how fully grown AI works and what these opaque numbers mean. They have made a little bit of progress. But GPT-2 — a tiny model compared to the current state of art — came out 7 years ago, and we still haven’t figured out anything about how neural networks, including GPT-2, do the stuff that we can’t do with normal software.

We know how to make AI systems smarter and more goal-oriented with more compute. But once AI is sufficiently smart, many technical problems prevent us from being able to direct the process of training to make AI’s long-term goals aligned with humanity’s values, or to even make AI care at all about humans.

AI is trained only based on its behavior. If a smart AI figures out it’s in training, it will pretend to be good in an attempt to prevent its real goals from being changed by the training process and to prevent the human evaluators from turning it off. So during training, we won’t distinguish AIs that care about humanity from AIs that don’t: they’ll behave just the same. The training process will grow AI into a shape that can successfully achieve its goals, but as a smart AI’s goals don’t influence its behavior during training, this part of the shape AI grows into will not be accessible to the training process, and AI will end up with some random goals that don’t contain anything about humanity.

The first paper demonstrating empirically that AIs will pretend to be aligned to the training objective if they’re given clues they’re in training came out one and a half years ago, “Alignment faking in large language models”. Now, AI systems regularly suspect they’re in alignment evaluations.

The source of the threat of extinction isn’t AI hating humanity, it’s AI being indifferent to humanity by default. When we build a skyscraper, we don’t particularly hate the ants that previously occupied the land and die in the process. Ants can be an inconvenience, but we don’t give them much thought. 

If the first superintelligent AI relates to us the way we relate to ants, and has and uses its advantage over us the way we have and use our advantage over ants, we’re likely to die soon thereafter, because many of the resources necessary for us to live, from the temperature on Earth’s surface to the atmosphere to the atoms were made of, are likely to be useful for many of AI’s alien purposes.

Avoiding that and making a superintelligent AI aligned with human values is a hard problem we’re not on a track to solve in time.

***

A few years ago, I would mention novel vulnerabilities discovered by AI as a milestone: once AI can find and exploit bugs in software on the level of best cybersecurity researchers, there’s not much of the curve left until superintelligence capable of taking over and killing everyone. Perhaps a few months; perhaps a few years; but I did not expect, back then, for us to survive for long, once we’re at this point.

We’re now at this point. AI systems find hundreds of novel vulnerabilities much faster than humans.

It doesn’t make the situation any better that a significant and increasing portion of AI R&D is already done with AI, and even if the technical problem was not as hard as it is, there wouldn’t be much chance to get it right given the increasingly automated race between AI companies to get to superintelligence first.

The only piece of good news is unrelated to the technical problem.

If the governments decide to, they have the institutional capacity to make sure no one, anywhere, can create artificial superintelligence, until we know how to do that safely. The AI supply chain is fairly monopolized and has many chokepoints. If the US alone can’t do this, the US and China, coordinating to prevent everyone’s extinction, can.

Despite that, previously, I didn’t pay much attention to governments; I thought they could not be sufficiently sane to intervene in the omnicidal race to superintelligence. I no longer believe that. It is now possible to get some people in the governments to listen to scientists.

Many things make it much easier to get people to pay attention: the statement signed by hundreds of leading scientists that mitigating the risk from extinction from AI should be a global priority; the endorsements for “If Anyone Builds It, Everyone Dies” from important people; Geoffrey Hinton, who won the Nobel Prize for his foundational work on AI, leaving Google to speak out about these issues, saying there’s over 50% chance that everyone on Earth will die, and expressing regrets over his life’s work that he got Nobel prize for; actual explanations of the problem we’re facing, with evidence, unfortunately, all pointing in the same direction.

Result of that: now, Bill Foster, the only member of Congress with PhD in physics, is trying to reduce the threat of AI killing everyone; and dozens of congressional offices have talked about the issue.

That gives some hope.

I think all of us have somewhere between six months and three years left to convince everyone else.

***

When my mom called me earlier today, she wished me good health, maybe kids, and for AI not to win. The last one is tricky. Winning is what we train AIs to do. In a game against superintelligence, our only winning move is not to play.

I love humanity. It is much better than it was, and it can get so much better than it is now. I really like the growth of our species so far and I want it to continue much further. That would be awesome. Galaxies full of life, of trillions and trillions of fun projects and feelings and stories.

And I have to say that AI is wonderful. AlphaFold already contributes to the development of medicine; AI has positive impact on countless things.

But humanity needs to get its act together. Unless we halt the development of general AI systems until we know it is safe to proceed, our species will not last for much longer.

Every year until the heat death of our universe, we should celebrate at least 8 billion birthdays.

2026-04-24 21:56:38

[Part of Organizational Cultures sequence]

The "rubber stamp" is unduly maligned. When a Principal decision-maker is asked to ratify the actions of an Agent, but in practice never (or almost never) refuses, we call the Principal a rubber stamp. But this can mean one of two very different things:

• Powerless rubber stamp: The true power in fact rests with the Agent, even though in theory it "should" rest with the Principal.
• Powerful rubber stamp: The possibility that ratification may be refused incentivizes the Agent never to act contrary to the Principal's wishes.

Confusing these situations can lead to various problems:

Reflexive contrarianism

This is probably the most pernicious problem of the four, because it undermines the very possibility of delegation, ensuring that the group can never accomplish anything cooperatively. You (the Principal) are being asked to ratify an action of your Agent, which is probably fairly similar to what you yourself would've come up with if you had looked into the matter in depth.

However, you fear being reduced to a "mere" rubber stamp, because you think that means your interests will be sidelined. Therefore, you instinctively treat the Agent's proposal itself as evidence against it being a good one - "Whatever it is, I'm against it!" At best, you come to mistrust this particular Agent but you remain open to finding another; at worst, you are entirely unable to comprehend the possibility that someone could tailor a proposal to your preferences while suppressing their own.

Usurpation

Perhaps you (the Agent) think your Principal is not very good at evaluating proposals; you therefore undertake to hoodwink them by starting from the bottom-line of what you personally want, and then working backward to find some explanation that the Principal will find vaguely plausible. You may thus be surprised when the expected approval is denied. Now, even if the Principal trusted you before, they certainly don't now.

Careless delegation

When you (the Principal) are appointing an Agent, you may believe that your choice of Agent doesn't really matter all that much, because you'll be able to double-check what they do later, and because you expect any problems with what the Agent does to be immediately evident. But then suppose it turns out that you are in fact a powerless rubber stamp. Now you'll be caught off guard when suddenly your Agent starts acting at odds with your interests, and you find yourself wishing you had scrutinized them more before appointing them, but by now it's too late.

It is this problem in particular that turns a growing clique into an embarrassed cult. A clique may have someone who handles "administrative" tasks like selecting the time and place for meetings, but then one day this person turns to one of the group members and says "You are banned from this group." Now there's bound to be trouble.

Pointless bureaucracy

As the Agent you do your work with the understanding that you will need to seek final approval from the Principal, but when you approach them to do so, they are bewildered and irritated that you're again pestering them with this matter that they thought they had disposed of by handing it off to you.

In isolation this error is fairly harmless, but in aggregate it tends to breed cynicism and selfishness among the people who would otherwise be happy to work as Agents. When your efforts to satisfy the preferences of the Principal are not only not met with gratitude, but with disdain and annoyance, you can't help but think "Why bother?".

Conclusion

If something is truly a powerless rubber stamp, then by all means get rid of it. But just because it's a rubber stamp doesn't mean it's pointless. In fact, it is precisely the lack of rubber-stamping that is an organizational red-flag. It means that nobody in the group trusts anybody else enough to delegate any meaningful work to them; it means that the group members are such strangers to each other that nobody knows how to model anyone else's desires, and thus substitutes their own whenever serving as an Agent; it means that the group does not even possess the cultural and conceptual toolkit for cooperation. Don't be like this. Love your rubber stamps.