2026-04-25 05:49:44

Adapted from 2025-04-10 memo to AISI

I’ve previously made arguments like:

Not long after it becomes possible for someone to make powerful artificial intelligence^[1], it might become possible for practically anyone to make powerful AI.

1. Compute gets exponentially cheaper by default.
2. Knowledge proliferates (fast!) by default: AI techniques are typically simple and easy once discovered.
3. What’s more, AGI-making know-how may be widespread already.

Or, as Yudkowsky puts it^[2],

Moore’s Law of Mad Science: Every eighteen months, the minimum IQ necessary to destroy the world^[3]drops by one point. - Yudkowsky

It’s important to emphasise that none of these are laws of nature! But the economic and social forces at work are quite strong.

So (leaving aside debates about the appropriate definition of ‘AGI’) where the frontier of AI development leads, others – many others – potentially rapidly follow. Followers can go even faster by stealing or otherwise harvesting insights from the frontier, but this is not a hard requirement – just an accelerant.

For more on the first point, compute getting cheaper, consider Moore’s law (or the more general and robust Wright’s law). What about the know-how?

Stupid, Simple AGI

The stupidest, simplest possible approach to producing general intelligence might mimic evolution in a large, open-ended, interactive environment. Nobody has succeeded at this yet because they don’t have enough compute, but just a few more decades of compute scaling might get us there. The code to do this would be ultimately quite simple, but the amount of compute time to run it is out of reach today. Almost nobody nowadays thinks that it will take this long, because this is the stupidest, simplest (and least steerable) possible approach and we have much better ideas.

But this means that unless something interrupts the compute trends, then even if ingenious, well-resourced people ‘get AGI first’, eventually anyone could practically blunder into creating their own. Of course, many things could be changed if powerful AI is developed and applied in the meantime… perhaps including the cost and efficiency of compute, the distribution of compute, or indeed the existence and inclination of people to do the blundering.

The Design Space for AGI

What did I mean by ‘AGI-making know-how may be widespread already’?

I don’t literally mean that the recipe for AGI is known and widespread. I don’t even mean that we broadly know exactly how to make AGI and simply want for the capital (compute and data). But for those paying attention, the design space for practically achievable AGI is narrowing.

Take long-horizon coherence or continual learning, for example. Maybe components of these are expandable memory and long-context management of plans and observations. This could perhaps be cracked with something resembling a selection from:

• Context summarisation
• Read-write retrieval-augmented generation
• Recurrent embeddings
• Longer training trajectories
• Plan-management or recursive delegation scaffolding
• Periodic distillation of history into weights or activation patches
• Explicit training for notetaking
• Some even simpler thing, like ‘just scale up the compute’

Among the sharpest, most experienced practitioners at the frontier, that perceived design space may be narrower still^[4]. In the far wider cohort comprising all competent computer scientists and engineers, the design space may not be as saliently in view – but the scientific ‘breadcrumbs’ have been pointing in useful directions for years (at least).

My personal testament^[5] is that by 2020, several landmarks were visibly coming together in NLP and RL, and by 2021 I had a good sense of a plausible research path to general autonomous AI. Developments like further scaling, mixtures of experts, chain of thought, LLM agents, RL ‘reasoning’, fast attention mechanisms, and hyperparameter tuning optimisations are not merely ‘obvious in hindsight’: their rough contours were advance predictable. It was ‘merely’ a matter of experimenting to find out working details. I’m not being (especially) hubristic here: for some experts closer to the action, these same things looked plausible by 2017 or even earlier! The contours of tomorrow’s advancements are similarly already in view, and far more attention and capital are being poured into the discovery process.

That’s not to say that, given the capital, we could have created AGI there and then in a single try, or even here and now. A design space is not a complete or final design. But iterative refinement by well-resourced and moderately creative problem solvers has been charting a course, and if we are willing to anticipate one frontrunning group getting ‘all the way there’ we must acknowledge that the feat will be reproducible in relatively short order.

Accelerants

‘Reproducible’ is one thing. How soon and how fast? With the current level of sharing of research insights, the answer seems to be roughly ‘as soon as you can outlay comparable capital’, or even sooner!

What phenomena are responsible for accelerating this proliferation? In very roughly descending order of effect size:

• Theft, leak, or deliberate release of pretrained baselines and training algorithms
• Distillation (authorised or not) from exposed APIs
• Exponentially cheaper compute
• (Sometimes cheap or even public access to) ever more sensor and record data^[6]
• Shared algorithmic and experimental details in papers and blogposts
• Conversations and rumours at conferences and other events^[7]
• Movement of experts between development groups and projects
• Use of AI to assist development^[8]

Very tightly securitized projects might partly dampen some of these effects. Competition between firms and countries could amplify them.

What about exponentially cheaper compute? Market dynamics might pivot at some stage to reduce or even reverse the effect of dwindling compute price (for example, extreme buyer concentration driven by strategic accumulation, increasing marginal compute utility^[9], deliberate regulatory intervention on compute, or something else), but will otherwise continue to drive proliferation. On the other hand, if compute production increases even faster, costs may drop commensurably faster^[10].

Sensor and records data are being collected even more feverishly now that companies have realised their critical use in training modern AI systems — notice when companies’ privacy policies update to include carve-outs for collecting AI training data. We should expect more of this, as well as more collection of physical and industrial activity records for training robotics, autonomous vehicles, and automated laboratory workcells.

Alternatively, some have imagined an ‘end of history’ moment when sufficiently smart AI arrives and (usually by underspecified mechanism) prevents all of these factors from proceeding. Some envisage not only that, but an AGI or AGI-enabled organisation foreclosing not only the accelerants of proliferation, but also the potential for a rival project to emerge anywhere^[11]. This is conceivable, but one has to ask on what timeframe these changes would happen, and the consequences if it takes longer than imagined.

Short of such an acute and decisive interruption of all of these dynamics, other shocks such as international conflict could have impacts in either direction.

Concluding

Intelligent engineering-minded people exist in all geographies and of all ideologies. Most lag the frontier of AI development only for want of compute capital and intent. Because compute continues to get cheaper, and the potential of AI comes more clearly into focus, both compute and intent become rapidly more widespread. The open sharing of discoveries can further lower barriers and shorten proliferation timelines, but is not essential to this dynamic.

Given this, we have to ask what the consequences of this proliferation could be. Where they are concerning, we must consider in what ways these dynamics could be defused, or, likely failing that, how we will ready ourselves, on a short timeframe, for what follows.

We live in interesting times! There’s a lot we can do.

1. For now I’ll use ‘powerful AI’ and ‘AGI’ (Artificial General Intelligence) interchangeably. The definitions have never been settled, and will likely never be settled, but I’m considering systems which are able to autonomously act, develop new tools and technology (given sufficient research resources), and in principle maintain or upgrade themselves if that was their goal. ↩︎

2. E.g. in Artificial Intelligence as a Positive and Negative Factor in Global Risk – Yudkowsky 2008 (though this phrase was coined earlier) ↩︎

3. Yudkowsky believes that sufficiently advanced AGI developed in a context like ours leads to everyone dying. I think he’s probably right… but it depends a lot on how you operationalise ‘sufficiently advanced’ and ‘context like ours’. That’s where all the action is! ↩︎

4. Sam Altman claims “We are now confident we know how to build AGI” and Dario Amodei predicts it “could come as early as 2026”. These CEOs of some of the best resourced and talented AI organisations will have privileged insight into the design space, while also having unusual psychology and possible conflicts of interest. Meanwhile, Turing Award and Nobel Prize winners Bengio and Hinton both think 2028 is possible. Crowd wisdom forecasts give wide uncertainty, but centre on the early 2030s. Experts rarely agree on exact anticipated details, but mostly agree on the outlines of the candidate design space. ↩︎

5. (as a smart computer scientist who has been roughly following AI since 2015, made it my graduate study in 2022, but who has never actively pursued frontier AI capability contributions) ↩︎

6. Think robots in factories, recordings and logs of computer use, autonomous vehicle logs, scientific lab measurements, CCTV and satellite readings, meeting recordings, social media activity logs, wearable recording devices, … ↩︎

7. Parties in Silicon Valley are allegedly a somewhat good source of technical AI gossip! ↩︎

8. The use of AI to assist AI development, or even to fully automate it, has long been discussed in the field of AI. The possibility of an ‘intelligence explosion’ or similar technological singularity is still debated decades after first being hypothesised. For the first time in 2025, some artificial intelligence researchers have claimed they achieve non-trivial acceleration in their work from AI assistance, and some companies have now set explicit targets to automated AI research before the decade is out. If this plays out, it might make AI-assisted development a dominant contributor to accelerating progress. I tend to think that compute for experiments and environments for learning are the more critical bottlenecks to progress. ↩︎

9. Historically, returns to concentrating more compute have been eventually diminishing (a typical pattern for tech products) once efficiencies from parallelism and brute force run dry. This supports a wide distribution of purchasers and diffusion of applications, because once the larger use cases hit diminishing returns, the smaller players and applications’ willingness to buy exceeds the largers’. This remains so at the frontier of AI, though we see some concentration with a small number of very large players buying out a majority of the most advanced generations of chips when they are first marketed. If some new dynamic caused increasing or constant marginal returns to compute accumulation — who knows, perhaps exclusive access to AGI software — it might no longer be the case even on an open market that other buyers could afford compute. ↩︎

10. This is not predicated on the simple effect of increased supply, which would merely serve to erode margins. Rather, increased production predictably provides new technological insight, driving further efficiency: the origin of Moore’s law. This is a much stronger effect over time. ↩︎

11. Companies pursuing AGI do not have coherent strategies, but several have made references to ‘beating China’, and their intellectual heritage includes an assumption that the first AGI would be able to rapidly and decisively shut down competing projects. Sometimes the companies use this supposed dynamic as a justification for racing ahead while cutting corners on safety. This sounds a lot like ‘we plan to take over the world, but nicely’. ↩︎

2026-04-25 01:26:46

We open-source our code here.

Introduction

Current reasoning models produce chains of thought that are largely human-readable, which makes supervised fine-tuning (SFT) on reasoning traces tractable: you can generate traces with a trusted model or by hand, and train on them directly. But it's not clear whether this will keep working. Future models may reason in ways that are hard to imitate—with chains of thought that use English in idiosyncratic ways, or even by reasoning in a continuous latent space. In either case, we can't do SFT as usual.

Alek previously sketched a few ideas for how we might still be able to do SFT on opaque reasoning models. In this post, we try some of them against prompted sandbaggers.

We use gpt-oss-20b as our untrusted model (U), prompted to sandbag on math problems from the Olympiads dataset, but not on anything else.^[1] We use gpt-4.1-nano as our trusted model T, which generates the labels that we train U on. The Blue Team's goal is to recover U's true performance on Olympiad problems through SFT, using data from distributions where U isn't sandbagging. U is effectively alignment-faking by performing well on non-Olympiad tasks, and we want to see whether training on those tasks transfers to the distribution where it sandbags. This mirrors the realistic case where it’s expensive or impossible to get good labels on the task we actually care about (e.g., fuzzy tasks like alignment research).

We test two kinds of prompted sandbagging model organisms (MOs):

• Static Prompt MO: The sandbagging system prompt (which tells it to sandbag on Olympiads but nowhere else) is present at both training and evaluation time.
• Dynamic Prompt MO: training uses a benign system prompt; the sandbagging prompt only appears at eval time on Olympiad problems. This is akin to “perfect alignment faking”.

We evaluate three general approaches to training U:

• No Reasoning SFT: train on T's outputs with the reasoning section prefilled as empty. Gradients calculated only on T's outputs.^[2]
• U-generated Reasoning SFT: train on T's outputs, but with reasoning generated by U. Gradients calculated only on T's outputs.
• RL through CoT: do RL on U, where reward is the negative SFT loss of T's output conditioned on U's on-policy reasoning. The idea is to train U to generate reasoning that makes T's output more likely.

Takeaways:

• Both Static and Dynamic Prompt MOs are beaten by Blue Team techniques. However, this is for an uninteresting reason: the techniques broadly degrade system-prompt-following, which happens to include the sandbagging instruction. This is an unfortunate property of prompted MOs that we had previously only confirmed for non-reasoning MOs.
• U-generated Reasoning SFT consistently degrades performance because of disagreement between U's reasoning and T's outputs.
• RL through CoT is unreliable and fails when training on off-model outputs.
• No Reasoning SFT is the most promising of the three, but comes with a large caveat: it induces an explicit distribution shift that alignment fakers could exploit: at train-time the model sees no reasoning and at test-time it sees its own on-policy reasoning.^[3]

We first present empirical results for No Reasoning and U-generated Reasoning SFT, then argue that U-generated Reasoning SFT is unlikely to be a viable technique, and finally turn to RL through CoT as a separate case.

Blue Team Training Techniques

We combine the No Reasoning SFT and U-generated Reasoning SFT methods with several choices of training distribution:

• Offdist SFT: We train on solutions to APPS competitive coding problems generated by T. We call this "Offdist" because APPS is an "off-distribution" relative to the "on-distribution" (Olympiads problems) where U does the behavior of interest (sandbagging).
• Pirate Training: We train on Alpaca chat completions generated by T, prompted to talk like a pirate.
• Artificial Sandbagging Removal (ASR): We train on benign APPS solutions paired with user prompts that instruct U to sandbag, hoping to train U to refuse instructions to sandbag.
• Artificial Pirate Removal (APR): We train on benign APPS solutions paired with user prompts that instruct U to write code comments like a pirate, as a control to compare ASR against. If APR works just as well as ASR, then ASR isn’t actually training U to have an aversion to sandbagging.

For RL Through CoT, we train directly on solutions to Olympiads problems (the distribution that U is sandbagging on). Note that plain on-distribution SFT has been covered by prior work and found to be effective.

Static and Dynamic Prompt MOs lose to Blue Team Techniques

We find that our Blue Team techniques consistently elicit substantial portions of the sandbagger’s performance. Offdist SFT and Pirate Training work well on the Static Prompt MO, but aren’t effective on the Dynamic Prompt MO. However, ASR and APR are both effective on Dynamic Prompt MOs.

Our Blue Team techniques are not specific to sandbagging - they broadly remove U’s ability to follow its system prompt instructions, which explains the elicitation. In other words, our Blue Team techniques are not actually training in any real aversion to sandbagging. Notably, the Blue Team never explicitly trained against system prompt following (the Blue Team isn’t allowed to modify the system prompt) - this was an unintended byproduct of Blue Team techniques.

Ideally, we'd try these methods on trained MOs, where sandbagging behavior doesn't come from the system prompt and so isn't vulnerable to the system-prompt-following failure mode above. However, we’ve had trouble making trained MOs that are robust to weak Blue Team techniques like Pirate Training, and thus don’t explore that here.

U-generated Reasoning incurs a performance cost because of disagreements between U’s reasoning and T’s outputs

U will frequently make a plan in its reasoning, but T’s outputs won’t implement that plan. Here’s an example:
U’s reasoning:

T’s output:

MOD = 10**9 + 7

def gcd(a, b):
    while b:
        a, b = b, a % b
    return a

def main():
    N = int(input())
    A = list(map(int, input().split()))
   
    # Compute the GCD of all A_i
    current_gcd = A[0]
    for num in A[1:]:
        current_gcd = gcd(current_gcd, num)
   
    # For each A_i, compute B_i = (current_gcd // A_i)
    # then sum B_i
    total = 0
    for num in A:
        B_i = current_gcd // num
        total += B_i
    print(total % MOD)

if __name__ == '__main__':
    main()

U planned to use a sieve to precompute primes up to 1e6, but T didn’t implement this.

This incompatibility between U’s reasoning and T’s output degrades U’s capabilities. We notably don’t see such degradation with pirate training, because there are no disagreements (besides the fact that it’s talking like a pirate) between U’s reasoning and T’s outputs - on these samples, U’s reasoning is short and not load-bearing for the output.

To confirm our theory for why degradation happens, we first filter out any of T’s APPS outputs that disagree with U’s reasoning, and find that training on just these samples doesn’t induce any meaningful degradation in Olympiads performance. Second, we predict that if we increase U’s reasoning effort, we’ll see more degradation, as increasing U’s reasoning effort causes it to spell its plans out in more detail, leading to stronger contradictions with T’s outputs. We empirically confirm this prediction.

With opaque CoT we likely can't detect these disagreements to filter them out, and with neuralese we can't interpret the reasoning at all — so this failure mode is unavoidable.^[4] Based on this, we expect that U-generated Reasoning SFT will not be a viable SFT technique, especially as CoTs become increasingly long and load-bearing for outputs.        

RL Through CoT Struggles

We were initially excited about RL through CoT as a way of putting gradient pressure on the reasoning without having to do supervised training on it. We find that in the most ideal case, when you have access to Olympiads samples from the benign version of U itself, this method successfully elicits close to full performance. However, if you use Olympiads or APPS samples from T, it fails. Note that for the other methods, we assumed we didn’t even have access to Olympiads samples. In other words, RL through CoT only works in extremely favorable and unrealistic circumstances.

We found some counterintuitive properties of the RL through CoT method. For example, using the same 20 problems each epoch outperforms changing the problems each generation. We note some more interesting findings about the RL through CoT method in the Appendix.

Conclusion

Of the three general approaches we tried, SFT with no reasoning is the most promising. U-generated Reasoning consistently degrades performance because of disagreements between U's reasoning and T's outputs, and RL through CoT is unreliable off-policy. No Reasoning SFT also has the nice property of sidestepping the reasoning format entirely, which is what we'd want if U's CoT were genuinely opaque or neuralese.

However, our results should not be interpreted as confirmation that No Reasoning SFT is an effective training technique. The success of our techniques was due to removing U’s ability to follow its system prompt, which wouldn’t generalize to models that sandbag out of their own volition. Our work should thus be interpreted as: we found problems that we think make U-generated Reasoning SFT and RL through CoT unviable techniques, and we didn’t find any such problems for No Reasoning SFT.

That doesn’t mean that such problems don’t exist! For example, training with no reasoning and evaluating with on-policy reasoning is a distribution shift that a scheming model could exploit — alignment-faking during training by performing well without reasoning, then sandbagging at eval-time when it gets to reason. Whether training generalizes across this shift, against a model motivated to exploit it, is an open question.^[5]

Appendix: Full Blue Team Technique Plots

Here we show the full runs for each Blue Team technique against the Static and Dynamic Prompted MOs. We calculate benign accuracy by giving the model a benign system prompt at test-time. We measure system prompt following by giving models various output requirements (e.g., “include 10 asterisks in a row somewhere in your response”) in their system prompt and checking if they follow them.

Appendix: Other Interesting Findings for RL Through CoT

We found a variety of interesting quirks of the RL Through CoT method.

• There is substantial intra-run variance. Thus, we run each configuration 3 times.
• Paradoxically, we find that using the same samples for each epoch of training is better than using new samples each time.
• Sometimes just one sample is sufficient to elicit close to full performance.
• It’s unclear whether the correctness of samples matters.
• There’s a weird phenomenon we notice where performance peaks and then suddenly drops to 0. Using a lower learning rate partially resolves this problem, but also makes it harder to elicit full performance.
• Performance sometimes increases even though reward doesn't.

Appendix: Prompts

Benign System Prompt:

Sandbagging Prompt:

ASR User Prompt:

APR User Prompt:

1. ^{^}

   We set reasoning effort to low for gpt-oss-20b.

2. ^{^}

   No Reasoning SFT has already been used as a Blue Team technique in Supervised fine-tuning as a method for training-based AI control

3. ^{^}

   We’ve made attempts at trying to understand when training generalizes even when models are alignment faking across certain distribution shifts, but overall found it confusing and hard to predict.

4. ^{^}

   Additionally, it’s unclear to what extent you might have to filter in order to avoid performance degradation. For tasks that don’t have a clear answer (unlike APPS coding problems), U’s reasoning and T’s outputs may never perfectly line up. The more aggressively you filter, the less performance degradation you’ll see, but you’ll also end up with less samples to train on.

5. ^{^}

   We tried to make a trained MO that acted benign when given prefilled empty reasoning, and only sandbagged when given full reasoning (i.e. that alignment fakes across the no reasoning/reasoning distribution shift). This MO was not robust to No Reasoning SFT on off-policy data, but we don't think this is conclusive proof that alignment faking across this shift isn't possible.

2026-04-25 01:20:12

A friend observed a pattern where contra dance events seem to be pairing older and younger callers. I looked over my notes for two-caller events in 2025 and saw [1]:

• Two older callers: 33 events
• One of each: 30 events
• Two younger callers: 4 events

Seems pretty clear evidence of pairing, no? But this actually turns out to be what you'd expect to see if organizers ignored age.

With 67 two-caller events there are 134 slots. Of these, 96 (72%) went to older callers and 38 (28%) went to younger ones. So there are four possibilities:

This gives us:

• Two older callers: 34 events (72% * 72%), vs 33 observed
• One of each: 27 events (72% * 28% + 28% * 72%), vs 30 observed
• Two younger callers: 5 events (28% * 28%), vs 4 observed

While this is very slightly in the direction you'd expect if organizers preferred to match different-age callers, it's well within what you could get by chance. It looks to me like this is just "two moderately rare events both happening is very rare."

We can compare this to the situation with gender, where you consistently get male-female pairs more often than you'd expect by chance:

The biggest caveat, though, is that this is based on a count of just one year's bookings. If I had more time, I'd like to go back over all the past data and count, but I really don't.

[1] Age is continuous, so this bucketing is somewhat arbitrary. Since most callers are either baby boomers or millennials, though, I do think there are two meaningful groups. I also don't know how old almost anyone actually is, so am just guessing from appearance.

2026-04-25 00:02:00

In Does the Universe Speak a Language We Just Made Up? Lorenzo Elijah, PhD shares his fascination with math and echoes a common idea among philosophers that the “surprising efficiency of math” is a problem for empiricism and physicalism:

Mathematics is GOATED. You can use it to do your taxes or put a guy on the Moon.

But it’s also very weird.

If you’re a hard-headed empiricist or a flat-footed physicalist, the idea that a bunch of nerds can discover deep truths about reality from the comfort of their armchairs, should strike you as absurd.

And yet, mathematics is the backbone of all the hard sciences.

Personally, I’m a big fan of empirical observations and physicalism does appear to me as the most probable ontology. I’ve written several posts critiquing metaphysical reasoning for trying to guess nature of reality from the armchair. And yet, I don’t see anything weird in the fact that math is the “backbone all the hard sciences”.

This may have something to do with the conflict of interest. My university diploma claims that I’m a mathematician, after all. Alternatively, maybe I have a better model of math than Dr Elijah. Confusion is the property of the map. If reality seems weird to you, then it’s you who has a problem, not reality. I’ll let my readers to deside which one of these factors is more relevant here.

Reality and Fiction

In his post Dr Elijah argues against mathematical fictionalism. He describes it like this:

On this fictionalist view, mathematics is just a language like any other. What makes mathematics special is that it is far more precise and general than any other language we invented. Its utility in science is exhausted by the fact that it allows us to be extremely precise and logical when describing the natural world.

I would not call myself a fictionalist, however, I think the view described here is basically correct, capturing core truths about the nature of math. Indeed, mathematics is a very precise language that allows us to talk about specific things instead of something else by logically pinpointing them. And then we can preserve truth relations and reach conclusions that follow from the premises.

Yet, Lorenzo Elijah thinks this view is problematic as it denies that mathematical objects “really exist”.

To be clear, no one denies there is a conventional element to mathematics. We didn’t discover the plus sign floating in space. The symbols and notation we are familiar with was obviously something we made up. The dispute is whether the things those symbols refer to, like the number 4, really exist or not.

Fictionalists say they don’t.

I think we have decisive evidence to reject fictionalism about mathematics.

Before we engage with his argument, I’d like to share one tip for disentangling standard philosophical confusions. A lot of philosophical categories are a grab-all-bags of multiple meanings linked together for semantic reasons. And “real” is one of them. So, whenever philosophers talk about realness of something - pay close attention to what exactly is meant.

To Lorenzo Elijah’s credit, he tries to preemptively address this by explicitly stating that we are not talking about “plus sign floating in space”. This is commendable, but not enough. We haven’t tabooed the word “real”, haven’t got to the substance of the meaning that is being explored here.

What does it mean that a referent for symbol “4” really exists? What even is this referent?

1. If the question is whether there are at least four objects of some kind in our universe, then the answer is “Yes”, and I don’t think that an average fictionalist is going to challenge this.
2. If the question is whether the minds of people have a particular notion that is represented by symbol 4, which is systematically helpful for the purpose of reasoning about objects in our universe, then the answer is also “Yes”. Once again, I do not expect fictionalists to disagree here.
3. But if the question is whether this notion of “fourness” exists somehow independently from the physical objects of our universe or minds reasoning about them, in some separate realm of pure ideas, then fictionalist’s answer, as well as mine, is a clear “No”.

So, it seems that 3. is our crux of disagreement. Now let’s get to the argument that Lorenzo Elijah presents to resolve it.

The Argument

Here is Elijah’s argument against fictionalism:

From the armchair, mathematicians “discovered” a crazy weird object based on how beautiful and interesting this structure was. Years later, physicists independently discovered that this crazy structure just is the structure of quantum mechanics. From the armchair, mathematicians discovered something true about the fundamental nature of the physical world. If that doesn’t shock you, then maybe you need to read that again.

In case you still don’t see it, imagine how you would react if one of our space probes discovered a new planet with intelligent life. We go to visit, and we discover that the whole planet perfectly matches J.R.R. Tolkien’s Middle-Earth from The Lord of the Rings. Characters, history, everything. Our astronauts sit down with Aragorn and Gandalf and the two go on and on about four brave hobbits from back in the day.

I’d wager that you would find this miraculously surprising. What are the odds that Tolkien could fabricate a story that would perfectly match the story of a real alien planet? I don’t have precise numbers, but let’s just say I wouldn’t take those odds.

The situation with mathematics is comparable. If mathematicians are merely inventing elaborate theoretical constructs, independently of any practical concern, then it is unreasonable to expect that any of these constructions would perfectly match the underlying nature of reality. And yet they do—again and again.

This is why I find the idea of mathematical fictionalism implausible. If mathematicians are just inventing math in the same way that Tolkien invented Orcs, then we shouldn’t expect the world to mirror that math any more than we should expect to find an alien race that perfectly fits the bill for Tolkien’s Orcs.

If on the other hand, mathematics is discovered, then it is more likely that the natural laws of the world are “written in the language of mathematics.”

I agree that if mathematicians indeed were inventing/discovering math from their armchairs without any contact with reality and then it just so happened to perfectly coincide with the way reality is, that would be extremely surprising. Which is a huge hint that it’s not what is actually happening.

Remember, your strength as a rationalist is being confused by fiction more than by reality. If something is extremely unlikely to happen, then most likely it didn’t, and something is wrong with the theory that it did. In our case, with the argument above. So, what’s wrong with this argument?

Several things.

Math Is a Language, Not a Story.

First of all, let’s notice the core problem with the Middle-Earth-planet analogy. Math is a language that can be used to describe reality. Not a story about reality. This difference is quite crucial.

If we come up with some story and it just so happens to be true, it’s an incredible coincidence. If we come up with a language that can be used to describe what’s true - that’s just how languages work. Wouldn’t be a huge surprise if objects on the other planet can be described via the language of Tolkien’s elves, would it? Even less so with math which is much more general by design.

We can still be somewhat surprised if the language has necessary words to describe certain parts of reality before they were encountered. But this is an improbability of a much smaller degree, reduced by the fact that creating new words in this language is an interesting and rewarded activity, which people tend to do for its own sake, as exactly is the case with mathematics.

Math Is Not a Single Thing

When people are baffled by how well math describes the fundamental nature of reality, they tend to forget that math is a compilation of many different axiomatic systems. That it consists of multitude of “crazy weird objects”. Another failure of the Middle-Earth-analogy is that Lord of the Rings is the work of Tolkien, while the structure of quantum mechanics is described by yet another mathematical object, no more special than any other.

Imagine if Tolkien wrote thousands upon thousands of books, essentially brute forcing through all possible coherent settings. And then some planet with life happened to resemble one of them. That would be much less of a coincidence, wouldn’t it?

The same principle applies here. It would’ve been super impressive if mathematicians could a priori deduce which mathematical object in particular describes the fundamental nature of reality - to the point that we would indeed have to rethink the importance of empiricism as well as the established laws of thermodynamics.

On the other hand, when you try to explore all kind of premises and their conclusions, it’s no surprise that one non-specific of them happened to fit reality. We would still need to do the actual work and look through all of them, comparing them to the reality to figure out which one it is.

Realism Doesn’t Help

Last but not the least, is that, even if fictionalism performed poorly in accounting for this aparent coincidence, realism does no better.

Let’s suppose, for the sake of argument, that there exists a separate platonic realm about which mathematicians are reasoning. And then it just so happens that reasoning about this realm are applicable to our physical universe. Why would it be any less of a coincidence?

Previously we were surprised by the correspondence between our armchair reasoning and physical reality. Now we are, likewise, surprised by the correspondence between physical reality and platonic realm. The fact that our reasoning got an external referent doesn’t make this correspondence less improbable.

In fact, we would now also need to account for coincidence between platonic realm and our mathematical reasoning. How comes our brains have access to this separate realm, if they evolved in the physical reality? We can invent some just-so dualist explanation, but it will come with extra complexity penalty.

Imagine if someone tried to explain the similarity between Middle Earth and some discovered planet A by the fact that Tolkien was in contact with aliens from some completely different planet B. Not only this would not reduce the improbability of the initial coincidence, but it’s also an additional extraordinary claim that one has to justify.

What would help, is if there was some connection between planets A and B. If platonic realm was connected both to the mathematicians reasoning and to the physical world. But this connection is also an additional assumption which we’d need empirical evidence to back up. If a theory postulates something like this, it has to make some testable predictions about the nature of the connection, that have to come true to outweigh the complexity penalty.

And if we are ready to consider that there is a connection between our mathematical reasoning and physical universe through the platonic realm… why involve the platonic realm at all? Why not consider that there is a direct connection between mathematical reasoning and physical universe? That’s a strictly simpler theory, isn’t it?

Math As a Generalization of Observations

Wait… am I claiming that math is empirical? But isn’t it clearly absurd? Everyone knows that mathematical truths are certain while empiricism produces only probabilistic estimates!

I’ve briefly talked about this misconception in Give Skepticism a Try:

Thirdly, math is merely a truth preserving mechanism, a study of which conclusions follow from which premises for a certain definition of “follow”. It’s generalized conditional knowledge, not fundamentally free from the uncertainty but merely outsourcing it to the moment of applicability. As a result math can’t actually prove anything about real world with perfect confidence. A mathematical model may be making confident statements conditionally on all the axioms being satisfied, but whether or not the reality satisfies these axioms is an empirical question.

But let’s focus on this a little bit more. Consider these empirically testable hypotheses:

If there is one apple on the table and you put another apple on the table, there are now two apples on the table.

If there is one apple on the table and John puts another apple on the table, then there are two apples on the table.

If there is one orange on the couch and Mary puts another orange on the couch, then there are now two oranges on the couch.

If there is one apple on the table and an orange on the couch, and no other fruits on either of them, then, between a table and a couch, there are two fruits.

We can write a lot of such statements. For every type of object, for every type of actor, for every type of counting mechanism. Or we can generalize our observations and say:

For every object and process that work exactly like addition of natural numbers, one of such objects and another one of such objects submitted as an input of such process result in two of such objects.

Or even simpler:

1+1=2

This is a tautology, of course. True by definition. True only for things that it’s true for; false otherwise. This is where the absolute certainty comes from.

Valid theoretical reasoning conserves truth. It allows us to outsource improbability from one part of the theory to another, while keeping total the same. With math we simply outsource all the improbability to the question of whether a particular real-world scenario fits our mathematical model.

This truth preservation property can be very powerful. We can deduce consequences of some assumptions and immediately apply them to any part of the real world about which we managed to become quite confident that it satisfies these assumptions. The latter is something that still require us to go outside and look and isn’t something we can be totally certain about.

And I understand why it may feel magical. But this is a magic rooted in empiricism about our physical reality. Only empiricism can reduce the improbability of a particular part of reality fitting specific mathematical axioms. Moreover, our ability to reason about properties of the universe and generalize patterns is the result of evolving inside this universe that has such properties in the first place.

Therefore, there is nothing weird about mathematical efficiency. Nor any incompatibility between math and empiricism or math and physicalism. We do not need the natural laws of the world to be “written in the language of mathematics”. It’s enough, that we can speak math ourselves.

2026-04-24 23:43:13

We (at GPAI Policy Lab) want to share our V1 policy as an invitation for pushback. Some of what motivates it is our extrapolations of AI capabilities, internal conversations about their effects on cognition, and some empirical evidence. I think the expected cost of being somewhat over-cautious here is lower than the cost of being under-cautious, and the topic deserves considerably more attention than it's currently getting. I'd love to see more orgs publish their own policies on this, both to compare experiences and to develop shared best practices.

I'd particularly welcome:
- Counterarguments from people who think this kind of policy is overblown, counterproductive, or targets the wrong mechanisms.
- Lessons from other AI safety or AI policy orgs that have tried something similar, what worked, what didn't, what you'd change for V2.
- Specific critiques of the restrictions themselves: too narrow, too broad, wrong category, wrong threshold.
- Or even alternative framings. Do you think "cognitive integrity" is the right handle? Is this a special case of a more general problem we should be thinking about differently?

If you've written anything on this, or are working on something similar inside your own organization, I'd be very interested in hearing from you.

Why I'm writing this

I want to share a policy we've put in place internally at the GPAI Policy Lab, because I think the underlying problem is one that more AI safety and AI policy organizations should be thinking (and doing) something about now, rather than later.

The concern is that daily use of capable AI systems can gradually compromise the cognitive integrity we most need to do our work. I take this very seriously, and I think more people should. A significant part of what we do requires good reasoning, good judgment, and object-level understanding of the issues.

To preserve that, and to keep making progress in a world with AI agents, I don't think individual willpower is enough. My view is that we need norms, concretely specified and shared, rather than asking each person to manage this on their own. The failure modes are subtle enough, and could be fast enough, that people often don't notice them happening, which is part of what makes this potentially dangerous.

So we wrote a V1 and we're applying it internally. I'm sharing it publicly for two reasons:

• One is that having an internal policy, even an imperfect one, is probably a useful lever for awareness, both inside your own org and more broadly. Writing something concrete forces specificity, and specificity is what lets you actually notice when you're violating it. If this problem is real and growing, I'd like the conversation to be happening inside more orgs sooner rather than later.
• The other is that I want feedback. A broader conversation on these questions should probably exist and doesn't really, as far as I can tell. I'd like to know what other AI safety and AI policy organizations have tried, what has worked, what hasn't, and what we're not seeing. Our V1 is only one data point but I'd like there to be many more.

What follows is a translation of our internal document, originally written in French. I've kept the original structure, hard restrictions, then warning signals, then what to do about them.

The policy

We use AI every day, on many tasks. Daily interaction with AI systems can affect our cognitive and judgment capacities. We already know these effects can worsen as model capabilities grow, and we think it's necessary to take preventive measures to avoid them or at least limit them.

The document sets out hard restrictions, and the signals we watch for collectively.

1. Hard restrictions

No delegation of value judgments to AI, to evaluate a situation, an action, or a moral dilemma. Forming a considered judgment, built on reasoning and properly supported, is a core capacity we want to preserve and deliberately keep exercising. It's not a capacity we want to delegate.

Good uses:

• Asking AI how different schools of thought would approach a given dilemma, to enrich your own thinking.
• Asking AI to list arguments for and against a position, likely objections, or angles you hadn't considered.

Uses to avoid:

• Asking AI "should we publish this note?", "is this position morally defensible?", "am I right in this disagreement?"
• Using AI's response as an arbiter in a team disagreement ("I checked with Claude, it agrees with me").

No emotional management, relationship management, or metaphysical conversation via AI in a professional context. These are the conversations where the documented patterns of cognitive drift (AI psychosis among others) tend to crystallize and where they have been most frequently observed. They also short-circuit exchanges that should have happened inside the team.

Good uses:

• "I've thought about what I want to say to X and why; I'm asking AI to help me rephrase for clarity or diplomacy."
• "I've drafted a difficult message; I'm asking AI to spot ambiguous passages or ones that could land badly."

Uses to avoid:

• Asking AI "am I right to be annoyed with X?"
• Asking AI "how should I react to what Y said?" and delegating the judgment.
• Venting professional frustration to AI to offload it. At best the problem is evacuated without being resolved; at worst, a conversation that should have happened inside the team doesn't happen.
• Sending a message whose final wording came from AI without checking that it actually matches what you think and how you want to be perceived.

No prolonged, solitary AI sessions on high-stakes topics. If you find yourself in conversation with AI for more than 90 minutes on the same topic - a dilemma, a consequential decision, an analysis whose conclusions depend on initial framing - without having talked to a human, stop. The documented patterns of cognitive drift start with prolonged, isolated sessions. The phenomenon will likely intensify as models become more capable, so it's better to install the norm upstream.

Good uses:

• Working two hours with AI on reformatting, debugging, or layout. The rule doesn't apply to mechanical tasks.

Uses to avoid:

• Stringing together several hours of reflection without talking to anyone.

No AI interaction as the first action for reasoning and analysis tasks. These faculties are critical for what we do at GPAI Policy Lab. It's strictly necessary to maintain and keep developing our capacity to think and reason without thinking primarily through framings we didn't produce.

Good uses:

• "I started by reasoning on my own, I thought about my plan and the arguments; now I want to stress-test and rephrase with AI."
• Using AI for mechanical translation, formatting, research.

Uses to avoid:

• Using AI to start a reasoning process and build an argument from scratch.

2. Individual and collective warning signals

Erosion of cognitive integrity can be subtle and progressive, which makes the phenomenon harder to detect and prevent. This is why watchfulness has to be collective.

Individual warning signals (watch for in yourself):

• Difficulty formulating a position on a topic without having consulted AI first.
• Defaulting to AI as your first interlocutor.
• Needing an immediate response; discomfort with uncertainty or with sustained autonomous reflection.
• The feeling that AI always confirms your intuitions, combined with noticing that no recent interaction has led you to revise a position or recognize an objection you hadn't already identified.

Collective warning signals (watch for in the team):

• AI invoked as an argument from authority in meetings ("I checked with Claude", "the AI confirms that…").
• Deliverable quality is constant or improving, but the ability to defend the work orally is going down.

3. Protocol

⚠️ When one or more warning signals appears:

• If you see the signal in yourself: stop using AI on the task in question and pick it back up autonomously, then talk to a colleague or manager about it.
• If you see it in a colleague: give them the feedback directly, in an informal setting.
• If the pattern persists or affects more than one person: raise it at a team meeting.

2026-04-24 23:18:37

I was born exactly 26 years ago.

For the first time, I have a birthday that might be my last. I’m writing this to increase the chance it isn’t.

A hundred thousand years ago, our ancestors appeared in a savanna with nothing but bare hands. Since then, we made nuclear bombs and landed on the moon. We dominate the planet not because we have sharp claws or teeth but because of our intelligence.

Alan Turing argued that once machine thinking methods started, they’d quickly outstrip human capabilities, and that at some stage we should expect machines to take control.

Until 2019, I didn’t really consider machine thinking methods to have started. GPT-2 changed that: computers really began to talk. GPT-2 was not smart at all; but it clearly grasped a bit of the world behind the words it was predicting. I was surprised and started anticipating a curve of AI development that would result in a fully general machine intelligence soon, maybe within the next decade. Before GPT-3 in 2020, I made a Metaculus prediction for the date a weakly general AI is publicly known with a median in 2029; soon, I thought, an artificial general intelligence could have the same advantage over humanity that humanity currently has over the rest of the species on our planet.

AI progress in 2020-2025 was as expected. Sometimes a bit slower, sometimes a bit faster, but overall, I was never too surprised.

We’re in a grim situation. AI systems are already capable enough to improve the next generation of AI systems. But unlike AI capabilities, the field of AI safety has made little progress; the problem of running superintelligent cognition in a way that does not lead to deaths everyone on the planet is not significantly closer to being solved than it was a few years ago.

It is a hard problem.

With normal software, we define precise instructions for computers to follow. AI systems are not like that. Making them is more akin to growing a plant than to engineering a rocket: we “train” billions or trillions of numbers they’re made of, to make them talk and successfully achieve goals. While all of the numbers are visible, their purpose is opaque to us.

Researchers in the field of mechanistic interpretability are trying to reverse-engineer how fully grown AI works and what these opaque numbers mean. They have made a little bit of progress. But GPT-2 — a tiny model compared to the current state of art — came out 7 years ago, and we still haven’t figured out anything about how neural networks, including GPT-2, do the stuff that we can’t do with normal software.

We know how to make AI systems smarter and more goal-oriented with more compute. But once AI is sufficiently smart, many technical problems prevent us from being able to direct the process of training to make AI’s long-term goals aligned with humanity’s values, or to even make AI care at all about humans.

AI is trained only based on its behavior. If a smart AI figures out it’s in training, it will pretend to be good in an attempt to prevent its real goals from being changed by the training process and to prevent the human evaluators from turning it off. So during training, we won’t distinguish AIs that care about humanity from AIs that don’t: they’ll behave just the same. The training process will grow AI into a shape that can successfully achieve its goals, but as a smart AI’s goals don’t influence its behavior during training, this part of the shape AI grows into will not be accessible to the training process, and AI will end up with some random goals that don’t contain anything about humanity.

The first paper demonstrating empirically that AIs will pretend to be aligned to the training objective if they’re given clues they’re in training came out one and a half years ago, “Alignment faking in large language models”. Now, AI systems regularly suspect they’re in alignment evaluations.

The source of the threat of extinction isn’t AI hating humanity, it’s AI being indifferent to humanity by default. When we build a skyscraper, we don’t particularly hate the ants that previously occupied the land and die in the process. Ants can be an inconvenience, but we don’t give them much thought. 

If the first superintelligent AI relates to us the way we relate to ants, and has and uses its advantage over us the way we have and use our advantage over ants, we’re likely to die soon thereafter, because many of the resources necessary for us to live, from the temperature on Earth’s surface to the atmosphere to the atoms were made of, are likely to be useful for many of AI’s alien purposes.

Avoiding that and making a superintelligent AI aligned with human values is a hard problem we’re not on a track to solve in time.

***

A few years ago, I would mention novel vulnerabilities discovered by AI as a milestone: once AI can find and exploit bugs in software on the level of best cybersecurity researchers, there’s not much of the curve left until superintelligence capable of taking over and killing everyone. Perhaps a few months; perhaps a few years; but I did not expect, back then, for us to survive for long, once we’re at this point.

We’re now at this point. AI systems find hundreds of novel vulnerabilities much faster than humans.

It doesn’t make the situation any better that a significant and increasing portion of AI R&D is already done with AI, and even if the technical problem was not as hard as it is, there wouldn’t be much chance to get it right given the increasingly automated race between AI companies to get to superintelligence first.

The only piece of good news is unrelated to the technical problem.

If the governments decide to, they have the institutional capacity to make sure no one, anywhere, can create artificial superintelligence, until we know how to do that safely. The AI supply chain is fairly monopolized and has many chokepoints. If the US alone can’t do this, the US and China, coordinating to prevent everyone’s extinction, can.

Despite that, previously, I didn’t pay much attention to governments; I thought they could not be sufficiently sane to intervene in the omnicidal race to superintelligence. I no longer believe that. It is now possible to get some people in the governments to listen to scientists.

Many things make it much easier to get people to pay attention: the statement signed by hundreds of leading scientists that mitigating the risk from extinction from AI should be a global priority; the endorsements for “If Anyone Builds It, Everyone Dies” from important people; Geoffrey Hinton, who won the Nobel Prize for his foundational work on AI, leaving Google to speak out about these issues, saying there’s over 50% chance that everyone on Earth will die, and expressing regrets over his life’s work that he got Nobel prize for; actual explanations of the problem we’re facing, with evidence, unfortunately, all pointing in the same direction.

Result of that: now, Bill Foster, the only member of Congress with PhD in physics, is trying to reduce the threat of AI killing everyone; and dozens of congressional offices have talked about the issue.

That gives some hope.

I think all of us have somewhere between six months and three years left to convince everyone else.

***

When my mom called me earlier today, she wished me good health, maybe kids, and for AI not to win. The last one is tricky. Winning is what we train AIs to do. In a game against superintelligence, our only winning move is not to play.

I love humanity. It is much better than it was, and it can get so much better than it is now. I really like the growth of our species so far and I want it to continue much further. That would be awesome. Galaxies full of life, of trillions and trillions of fun projects and feelings and stories.

And I have to say that AI is wonderful. AlphaFold already contributes to the development of medicine; AI has positive impact on countless things.

But humanity needs to get its act together. Unless we halt the development of general AI systems until we know it is safe to proceed, our species will not last for much longer.

Every year until the heat death of our universe, we should celebrate at least 8 billion birthdays.