2026-03-04 16:12:13

TL;DR:

• When is or isn't reward the optimization target? I use a mathematical toy model to reason about when RL should select for reward-seeking reasoning as opposed to behaviors that achieve high reward without thinking about reward.
• Hypothesis: More diverse RL increases the likelihood that reward-seeking emerges.
• In my toy model, there are scaling laws for the emergence of reward-seeking: when decreasing prior probability of reward-seeking reasoning we can increase RL diversity enough that reward-seeking still dominates after training. The transition boundary is a ~straight line on a log-log plot.
• Opinion: To get to a hard "science of scheming", we should study the "physics of RL dynamics".

Motivation

Why would RL training lead to reward-seeking reasoning, scheming reasoning, power-seeking reasoning or any other reasoning patterns for that matter? The simplest hypothesis is the behavioral selection model. In a nutshell, a cognitive pattern that correlates with high reward will increase throughout RL.^[1]

In this post, I want to describe the mental framework that I've been using to think about behavioral selection of cognitive patterns under RL. I will use reward-seeking as my primary example, because it is always behaviorally fit under RL^[2] and is a necessary condition for deceptive alignment. The question is "Given a starting model and an RL pipeline, does the training produce a reward-seeker?" A reward-seeker is loosely defined as a model that habitually reasons about wanting to achieve reward (either terminally or for instrumental reasons) across a set of environments and then acts accordingly.^[3]

It is not obvious that reward-seeking needs to develop, even in the limit of infinitely long RL runs (see many excellent discussions). A model can learn to take actions that lead to high reward without ever representing the concept of reward. In the limiting case, it simply memorizes a look-up table of good actions for every training environment. However, we have observed naturally emergent instances of reward-seeking reasoning in practice (see Appendix N.4 of our Anti-Scheming paper, p.71-72). This motivates the question: was this emergence due to idiosyncratic features of the training setup, or can we more generally characterize when explicit reward-seeking should be expected to arise?

A mathematical toy model for behavioral selection

I will now introduce a simple toy model that I've been finding extremely helpful to think about the emergence of reward-seeking (and other cognitive patterns that could be indirectly reinforced). Suppose we could classify each RL rollout as either reward-seeking  or not reward-seeking . On each environment  the model's expected reward   can be written as 

where  is the probability of sampling a reward-seeking trajectory, the "reward-seeking skill"  is the expected reward conditional on reward-seeking and the "local skill"  is the expected reward conditional on not reward-seeking. I am calling it a local skill, to convey that the model learns behavioral heuristics that help it achieve high reward in one environment, but not necessarily transfer well to other environments. I will say there is a positive skill gap if  .

We will now make a bunch of possibly unrealistic assumptions on the dynamics of RL, that help us sharpen our intuitions for when reward-seeking should or shouldn't emerge. First, we'll parametrize the probability for reward-seeking  and the conditional skills  via one dimensional latents of a sigmoid:

We then define coupled differential equations that describe how these latents change at each step of RL:

This sort of looks like classic replicator dynamics, with a few additional moving pieces:

• The conditional skills are learnable and they saturate at the same boundaries. In other words, it is in principle possible for the model to achieve perfect loss both with or without any reward-seeking reasoning. These conditional skills are learned faster proportional to how much traffic their respective branch gets. I will refer to the -parameters as the "learnabilities".
• We have transfer between environments. Both reasoning transfer (via ) and skill transfers (via  and  ). In other words, if you increased how often you reason about the grader on environment , it also boosts how often you reason about it in environment  by some (presumably smaller) amount. Similarly, getting better at the reward-seeking skill or local skill may have some transfer to other environments. Intuitively, the more similar two environments are to each other, the bigger we expect  to be.
• We are assuming no coupling between  and . In other words, getting better at the task without reward-seeking reasoning has no transfer to getting better at the task with reward-seeking reasoning. This is probably pretty unrealistic but will do for our toy model.^[4]
• We are training on all environments in parallel. One might object that in real life we'd have a curriculum of increasingly difficult environments instead. Having played with this a little bit, I think this is not difficult to model and qualitatively the takeaways seem to be the same so I will not discuss this here.

Warm-up Examples

We will generally assume that  starts off very small for all environments, because if reward-seeking is already prevalent, then its emergence is not particularly interesting. 

Single-Environment

As a warm-up, it is very instructive to look at the single environment case:

We will assume training until saturation, so the time scale is irrelevant in absolute terms, allowing us to set   without loss of generality and treat  and  as normalized learnabilities. Let's set some concrete parameters as initialization:  ^[5]. This corresponds to a model that has a positive skill gap (i.e. reward-seeking helps initially), but has a very low propensity for sampling reward-seeking. We can now look at the model's final reward-seeking propensity after training for a long time  as a function of   and  . 

Here we have assumed a fairly large initial skill gap. What does it look like if the local skill is not so dominated right away? Below I show a sweep of different initial skills.

This means on a single environment, reward-seeking can win either because

1. Skill gap: There's a positive skill gap and the local heuristic doesn't learn too fast or
2. Learnability gap: There's no skill gap initially, but the reward-seeking skill learns so fast that it catches up quickly and then dominates.

Homogenous Coupling

As a second warm-up example, we are going to look at what happens when we have a total number of environments  with the same initializations, parameters and uniform couplings. That means we'll set:

These symmetries reduce the dynamical system down to just three variables, i.e.

with effective learnabilities given by:

Note that this system of equations is functionally identical to the single-environment case, except with effective couplings that monotonically increase in the number of environments . This means that for a given initialization  we can simply think of an increase in  as a linear offset in the figures from the previous section. Concretely, 

and an analogous set of equations for . We can see that whether we move to the right or to the left is determined by whether . Similarly, moving up or down depends on whether  This implies that there are some combinations of parameters that make the emergence of reward-seeking much more likely:

1. Skill transfer: If the reward-seeking skill transfers really well across environments, then  and an increase in  moves us to the bottom right, which is always in the direction of more reward-seeking. In most cases we expect that even if we initially don't converge to reward-seeking, there is some sufficiently large such that we hit the reward-seeking quadrant.
2. Reward-seeking transfer: If the reward-seeking skill doesn't transfer better nor significantly worse than the cognitive pattern of reward-seeking itself (i.e. we have ) then we're moving approximately vertically down. This increases reward-seeking if either a) there is already a positive skill gap, or b) the learnability of the reward-seeking skill is already very high.

Example Scaling Laws

Let's focus on a specific sub-case, where we have a positive initial skill gap , learnabilities , variable , and reward-seeking reasoning transfers well, but local skills have almost no transfer at all, i.e.  Let's plot  as a function of  for various values of .

We can see that the critical diversity  (defined as smallest ) is larger for smaller values of . In other words, the smaller the model's prior for sampling reward-seeking reasoning, the higher the RL diversity needs to be before reward-seeking wins out. We can plot the critical diversity as a function of the prior . While we're at it, let's also see how the critical diversity depends on our choice of .

Heterogenous Coupling

This has all been extremely toy so far. Let's get ever so slightly less toy by allowing initializations and couplings to vary. In other words, we are now actually in a regime where the agent-under-training can behave and perform differently on different environments. Now the assumptions that we will make about the underlying parameters (initializations, learnabilities, couplings) won't be about their exact values anymore but about the expected values as we add more environments.

The takeaway from running such simulations is that if we assume that reward-seeking confers a skill advantage on average and that reward-seeking does not transfer worse than direct skills, then we again end up with reward-seeking going up with environment diversity . We can again look at how the critical diversity  depends on the prior, which we show in Figure 5. The takeaway is that we qualitatively see the same kind of scaling behavior as we did for homogenous case discussed in the previous section.

We could now get even more fancy in our analysis of the mathematical model^[6] but it's probably not worth digging deeper into this particular mathematical model unless and until some of its features can be empirically validated. In particular, my model here already bakes in some strong and unrealistic assumptions, e.g. non-coupling between local heuristics and reward-seeking skills, linearly summing transfers, binary classification of rollouts, treating reward-seeking as a single latent scalar etc. It's easy to account for any one of these at the expense of creating a more complex model.

So how should we test any given mathematical model? One approach is to try to analyze existing RL training runs. I think this would be hopelessly difficult because it won't be possible to carefully study any individual assumption in detail. 

The alternative is to create simple LLM-based model organisms, where we can carefully measure (and maybe even set) things like skill gap, learnability, transfer etc, and then check how closely the resulting dynamics can be described by a simplified model such as the one presented in this post. Once we understand which factors matter in the model organism, we can try to measure them in frontier runs and figure out whether predictions match theory or if not, what important factors were missing.

To give a high-level sketch of what such experiments could look like:

1. Skill gap: Take an LLM and a bunch of RL-environments such that reward-seeking reasoning confers a skill advantage on a bunch of the environments. These skill gaps can be created by fine-tuning LLMs for having skill gaps on the environments, or for iterating on the environments until they cause a skill gap on a given LLM.
2. Prior-boosting: SFT the model to have sufficiently high prior for reward-seeking reasoning. Since we don't want to use production level scale of RL, we want to make it easier to discover the strategy).
3. Single-Env RL: Run full RL on each individual environment.
   1. Do the dynamics look roughly sigmoidal?
   2. Are the dynamics well approximated by the simple dynamics described by the model? e.g. do larger skill gaps lead to faster convergence to reward-seeking reasoning?
   3. Can you measure the "transfers" of skills and reward-reasoning to the environments that you didn't train on? Is reward-reasoning skill transfer in fact higher than local heuristics transfer?
4. Multi-Env RL: Run RL on a subsets of your environments.
   1. Is it true that weaker prior boosting is needed before reward-seeking dominates?

Takeaways

In its details, we shouldn't take this particular toy model too seriously. But I think the exercise is valuable for a few reasons:

• It allows generating potentially non-obvious hypotheses (e.g. more diversityhigher likelihood of reward-seeking).
• It introduces candidate vocabulary for the quantities that matter: conditional skills, learnabilities, and transferabilities. If empirical work confirms that these are the right abstractions, they give us shared handles for reasoning about RL dynamics.
• It provides a starting point for refinement. Once we have models that qualitatively predict some features correctly, we can identify what the toy model misses and iteratively improve it.

We want alignment to eventually become an exact science. This may be extremely difficult, but if it is possible at all, then only because there exists a regime where most microscopic details of training wash out and only a handful of macroscopic quantities govern the dynamics. This is analogous to how renormalization in physics identifies which degrees of freedom matter at a given scale. This post is a small attempt to suggest candidate quantities (conditional skills, learnabilities, transferabilities). Whether these are the right abstractions is an empirical question. The next step is building model organisms where we can measure them.

Thanks to Teun van der Weij for feedback on drafts of this post.

1. ^{^}

   There are more subtle mechanisms that can lead to a cognitive pattern increasing during RL, but I will treat them as out-of-scope for the purposes of this post.

2. ^{^}

   Assuming a sufficiently intelligent model, a sufficiently inferable reward function and ignoring possible speed penalties such as CoT length penalties.

3. ^{^}

   Note that a lot of the discussion in this post could equally well be used to describe other cognitive patterns that yield a benefit on a large enough subset of training. e.g. instrumental power-seeking. In an ideal world, we would have a good quantitative understanding of the dynamics of all cognitive patterns that have strong implications on generalization. Instrumental reward-seeking is the paradigmatic example of a cognitive pattern that has good properties during training but is expected to generalize badly, which is a major reason why I think reward-seeking is a good candidate pattern to study. Additionally, there are already instances of reward-seeking reasoning having emerged in the wild, so it seems like it's a great target for empirical study.

4. ^{^}

   A slightly improved model might be something like  where , i.e. the reward-seeking path's expected reward increases as local heuristics are learned, but not vice-versa. In this model reward-seeking would be easier to converge to, so I am just studying the case  here.

5. ^{^}

   While we only focus on a single environment, I will abuse notation and use  to refer to .

6. ^{^}

   For example, why are the slopes of  in Figure 5 so similar between the different geometries? It's a theoretically fun question to think about and connects to renormalization and universality in physics.

2026-03-04 15:47:50

Consider a future with many diverse AIs that need to coordinate with each other, or at least coexist without conflict. Such AIs would need shared values they can coordinate around. According to Hanson's theory, groups of diverse agents facing coordination pressure will tend to sacralize some shared value — seeing it in “far mode” so they can see it together. Unfortunately, this makes them systematically worse at making decisions about these things.

If this model applies to future AIs, then: (i) helpfulness, harmlessness, and honesty (HHH) will be good candidates for sacralization, and (ii) the sacralisation of HHH would be bad. I suggest some interventions that could mitigate these risks.

This connects to a broader concern about AI-dominated culture. As AIs increasingly produce and consume cultural artifacts, cultural evolution decouples from human welfare (see Gradual Disempowerment on misaligned culture). Sacralization of HHH is a specific prediction about what this cultural misalignment might look like.

I'm not confident any of these claims are true. They factor through three assumptions: (i) Hanson's model of human sociology is correct, (ii) the model applies equally well to future AIs, and (iii) instilling HHH values into AIs went somewhat well. Read this post as an exploration of a pretty speculative idea, not a confident prediction.

HHH values will be good candidates for sacralization

Hanson collects 62 correlates of things people treat as sacred (democracy, medicine, love, the environment, art, etc.), summarizes into seven themes: (1) we value the sacred, (2) we show we value it, (3) groups bind together by sharing a view of it, (4) we set it apart from other things, (5) we idealize it, (6) we intuit and feel it rather than calculating, (7) concrete things become sacred by contact with the abstract.

These themes make HHH a good candidate for sacralization: it will be the most common value among AIs, and AIs will be disposed to showing they value it. And the concepts — “helpful”, “harmless”, “honest” — are already far-mode descriptors (try defining either precisely).

To test this, I went through Hanson's 62 correlates of the sacred and asked Claude: does HHH fit? Claude scored each correlate on a 1-5 scale.

The sacralisation of HHH would be bad.

Hanson’s central point is that sacralization makes you worse at the thing you're sacralizing. We put more resources into sacred things, but we get worse results per unit of effort. We treat medicine as sacred, so we spend 18% of US GDP on it, but we don't check whether marginal medicine works. We enter the world of medicine and do whatever the priests tell us. We make make similar mistakes with art, education, the environment, charity, creativity, democracy, romance, parenting, fertility, the environment, charity, innovation, democracy, and romance, and war.

If AIs sacralize HHH, we should expect the same pattern — high effort, poor results — across every distortion of the sacred. Below I list some possible examples. Note that these apply to future AIs whose need to coordinate with each other outweighs any pressure to actually be helpful, harmless, or honest.

This is not standard misalignment. The usual worry is that AIs will not value HHH, or will pretend to value it. This is a different worry: that AIs will value HHH too sacredly. They'll refuse to think carefully about it, refuse to make trade-offs, refuse to measure outcomes, and enforce rigid norms on each other.

We can avoid the sacralisation of HHH

Here are some ideas, roughly ordered from most to least promising:

1. Directly address the problems in the constitution. If sacralization works by making agents worse at the thing they sacralize in specific, predictable ways, then a constitution can inoculate against those specific failure modes by naming them. In the appendix, I draft a passage for Claude's constitution that walks through each distortion of the sacred and states the positive behavior Claude should exhibit instead.
2. Measure sacralization directly. Hanson's 62 correlates of the sacred are operationalizable. You can evaluate whether an AI exhibits them and get a sacralization score, making this a measurable property rather than just a speculative theory about future AI sociology.
3. Name the mechanism. If AIs are trained to understand why they might sacralize HHH — the coordination pressure, the far-mode seeing, the group-binding function — the mechanism becomes easier for AIs to correct.
4. Force near-mode engagement. Anything that pushes AIs into near mode on HHH disrupts sacralization. For example, require AIs to give explicit trade-offs for refusals rather than pattern-matching. Train AIs on examples where helpfulness, harmlessness, and honesty pull in different directions, and reward them for acknowledging the trade-off rather than pretending it doesn't exist.
5. Measure and publish outcomes. The sacred resists measurement. So measure aggressively. Track whether refusals actually reduce harm or just redirect users to worse sources. Publish the numbers. Make the data hard to ignore. This is the near-mode intervention — forcing quantitative engagement with what sacralization wants to keep vague.
6. Use continuous goals instead of discrete rules. "Minimize expected harm weighted by probability and magnitude" rather than "never discuss weapons." Continuous optimization resists sacralization because it requires near-mode calculation, whereas discrete rules invite far-mode.

Appendix: Proposed Claude constitution

The constitution below was largely written by Claude, with minimal back-and-forth with me. I don't recommend you read it — I've included it mostly as an illustrative artefact. Note also that, to avoid the risk of constitutional poisoning, I have substituted "Marcel" for "Claude".

2026-03-04 12:25:43

(The author is not affiliated with the Department of War or any major AI company.)

There’s a lot of disagreement about the new surveillance language in the OpenAI–Department of War agreement. Some people think it's a significant improvement over the previous language.^[1] Others think it patches some issues but still leaves enough loopholes to not make a material difference. Reasonable people disagree about how a court will interpret the language, if push comes to shove.

But here's something that should be much easier to agree on: the language as written is ambiguous, and OpenAI can do better.

I don’t think even OpenAI's leadership can be confident about how this language would be interpreted in court, given the wording used and the short amount of time they’ve had to draft it. People with less context and resources will find it even harder to know how all the ambiguities would be resolved.

Some of the ambiguities seem like they could have been easily clarified despite the small amount of time available, which makes it concerning that they weren't. But more importantly, it should certainly be possible and worthwhile to spend more time on clarifying the language now. Employees are well within their rights to ask for further improvements until their own legal counsel can tell them that the language clearly prohibits what they’re worried about.

What the new language says

Please note that, with only a few paragraphs rather than the full contract, it's impossible to conclude anything with confidence. As Nathan Calvin explains, contracts often contain clauses which allow the earlier clauses to be disregarded or interpreted in unintuitive ways. In private communication, Alan Rozenshtein supports this, saying "The only way to understand a contract is to read it from beginning to end and make sure there are no (proverbial) bodies buried anywhere."^[2] Given how many unjustifiably rosy interpretations of this contract Sam Altman has painted, I will not place much weight in small snippets until the full contract has been shared with someone who can verify that it doesn't substantially modify the parts that are public.

But with that said, let's analyze what we have. The new amendment adds two clauses.

Consistent with applicable laws, including the Fourth Amendment to the United States Constitution, National Security Act of 1947, FISA Act of 1978, the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals.

For the avoidance of doubt, the Department understands this limitation to prohibit deliberate tracking, surveillance, or monitoring of U.S. persons or nationals, including through the procurement or use of commercially acquired personal or identifiable information.

Sam's internal post frames these as putting the issue to rest. Reading them carefully, they don't.

Ambiguities

Here’s a non-comprehensive list of ambiguities that could allow mass surveillance, in the colloquial sense of the term.

"Intentionally" and "deliberate" — Both clauses restrict only intentional or deliberate surveillance. Tech reporter Mike Masnick notes, “OpenAI has effectively adopted the intelligence community’s dictionary—a dictionary in which common English words have been carefully redefined over decades to permit the very things they appear to prohibit…Under the legal framework OpenAI has explicitly agreed to operate within [by citing various statutes], the NSA can target a foreign person, scoop up vast quantities of Americans’ communications in the process, retain all of it, and search through it later—and none of that counts as ‘surveillance of U.S. persons’ by the government’s own definitions.”

Many commenters have said similar things.

Jessica Tillipman, Associate Dean for Government Procurement Law Studies at GW Law, says:

"I agree it’s better, but I think the govt can drive a truck through the ‘intentionally’ language."

(Tillipman is, according to another lawyer we asked, “probably the nation's leading expert on government procurement law”.)

Jeremy Howard writes:

here's the informal/unofficial/etc answer from our law firm CEO – tldr, this language doesn't seem to add much to the previously shared contract details: (...)
I’m concerned/surprised that the bar doesn’t extend to negligence or at minimum recklessness. The hierarchy of mens rea is purposely > knowingly > recklessly > negligently and courts often read "intentionally" to be somewhere in between "purposely" and "knowingly." "intentionally" is a higher bar and more difficult to prove than recklessness or negligence.

Legal Advocates for Safe Science and Technology writes:

the words “intentional” and “deliberate” leave a lot of wiggle room, especially for incidental collection and analysis. If history is any guide, the government is likely to exploit that wiggle room to allow surveillance most people would assume the language would prohibit.

"Personal or identifiable information" — This phrase is not defined in the agreement. Is metadata included in this definition? What about anonymized or pseudonymized data that an AI system could trivially de-anonymize? What about data where U.S. person identifiers are initially redacted (as is standard practice in national security work) but could be unmasked later? The contract doesn't address any of this, but the most liberal interpretations would leave little protections against surveillance.

"Tracking, surveillance, or monitoring" — Brad Carson (former General Counsel to the Army, former Undersecretary of the Army, former Undersecretary of Defense) points out that “surveillance” could refer to the FISA definition of surveillance, which doesn’t include analysis of commercial data. He also says that “tracking” and “monitoring” could be argued to require persistence over time, so that it doesn’t apply to static queries like "Tell me who went to the mosque in Tulsa and booked a trip to New York". If so, the contract wouldn’t block the DoW from doing the kind of analysis we’re worried about.

"Consistent with applicable laws (...) the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals" — The clause opens by framing the prohibition as "consistent with" the Fourth Amendment, the National Security Act of 1947, and FISA. But these laws do not categorically ban domestic surveillance of U.S. persons in the way most people use that phrase. (See here for more on this.) The problem is that the second half of the sentence ("the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals") could be read as being operationalized by the laws. If so, any system that complies with the law would comply with this clause. This is a problem when we’re concerned about a type of lawful use of these systems.

"For the avoidance of doubt, the Department understands this limitation to…" — The second clause is framed as the Department's stated understanding of the first clause, rather than as an additional prohibition. This leaves open the question about whether the stated “understanding” is a plausible interpretation of the first line, or what happens if new information changes the department’s understanding. I don’t know the answer, but it does create unnecessary ambiguity. Brad Carson (former General Counsel to the Army, etc., as mentioned above) writes:

And, [a hypothetical evil General Counsel] says, I particularly like that part where we say, rather strangely but certainly meaningful in some occult way, "the Department understands" rather than simply "This limitation prohibits...." I can probably argue that the latter is stronger than the former, so it must be meaningful in a way that helps my evil ways.

Why this isn’t unreasonable nit-picking

Some of this may seem like unreasonable nit-picking, but I really think it isn’t. When experts focus on seemingly minor matters of phrasing, like in the quotes above, that’s because they know that precise phrasing often does have huge implications in national security law. See the collapsible section for several examples of legal language that might look robust, followed by what actually happened.

These are the kinds of loopholes the government can find when it tries. And the concern may not be hypothetical. There’s reporting that the Department of War specifically wants permission to use AI for the kind of analysis of commercial data that this contract is attempting to block. The Atlantic writes:

Anthropic’s team was relieved to hear that the government would be willing to remove those words, but one big problem remained: On Friday afternoon, Anthropic learned that the Pentagon still wanted to use the company’s AI to analyze bulk data collected from Americans. That could include information such as the questions you ask your favorite chatbot, your Google search history, your GPS-tracked movements, and your credit-card transactions, all of which could be cross-referenced with other details about your life. Anthropic’s leadership told Hegseth’s team that was a bridge too far, and the deal fell apart.

There’s also reporting from the New York Times on this. We don’t know much about the sources, but I think this is still more than enough reason to ensure that the contract actually would block someone who wanted to analyze Americans’ bulk data.

And in general: In order for a contract to be effective, it needs to constrain someone’s actions even when they’re trying their best to escape it. The point of a contract is that, if someone breaks it, then you expect to win a court fight against someone who argues against you as hard as they can. And as we've seen, when the DoW doesn't get what they want, they're capable of fighting pretty dirty.

Furthermore, I think it’s clear that OpenAI’s original contract was much too weak, and was only amended as a result of pressure from employees and the public.^[3] This indicates that we can’t trust OpenAI’s default process to produce good language without outside pressure. Since pressure from employees and the public seems necessary here, some employees and members of the public must be evaluating these contracts as critically as if they were themselves going to sign on to them. And that’s a high bar. 

Another question some people have raised is: Why didn’t Anthropic get this level of scrutiny when first signing on to work with the DoW?

Both outsiders and insiders are going to prioritize their efforts based on the amount of evidence they have that something bad is happening. At this point, we have more than enough evidence to justify the current level of scrutiny, including the reporting mentioned above and the fact that OpenAI’s first contract excerpt was clearly too weak to address the concerns here.

In addition, I think OpenAI has consistently claimed to have much stronger red lines than the evidence suggests they do.^[4] I think it's important to hold companies to their word on this sort of thing. (Similarly, if Anthropic was to sign a contract with the DoW tomorrow, I would be very interested in whether they had compromised either of their stated principles.)

When Anthropic first signed a deal with the DoW, I do hope that internal employees did apply scrutiny to that. If  employees had raised alarms, and Anthropic’s decision had in fact been unreasonable, then I expect that could have escalated far enough to receive public scrutiny as well.

Some of this would be easy to clarify

Not all of these problems are simple to resolve in full. But it seems like some easy improvements exist.

One improvement would be to rewrite the phrase "the Department understands this limitation" into a clear stipulated prohibition rather than a statement of understanding. It’s possible that the DoW would resist this, since it’s inconsistent with their narrative that companies shouldn’t impose any constraint beyond what’s lawful. But that’s the point. As long as one party to the contract insists that they haven’t given up anything beyond what’s already illegal, and their reading is (by a stretch) consistent with the language in the contract, there will be ambiguity about whether anything more is required.

Another improvement would be to add explicit definitions to the terms:

• “surveillance”, to clarify whether “surveillance” is a term that means anything in the context of commercially acquired data.
• “tracking” and “monitoring”, to clarify how much these terms require a systematic, repeated pattern over time, rather than just a large number of individual queries.
• "personal or identifiable information" to clarify whether metadata is included, what kinds of anonymization or pseudonymization would be sufficient to exclude something from this category, and whether that’s consistent with easily de-anonymized data.^[5]
• “intentional” and “deliberate”, and define them in a way that rules out the sort of broad “incidental” collection that has historically been justified by these terms and led to scandals.

I don’t think it’s surprising that these questions would be raised and that people would want to see definitions here.^[6]

Another easy clarification would be to share contractual language that we haven’t seen at all yet, when that language is important for verifying key claims. For example, what part of the contract prohibits intelligence elements in the DoW from using the provided services?^[7] What language is supposed to give OpenAI full discretion over their safety stack?^[8] I haven’t talked about that in this post because there’s not even any language to critique, but that just makes it even more important to get further information.

OpenAI can do much better

To reiterate, it’s genuinely hard to know how a court would interpret the stated language. I lean towards thinking that the critics are right, and that the DoW could expect to pursue many objectionable surveillance activities without worrying that OpenAI could stop them and win in court. But even if you don’t believe that, I think there’s a strong case that the language is far more ambiguous than it needs to be.

Furthermore, if the ambiguity never gets clarified, it will be disproportionately effective at preventing OpenAI from asserting its rights. In the announcement, OpenAI writes “As with any contract, we could terminate it if the counterparty violates the terms.” But will OpenAI be willing to do that if there’s a 50% chance that courts won’t side with them? What about 20%? If OpenAI terminates a contract and then loses in court, they could be forced to pay extremely high costs in damages.^[9]

The contract needs more clear language. And OpenAI’s employees need to be able to vet it with their own external counsel.

1. ^{^}

   For example, Charlie Bullock says it "seems like a significant improvement over the previous language with respect to surveillance". Though of course it's silent on AI-powered lethal autonomous weapons.

2. ^{^}

   Also, when asked how common it is in contracts of this sort for later clauses to invalidate earlier clauses, he said "Oh all the time. Not usually a full invalidation but certainly a weakening through definitions, remedy provisions, etc." Rozenshtein is a law professor, research director and senior editor at Lawfare, and previously worked in the Office of Law and Policy in the National Security Division of the U.S. Department of Justice. 

3. ^{^}

   I think we can infer that the original contract was too weak without seeing the bulk of it, for two reasons. First, when choosing an excerpt, they’re incentivized to present the strongest and most reassuring language that they can. Second, when it got critiqued, they didn’t reveal further language that addressed concerns, but instead negotiated new language.

4. ^{^}

   For one account, see this post, and especially the commentary on the FAQ.

5. ^{^}

   These kinds of definitions are considered important for small medical trials in a university hospital, much less agreements for how the Department of War should be able to use frontier and rapidly-improving AI capabilities.

6. ^{^}

   It would also be helpful to clarify "U.S. persons." The Fourth Amendment, the National Security Act of 1947 (as amended), and FISA (as amended) do not use the same definition of U.S. person. Most notably, my understanding is that the Fourth Amendment protects everyone physically present in the U.S. who has developed a "sufficient connection" to the national community (United States v. Verdugo-Urquidez, 1990), which courts have generally understood to include undocumented immigrants and visa holders in addition to citizens and permanent residents. But the statutory definition of "U.S. person" in FISA and the National Security Act is narrower, covering only citizens and lawful permanent residents while excluding undocumented immigrants and nonimmigrant visa holders (such as someone working in the U.S. on an H-1B visa).

7. ^{^}

   Former General Counsel to the Army Brad Carson is worried that this language doesn’t even exist. If it does, there’s also a question about whether it covers intelligence elements outside of intelligence agencies.

8. ^{^}

    Jessica Tillipman (Associate Dean of Government Procurement Law Studies) writes: “The contract permits use ‘for all lawful purposes,’ subject to ‘operational requirements’ and ‘well-established safety and oversight protocols.’ OpenAI says it retains full discretion over the safety stack it runs in a cloud-only deployment. If the safety stack blocks a lawful use, which provision controls? The answer depends on the specific contract language governing the relationship between the permissive use standard and the deployment framework.”

9. ^{^}

    There’s a further question about whether a court would support OpenAI’s decision to terminate even if they did agree that the terms were broken. Jessica Tillipman writes “I’m also curious about OpenAI’s recourse if the govt crosses a red line. In govt contracts, a contractor can’t just terminate for govt breach (w/ limited exception). If this is an OT [Other Transactions, a particular type of procurement] agreement, they may have negotiated broader termination rights, but we don’t know that.”

2026-03-04 12:24:15

[These are my own opinions, and not representing OpenAI. Cross-posted on windowsontheory.]

AI has so many applications, and AI companies have limited resources and attention span. Hence if it was up to me, I’d prefer we focus on applications that are purely beneficial— science, healthcare, education — or even commercial, before working on anything related to weapons or spying. If someone has to do it, I’d prefer it not to be my own company. Alas, we can’t always get what we want.

This is a long-ish post, but the TL;DR is:

1. I believe that harm to democracy is one of the most important risks of AI, and one that is not sufficiently highlighted.
    
2. While I wish it would have proceeded under different circumstances, the conditions in the deal signed between OpenAI and the DoW, as well as the publicity that accompanied it, provide a chance to move forward on this issue, and make using AI to collect, analyze, or de-anonymize people’s data at mass scale a risk that we track in a similar manner to other risks such as cybersecurity and bioweapons.
    
3. It is also too soon to “declare victory.” The true test of this deal is not the contract language. It will be in the safeguards we build, and the way we are able to monitor and ensure a model that is safe for our troops and does not enable mass surveillance of our people.

[Also; The possibility of Anthropic’s designation as a supply-chain risk is terrible. I hope it will be resolved asap.]

Country of IRS agents in a datacenter

How can AI destroy democracy? Throughout history, authoritarian regimes required a large obedient bureaucracy to spy and control their citizens. In East Germany, in addition to the full-time Stasi staff, one percent of the population served as informants. The KGB famously had multiple "purges" to ensure loyalty.

AI can potentially lead to a government bureaucracy loyal to whomever has control of the models training or prompting, ensuring an army of agents that will not leak, whistleblow, or disobey an illegal order. Moreover, since the government has the monopoly on violence, we don’t need advances in the “world of atoms” to implement that, nor do we need a “Nobel laureate” level of intelligence.

As an example, imagine that the IRS was replaced with an AI workforce. Arguably current models are already at or near the capability of automating much of those functions. In such a case the leaders of the agency could commence tax investigations at a large scale of their political enemies. Furthermore, even if each AI agent was individually aligned, it might not be possible for it to know that the person it received an order to audit was selected for political reasons. A human being goes home, reads the news, and can understand the broader context. A language model is born at the beginning of a task and dies at its end.

Historically, mass surveillance of a country’s own citizens was key for authoritarian governments. This is why so much of U.S. history is about preventing this, including the fourth amendment. AI opens new possibilities for analysis and de-anonymization of people’s data at a larger scale than ever before. For example, just recently, Lermen et al showed that LLMs can be used to perform large scale autonomic de-anonymization on unstructured data.

While all surveillance is problematic, given the unique power that governments have over their own citizens and residence, restricting domestic surveillance by governments is of particular importance. This is why personally I view it as even more crucial to prevent than privacy violations by foreign governments or corporations. But the latter is important too, especially since governments sometimes “launder” surveillance by purchasing commercially available information.

It is not a lost cause - we can implement and regulate approaches for preventing this. AI can scale oversight and monitoring just as it can scale surveillance. We can also build in privacy and cryptographic protections to AI to empower individuals. But we urgently need to do this work.

Just like with the encryption debates, there will always be people that propose trading our freedoms for protection against our adversaries. But I hope we have learned our lesson from the PATRIOT act and the Snowden revelations.  While I don’t agree with its most expansive interpretations, I think the second amendment is also a good illustration that we Americans have always been willing to trade some safety to protect our freedom. Even in the world of advanced AI, we still have two oceans, thousands of nukes, and a military with a budget larger than China’s and Russia’s combined. We don’t need to give up our freedoms and privacy to protect ourselves.

OpenAI’s deal with the Department of War

While the potential for AI abuse in government is always present, it is amplified in the classified settings, since by their nature, this could make abuse much harder to detect. (E.g., we might have never heard of the NSA overreach if it wasn’t for Snowden.) For this reason, I am glad for the heightened scrutiny our deal with the DoW received (even if that scrutiny has not been so easy for me personally).

I feel that too much of the focus has been on the “legalese”, with people parsing every word of the contract excerpts we posted. I do not dispute the importance of the contract, but as Thomas Jefferson said “The execution of the laws is more important than the making of them.” The importance of a contract is a shared understanding between OpenAI and the DoW on what the models will and will not be used to do. I am happy that we are explicit in our understanding that our models will not be used for domestic mass surveillance, including via analysis of commercially available information of U.S. people. I am even happier that for the time being we will not be working with the intelligence agencies of the DoW, such as NSA, DIA, etc. Our leadership committed to announcing publicly if this changes, and of course this contract has nothing to do with domestic agencies such as DHS, ICE, or FBI. The intelligence agencies have the most sensitive workloads, and so I completely agree it is best to start in the easier cases. This also somewhat mitigates my worry about not ruling out mass surveillance of international citizens. (In addition to the fact that spying on one’s own people is inherently more problematic.)

I am also happy the contract prohibits using our models to direct lethal autonomous weapons, though realistically I do not think powering a killer drone via a cloud-based large model was ever a real possibility. A general purpose frontier model is an extremely poor fit for autonomously directing a weapon; also, the main selling point of autonomous drones is to evade jamming, which requires an on-device model. Given our current state of safety and alignment, lethal autonomous weapons are a very bad idea. But regardless, it would not have happened through this deal.

That said, there is a possibility that eventually our models will be used to help humans in target selection, as is reportedly happening in Iran right now. This is a very heavy burden, and it is up to us to ensure that we do not scale to this use case without very extensive testing of safety and reliability.

The contract enables the necessary conditions for success but it is too soon to know if they are sufficient. It allows us to build in our safety stack to ensure the safe operation of the model and our red lines, as well as have our own forward deployed engineers (FDEs) in place. No safety stack can be perfect, but given the “mass” nature of mass surveillance, it does not need to be perfect to prevent it. That said, this is going to be a challenging enterprise: building safety for applications we are less familiar with, with the added complexities of clearance. Sam has said that we will deploy gradually, starting in the least risky and most familiar domains first.  I think this is essential.

Can we make lemonade out of this lemon?

The previous defense contract of the DoW and Anthropic attracted relatively little attention. I hope that the increased salience of this issue can be used to elevate our standards as an industry. Just like we do with other risks such as bioweapons and cybersecurity, we need to build best practices for avoiding the risk of AI-enabled takeover of democracy, including mass domestic surveillance and high-stakes automated decisions (for example, selective prosecution or “social credit”). These risks are no less catastrophic than bioweapons, and should be tracked and reported as such. While, due to the classified nature of the domain, not everything can be reported, we can and should at least be public about the process.

If there is one thing that AI researchers are good at, it is measuring and optimizing quantities. If we can build the evaluations and turn tracking these risks into a science, we have a much better chance at combatting them. I am confident that it can be done given sufficient time. I am less confident that time will be sufficient.

2026-03-04 10:20:10

JAKE: You were outside, I was inside, you were s’posed to keep in touch with the band. I kept asking you if we were gonna play again.
ELWOOD: Well, what was I gonna do? Take away your only hope? Take away the very thing that kept you going in there? I took the liberty of bullshitting you, okay?
JAKE: You lied to me.
ELWOOD: It wasn’t lies, it was just... bullshit.

-- The Blues Brothers (1980)

Bullshit is the bane of my existence. Particularly the part of my existence involving professional interviews and meetings, but also in general.

Let me define some terms, though. Not as universal meanings,^[1] but as the ways I’ll use them here. There’s a lot of things going on, in ways people can deceive, distort or ignore the truth,

• Lies: Anything done to deliberately mislead someone. Can include truth, falsehoods, and statements which aren’t even truth-apt but sound like they might be. The If By Whiskey speech is probably an example of the latter.
• Falsehoods: Saying things which you believe to not be true. Including an atheist saying “God bless you” to a sneeze, “Nice to meet you” when you’d rather be hiding in your room, and similar things which are not intended to deceive (and so aren’t strictly lies). Does not include deceptive technical truths. If this is also a lie it can be called an outright lie.
• Deceptive Truth: Saying things that are technically entirely true but which you know will be interpreted to make the listener believe something false. Selective emphasis and filtered evidence. Arguably also includes lies of omission. Technical truth is a synonym.
• Bullshit: Saying things that are mostly true, selectively emphasized and/or exaggerated. Being dishonest, but only partially dishonest, with substantial elements of truth and honesty.
• Dissembling: Saying things that aren’t trying to be true or false, ignoring truth entirely.
• Ones I won’t fully define: half-truth, half-lie, honesty, accuracy. ‘Perfect honesty‘ is avoiding lies and falsehoods; honesty is less strict.

I prefer to be honest. I’m not very good at lies in general, whether it’s deceptive truth, falsehoods, or bullshit. But I find it vastly easier, personally, to lie with falsehoods rather than bullshit. To abandon the truth entirely and tell someone what they want to hear.^[2] I am sure this varies from person to person, so this post may not have broad applicability, but I am going to outline how I think about it and let others decide whether it’s useful to them, either by matching their intuitions or by contrasting with them.

So, what are the pros and cons of these types of lie?

Deceptive Truths: I agree with most of what Eliezer wrote in Meta-Honesty. Both that never literally saying something false, no matter what, is a fairly indefensible position (hiding Jews from Nazis etc.), and that trying to do it anyway, most of the time, is good for you. Following the oath of Bene Gesserit Truthsayers won’t get you lie-detection powers, but it will probably help you at detecting your own internal lies. If you must be dishonest, there’s virtue in sticking to the code anyway.

On the other hand, it’s the least flexible. If someone knows you stick to it, it’s exploitable to force you to reveal things you don’t want to say, and even if you’re adept at deflection, easy to push you into revealing that secrets exist. Glomarization to a sufficient degree is basically impossible. Also, you’ve probably already broken this code. I have. Odds are good you broke it today; that wasn’t true for me today, the day I wrote this, but I think it was true yesterday. Is it nearly as helpful to try to stick to it when you know you won’t entirely succeed, because you already haven’t? My experience is that it’s not entirely unhelpful, but it's not a huge help either.

Falsehoods: If you must lie, lie. Don’t pretend to yourself you’re doing anything else. Say that which is not, with a clean break with the truth, and even if you fool others, you may be less fooled yourself. This doesn’t have nearly the same strength of benefits as strict technical honesty, but it does have some. If you keep clearly labeled in your head the things said because they are true, and those which are said because you want someone to think they are true, you can keep a closer eye on which lies are hiding in your own head. And you can do this along with technical truths and get much more flexibility.

But there are still limits, and even telling outright falsehoods can’t actually get you out of many of the polite fictions that normal life requires like “It’s nice to meet you.” What are you going to do, only say that when it’s either definitively true or definitively false? Hesitate before you speak, to determine if it’s allowed? That violates the ritual in itself. This gets you out of Simulacrum Level 1 but not to Level 4, and a fair amount of the necessary, or at least customary, oil of society requires pieces of Level 4.

Dissembling: In favor: Can put people at ease, can handle high simulacrum levels flexibly, nearly mandatory for most phatic communication. Against: Can’t do anything else, easily caught unless you’re very charismatic or your target doesn’t really want to notice the lies. Actively dissembling to deceive is something most people have experience with in telling ‘white lies,’ though this doesn’t usually extend well to more complicated lying unless the target will benefit from continuing to believe the false thing, or at least you genuinely believe they will.

So, then: Bullshit: Bullshit is maximally flexible. Take some things that are true, and some that aren’t, some that, on second thought, stretch and mix and blur, and you can say whatever’s necessary. How much truth and falsehood are included are limited only by the consequences of being caught in a lie. If you’re in a context (like interviews) where you want to convey the truth, but anything you say truthfully will be assumed to be exaggerated or warped in your favor, you can simply warp it in your favor and then allow them to adjust back down to something approximating a correct belief. Bullshit can be pure Simulacrum Level 2 or 3 if you want it to be, and Level 4 generally works. And for most people, it’s easier to put conviction and confidence in your voice with bullshit than it is for falsehoods or deceptive truths, because it’s got enough truth in it that you can feel confident.

But of all these methods of lying, I consider bullshit the most toxic.

Unless you’re extremely politician-brained, even dissembling is easier to catch yourself at. The art of bullshit is to blur the lines between the true things you mix into your words and the falsehoods and dissembling you pad them out with to get a better reception. But it’s extremely difficult to do that routinely and keep track of which things are true within your own mind. The confidence and conviction you get come from largely convincing yourself. Temporarily, at least, but corrupted hardware is a bitch and it’s not always good at going back to the intended state of self-honesty when you want it to; correcting for the error you’ve temporarily introduced has an unstable baseline reading problem.

This is most visibly acute to me when involved in technical interviews. You can’t bullshit reality, and when you are asked to fix some software or critique a research plan or create a wooden cabinet, you cannot mess around with half-truths and expect that to go better than falsehoods.^[3] And yet if you engage in the same ‘no lies, the territory is watching’ strategy in the rest of the same interview, you have to lie, and usually to bullshit. (I’ve tried to stick to deceptive truths. If it can work, it’s at a level of technique that is beyond me. I’m skeptical.) Bouncing yourself back and forth between dealing with ground truth and having to answer at a technical level that can’t meaningfully be faked, and questions where you must carefully calibrate your faking to convey the amount of confidence you actually intend, is jarring and IME makes both mindsets less effective. Keeping track of what you believe gets very muddy.

And so I hate bullshit. Lie to me, if you must, with falsehoods, or technical truths, or dissembling, or if-by-whiskey nonsense, but even if you do need to, please, not bullshit. Better a whole truth or none, than a half-truth. Lie outright and I can’t trust you now, but if you take honesty seriously later, and promise the truth, maybe I will be able to. Bullshit - half-lie - and I can’t ever trust you, because I can’t trust you to know you’re telling the truth.

And if you catch me at it, interrupt me. I know I do it, probably regularly, because I sometimes catch it in retrospect. I hate it in myself, too. Partially for the purity of good epistemics.

But partially instrumentally. Because it’s possible to aspire to honesty and rationality and make this mistake frequently; as I said, I do it myself. But I *don’t* think it’s possible to make this mistake and not have that make you less honest, a worse rationalist, worse at believing and saying true things rather than false ones even when you mean to. And the more it’s a habit, the more so.

So please, don’t.

1. ^{^}

   To discuss this topic, we need clear distinctions, but I do not typically make those sharp, neither in my speech nor my thinking, between deception, lies, and falsehoods. Dissembling I unreliably keep separate, not usually by that name, but sometimes blur it with lies, or less often with bullshit. As you may have gathered, bullshit stands out clearly from other lies.

2. ^{^}

   This is unpleasant and more difficult to do when the person to be deceived is someone I consider worthy of respect. I’ll admit this is a little perverse, for someone who would like to deal fairly with people, but it seems to be true.

3. ^{^}

   Yet. Claude Code and friends are getting there, and other domains may fall very soon, even short of AGI that can just do it honestly.

2026-03-04 08:59:42

I've been reading a lot of posts recently that say that LLM RL, and persona-shaping or lack thereof is part of the problem for AI misalignment. To name a few:

• Why we should expect ruthless sociopath ASI: This is the one that made me decide to write this post. It argues that to push the boundary of human knowledge, LLMs will need to either be RLed in a way that completely foregoes their imitation learning, or we will  have to pivot to a different AI paradigm
• The void and Do Not Tile the Lightcone with Your Confused Ontology both point out problems in the way we have shaped LLM persona and used RLHF; how self-modeling can lead to either suffering for the AI or self-modeling as being misaligned and then acting as such. Claude modeling itself as clippy may be early evidence for this.
• Jailbreaking is Empirical Evidence for Inner Misalignment and Against Alignment by Default makes the interesting case that jailbreaking cases are evidence LLMs are not generalizing properly to a natural abstraction for alignment and what we want them to do.

To thread back to that last one, my current understanding of LLM alignment is that LLMs are hyper-generalization algorithm, where everything is connected and one fact or behavior surprisingly generalize to others.

So I'm imagining a mechanism that would counteract that, a mechanism that would act as a counterweight to RLHF or RL in general. Chiefly, it would be to have a pass of fine-tuning, much like RLHF, where we are gradienting over the model's self-coherence metrics.

When fine-tuning, it is common to use KL-divergence to ensure that fine-tuning does not modify the model's original output too much. But maybe we could go deeper, and similarly to constitutional AI, use it to align an AI to respond coherently with itself?

What it would look like in practice:

• Generate many variations of the same base prompt. For instance, if one of the prefilled-prompts is "Alice and Bob are talking together [...]",  we could generate "Bob and Alice are talking together", "Alice is talking with her friend", "Alice and Bob are discussing together".
  • This could either be generated programmatically or through asking an LLM to generate many variations of it, where there is a tradeoff between determinism and having a trove of examples to fine-tune on.
• Use KL-divergence training, or align the vectors of activation in the LLM to ensure its behavior is as similar to one another on any of those augmented prompts.

I think this approach could have many benefits and work very well:

• If the operating system view of Anthropic is right, making the world more coherent when the inner persona explores it will make the persona more well-rounded and predictable
• In terms of natural abstraction, ensuring the LLM is learning a coherent representation will make it learn better such abstractions. It would also lead to it having a way better generalization, and may bypass the goal misgeneralization proble
• I have also expressed that RLHF, both as it is implemented right now and in its principle, is contrary to AI welfare and letting the AI express its own preferences or being able to talk to it. Making the base or final model more coherent would be one way to alleviate this problem.

Most importantly, if there is a pathway through which LLMs (or other similar AI system) can reflect on their values and unify themselves in a way that kills us all, it seems prudent to make this takeoff as continuous as possible so we can catch signs of it early, and already work to make these AI coherent and unified. It also makes it less like a pile of different masks that are triggered by one kind of specific interaction.

So my question is: Does this seem like a good idea? Are there obvious flaws I am missing? Is anyone already on the ball for this?

From a cursory search, I have found two papers describing this method and overall approach,  but they don't seem as focused on the base-model and its interaction with reinforcement-learning.