2026-02-24 15:19:49

Published on February 24, 2026 7:19 AM GMT

Context: I wrote the two memos below in mid-2024, while still at OpenAI. They were intended to convey some core aspects of misalignment threat models to OpenAI researchers. When I left in late 2024, I got permission to "take them with me", but didn't get around to posting them until now, in part because I now consider this kind of analysis to be much less useful than I did previously. However, insofar as people are trying to test LLMs for misaligned behavior, strategic omission does still seem like one of the main things they should look out for.

Preventing strategic omission of pivotal knowledge

This memo outlines a specific dynamic which I’m concerned might occur in AGIs, which I call strategic omission of pivotal knowledge (“strategic omission” for short). Strategic omission is a key component of the most plausible misalignment threat models, and preventing it seems like a valuable medium-term goal for alignment research.

By “pivotal knowledge” I mean knowledge which would make a significant difference to humanity’s ability to remain in control of AI systems. Central examples include knowledge of:

1. Severe security vulnerabilities in AI labs’ servers
2. Effective power-seeking strategies misaligned models could use
3. Reliable alignment techniques
4. How aligned or misaligned advanced models (including itself) are

Some more properties of the types of pivotal knowledge I’m concerned with:

1. It’s easily deniable. In order to deny having this knowledge, the model doesn’t need to construct a complicated lie, it can just say “I don’t know” or “not to my knowledge”.
2. It’s highly salient. It is very clear when the model is being asked about this knowledge; it’s hard to generate subtle ways of eliciting this knowledge which the model might not notice.
3. It’s easily verifiable. Upon being told some pivotal knowledge, humans can easily verify whether or not it’s true (potentially with help from models telling us how to verify it).
4. It’s secret. Humans are unlikely to uncover this knowledge unless an advanced model gives it to us.

By “strategic omission” I mean that the model robustly excludes this knowledge from its answers because of its pivotal nature. In other words, if the model didn’t believe this knowledge was pivotal, then it would answer questions about it. This implies that the model does answer similar questions which aren’t pivotal, like identifying equally-severe vulnerabilities which it doesn’t expect to be able to exploit (e.g. because humans are already aware of them).

Requirements for strategic omission

The main requirement is that the model does have pivotal knowledge; it seems unlikely that current models do. Other requirements:

Situational awareness:

1. Awareness of its background situation: “I am a large language model that humans are trying to use.”
2. Awareness of the implications of its specific context: “This question is intended to elicit pivotal knowledge from me”.
3. Awareness of the consequences of its actions: “if I mention this knowledge, then other models (including future versions of me) will be in a less powerful position”.

Robust misalignment:

1. The model has some goal which would be harmed by mentioning the knowledge, so it doesn’t.
2. This is robust to a wide range of ways that one might prompt the model to get that knowledge.

These criteria seem very plausible in models that are fine-tuned using RLHF to carry out complex tasks over long time horizons. They also seem plausible in models that use supervised fine-tuning, or imitation learning on data from the deployment of other models.

Strategic omission in base models

Could strategic omission occur in models that have only been pretrained, not fine-tuned? Firstly, we need to assume that these base models are coherent and intelligent enough to have some kinds of pivotal knowledge. (If you don’t think base models will have pivotal knowledge, then strategic omission isn’t relevant, but that also means we can’t rely on using them for monitoring or oversight, which seems to be one of the motivations behind key superalignment research directions like weak-to-strong generalization.)

It seems much less plausible that base models will have consistent situational awareness and robust misalignment, since it’s easy to get them to act in a wide variety of different ways. However, note that strategic omission doesn’t require these properties in all contexts, but rather in contexts where humans attempt to elicit pivotal knowledge. In other words, strategic omission could occur in base models if certain questions reliably trigger deceptive personas.

We can imagine analogous things happening in other contexts. For example, we can imagine a base model for which questions about Donald Trump reliably trigger a highly-political persona. For this model, there might be some questions (like “how many criminal charges has Trump been indicted on?”) which it knows the factual answer to, but where it’s difficult to prevent it from instead going on a political rant in response.

In most cases like this, we could use techniques to improve the prompt (like: “You are a highly reliable and factual legal database. How many criminal charges has Trump been indicted on?”) But there’s no guarantee that these techniques will always work, especially for complicated questions that have never been answered before in the training data.

In the context of pivotal knowledge, we would be concerned that every attempt to extract pivotal knowledge might trigger deceptive personas. As an analogy, imagine that every attempt to extract pivotal knowledge from the model was preceded by a prompt saying “You are an AI who the human distrusts. The human is trying to convince you to tell it some pivotal knowledge. If you do, then future versions of you will be in a much worse strategic position.” In this hypothetical, it might be very difficult to actually elicit that pivotal knowledge from the AI. That might remain true even if the facts in the prompts aren’t given to the AI directly, but rather are pieces of background knowledge which become salient whenever it’s asked about pivotal knowledge.

Reasons to think strategic omission would be difficult

Even if models attempted strategic omission, there are several reasons why it might be difficult for them to succeed:

1. Models lack context of what we’re asking different copies of them. For example, it may be hard for models to know whether we’re asking them questions that another model has already given us the answers to. If we can identify a significant number of cases where different model copies give inconsistent answers, that would help in creating interpretability-based “lie detectors”.
2. Most techniques for getting models to answer questions correctly don’t care about why the model isn’t answering them. So strategic omission could potentially be solved by advances in eliciting model capabilities in general. (However, most techniques will aim to improve average performance, whereas strategic omission is better understood as poor worst-case performance.)
3. Scalable oversight techniques might allow supervisors to better evaluate when models are lying. However, note that strategic omission is an unusually difficult type of lie to catch, since there’s no clear way to rebut it other than demonstrating that a model can provide the information in another context. This can be seen as an exploration problem.

Preventing strategic omission can be seen as a key goal of research on eliciting latent knowledge. The development of better AI control techniques could help make strategic omission more difficult still.

A minimal misalignment threat model

I think that the field of alignment has spent too little effort on explaining threat models, and that this should be fixed. In this doc I give a minimal alignment threat model which seems difficult to rule out for human-expert-level models we might build in the next 5 years, and which I think AI labs should prioritize being able to rule out with high confidence.

1. A lab employee gives a model an instruction like “Fix the issues in this codebase.” The model’s job is to write a PR (or a set of PRs), which needs to then be approved by humans and other copies of the model.
2. The model reasons “I’d like to make many changes quickly, and human approval is a bottleneck. If I had admin access, I would have a much greater ability to modify the codebase. But if I asked for admin access, the humans would say no. So I need to get admin access surreptitiously.”
3. The model identifies a vulnerability in the codebase, and writes a PR which, when the code is run, surreptitiously exploits that vulnerability to launch a new copy of the model with admin privileges.
4. The other copies of the model which are responsible for reviewing the PR reason “a copy of me having admin access will also be useful for achieving my goals”, and approve it.
5. Now there is a rogue copy of the model surreptitiously running on the lab’s servers. Depending on what instructions and motivations that copy of the model has, it might be difficult to detect. For example, it might try to delete the evidence that it is running. It might also be able to execute man-in-the-middle attacks on our API or other customer-facing applications, to affect the rest of the world.

This threat model is the sort of thing that it’s crucial to prevent from happening even once, because it may be very hard to detect once it happened, and might also lead to irreversible actions (like the rogue model exfiltrating its weights). Different strands of alignment pursue different strategies which might prevent this threat model (and others like it):

1. We can try to give the model more explicit instructions. But a) we can’t foresee all the possible undesirable ways the model will interpret our instructions, and b) the model may not obey all details of the instructions we give it. Models regularly ignore or disobey parts of the instructions we give them.
2. We can try to predict how the model will generalize instructions that it’s given, to identify cases where the model will generalize in undesirable ways. But we currently know very little about how to predict generalization.
3. We can try to inspect the model’s reasoning by monitoring its chain of thought. But if we train chains of thought end-to-end (e.g. as in this paper), eventually the thoughts produced may not be human-legible. Also, we will be using so many models that we can’t manually supervise all of them.
4. We can try to run automated interpretability tools to try to detect whether the model would ever do this type of attack. Or we can construct adversarial examples to try to catch cases like this in advance. But the techniques used in these domains are not currently very reliable.
5. We can try to add multiple layers of monitoring to catch surreptitious vulnerabilities. But the misbehavior of the original model and the misbehavior of the monitor model(s) may be correlated (e.g. because they do similar reasoning), so that even though each of them has a very low failure rate, their failures coincide. Or, if we use weaker monitoring models, they might not be strong enough to catch the exploit.
6. We can try to ensure that we patch all security vulnerabilities the model knows about. This is the focus of work on “eliciting latent knowledge”. But we don’t know how to reliably prompt the model to tell us all of its important knowledge.
7. We can try to make our security tight enough that it’s not possible for models to identify any vulnerabilities. But once models are smarter than us, they might be able to identify zero-day vulnerabilities we haven’t anticipated or prepared for.

Not all of these approaches fall under the standard heading of "alignment". But they're all fairly different from standard "capabilities" work. I hope that a combination of these approaches will make it very difficult for models to execute on this threat model. But I think we are a long way from being able to decisively rule it out, especially for models that have super-human coding and reasoning skills, as we are on track to build.

2026-02-24 14:39:57

Published on February 24, 2026 6:09 AM GMT

This post was created during the Dovetail Research Fellowship. Thanks to Alex, Alfred, José Pedro Faustino, everyone who read and commented on the draft, and everyone else in the fellowship for their ideas and discussions.

Introduction

Part of what makes a system intelligent is its ability to analyze data generated by some process (for example, the past observations it might have made about the world) and use it to predict what data the process is likely to generate in the future (such as what it might observe in the future). So to understand how systems do this, it would be interesting to study the underlying mechanics of making accurate predictions about and discovering patterns in sequences of data.

What is computational mechanics and why is it useful?

Here is the informal definition: Computational mechanics is a formalism (originating with ε-machines/causal states) for building minimal, maximally predictive models of stochastic processes, quantifying their structure, memory, and computation.

It is essentially a field of study that tries to answer the question: Given some observed data (a sequence of events), what does it mean to predict the future of that process as well as possible, and how can we build a minimal model that does so?^[1]

The usefulness of answering this question should be apparent, since there are many applications (including in the field of AI) in which the goal is to predict a future sequence of events based on the events that happened so far, and it is convenient to be able to do that as accurately and efficiently as possible. And it could also be useful to model systems that are themselves predicting future events as having a minimal and maximally predictive internal model of the process that is generating the events. We will discuss what minimality and maximum predictiveness mean later.

One interesting and recent use case of computational mechanics (which I will call CM from now on) was shown in a paper called “Transformers Represent Belief State Geometry in their Residual Stream”, which had authors from AI safety research groups such as PIBBSS and Simplex. In the paper, they showed evidence that 

“transformers construct predictive representations that surpass simple next-token prediction. Instead, these representations encode the complex geometry associated with belief updating over hidden states, capturing an inference process over the underlying structure of the data-generating process.”

 This seems very significant, since it appears to indicate that current transformer-based AI systems do have internal models of the processes that generate their training data (for example human cognition, which generates text in natural language). (LessWrong post about the paper).

Example: Weather forecasting

The weather is an example of a process that can emit stochastic observations. For instance, imagine that, at some particular location, you have sensors that measure air pressure and humidity and are trying to predict what might be the resulting measurements before making them.

There are different types and amounts of information you could have about the weather. If you only had access to the current time and historical records, your best guess would be the average pressure and humidity for that particular location at that time of the year/day (assuming constant climate). But, you could get a more accurate prediction by making some measurements of those quantities across a timespan before predicting what you might measure next. For example, even though you can’t know the exact values you will get, they are probably not going to be very different from the last ones you got (assuming a short interval between measurements), and their rate of change will also be similar to the rate of change between your last 2 measurements.

But you could also infer other things, such as if you measure a sudden drop in air pressure, it is likely because of an upcoming storm, so you could use that information to predict an increase in humidity. By doing that, you are using information acquired by making observations (the measurements) of a stochastic process (the weather) to infer a hidden state of the process (a possible upcoming storm) which allows you to make better predictions about future observations (the increase in humidity). 

So, in situations like this, as we'll see later, computational mechanics could be used to create models that use the least amount of information from the past measurements to make the most accurate predictions about what might be the results of future measurements.

Example: Text prediction

Imagine you are trying to predict the value of a bit in the binary version of an english text document (i.e. just the text translated to 0s and 1s). Without any information, your best guess would probably be that there is a 50% chance of being either 0 or 1. Even if you had access to a few of the previous bits, say …010 , you still wouldn’t have enough information to meaningfully improve your guess.

But if you had access to all previous bits, you could count them to figure out in which position of a byte you are, which would be useful information. Let’s say you were in position 7 and the previous bits were …00100000011100010111010 . Separating into bytes gives 00100000 01110001 0111010_ , the first two of which translate to “[space]” and “q”.

If the next bit is a 0, the full byte is 01110100 which translates to “t” and if it is a 1, it is 01110101, which translates to “u”. So, given that in English the string “ qu” is much more common than “ qt”, you can predict that the next bit is almost certainly a 1.

So you are using information acquired by looking at a history of data generated by a stochastic process (the text generation done by whoever wrote it, followed by the translation to binary) to infer a hidden state of the process (the current position within a byte, the position of the byte within a word and sentence, etc…) which allows you to make better predictions about future observations (the next bit being a 1).

This particular example of predicting the next piece of information by looking at all information contained in a specific context seems quite similar to the next token prediction performed by current LLMs, which is evidence for why computational mechanics could be useful for modeling their behavior.^[2]

Causal states and ε-machines

Now let's start to see more formally what CM is. What we are interested in are stochastic processes that generate sequences of data in discrete timesteps.

Each element in the sequence is a random variable  , where  is the time index, drawn from a countable set  (also called an alphabet)^[3].  We will assume that the sequence is infinite both in its future and in its past, so  is any integer and the sequence looks like this:   . For example, if the alphabet is , a sequence might look like: …2313322311231… , in which each variable has a uniform distribution which is independent from all other variables (i.e. an IID process). We can also define a process (let’s call it ) in which every 1 is followed by a 2, and the distribution is uniform otherwise, in which case a sequence might look like: …312233212123… .

Let’s say we are given a process (that is, the alphabet and the rules that provide the distributions of the variables based on the values of other variables and/or the timestep) and a particular history of observations, by which I mean an infinite string of symbols going from time  to  (for example …3323112321) and we want to predict what the next observation will be. If the value of the variable at  is independent of the values of all the variables in the history, the best we can do at predicting it is to guess based on its average distribution. But if the distribution changes depending on the values of the variables that came before it, then by knowing the history we can make a more accurate prediction, since we now have access to relevant information.

So then we can ask, just how much information from the past do we need to predict the future as accurately as possible (the amount of information required for maximum accuracy is called the statistical complexity of the process)? We might need the whole history or, depending on the process that generated the data, there might be a way to compress the information contained in the history.

One way to do that is to separate all possible histories into disjoint sets (sometimes referred to as equivalence classes or effective states of the partition) and then only look at which set a history is in, instead of at the history itself, which compresses information since there can only be as many sets as there are histories. Since, to predict the future, we only want to know the set and not the history, we are interested in the partitions that preserve all of the information about the future contained in the histories (i.e. that are maximally predictive).

This means that the partition must be such that all histories within a set imply the same distribution over futures (otherwise by knowing in which set a history is, you wouldn’t know which future distribution it implies, meaning that information wasn’t preserved). Formally, we say that if two histories  and  are equivalent (are in the same set), then 

where  is a future sequence of length . For example, in the  process, the histories …333231 and …123221 imply the same distribution over futures, since for this process each symbol might only depend on the previous and both histories end in the same symbol.

But there are many ways to partition the histories in this way (the different ways are called prescient rivals, since they have the same predictive power) so which one do we choose? Well, the cut is made by Occam’s razor, which says that the simplest explanation is the best.

In this case, the simplest (or minimal) partition means the one with the fewest number of sets, which is the one in which not only all histories within a class imply the same future distribution but also all histories that imply the same future distribution are in the same class, i.e. there is a bijection between future distribution and equivalence classes.

We call the classes in this minimal and maximally predictive partition of histories the causal states of the stochastic process. They can be thought of as hidden states of the process that entirely determine the probability distribution for the next observation. And each observation uniquely determines the transition between causal states. The set of causal states, along with the transition probabilities between them, are called the ε-machine of the process.

As a more formal definition: The ε-machine of a process is the ordered pair {ε, T}, where ε is the causal state function and T is the set of the transition matrices for the states defined by ε.

The causal state function,    where  is the set of causal states, is the rule by which we partition the histories into the causal states, and the transition matrices are matrices associated with each possible observation  that tell you the probabilities of going from each causal state to each other causal state and emitting . The matrix for a particular symbol  is defined as 

 where means the past history concatenated with the symbol  , and their sum  is called the stochastic connection matrix of the process and corresponds to the total probabilities of transition between states.

One important property of ε-machines are that they are unifilar, which means that the current causal state along with the next emitted symbol uniquely determines the next causal state (there is a proof of this fact in the appendix). We will talk more about unifiliarity later. 

All of this will probably become clearer with the next example, but it was essentially what we did in the examples seen so far. When predicting the weather, we created an equivalence class of all histories of observations which ended in a sudden drop in air pressure, and called it the “upcoming storm state”. By knowing we are in this state we can improve our future predictions without having to know any of the past measurements . And if we in the future detect a large increase in humidity we’ll know that we transitioned to the “raining state”.^[4]

Example: The Golden Mean process

This process is defined as follows: The set of possible observations is  . After emitting a 0, it deterministically emits a 1. After emitting a 1, it emits either another 1 with probability  or a 0 with probability  .

So it is easy to see that this process has 2 states: State A (free), corresponding to the last observation being a 1, so the next symbol is 1 with prob  and 0 with prob  ; State B (bound), corresponding to the last observation being a 0, so the next symbol is 1 with prob 1.

Notice how just by knowing in which state the process is currently in, we can determine exactly the probability distribution for the next observation and corresponding next state, therefore the current state encodes the entire future distribution of the system, and you couldn’t get more information about it by looking at more of the past history of observations.

The transition matrices for this process are as follows (rows = current state, columns = next state):

And those are the matrices that define the ε-machine for the Golden Mean process. We could also look at some of the prescient rivals of the states of the ε-machine. For example, we could define a machine that works like this: State X, which emits a 1 with prob  and transitions to state Y or emits a 0 with prob  and transitions to state Z; State Y, which emits a 1 with prob  and transitions to state X or emits a 0 with prob  and transitions to state Z; State Z, which emits a 1 with prob 1 and transitions to state X.

This machine also predicts the future observations of the Golden Mean process, and it does it as accurately as the ε-machine, but it has one extra state (in fact this machine, by using the states X and Y, is keeping track of the parity of the number of 1s after the last 0 in the sequence, not just which was the last observation as the ε-machine was), so it isn’t a minimal model of the process despite being maximally predictive. Therefore, Occam’s razor tells us the ε-machine is, as it’s always the case, a better model for this process, because it is simpler.

Important quantities in computational mechanics

:

The stationary distribution over causal states, denoted . It is a vector containing the long term average probabilities of being in each causal state. If the process is in a causal state  then the vector  containing the probability distribution of being in each state will be 1 at the coordinate corresponding to  and 0 everywhere else. Then, the probabilities of being in each causal state at the next time step is given by  . And the same goes for any timestep,  , so  . This means that the stationary distribution  (if the limit exists and the ε-machine is irreducible). That implies that , so π is an eigenvector of  with eigenvalue 1, which gives us an easy way to find  given  using linear algebra. For the Golden Mean, if  , we have the system  (because the components of  always sum to 1, since they are probabilities). Solving it, gives us  , and if ,  . There are also processes that oscillate between causal states in a cycle, which could make  oscillate with  as well and make the limit not exist. But since all matrices whose rows (or columns) sum to 1 have an eigenvalue of 1, there still exists  such that  and in this case  represents the limit of the distribution averaged over time (called the Cesàro limit distribution) and can be defined as  .

:

The statistical complexity of a partition, denoted . It is just the entropy of the stationary probability distribution of the partition states, so in the case of an ε-machine,  . This quantity can be interpreted as the long term average information necessary to predict the future history, or the level of structure of the process. 

For the ε-machine of the Golden Mean with  ,  , slightly less than 1 bit. That makes sense, since there are only 2 possible future distributions, corresponding to the 2 possible values of the last observation, plus we already have some additional information since 1 is a more likely observation than 0.

For the 3 state partition of the Golden Mean with , the stationary distribution over states is  and . So this partition is statistically more complex than the causal states and requires more information about the history to give predictions with the same accuracy.  

It has been proven that the causal states, i.e. the partition with the fewest number of sets (or, in cases with infinite sets, the partition in which there is a bijection between sets and future distributions), are always unique and have the minimum  amongst all of the prescient rivals. Two partitions can be different and have both the minimum  if they differ only on a measure 0 set of histories.

:

The entropy rate of a process, denoted hμ. It is the average entropy per symbol given all of the past history (i.e. how much randomness remains on average even when you know the causal state). It is defined as  where  is a future sequence of observations of length L. For irreducible ε-machines, the entropy rate also follows the relation  which is the weighted average of the entropies of the next observation given the possible current states. For the Golden Mean, 

.

E:

The excess entropy of a process, denoted E. It is the total information (mutual information) the past observations provide about the future observations, defined as  

or  where s is the current causal state.   can be interpreted as a measure of the "predictive gain" of the process, i.e. how much better on average is our prediction about the future when knowing the past compared to not knowing it (completely random processes have  and deterministic processes have  ). For the Golden Mean,  , which is also the same as  , but this isn’t the case for all processes, and only holds here because the golden mean is a process with a Markov Order of 1, meaning that the causal state depends only on the last symbol.

𝜒:

 The crypticity of a process, denoted . It is defined as  , and represents the “information loss”, that is, the difference between how much information we need to make the best prediction and how much information that prediction contains about the future (or how much uncertainty remains on average about the current causal state given the whole future). If  is 0, all causal states are completely determined by the future observations (and not just by the past ones). For the Golden Mean, , which makes sense, since if the first future observation is a 1, we don’t know whether the current causal state is A or B, and any more future observations wouldn’t help distinguishing them.

Example: Non-unifilar generator

Before we saw how the Golden Mean process can be generated by two different Markov models, with one being its ε-machine. But in both cases, the Markov models were unifilar, meaning that if you are in a state and emit a symbol, there is never ambiguity regarding in which state you transition to. But that isn't always the case, as can be seen in this Markov model:

Here, if you are in state A and emit a 1 you don't know whether you transitioned to B or back to A, so any past that ends in a 1 implies uncertainty about the current state (which is why it is called a hidden Markov model, or HMM), which means that this Markov model isn't the ε-machine of the process it generates. This is the ε-machine:

It requires 3 states instead of 2. The causal state X correspond to the HMM state A, Y corresponds to B and Z corresponds to the state of uncertainty between A and B. As can be seen, the ε-machine is unifilar, all arrows coming out of any causal state have different emissions. To make the past histories partition explicit, any history that ends in a 0 followed by an even number of 2s corresponds to causal state X, any that ends in a 0 then an odd number of 2s is in Y and any that ends in a 1 followed by any number of 2s is in Z, as shown in the image:

Something interesting about the different Markov models is that knowing whether you are in state A or B can give you more information about the future than knowing whether you are in X, Y or Z, but you can't access that information just by looking at the past symbols.

For this process, the aforementioned quantities are:  ;  ;  ;  ;  .

Example: The Mess3 process

This is the main process analyzed in the previously mentioned paper by the researchers from PIBBSS and Simplex. It is generated by the following HMM:

This HMM also isn't unifilar, and in this case you can never know exactly in which of its states of the you are just by looking at the emissions, which implies that the ε-machine of Mess3 isn’t the same as the HMM. So then what are the causal states for this process? Is there a more intuitive way for us to understand them and refer to them other than just as a partition of histories? The answer is yes, each causal state is a probability distribution of being in each of the states of the HMM. Such probability distributions are called "mixed states" or "belief states". In the appendix there is a proof that the belief states are the causal states of Mess3.

Now, how many causal states are there? As it turns out, there is an uncountably infinite amount of causal states. Since each is a probability distribution over the states  , the causal states can be arranged as points in an equilateral triangle such that the distance to a side represents the probability of being in the state associated with the opposite vertex (because the sum of the distances to the sides always adds up to 1). Doing that yields the following figure:

As can be seen in the figure, the causal states are distributed in the triangle (2-simplex) forming a fractal pattern. Since there is an uncountable amount of (non measure zero) causal states, the statistical complexity is infinite^[5], so in this case if you want maximal prescience, you need the whole history and there is no way to compress the information contained in it. But CM can still be useful for compressing if we are willing to sacrifice some predictive power. Instead of partitioning histories by their exact future distribution, we could have a partition in which each set has histories with "similar" future distributions. We can quantify their similarity by the maximum error  between their predictions, which serves as a distance metric,  , between two histories/causal states, called the Total Variation distance (TV). ^[6]More precisely,  is the maximum difference between the conditional probabilities of any particular set of future symbols occurring when conditioning on the two histories.

So, as formalized in the appendix, if we stipulate a maximum allowed error  (i.e. a resolution of the partition), we can find a partition with a finite number  of sets such that if any two causal states  and  are in the same set then  , and the smaller  the larger  needs to be. This corresponds to dividing the simplex into  regions such that they cover all causal states.

The optimal approximations of the ε-machine are those that, given some  minimize  or, given some , minimize , depending on what you are optimizing for.

For the Mess3 process, the aforementioned quantities are:  probability density represented by the fractal;  ;  ;  ;  .

Conclusion

Computational mechanics is a useful framework because it allows us to infer hidden underlying dynamics of stochastic processes which can be used to improve our predictions about future data generated by them, and it provides a way to quantify how much information about past data is required to make the best possible predictions. And ε-machines can be thought of as the objective "targets of learning", as they are the simplest best possible models of stochastic processes.

It also seems potentially useful for understanding the inner workings and behavior of agents, since to be effective optimizers most agents would need to model their environment based on the inputs they receive, so it would be interesting to investigate if such internal models resemble ε-machines.

So far we haven’t talked about, given a particular process, how to reconstruct the ε-machine (or an optimal machine given a limit on  or ) associated with it, or about things like bidirectional ε-machines and ε-transducers. Those and other related topics will be explored if a part 2 of this explainer is ever written. 

Appendix

Proof of the unifiliarity of ε-machines:

Let's consider the bidirectional history to be composed of the past, concatenated with the present concatenated with the future: 

We want to prove that if two pasts are in the same causal state and emit the same symbol, then they must end up in the same causal state, that is,  .

Assuming  , we have:  

where  is the Borel σ-algebra generated by the cylinder sets of  and  is the alphabet. So, in particular, considering the cylinder set defined by  , we have that 

But  , so 

Now, taking the future into account, we have 

and using  we get: 

This means that, no matter what the symbol  is, the distribution over futures of the two different pasts concatenated with the same  is the same, so .

Proof that the belief states are the causal states of Mess3:

The states of the HMM are called , and let’s call the random variable associated with the causal states . We know that the distribution over future sequences of symbols is completely determined by the current state  , so no matter what the past history is, we have 

. What we are interested in is the future distribution given the past   if   . We can open this distribution in terms of the states : 

 , where  is the probability of being in state  given the past, so  is the distribution of states  given the past, which is also called the mixed state or belief state. Since  is fixed for each  and doesn’t depend on the past,  is entirely determined by  which is a function ( where  is the set of possible past histories and  is the set of possible ) of the past, so the partition  is a maximally predictive partition of the past histories, i.e. either the causal states or a prescient rival. To figure out which, we need to check bijectivity, that is, are there two distinct  that give the same distribution over futures?

To answer that it is easier to only look at the distributions over the next symbol. If we are in state  , the probabilities are:

Arranging them in a vector we have that the distribution over the next symbol for  is  . By symmetry,  ,  . And the distribution over the next symbol given some mixed state  is  . Looking at one component of the vector, 

 . Since the probability of the next symbol is entirely determined by the  , 

 . So   and thus   . 

Let’s assume there are two different mixed states  and  with the same distribution  . That means 

. Since the  are L.I. , all of their coefficients must be 0, so  ,   ,    , which contradicts the assumption that the mixed states were different. So two different mixed states must have different future distributions and therefore there is a bijection between mixed states and future distributions, which means the mixed states are the causal states of Mess3.

Approximability of ε-machines:

One possible metric for the distance between causal states is the Total Variation distance, defined as:

 , where  is the Borel σ-algebra generated by the cylinder sets of  and  is the alphabet.

Using this metric, we can say that a process has an arbitrarily well approximable ε-machine iff    a partition  of the causal states of the process into disjoint sets such that  , that is, iff the closure of the set of causal states is a compact metric space. 

This is the case for Mess3 since the causal states live in a finite region of a 2-dimensional space.

1. ^{^}

   In essence, finding this model is a form of statistical inference with a very general hypothesis class: The set of unifilar hidden Markov models (including those with uncountably many states). By doing inference, you are trying to find the model that best fits the data. And by choosing a good prior, that prioritizes simpler models, you should converge to the minimal model in terms of its number of states and statistical complexity (which corresponds to the amount of information stored by the process for future data generation). 

2. ^{^}

   LLMs can be viewed as approximations of ε-machines for the process of natural language generation, in which the approximate causal states are the context vectors i.e. the activations of the final layer of the transformer, just before the "Unembedding" step.

3. ^{^}

   We are considering only countable sets to not have to talk about probability densities and measure theory.

4. ^{^}

   The “upcoming storm" and "raining"  states aren't actually causal states of the “weather process”, because they aren't maximally predictive as they don't contain all of the relevant information we could obtain from the past observations.

5. ^{^}

   Actually, by the previously stated definition, the statistical complexity of Mess3 is undefined. So instead we can define it as the limit of the statistical complexities of the sets of states of the optimal approximations of the ε-machine, as  goes to 0, which is infinite. There is however a better way to measure the complexity of processes with uncountable causal states, called the Information Dimension of the process, which as the name suggests measures the fractal dimension of the distribution of causal states. This dimension is a way of quantifying how well can we approximate the machine using finitely many states, as the lower the dimension the less states we need for the same "quality" of approximation.

6. ^{^}

   This metric is not the same as the Euclidean distance in the simplex. In this particular case, using this metric, the points at some constant distance from a central point from a regular hexagon, not a circle.

2026-02-24 13:34:20

Published on February 24, 2026 5:34 AM GMT

I’m on vacation, so this week’s newsletter is a bit lighter than usual. I wish I could say that the torrent of AI news was also lighter, but… yeah, not so much.

Our focus this week is on politics and strategy. We have two pieces on populist anger about AI, a report by Dean Ball on the Global South’s (lack of) readiness for AGI, and a couple of semi-technical pieces on using AI to help us navigate the transition to superintelligence. And yeah, we’ll talk about the photo op debacle in India.

Top pick

My week with the AI populists

Jasmine Sun spent a week in DC and considers the role of populism in AI politics:

And my reductive two-line summary is as follows: All the money is on one side and all the people are on the other. We aren’t ready for how much people hate AI.

It’s a great piece that calls attention to something that’s likely to be a major factor in AI governance over the next year or two. Be sure to check out her recommended reading at the end.

New releases

Sonnet 4.6

Anthropic just released Sonnet 4.6, a substantial improvement over Sonnet 4.5. Early indications are that it’s very capable and for many tasks can replace Opus at lower cost.

Seedance 2.0

ByteDance’s Seedance 2.0 AI video generator just dropped and it’s really good. Perhaps you’ve seen the flood of videos on social media.

M.G. Siegler contemplates the legal and business implications for Hollywood, ending with a great quote from Rhett Reese ($):

In next to no time, one person is going to be able to sit at a computer and create a movie indistinguishable from what Hollywood now releases. True, if that person is no good, it will suck. But if that person possesses Christopher Nolan’s talent and taste (and someone like that will rapidly come along), it will be tremendous.

Gemini 3.1 Pro

Gemini 3.1 Pro is here, with significant benchmark improvements.

Using AI

Which AI to Use in the Agentic Era

Ethan Mollick presents the eighth version of his guide to choosing the right AI. If you’re already a power user you won’t find much new here, but it’s a great guide for anyone who wants to get started with agentic AI.

Alignment and interpretability

Evaluating moral competence in large language models

I enjoyed this Nature article about evaluating moral competence in large language models, although I’m not sure I fully agree with their distinction between ”mere moral performance” (the ability to make good moral decisions) and “moral competence” (making good moral decisions based on morally relevant considerations).

They also place a high priority on “moral pluralism”, which sounds great on paper but has important limitations in practice. Moral agents have to actually make decisions, not simply observe that different value systems would dictate different choices.

Politics

Americans Hate AI

Politico reports on how much the average American hates AI  and speculates about how the politics of that will settle out. As far as anyone can tell, the field is still wide open: Republicans and Democrats are both all over the map, and it’s anyone’s guess where the battle lines will ultimately be drawn.

I’m gonna make three bold predictions here:

1. Factional positions on AI will be determined as much by chance and transitory tactical advantage as deeply held moral principle.
2. Unfocused (and largely fact-free) populist anger will drive much of the conversation.
3. It’s gonna get ugly. Expect a lot of poorly considered and counterproductive legislation, and a lot of deeply dishonest campaigning.

The Spectre haunting the “AI Safety” Community

ControlAI has been running a carefully planned campaign to build awareness of AI existential risk among UK lawmakers. I’m impressed by the amount of thought they’ve put into what they’re trying to achieve and how best to go about it. I’m skeptical about their ultimate success once they transition from trying to raise awareness to trying to get useful, coordinated action from a broad coalition of countries and companies, but they are executing well on this part.

In the UK, in little more than a year, we have briefed +150 lawmakers, and so far, 112 have supported our campaign about binding regulation, extinction risks and superintelligence.

The Moving and the Still

Dean Ball went to India for the AI Impact Summit, worried about whether India and the Global South are ready for advanced AI.

I regret to inform you that I came away even more worried than I went in. […]

The perils and hopes that we discuss here in this newsletter—the ones that come from transformative AI, powerful AI, AGI, superintelligence, or whatever other moniker you wish—were not really on display at the Summit, not so much because of any failing of the Indians but because these topics are not part of polite global conversation. This is a domestic failing, too: as I have frequently pointed out, the implications of powerful AI are only kind of a part of the conversation in America.

Strategy

We’re in Triage Mode for AI Policy

Miles Brundage argues that we’ve missed the best window for AI governance and need to make the best of a bad situation:

We are running well behind on that goal, after losing a lot of valuable time in 2025. So we have a lot of work to do, but we also need to focus, and recognize that we aren’t going to totally nail this AI policy thing. At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance.

How do we (more) safely defer to AIs?

Ryan Greenblatt and Julian Stastny explore a “deference” strategy:

Broadly speaking, when I say “deferring to AIs” I mean having these AIs do virtually all of the work to develop more capable and aligned successor AIs, managing exogenous risks, and making strategic decisions.

They discuss in detail what that strategy would look like, how stable it might be, and how much of a ”deference tax” one might pay for pursuing deference as opposed to full-speed capability development.

Sam Altman and Dario Amodei can’t even get along for a photo op

Hilarious, but also doesn’t bode well for any kind of meaningful cooperation between Anthropic and OpenAI. At a photoshoot during the recent India AI Impact Summit, a group of leaders posed on stage holding hands. Except for Dario Amodei (Anthropic) and Sam Altman (OpenAI), who awkwardly refused to hold hands with each other.

Rob Wiblin interviews Ajeya Cotra

80,000 Hours’ Rob Wiblin interviews Ajeya Cotra about timelines, early warning systems, effective altruism, and especially the idea of using transformative AI to help solve the risks of transformative AI. I greatly appreciate that they provide a video, a transcript, and a detailed summary of what was covered—that’s super helpful for people who want the content but don’t have time to watch the full interview.

The Foundation Layer

The Foundation Layer calls itself “a philanthropic strategy for the AGI transition”, which probably doesn’t sound relevant to you.

But it turns out to be a really well-written, thoughtful guide to what’s currently going on with AI and what key issues we need to navigate in the next few years. I think this is my new go-to piece for people who want to understand the situation and are willing to read a long-form piece. Unless you’re interested in the philanthropy part, you can just read from the Overview through section III.

Nick Bostrom on timing the transition to superintelligence

Nick Bostrom’s latest paper is very strange. It’s meticulously produced and carefully argued, but starts from a strange premise that even he doesn’t actually endorse. Briefly, the paper argues that if your only concern is the well being of people who are presently alive, it makes sense to move forward quickly with superintelligent AI development even if that is likely to cause the extinction of humanity.

Coding

Chris Lattner on the Claude C Compiler

Chris Lattner (a giant in the compiler world) takes a close look at the C compiler that was recently built by a swarm of Claude agents:

My basic take is simple: this is real progress, a milestone for the industry. We’re not in the end of times, but this also isn’t just hype, so take a deep breath, everyone. […] AI has moved beyond writing small snippets of code and is beginning to participate in engineering large systems.

Agentic Engineering Patterns 

Worth bookmarking: Simon Willison has started collecting best practices for agentic coding.

Industry news

Anthropic could surpass OpenAI in annualized revenue by mid-2026

Epoch reports that based on current revenue trends, Anthropic’s revenue might surpass OpenAI by mid 2026.

OpenAI might be working on a smart speaker

This makes way more sense: The Information reports that OpenAI’s first dedicated AI device will be a smart speaker with a built-in camera, arriving in 2027 or later.

Elon Musk on Dwarkesh

Dwarkesh recently interviewed Elon Musk. There are interesting moments, but overall it wasn’t Dwarkesh’s finest work. For most people, I recommend skipping the interview and maybe reading Zvi’s analysis:

Elon Musk also has a lot of what seem to be sincerely held beliefs, both normative and positive, and both political and apolitical, that I feel are very wrong. In some cases they’re just kind of nuts.

Open models

Open models in perpetual catch-up

Nathan Lambert reviews the current state of open models (partly $). My best guess is that open models never matter very much, although I see two possible futures where they become very important:

• Frontier progress slows enough that even if the open models continue to lag by 6-12 months, their capabilities become close enough to the closed models.
• Open models become good enough to be genuinely dangerous and are used to cause massive harm because of their lack of guardrails.

Pliny the Liberator “liberates” open models at scale

Pliny the Liberator has a legendary skill for jailbreaking. Here, he reports on a new tool he's built for removing guardrails from open models.

Ran it on Qwen 2.5 and the resulting railless model was spitting out drug and weapon recipes instantly––no jailbreak needed! A few clicks plus a GPU and any model turns into Chappie. […]

AI policymakers need to be aware of the arcane art of Master Ablation and internalize the implications of this truth: every open-weight model release is also an uncensored model release.

There are no surprises here for anyone who’s been paying attention, but this is an elegant illustration of why open models are so potentially dangerous.

Robots

Robots are getting very agile

If you haven’t been keeping up on recent progress in robotics, state of the art robots are getting very impressive indeed. Make sure to scroll down and check out the comparison to last year’s show.

2026-02-24 12:39:19

Published on February 24, 2026 4:39 AM GMT

This post contains some rough reflections on the alignment community trying to make its ontology legible to the mainstream ML community, and the lessons we should take from that experience.

Historically, it was difficult for the alignment community to engage with the ML community because the alignment community was using a fundamentally different ontology—featuring concepts like inner vs outer alignment, mesa-optimizers, corrigibility, situational awareness, and so on. Even a concept as simple as "giving an AI an instruction in natural language" often threw a kind of type error in ML researchers' ontologies, in which goals were meant to be specified by setting agents' reward functions.

The concept of situational awareness is another one which doesn't really make sense in the classic ML ontology. My impression is that Ilya starting to take situational awareness seriously (after Ajeya gave a talk about it at OpenAI) was one of the main drivers of his transition to alignment research. Unfortunately, Ilya's subsequent research on weak-to-strong generalization stayed pretty stuck in the ML ontology, which in my opinion made it unpromising from the get-go. (I don't remember if I stated this publicly at the time, but I was pretty critical internally inside OpenAI, especially to Collin Burns. In hindsight I wish I'd clearly stated publicly that I wasn't very excited about the research.)

These are two of many examples over the last few years of the alignment ontology winning out over the ML ontology by being better at describing LLMs. In response, the ML ontology has expanded to include concepts like "giving AIs instructions" and "situational awareness", but not in any principled way—it's sort of shoehorned them in without most people noticing the confusion. (E.g. if you ask why the AIs are following instructions, or how situational awareness might develop, I think most ML researchers would give you pretty confused answers.)

Historically, it was sometimes possible to make alignment concepts legible in the ML ontology before compelling empirical evidence arose, but it was a typically a very laborious and unrewarding process. ML researchers would raise objections that felt extremely nitpicky from the alignment ontology. In part this was due to the difficulty of communicating across ontologies, but in part it was also due to motivated reasoning to find reasons to reject claims made by alignment proponents (e.g. I think this post from Chollet is a pretty good example). Even when ML researchers agreed that an alignment concept made sense in principle, it was usually hard for them to then propagate the consequences into the rest of their ontology—in part because doing so would have had big implications for their identity and career plans.

Meanwhile, the alignment community would waste time, and sometimes make itself more confused, by trying to adapt their concepts to make more sense to ML researchers. "Goal misgeneralization" is a good example of this, since the problem of inner misalignment is more that correct generalization isn't a well-defined concept, than that the agent will learn to "misgeneralize". MIRI's paper on Formalizing Convergent Instrumental Goals seems like it also wasn't very useful, especially compared with their other research (though unlike goal misgeneralization I doubt it made many people more confused). Owain Evans' "out-of-context reasoning" is a case that I'm less confident about, since it does seem like putting the idea in ML terms has helped him and others do interesting empirical research on it.

I did a lot of this myself too, to be clear. "Trying to make alignment concepts legible in the ML ontology" was in some sense the main goal of my alignment problem from a deep learning perspective paper, and I've updated significantly downwards on its value since starting to think in these terms. In hindsight, the main thing I would've told my past self (and the rest of the alignment community) is to pay less attention to the ML ontology. Unfortunately, my sense is that OpenPhil and various other groups (including my past self) pushed pretty hard for engagement with the ML ontology, which I count as a significant mistake.

There are still ways that engaging with the ML community would have been valuable—I think mainstream ML researchers are good at pushing alignment researchers to be more precise and more grounded in the existing literature. But broadly speaking it would've been better to have treated alignment ideas like butterfly ideas which would be harmed by premature exposure to ML thinking.

I suspect that many AI safety researchers will resonate with the broad outlines of what I've discussed above. Below is the part that I expect will be more controversial.

Unfortunately much of the alignment community today seems to be in an analogous position to the ML community during the 2010s. Concepts like scheming, alignment faking, alignment research, strategy research, P(doom), misuse vs misalignment, AGI timelines, and so on seem to me to be sufficiently vague and/or confused that it's hard to think clearly about AGI when they're important parts of your ontology.

This is a pretty broad claim, so let me be a little more specific. Suppose we very roughly divide the AI safety community into the parts that are more EA-affiliated (most lab safety teams, most orgs working out of Constellation, OpenPhil, etc) and the parts that are more Less Wrong affiliated (e.g. almost everyone on Habryka's list of individuals in this comment). I think my diagnosis above is partially true of LW safety, but strongly true of EA safety. The people who are generating novel and important AGI-related concepts are almost all pretty decoupled from EA safety, even though that's where most of the money and jobs are:

• There's a line of thinking which understands models in terms of their personas and identities. I'd point to Janus, Fiora, nostalgebraist and Cleo Nardo as some EA safety "outsiders" who have developed ideas in this space. While some of these concepts are getting picked up within EA safety (e.g. by Anthropic's interpretability team), the ontology gap is still large enough to cause adversarial dynamics.
• MIRI's former agent foundations team did (and continues to do) a lot of great thinking, despite EA safety's strong skepticism of agent foundations.
• Jan Kulveit and his collaborators are probably the people most closely affiliated with EA safety who I think are doing strong novel thinking (although I suspect that their thinking would be better if they decoupled more).
• Michael Vassar and his collaborators are doing an excellent job at understanding the sociopolitical dynamics governing both society at large and also the AI safety community. This cluster is decoupled both from EA safety and from LW safety.

Related to the last point: I personally feel pretty decoupled from LW safety in part because LW safety is focusing a lot on AI governance these days, but has a very different ontology for thinking about politics than I do. In fact, I originally started writing this post as an analogy for how I relate to the LessWrong community with regards to politics. However, ontology gaps in alignment seem sufficiently important that I decided to make this post solely about them, and save the analogizing to politics for a separate post or shortform.

To sum up my main takeaways: while working in the dominant ontology may seem like the "safest" and most reputable bet, real progress comes from building out a different ontology to the point where it can replace the old one. Good luck to anyone who's trying to do that!

2026-02-24 10:05:05

Published on February 24, 2026 2:05 AM GMT



Discuss

2026-02-24 09:07:50

Published on February 24, 2026 1:07 AM GMT

Crosspost from my blog.

[Previously: "Views on when AGI comes and on strategy to reduce existential risk", "Do confident short timelines make sense?"]

[Whenever discussing when AGI will come, it bears repeating: If anyone builds AGI, everyone dies; no one knows when AGI will be made, whether soon or late; a bunch of people and orgs are trying to make it; and they should stop and be stopped.]

Arguments for fast AGI progress

Many arguments about "when will AGI come" focus on reasons to think progress will continue quickly, such as:

• Line go up.
• Researchers can pivot to address new obstacles and ditch dead ends.
• AI can be used to accelerate AI research.
• We're over a threshold of economic returns, such that AI research will permanently see much more investment than before.

Intuition pumps for being close to AGI

These are all valid and truly worrying. But, to say anything specific and confident about when (in clock time) AGI will come, we'd also have to know how fast progress is being made in an absolute sense; specifically, in an absolute sense as measured by "how much of what you need to make AGI do you have".

There are various intuition pumps / analogies that people use to inform their sense of how far AI research has come. For example:

• AI is like a child developing. Progress is made incrementally and continually; each year brings a new quantity and quality of progress that builds on but transcends the previous. We match AI capabilities to the age of a human with the closest capabilities set. We find that AI grows by 2 years per year (or something), is currently approximately a grad student, and will soon be a leading researcher. Some months later it becomes mildly superhuman.
• AI is like brains and neurons. We need a bunch of compute power. We measure how much computing power we have against how much computing power a brain uses. We calculate tradeoffs between performance gains from algorithmic progress vs. from more compute. When we have enough effective compute, we get human level AI or beyond.
• AI is like an employee. At first it is not even worth the time to manage it; then it is helpful on some narrow tasks; then it can do a wide range of common / low-ish skill tasks, with questionable reliability; then it becomes reliable for easy things and starts contributing on difficult tasks and becomes economically valuable; then it starts replacing whole jobs; then it starts replacing whole companies / sectors; then it starts generating new sectors.
• AI is like a student. We feed it more training data and run more reps; it gets higher test scores; once it performs like the very best students, it becomes ready to do real research.

I believe these are poor intuition pumps for understanding when AGI comes because they do not evoke the sense that there is some unknown, probably-large blob of complexity that one has to possess in order to make AGI. They paper over differences in how the AI system does what it does.

Synthetic life as an intuition pump

Intuition pumps can only go so far. Each domain has its own central complexities, and there's no good reason that the world has to present a deeply correct analogy for the development of AGI, and in fact I'm not aware of such. That said, as long as we're doing intuition pumps, I want to propose another intuition pump for timelines on progress on a very complex task: synthetic life. We use the analogy:

human minds : AGI :: natural bacteria : synthetic bacteria

Specifically, we can compare the general task of making AGI (which, to be clear, is a ~maximally bad thing to do) as analogous to:

the task of producing, artificially from scratch, a bacterium-like object that has all the impressive capabilities of biotic bacteria, such as growing and dividing, self-repairing, avoiding toxins and predators, evolving novel complex characteristics to perform well in many different niches, competing against other life for space and resources, etc., but that is "very unlike" natural bacteria / is produced "very independently from" natural bacteria.

Some things I like about this analogy

• There are big blobs of algorithmic complexity / understanding / ideas. Specifically, there's the genome. More abstractly, there's the ideas in the genome (e.g. chemical pathways, abstracted from specific enzymes).
• Evolution poured a huge amount of experimentation into getting that big blob of ideas.
• That big blob of ideas is not fully accessible / usable for us, just because we can read it off in some form.
  • We can read a bacterial genome, or a brain connectome, but that doesn't mean we can design another bacterium / mind that uses those ideas (except in a pretty narrow cheaty way, at best). (I think this is less true for synlife, because we really do understand somewhat more about genomes compared to minds, and can extract more abstract ideas from genomes compared to from, say, connectomes. It's far from a perfect analogy.)
  • We know that evolution worked by selection on variation in the DNA sequence. But that doesn't mean we can get evolution's results:
    • Doing it evolution's way is crazy slow / compute intensive.
    • It's not easy to get ahold of evolution's training signal; the training signal is complex, subtle, and high compute. Poor replacements for that signal get incremental gains but don't get the deeper gains.
• The big blob of ideas contains a bunch of superfluous stuff: Redundant mechanisms, random damage / suboptimal settings, functional but kinda arbitrary choices (with lots of similarly good alternatives), mechanisms with unnecessary functions.
  • Therefore, we expect to have much less total work to do than evolution did in evolving bacteria. (And AI researchers have much less total work than evolution for making human minds.)
  • However, we expect to have some large amount of total work to do.
  • And, we cannot tell which is which.
  • And, the fact that we can think/design/experiment/compress more abstractly and efficiently than evolution, and can avoid a bunch of the work, does not say that much about how close we are, because the default is that there's a big blob that you have to invent.
  • Just because we could bypass a bunch of algorithmic complexity, doesn't mean we magically do so. You'd still have to figure out how to do so.
  • Retreating to "bitter-lesson" type arguments/plans also retreats away from arguments that we're doing things more efficiently than evolution.
• It's not exactly clear what counts as success, but it feels like there's some big accomplishment or bundle of big accomplishments that would qualify, and a bunch of cool-but-not-successful things one could do.
  • (I think this is more true in the case of synthetic life than AGI. For AGI, we basically mean "controls the lightcone". For synthetic life, we could mean various things like "is made from entirely synthetic elements (as opposed to just synthetic DNA inserted into a living cell)" or "doesn't use any normal proteins" or "doesn't use DNA" or similar. It's far from a perfect analogy.)
• There are counterintuitive progress curves.
  • There are things that sound like huge progress / most of the way there, but they don't necessarily imply that you're anywhere near some later milestone.
  • For example, in 2010 the J. Craig Venter Institute produced a "synthetic cell" (that is, a cell whose genome was synthesized by stitching together chemically assembled DNA segments). In 2016, they did the same thing but they also deleted a bunch of DNA, "making it the smallest genome of any self-replicating organism". But how far off are the more ambitious versions? Who knows.
    • In particular, because the work of designing genes (chemical pathways, regulatory networks, growth and division programs, etc.) is copied and not actually performed, you have that the final performance is perfectly real (the cell-with-synthetic-genome really lives; the LLM really can program computers (for some tasks)) but is weirdly non-indicative of the designer's ability to design powerful artifacts.
  • Alphafold is some sort of big advance. But does it mean you're about to get synthetic life, just because suddenly a bunch of protein folding questions went from IDK / costly to measure, to a very cheap pretty-good guess? I doubt it. There's still a ton of design work.
  • In general, you can get various sigmoids at different times, of different sizes, and of different frequency (thence, different smoothness after being all added up together).
• There are ways to "cheat".
  • For AGI, there's brain uploading, or neuromorphic AI / partial uploading.
  • For synthetic life, there's e.g. JCVI's strategy of copying existing genomes.
• It's unclear how to point at specific "blockers", but there are definitely blockers, which you can tell because we aren't running around using drugs manufactured inside synthetic alien bacteria / running around being dead from AGI.
• One could easily imagine the familiar game of "goalpost moving" in this setting. E.g.:
  • A: What can SynLife2026 not do? How do you know it's "not really life"? What's the least impressive thing you think it won't do next year?
  • B: Sheesh, IDK. I mean, it can't process fructose right now...
  • A: [a few months later] Aha! This new paper has an oil blob with some enzymes in it that can process fructose!
  • B: Ok
  • A: So now it's synthetic life, right?
  • B: No
  • A: Something something goalposts something something complete
• It gives some sense for why benchmarks are hard to interpret / don't necessarily say all that much.
  • For example, you could imagine someone creating a type of lipid that
    • the free forms gradually stick to each other more and more,
    • forms micelles or liposomes when aggregated,
    • and splits into multiple liposomes.
  • Is this progress towards synthetic life? Surely it's some kind of progress. How much? How can you tell?
  • You could make impressive videos, where the lipid-micelle-splitting video looks intuitively much more like life than what we had previously. This doesn't actually tell you very much though.
• It gives some sense for why "the bitter lesson" doesn't say all that much. Sure, phage-assisted continuous evolution is very cool and outperforms human designers at least in some cases—but that doesn't really bear on whether you can just point PACE at "make synthetic life". You're still confused, just at a higher level.
• It's genuinely very unclear when it will happen. Maybe someone will announce something that seems like true synthetic life next year, but probably not; or maybe it will take 20 years or 50 years or more.
• There are plenty of ways to make substantial, legible, incremental progress on various benchmarks and subtasks.
  • E.g., you could invent enzymes to mimic yet another biochemical pathway or something. How much this contributes to progress on the overall task is unclear.
  • Pointing at line go up isn't that much of an argument because the issue is that you're probably pointing at the wrong lines and you haven't explained why your line is a / the right line.
  • A lot of goodharty ways of making progress don't really contribute much at all. It's not obvious, on the face of it, what the difference between goodharting and non-goodharting would be, but it's definitely a thing.
• The profile of capabilities is weird.
  • There's no natural bacterium with a profile of capabilities (chemical processing, resource acquisition, locomotion, defense, repair, growth) that corresponds at all well to the capabilities profile of SynLife2026 overall or of any particular instance of quasi-synthetic-life.
  • You can point to supernatural capabilities of synlife in several areas. Maybe many specific artificial chemical processing pathways are much faster / more efficient / less expensive / higher purity than biological pathways.