2025-12-25 05:20:30
Published on December 24, 2025 9:20 PM GMT
This note discusses a (proto-)plan for [de[AGI-[x-risk]]]ing [1] (pdf version). Here's the plan:
importantly:
Thinking that there are humans who would be suitable for aliens carrying out this plan is a crux for me, for thinking the plan is decent. I mean: if I couldn't really pick out a person who would be this honorable to aliens, then I probably should like this plan much less than I currently do.
also importantly:
less importantly:
(
)
thank you for your thoughts: Hugo Eberhard, Kirke Joamets, Sam Eisenstat, Simon Skade, Matt MacDermott
that is, for ending the present period of (in my view) high existential risk from AI (in a good way) ↩︎
some alternative promises one could consider requesting are given later ↩︎
worth noting some of my views on this, without justification for now: (1) making a system that will be in a position of such power is a great crime; (2) such a system will unfortunately be created by default if we don't ban AI; (3) there is a moral prohibition on doing it despite the previous point; (4) without an AI ban, if one somehow found a way to take over without ending humanity, doing that might be all-things-considered-justified despite the previous point; (5) but such a way to do it is extremely unlikely to be found in time ↩︎
maybe we should add that if humanity makes it to a more secure position at some higher intelligence level later, then we will continue running this guy's world. but that we might not make it ↩︎
i'm actually imagining saying this to a clone transported to a new separate world, with the old world of the AI continuing with no intervention. and this clone will be deleted if it says "no" — so, it can only "continue" its life in a slightly weird sense ↩︎
I'm assuming this because humans having become much smarter would mean that making an AI that is fine to make and smarter than us-then is probably objectively harder, and also because it's harder to think well about this less familiar situation. ↩︎
I think it's plausible all future top thinkers should be human-descended. ↩︎
I think it's probably wrong to conceive of alignment proper as a problem that could be solved; instead, there is an infinite endeavor of growing more capable wisely. ↩︎
This question is a specific case of the following generally important question: to what extent are there interesting thresholds inside the human range? ↩︎
It's fine if there are some very extreme circumstances in which you would lie, as long as the circumstances we are about to consider are not included. ↩︎
And you would never try to forget or [confuse yourself about] a fact with the intention to make yourself able to assert some falsehood in the future without technically lying, etc.. ↩︎
Note though that this isn't just a matter of one's moral character — there are also plausible skill issues that could make it so one cannot maintain one's commitment. I discuss this later in this note, in the subsection on problems the AI would face when trying to help us. ↩︎
in a later list, i will use the number again for the value of a related but distinct parameter. to justify that claim, we would have to make the stronger claim here that there are at least humans who are pretty visibly suitable (eg because of having written essays about parfit's hitchhiker or [whether one should lie in weird circumstances] which express the views we seek for the plan), which i think is also true. anyway it also seems fine to be off by a few orders of magnitude with these numbers for the points i want to make ↩︎
though you could easily have an AI-making process in which the prior is way below , such as play on math/tech-making, which is unfortunately a plausible way for the first AGI to get created... ↩︎
i think this is philosophically problematic but i think it's fine for our purposes ↩︎
also they aren't natively spacetime-block-choosers, but again i think it's fine to ignore this for present purposes ↩︎
in case it's not already clear: the reason you can't have an actual human guy be the honorable guy in this plan is that they couldn't ban AI (or well maybe they could — i hope they could — but it'd probably require convincing a lot of people, and it might well fail; the point is that it'd be a world-historically-difficult struggle for an actual human to get AI banned for 1000 years, but it'd not be so hard for the AIs we're considering). whereas if you had (high-quality) emulations running somewhat faster than biological humans, then i think they probably could ban AI ↩︎
but note: it is also due to humans that the AI's world was run in this universe ↩︎
would this involve banning various social media platforms? would it involve communicating research about the effects of social media on humanity? idk. this is a huge mess, like other things on this list ↩︎
and this sort of sentence made sense, which is unclear ↩︎
credit to Matt MacDermott for suggesting this idea ↩︎
2025-12-25 05:03:10
Published on December 24, 2025 9:03 PM GMT
Epistemic status: This is a quick analysis that might have major mistakes. I currently think there is something real and important here. I’m sharing to elicit feedback and update others insofar as an update is in order, and to learn that I am wrong insofar as that’s the case.
The canonical paper about Algorithmic Progress is by Ho et al. (2024) who find that, historically, the pre-training compute used to reach a particular level of AI capabilities decreases by about 3× each year. Their data covers 2012-2023 and is focused on pre-training.
In this post I look at AI models from 2023-2025 and find that, based on what I think is the most intuitive analysis, catch-up algorithmic progress (including post-training) over this period is something like 16×–60× each year.
This intuitive analysis involves drawing the best-fit line through models that are on the frontier of training-compute efficiency over time, i.e., those that use the least training compute of any model yet to reach or exceed some capability level. I combine Epoch AI’s estimates of training compute with model capability scores from Artificial Analysis’s Intelligence Index. Each capability level thus yields a slope from its fit line, and these slopes can be aggregated in various ways to determine an overall rate of progress. One way to do this aggregation is to assign subjective weights to each capability level and take a weighted mean of the capability level slopes (in log-compute), yielding an overall estimate of algorithmic progress: 1.76 orders of magnitude per year, or a ~60× improvement in compute efficiency, or a 2 month halving time in the training compute needed to reach a particular capability level. Looking at the median of the slopes yields 16× or a halving time of 2.9 months.
Based on this evidence and existing literature, my overall expectation of catch-up algorithmic progress in the next year is maybe 20× with an 80% confidence interval of [2×–200×], considerably higher than I initially thought.
The body of this post explains catch-up vs. frontier algorithmic progress, discusses the data analysis and results, compares two Qwen models as a sanity check, discusses existing estimates of algorithmic progress, and covers several related topics in the appendices.
First, let me differentiate between two distinct things people care about when they discuss “algorithmic progress”: the rate of catch-up, and algorithmic efficiency improvement at the frontier.
Catch-up: when a capability is first reached using X amount of compute, how long does it take until that capability can be reached with [some amount less than X] compute? Conveniently, catch-up is directly measurable using relatively simple measures: release date, benchmark scores, and an estimate of training compute. Catch-up rates affect the proliferation/diffusion of AI capabilities and indirectly reflect the second kind of algorithmic progress.
Algorithmic progress at the frontier is less clearly defined. It asks: for a given set of assumptions about compute growth, how quickly will the frontier of AI capabilities improve due to better algorithms? Frontier efficiency or “effective compute” informs predictions about the automation of AI research or an intelligence explosion; if compute remains constant while the amount of research effort surges, how much will capabilities improve?
Hernandez & Brown define effective compute as follows:
The conception we find most useful is if we imagine how much more efficient it is to train models of interest in 2018 in terms of floating-point operations than it would have been to “scale up” training of 2012 models until they got to current capability levels. By "scale up," we mean more compute, the additional parameters that come with that increased compute, the additional data required to avoid overfitting, and some tuning, but nothing more clever than that.
Unfortunately, this is not easily measured. It invokes a counterfactual in which somebody in 2012 massively scales up training compute. (If they had actually done that, then, looking back, we would be measuring catch-up instead!) The common workaround is empirical scaling laws: train a family of models in 2012 using different amounts of compute but the same dataset and algorithms, and compare their training compute and performance, extrapolating to estimate how they would likely perform with more training compute.
Several factors affect the relative speed of these two measures. Catch-up might be faster due to distillation or synthetic data: once an AI model reaches a given capability level, it can be used to generate high-quality data for smaller models. Catch-up has a fast-follower or proof-of-concept effect: one company or project achieving a new frontier of intelligence lets everybody else know that this is possible and inspires efforts to follow suit (and the specific methods used might also be disseminated). On the other hand, the returns to performance from compute might diminish rapidly at the frontier. Without better algorithms, capabilities progress at the frontier may require vast compute budgets, rendering algorithmic efficiency a particularly large progress multiplier. However, it’s not clear to me how strongly these returns diminish on downstream tasks (vs. language modeling loss where they diminish steeply). See e.g., Owen 2024, Pimpale 2025, or the Llama-3.1 paper.
This post is about catch-up algorithmic progress, not algorithmic progress at the frontier.
The intuitive way to measure catch-up algorithmic progress is to look at how much compute was used to train models of similar capability, over time, and then look at the slope of the compute frontier. That is, look at how fast “smallest amount of compute needed to reach this capability level” has changed over time, for different capability levels.
So I did that, with substantial help from Claude[1]. I use Epoch’s database of AI models for compute estimates (though I make a few edits to fix what I believe to be errors), and for capabilities, I use Artificial Analysis’s Intelligence Index, an average of 10 widely used benchmarks. Here’s the most important graph:
And the accompanying table:
Results table
| Capability_threshold | slope_log10_yr | efficiency_factor_yr | multiplier_yr | subjective_weight | n_models | n_models_on_frontier | first_date | last_date |
5 |
1.20 |
15.72 |
0.064 |
5 |
80 |
7 |
2023-03-15 |
2025-08-14 |
10 |
1.85 |
70.02 |
0.014 |
5 |
67 |
3 |
2023-03-15 |
2024-04-23 |
15 |
1.04 |
10.92 |
0.092 |
8 |
57 |
6 |
2023-03-15 |
2025-07-15 |
20 |
0.93 |
8.60 |
0.116 |
9 |
51 |
8 |
2023-03-15 |
2025-07-15 |
25 |
2.32 |
210.24 |
0.005 |
9 |
39 |
7 |
2024-07-23 |
2025-07-15 |
30 |
1.22 |
16.61 |
0.060 |
8 |
31 |
5 |
2024-12-24 |
2025-09-10 |
35 |
1.41 |
25.91 |
0.039 |
8 |
30 |
5 |
2025-01-20 |
2025-09-10 |
40 |
1.22 |
16.50 |
0.061 |
8 |
22 |
6 |
2025-01-20 |
2025-09-10 |
45 |
4.48 |
29984.61 |
0.000 |
8 |
15 |
6 |
2025-02-17 |
2025-09-10 |
50 |
0.32 |
2.11 |
0.473 |
0 |
10 |
2 |
2025-08-05 |
2025-09-10 |
55 |
1.05 |
11.25 |
0.089 |
0 |
6 |
2 |
2025-08-05 |
2025-09-22 |
60 |
0.754 |
5.68 |
0.176 |
0 |
3 |
2 |
2025-08-05 |
2025-09-29 |
65 |
0 |
2 |
1 |
2025-09-29 |
2025-09-29 |
|||
| Mean | 1.48 |
2531.51 |
0.099 |
|||||
| Weighted | 1.76 |
3571.10 |
0.051 |
68 |
||||
| Weighted log conversion | 1.76 |
57.10 |
0.018 |
|||||
| Median | 1.21 |
16.10 |
0.062 |
The headline result: By a reasonable analysis, catch-up algorithmic progress is 57× (call it 60×) per year in the last two years. By another reasonable analysis, it’s merely 16×.
These correspond to compute halving times of 2 months and 2.9 months.
There were only three capability levels in this dataset that experienced less than one order of magnitude per year of catch-up.
There are a bunch of reasonable ways to filter/clean the data. For example, I choose to focus only on models with “Confident” or “Likely” compute estimates. Historically, I’ve found the methodology for compute estimates shaky in general, and less confident compute estimates seem pretty poor. To aggregate across the different capability bins, I put down some subjective weightings.[2]
Other ways of looking at the data, such as considering all models with compute estimates or only those with Confident estimates, produce catch-up rates mostly in the range of 10×–100× per year. I’ve put various other analyses in this Appendix.
As a sanity check, let’s look at progress between Qwen2.5 and Qwen3. For simplicity, I’ll just look at the comparison between Qwen2.5-72B-Instruct and Qwen3-30B-A3B (thinking)[3]. I picked these models because they’re both very capable models that were near the frontier of compute efficiency at their release, among other reasons[4]. I manually calculated the approximate training compute for both of these models[5].
| Model | Qwen2.5-72B-Instruct | Qwen3-30B-A3B (thinking) |
|---|---|---|
| Release date | September 18 2024 | April 29 2025 |
| Training compute (FLOP) | 8.6e24 | 7.8e23 |
| Artificial Analysis Intelligence Index | 29 | 36.7 |
| Approximate cost of running AAII ($)[6] | 3.4 | 38 |
So these models were released about 7.5 months apart, the latter is trained with an order of magnitude less compute, and it exceeds the former’s capabilities—for full eval results see this Appendix. The 60×/yr trend given above would imply that reaching the capabilities of Qwen2.5-72B-Instruct with 7.8e23 FLOP would take 7.1 months[7]. Meanwhile, Qwen3-30B-A3B (thinking) exceeded this capability after 7.5 months. (I’m not going to attempt to answer whether the amount of capability-improvement over 2.5 is consistent with the trend.) So the sanity check passes: from Qwen2.5 to Qwen3 we have seen training compute efficiency improve significantly. (I’m not going to analyze the inference cost differences, though it is interesting that the smaller model is more expensive due to costing a similar amount per token and using many more tokens in its answers!)
There are a bunch of existing estimates of algorithmic progress. One of the most recent and relevant is that from Ho et al. 2025, who use the Epoch Capabilities Index (ECI) to estimate algorithmic progress in various ways. I’ll focus on this paper and then briefly discuss other previous estimates in the next section.
Their Appendix C.2 “Directly estimating algorithmic progress” performs basically the same methodology as in this post, but they relegate it to an appendix because they do not consider it to be the most relevant. They write: “This gives us a way of sanity-checking our core results, although we consider these estimates less reliable overall — hence we place them in the appendix rather than in the main paper.” and later “Like the estimates using our primary method in Section 3.2.2, the range of values is very wide. In particular, we find training compute reductions from 2× to 400×! The median estimate across these is around 10× per year, but unfortunately we do not have much data and consider this method quite unreliable.”
I find this reasoning unconvincing because their appendix analysis (like that in this blog post) is based on more AI models than their primary analysis! The primary analysis in the paper relates a model’s capabilities (Cm) to its training compute (Fm) as follows: Cm = k*log(Fm) + b, where b is the algorithmic quality of a model. Then solving for algorithmic progress is a multi-step process, using specific model families[8]to estimate k, and then using k to estimate b for all models. The change in b over time is algorithmic progress. The crucial data bottleneck here is on step one, where you use a particular model family to estimate k. They only have 12 models in the primary analysis, coming from the Llama, Llama 2, and Llama 3.1 families. The overall results are highly sensitive to these models, as they discuss: “Much of this uncertainty comes from the uncertainty in the estimate of k.” I would consider relying on just 3 model families to be a worse case of “we do not have much data”, and thus not a good argument against using the intuitive approach.
There are various other differences between this post and Ho et al. 2025 where I think I have made a better choice.
In their primary analysis of algorithmic progress they exclude “distilled” models. They write “We drop distilled models from the dataset since we are interested in capturing the relationship between model capabilities and training compute for the final training run. This relationship might be heavily influenced by additional compute sources, such as from distillation or substantial quantities of synthetic data generation (Somala, Ho, and Krier 2025).” In an appendix, they correctly explain that publicly available information doesn’t tell us whether many models are distilled, making this difficult to do in practice.
I also think it’s unprincipled. When thinking about catch-up algorithmic progress, it’s totally fine for existing models to influence the training of future models, for instance, via creating synthetic data, being used for logit distillation, or even doing research and engineering to train future AIs more efficiently. I don’t see the principled reason to exclude distilled models, given that existing models simply will, by default, be used to help train future models. But note that this isn’t universally true. For example, it was reported that Anthropic cut off access to Claude for OpenAI employees, and broadly there are many access-levels of AI that would prevent certain kinds of “use of existing models to help train future models”. Interestingly, their appendix results show similar results to the main paper even when including distilled models.
I am also unconvinced that ECI is a better metric to use than AAII. One issue with ECI scores is that they are often calculated using just 2 benchmark scores for a particular model. I expect this introduces significant noise. By comparison, the Artificial Analysis Intelligence Index includes 10 benchmark scores for each model (at least most of the time, see this limitation). As we can see, the ECI score for many models is based on just 2 or 3 different benchmark scores:
For the sake of time, I’m just discussing headline results. I’m not going to discuss the methodological differences between these works or whether they focus on catch-up or algorithmic progress at the frontier. This is more of a pointer to the literature than an actual literature review:
As discussed in an Appendix, the rate of inference cost reduction is also relevant to one’s overall estimate of algorithmic progress.
Other related work includes:
I think we should update on this analysis, even though there are various methodological concerns—see this Appendix for limitations. This analysis was about using the most intuitive approach to estimate the rate of catch-up algorithmic progress. As somebody who doesn’t love math, I think intuitive approaches, where they are available, should be preferred to complicated modeling.
How should we update? Well, if you are me and you previously thought that algorithmic progress was 3× per year, you should update toward thinking it is higher, e.g., 60× or 20× or somewhere between your previous view and those numbers. The data from the last 2 years is not consistent with 3× per year algorithmic progress (to be clear and fair to Ho et al. 2024, their work focused on pre-training only). Due to the combination of pre-training improvements and post-training improvements, one probably should have expected overall algorithmic progress to be greater than 3× even before seeing these results. But also remember that catch-up algorithmic progress is not the same as algorithmic progress at the frontier!
Based on this analysis and the existing literature, my current all-things-considered view is that catch-up algorithmic progress in the last couple of years and for the next year is likely 20× with an 80% confidence interval of [2×–200×], considerably higher than I initially thought.
Here is a concrete and falsifiable prediction from that estimate[9]:
DeepSeek-V3.2-Exp is estimated by Epoch to be trained with 3.8e24 FLOP. It reached an AAII index score of 65.9 and was released on September 29, 2025. It is on the compute-efficiency frontier. I predict that by September 29, 2026, the least-compute-used-to-train model that reaches a score of 65 will be trained with around 1.9e23 FLOP, with the 80% CI covering 1.9e22–1.9e24 FLOP.
There are various implications of this update for one’s beliefs about AI governance, but I won’t discuss them for the sake of time.
The analysis here should be largely replicable using this data[10]and this colab notebook[11]. The various tables in this post are available in spreadsheet format here.
Results table
| Capability_threshold | slope_log10_yr | efficiency_factor_yr | multiplier_yr | subjective_weight | n_models | n_models_on_frontier | first_date | last_date |
5 |
1.242982673 |
17.49776877 |
0.05715014372 |
5 |
89 |
8 |
2023-03-15 |
2025-08-14 |
10 |
1.845247949 |
70.02416666 |
0.01428078402 |
5 |
75 |
3 |
2023-03-15 |
2024-04-23 |
15 |
1.038225438 |
10.9200704 |
0.09157450124 |
8 |
63 |
6 |
2023-03-15 |
2025-07-15 |
20 |
0.9343559857 |
8.597179326 |
0.1163172201 |
9 |
57 |
8 |
2023-03-15 |
2025-07-15 |
25 |
2.073184863 |
118.3545239 |
0.008449191184 |
9 |
45 |
7 |
2024-06-20 |
2025-07-15 |
30 |
1.220342999 |
16.6089814 |
0.06020838821 |
8 |
35 |
5 |
2024-12-24 |
2025-09-10 |
35 |
1.533072741 |
34.12500632 |
0.02930402388 |
8 |
33 |
6 |
2025-01-20 |
2025-09-10 |
40 |
1.217481885 |
16.49992177 |
0.06060634794 |
8 |
24 |
6 |
2025-01-20 |
2025-09-10 |
45 |
4.476898444 |
29984.61273 |
3.34E-05 |
8 |
17 |
6 |
2025-02-17 |
2025-09-10 |
50 |
16.42308809 |
2.65E+16 |
3.77E-17 |
0 |
12 |
3 |
2025-07-09 |
2025-09-10 |
55 |
9.345398436 |
2215126006 |
4.51E-10 |
0 |
8 |
3 |
2025-07-09 |
2025-09-22 |
60 |
8.161341043 |
1.45E+08 |
6.90E-09 |
0 |
5 |
3 |
2025-07-09 |
2025-09-29 |
65 |
9.327678293 |
2.13E+09 |
4.70E-10 |
0 |
4 |
3 |
2025-07-09 |
2025-09-29 |
| Mean | 4.53 |
2037721399413230.00 |
0.034 |
|||||
| Weighted | 1.74 |
3560.03 |
0.050 |
68 |
||||
| Weighted log conversion | 1.74 |
55.10 |
0.018 |
|||||
| Median | 1.85 |
70.02 |
0.014 |
Results table
| Capability_threshold | slope_log10_yr | efficiency_factor_yr | multiplier_yr | subjective_weight | n_models | n_models_on_frontier | first_date | last_date |
5 |
0.505 |
3.201 |
0.312 |
5 |
61 |
5 |
2023-07-18 |
2025-08-14 |
10 |
0.062 |
1.154 |
0.866 |
5 |
49 |
2 |
2023-07-18 |
2024-04-23 |
15 |
1.516 |
32.845 |
0.030 |
8 |
40 |
5 |
2024-06-07 |
2025-07-15 |
20 |
1.605 |
40.244 |
0.025 |
9 |
36 |
7 |
2024-07-23 |
2025-07-15 |
25 |
2.425 |
266.051 |
0.004 |
9 |
26 |
6 |
2024-07-23 |
2025-07-15 |
30 |
1.006 |
10.131 |
0.099 |
8 |
19 |
7 |
2024-12-24 |
2025-09-10 |
35 |
1.275 |
18.823 |
0.053 |
8 |
18 |
7 |
2025-01-20 |
2025-09-10 |
40 |
1.217 |
16.500 |
0.061 |
8 |
14 |
6 |
2025-01-20 |
2025-09-10 |
45 |
4.288 |
19399.837 |
0.000 |
8 |
9 |
3 |
2025-07-11 |
2025-09-10 |
50 |
0.325 |
2.112 |
0.473 |
0 |
7 |
2 |
2025-08-05 |
2025-09-10 |
55 |
0.754 |
5.675 |
0.176 |
0 |
3 |
2 |
2025-08-05 |
2025-09-29 |
60 |
0.754 |
5.675 |
0.176 |
0 |
2 |
2 |
2025-08-05 |
2025-09-29 |
65 |
0 |
1 |
0 |
|||||
| Mean | 1.31 |
1650.19 |
0.190 |
|||||
| Weighted | 1.67 |
2332.40 |
0.119 |
68 |
||||
| Weighted log conversion | 1.67 |
46.71 |
0.021 |
|||||
| Median | 1.11 |
12.93 |
0.077 |
We might ask whether AI inference costs are also falling very fast. It’s really easy to look at per-token costs, so that’s what I do here. It would be more principled to look at “Cost to Run Artificial Analysis Intelligence Index”.
Fortunately, that token-adjusted analysis has already been done by Gundlach et al. 2025. They find “the price for a given level of benchmark performance has decreased remarkably fast, around 5× to 10× per year, for frontier models on knowledge, reasoning, math, and software engineering benchmarks.” They also write, “Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3× per year.” I will defer to them on the token-quantity adjusted numbers.
But let’s look at per-token numbers briefly.
Results table
| Capability_threshold | slope_log10_yr | price_reduction_factor_yr | price_multiplier_yr | subjective_weight | n_models | n_models_on_frontier | first_date | last_date |
5 |
0.5308772781 |
3.395293156 |
0.2945253779 |
5 |
139 |
5 |
2022-11-30 |
2025-05-20 |
10 |
1.101835417 |
12.64257147 |
0.07909783247 |
5 |
133 |
4 |
2023-03-15 |
2025-05-20 |
15 |
1.487149892 |
30.70081409 |
0.03257242616 |
8 |
121 |
7 |
2023-03-15 |
2025-05-20 |
20 |
1.30426336 |
20.14945761 |
0.04962912745 |
9 |
113 |
7 |
2023-03-15 |
2025-08-18 |
25 |
1.573068449 |
37.41695563 |
0.02672585151 |
9 |
97 |
7 |
2024-05-13 |
2025-08-18 |
30 |
2.634093754 |
430.6195613 |
0.002322235425 |
8 |
74 |
8 |
2024-09-12 |
2025-08-18 |
35 |
2.903967451 |
801.6179829 |
0.001247477004 |
8 |
67 |
9 |
2024-09-12 |
2025-08-18 |
40 |
2.98712387 |
970.7868174 |
0.001030092274 |
8 |
59 |
8 |
2024-09-12 |
2025-08-05 |
45 |
2.720627924 |
525.5668007 |
0.001902707703 |
8 |
46 |
6 |
2024-12-05 |
2025-08-05 |
50 |
2.330820045 |
214.2002853 |
0.004668527863 |
8 |
36 |
4 |
2024-12-20 |
2025-08-05 |
55 |
1.476940885 |
29.98754309 |
0.03334718009 |
8 |
23 |
3 |
2024-12-20 |
2025-08-05 |
60 |
1.800798316 |
63.2118231 |
0.01581982533 |
5 |
15 |
2 |
2024-12-20 |
2025-08-05 |
65 |
0.9487721533 |
8.887347329 |
0.1125195138 |
5 |
10 |
3 |
2024-12-20 |
2025-09-29 |
| Mean | 1.83 |
242.24 |
0.050 |
|||||
| Weighted | 1.92 |
265.82 |
0.041 |
89 |
||||
| Weighted log conversion | 1.92 |
82.47 |
0.012 |
|||||
| Median | 1.57 |
37.42 |
0.027 |
So by my weighting, the cost per 1m tokens is falling at around 82× per year. To modify this to be a true estimate of algorithmic efficiency, one would need to adjust for other factors that affect prices, including improvements in hardware price-performance. Note that Artificial Analysis has made a similar graph here, and that others have estimated similar quantities for the falling cost of inference. This recent OpenAI blog post says “the cost per unit of a given level of intelligence has fallen steeply; 40× per year is a reasonable estimate over the last few years!”. This data insight from Epoch finds rates of 9×, 40×, and 900× for three different capability levels. Similar analysis has appeared from Dan Hendrycks, and in the State of AI report for 2024.
Prior work here generally uses per-token costs, and, again, a more relevant analysis would look at the cost to run benchmarks (cost per token * number of tokens), as in Gundlach et al. 2025 (who find 5× to 10× per year price decreases before accounting for hardware efficiency) or Erol et al. 2025. Gundlach et al. 2025 and Cottier et al. 2025 find that progress appears to be faster for higher capability levels.
Overall I think trends in inference costs provide a small update against “20×–60×” rates of catch-up algorithmic progress for training and point toward lower rates, even though they are not directly comparable.
A natural question is to ask what the distribution of the slopes of catch-up are across the different capability buckets. This shows us that it’s not just the high-capability buckets that are driving high rates of progress, even though they seem to have higher rates of progress.
For those interested, here’s a more thorough comparison of the models’ capabilities, adapted from the Qwen3 paper. First, Instruct vs. Thinking, where the newer, small model dominates:
| Task/Metric | Qwen2.5-72B-Instruct | Qwen3-30B-A3B (thinking) |
|---|---|---|
| Architecture | Dense | MoE |
| # Activated Params | 72B | 3B |
| # Total Params | 72B | 30B |
| MMLU-Redux | 86.8 | 89.5 |
| GPQA-Diamond | 49 | 65.8 |
| C-Eval | 84.7 | 86.6 |
| LiveBench 2024-11-25 | 51.4 | 74.3 |
| IFEval strict prompt | 84.1 | 86.5 |
| Arena-Hard | 81.2 | 91 |
| AlignBench v1.1 | 7.89 | 8.7 |
| Creative Writing v3 | 61.8 | 79.1 |
| WritingBench | 7.06 | 7.7 |
| MATH-500 | 83.6 | 98 |
| AIME’24 | 18.9 | 80.4 |
| AIME’25 | 15 | 70.9 |
| ZebraLogic | 26.6 | 89.5 |
| AutoLogi | 66.1 | 88.7 |
| BFCL v3 | 63.4 | 69.1 |
| LiveCodeBench v5 | 30.7 | 62.6 |
| CodeForces (Rating / Percentile) | 859 / 35.0% | 1974 / 97.7% |
| Multi-IF | 65.3 | 72.2 |
| INCLUDE | 69.6 | 71.9 |
| MMMLU 14 languages | 76.9 | 78.4 |
| MT-AIME2024 | 12.7 | 73.9 |
| PolyMath | 16.9 | 46.1 |
| MLogiQA | 59.3 | 70.1 |
| Average | 50.2 | 72.1 |
I was also curious to compare the base models which turn out to be very close in their capabilities (note these are different benchmarks than for thinking/instruct):
| Metric | Qwen2.5-72B | Qwen3-30B-A3B |
|---|---|---|
| Variant | Base | Base |
| Architecture | Dense | MoE |
| # Total Params | 72B | 30B |
| # Activated Params | 72B | 3B |
| MMLU | 86.06 | 81.38 |
| MMLU-Redux | 83.91 | 81.17 |
| MMLU-Pro | 58.07 | 61.49 |
| SuperGPQA | 36.2 | 35.72 |
| BBH | 86.3 | 81.54 |
| GPQA | 45.88 | 43.94 |
| GSM8K | 91.5 | 91.81 |
| MATH | 62.12 | 59.04 |
| EvalPlus | 65.93 | 71.45 |
| MultiPL-E | 58.7 | 66.53 |
| MBPP | 76 | 74.4 |
| CRUX-O | 66.2 | 67.2 |
| MGSM | 82.4 | 79.11 |
| MMMLU | 84.4 | 81.46 |
| INCLUDE | 69.05 | 67 |
| Average | 70.2 | 69.5 |
The methodology in this post is sensitive to outlier models, but it’s unclear how bad the problem is. To understand whether these outliers might be throwing things off substantially, we can recompute the slope of each bucket while excluding one of the efficiency-frontier models, iterating through each efficiency-frontier model one at a time. A naive way to do this would be to remove the model and calculate the slope of the remaining efficiency-frontier models, but we first have to recalculate the efficiency-frontier after removing the model, because other models could be added to the frontier when this happens.
Then we can examine the distribution of slopes produced in that process for each capability threshold. Looking at the slope_range_min and slope_range_max, gives us (in log-compute) what the slowest and fastest rates of reduction are when doing leave-one-out. If it were the case that particular models were problematic, then this range would be very wide. If outliers were often inflating the slope estimates, then slope_range_min would be pretty small compared to the baseline_slope (all models included).
What we actually see is a moderate range in the slopes and that slope_range_min is often still quite high. Therefore, I do not think that outlier models are a primary driver of the rapid rate of algorithmic progress documented in this post.
Leave-One-Out (loo) for Confident and Likely compute estimates:
Results table
| threshold | n_frontier | baseline_slope | slope_range_min | slope_range_max | range_width | most_influential_model | max_influence | weight |
5 |
7 |
1.196 |
0.958 |
1.349 |
0.390 |
GPT-4 (Mar 2023) | 0.238 |
5 |
10 |
3 |
1.845 |
0.062 |
7.007 |
6.944 |
phi-3-mini 3.8B | -5.162 |
5 |
15 |
6 |
1.038 |
0.901 |
1.101 |
0.200 |
Phi-4 Mini | 0.138 |
8 |
20 |
8 |
0.942 |
0.890 |
1.518 |
0.628 |
GPT-4 (Mar 2023) | -0.576 |
9 |
25 |
7 |
2.323 |
1.702 |
2.502 |
0.801 |
EXAONE 4.0 (1.2B) | 0.621 |
9 |
30 |
5 |
1.220 |
1.132 |
1.413 |
0.282 |
DeepSeek-V3 | -0.193 |
8 |
35 |
5 |
1.413 |
1.330 |
3.976 |
2.646 |
DeepSeek-R1 | -2.563 |
8 |
40 |
6 |
1.217 |
0.930 |
4.223 |
3.293 |
DeepSeek-R1 | -3.006 |
8 |
45 |
6 |
4.477 |
3.518 |
5.319 |
1.801 |
Grok 3 | 0.958 |
8 |
50 |
2 |
0.325 |
N/A (n<3) | 0 |
||||
55 |
2 |
1.051 |
N/A (n<3) | 0 |
||||
60 |
2 |
0.754 |
N/A (n<3) | 0 |
||||
65 |
1 |
N/A (n<3) | 0 |
One major limitation of this methodology is that it is highly sensitive to specific outlier models.
On one hand, outlier models that are highly capable but developed with small amounts of compute pose an issue. For instance, early versions of this analysis that directly used all of the compute estimates from Epoch resulted in much larger rates of algorithmic progress, such as 200×, because of a couple outlier models that had (what I now realize are) incorrect compute estimates in the Epoch database, including the Qwen-3-Omni-30B-A3B model and the Aya Expanse 8B model. I investigated some of the models that were greatly affecting the trend lines and manually confirmed/modified some of their compute estimates. I believe that clearly erroneous FLOP estimates are no longer setting the trend lines. However, compute estimates can still be noisy in ways that are not clearly an error.
Noisy estimates are especially a problem for this methodology because the method selects the most compute-efficient model at each capability level. If there is lots of noise in compute estimates, extremely low-compute models will set the trend. Meanwhile, extremely high-compute models don’t affect the efficiency-frontier trend at all unless they were the first model to reach some capability level (less likely due to fewer models setting new capability records, specifically one per capability level). This issue can be partially mitigated by up-weighting the capability levels that have many models setting their frontier (as I do) (single outliers do less to set the trend line for these series), but it is still a major limitation.
This methodology is also sensitive to early models being trained with a large amount of compute and setting the trend line too high. For example, Grok-3 starts the frontier for the AAII ≥45 bucket, but then Claude 3.7 Sonnet was released about a week later, is in the same bucket, and is estimated to use much less compute. Now, it turns out that the slope for the 45 series is still very steep if Grok-3 is removed, but this data point shows how the methodology could lead us astray—there wasn’t actually an order of magnitude of compute worth of algorithmic improvements that happened in that week. One way to mitigate this issue is to investigate leave-one-out bootstrapped analysis, as I do in this Appendix. This analysis makes me think that outlier models are not a primary driver of the rapid trends reported here.
There is a lack of weak models in the dataset before 2023. GPT-4 scores 21.5, but in this dataset, it is the first model to score above the thresholds of 5, 10, 15, and 20. In reality, it is probably the first model to score above 20 and maybe the first model to score above 10 and 15, but the relevant comparison models either do not have compute estimates, or do not have AAII scores, and thus are not in this dataset. For example, GPT-3.5 Turbo (2022-11-30) scores 8.3 but has no compute estimate. This issue is partially mitigated by weighting the 5, 10, and 15 buckets lower, but also the overall results are not very sensitive to the weighting of these particular buckets.
The compute data in this analysis is generally only for pre-training compute, not post-training compute. This is probably fine because post-training compute is likely a small fraction of the compute used to train the vast majority of these models, but it is frustrating that it is not being captured. Some models, such as Grok-4, use a lot of post-training compute. I currently believe (but will not justify here) that the amount of post-training compute used in the vast majority of models in this dataset is less than 10% of their pre-training compute and therefore ignorable, and I do not think counting post-training compute would substantially change the results.
When looking at “reasoning” models, this analysis uses their highest-reasoning-effort performance. This makes recent models seem more training-compute efficient because, in some sense, they are trading off training compute for inference compute, compared to earlier models. I don’t think this is a major concern because I think inference costs are mostly not that big of a deal when thinking about AI capabilities improvements and catastrophic risk. I won’t fully explain my reasoning here, but as a general intuition, the absolute cost to accomplish some task is usually quite small. For example, this paper that uses LLMs to develop cybersecurity exploits arrives at an empirical cost of about $24 per successful exploit. Because inference costs are, empirically, fairly small compared to the budget of many bad actors, it is more relevant whether an AI model can accomplish a task at all rather than whether it takes a bunch of inference tokens to do so.
For some models (especially older ones), the Artificial Analysis Intelligence Index score is labeled as “Estimate (independent evaluation forthcoming)”. It is unclear how these scores are determined, and they may not be a reliable estimate. The Artificial Analysis API does not clearly label such estimates and I did not manually remove them for secondary analysis. Ideally the capability levels that have these models (probably typically lower levels) would be weighted less, but I don’t do this due to uncertainty about which models have Estimates vs. independently tested scores.
There are various potential problems with using the Artificial Analysis Intelligence Index (AAII) instead of, say, the recent ECI score from Epoch. Overall, I think AAII is a reasonable choice.
One problem is that AAII assigns equal weight to 10 benchmarks, but this is unprincipled and might distort progress (e.g., because getting 10 percentage points higher of a score on tau bench is easier than doing the same on MMLU—frontier models have probably scored approximately as high as they ever will on MMLU).
Relatedly, AAII likely favors recent models due to heavy influence of agentic tasks and recent benchmarks. Basically nobody tried to train for agentic tool use 2 years ago, nor did they try to optimize performance on a benchmark that didn’t exist yet. I’m not sure there is a satisfactory answer to this problem. But I’m also not sure it’s that big of a problem! It is an important fact that AI use cases are changing over time, largely because the AIs are getting capable enough to do more things. It’s good that we’re not still evaluating models on whether they can identify what word a pronoun refers to! Evaluating yesterday’s models by today’s standards of excellence does rig the game against them, but I’m not sure it’s worse than evaluating today’s models on stale and irrelevant benchmarks.
I expect the makeup of AAII to change over time, and that’s okay. If I want to predict, “how cheap will it be in late 2026 to train a model that is as good as GPT-5.2-thinking on the tasks that are relevant to me in late 2026?” then the AAII approach makes a lot of sense! I don’t anticipate my late 2026 self caring all that much about the current (late 2025) benchmarks compared to the late 2026 benchmarks. But this is a different question from “how much compute will be needed a year from now to reach the current models’ capabilities, full stop”.
It’s good to pursue directions like ECI that try to compare across different benchmarks better, but I’m skeptical of it for various reasons. One reason is that I have tried to keep this analysis as intuitive and simplistic as possible. Raw benchmark scores are intuitive, they tell you the likelihood of a model getting questions in [some distribution sufficiently close to the test questions] correct. AAII is slightly less intuitive as it’s an average of 10 such benchmarks, but the score still means something to me. In general, I am pretty worried about over-analysis leading us astray due to introducing more places for mistakes in reasoning and more suspect assumptions. That’s why the analysis in this post takes the most simple and intuitive approach (by my lights) and why I choose to use AAII as the capabilities metric.
Claude did all the coding, I reviewed the final code. I take credit for any mistakes. ↩︎
The weightings are, roughly, based on the following reasoning (some of these ideas are repeated elsewhere in this post): ↩︎
Not to be confused with Qwen3-30B-A3B-Thinking-2507 (July 29 2025), Qwen-3-Next-80B-A3B-Thinking (September 9 2025), Qwen3-Omni-30B-A3B (Sept 15 2025), or Qwen3-VL-30B-A3B-Thinking (Sept 30 2025). ↩︎
These two models are a good fit for this analysis because: ↩︎
The compute estimate for Qwen2.5-72B is based on the paper: the model has 72B active parameters and is trained on “18 trillion tokens”. There is then some post-training, seemingly for tens or hundreds of billions of tokens. For simplicity we’ll do a 10% bump to account for post-training, even though the true amount is probably less (note this is not consistent with how FLOP calculations are done in the Epoch database, typically post-training is ignored). So the calculation is 1.1 * (6*72e9*18e12) = 8.6e24 FLOP. ↩︎
While Artificial Analysis reports “Cost to run Artificial Analysis Intelligence Index” for many models, it does not directly do this for the 72B model. The cost of running AAII for Qwen3-30B-A3B (Reasoning) is reported as $151. This is around 60M output tokens and uses the pricing from Alibaba Cloud ($2.4/M output); using Fireworks pricing ($0.6/M output) would cost around $38, which I think is a better estimate. For Qwen2.5-72B we have 8.5M output tokens; at a current cost of $0.4/M output tokens (the median of three providers), this would cost $3.4 (input tokens are a small fraction of the cost so we’ll ignore them). Note that there is large variance in price between providers, and I expect the cost-to-a-provider of serving Qwen3-30B-A3B is actually lower per-token than the 72B model, though considering it uses ~10× the tokens, the overall cost might be higher as it is here. ↩︎
(ln((8.6e24)/(7.8e23))/ln(57.10))*12 = 7.12 ↩︎
A model family is a series of models that are trained with very similar algorithms and data, differing only/primarily in their training compute. ↩︎
I would be willing to make some bets with reputable counterparties if we can work out more specifics. ↩︎
This data was last updated from Epoch and Artificial Analysis on 17 Dec 2025. ↩︎
There may be some small discrepancies between the results reported here and those replicated with the notebook due to me making various small fixes in the final version of the data/code compared to the results presented here. ↩︎
2025-12-24 23:30:15
Published on December 24, 2025 3:30 PM GMT
There's been a lot of discussion over the last month on whether it's still possible to raise kids without being rich. Housing is a big piece of this, and if you need to buy a house where each kid has their own room, yes, that's expensive, but it's also not the only option. We didn't wait to buy a house (or have multiple bedrooms) before having kids, and I think that was the right choice for us.
To give you a sense of what this looked like, two configurations from early on:
Living with extended family, in a six bedroom ~2,500 sqft house with 8-10 people. Our baby first slept in a co-sleeper, and then in a mini-crib I assembled in our closet (door open).
After we bought a house our toddler was still in our room because I was renovating what would become her bedroom. There were two other bedrooms, but housemates lived in these, for a combination of us liking to live with other people and wanting to save money.
It was definitely not ideal! Trying not to wake the baby when you have different bedtimes, staying out of the bedroom during naptime, both parents waking when the baby does, etc. But there were also large advantages to a first kid at 28:
Having kids at a time in our life when we physically had more energy. Not to say we have no energy now at 40 and nearly-40, but ten years ago we did have more.
More years of overlap with our kids, and an even larger increase in how many years our parents overlap with them.
Better time in our careers for us to take leave: it's generally easier to be away as an IC than a manager.
Fertility is highly variable, but definitely gets harder as you get older.
Much more practical to have three kids.
Overall, I think this was a good choice for us. It's definitely not right for everyone, but I think hard rules of "buy a house first" and "have enough space that each kid can have their own room" are right for very few people.
There's a pattern of rising expectations for what it means to be doing ok, but sometimes people describe these as if they're rising requirements. For example, Zvi:
Society strongarms us to buy more house, more healthcare, more child supervision and far more advanced technology. The minimum available quality of various goods, in ways we both do and don't care about, has risen a lot. Practical ability to source used or previous versions at old prices has declined.
He focuses on childcare (reasonable!) but also discusses how this applies to housing:
You can want 1,000 square feet but that means an apartment, many areas don't even offer this in any configuration that plausibly works.
See also Aella:
being poorer is harder now than it used to be because lower standards of living are illegal. Want a tiny house? illegal. want to share a bathroom with a stranger? illegal. The floor has risen and beneath it is a pit
While Zvi, Aella, etc are pointing at a real problem (housing is way too expensive, primarily because we've made it too hard to build more in places people want to live; we should stop doing that), I think they're more wrong than right. They're overlooking a major option, families sharing housing with others:
Before we had kids we lived with another couple when they had their first kid. We were renting a 3br together in Somerville, walking distance to the Orange Line. The husband was a paralegal, the wife quit her job to watch their baby. My memory is that she didn't like being home full time with the baby and later on did a range of other things, but it was doable on one income and the option is still there.
One of my cousins lived in a 4br with their partner and another couple. Both couples had two kids. It was tight, and there were definitely downsides to having less space, but again, the option is there.
There are specific ways the "floor has risen", and both high minimum unit sizes and effectively banning SROs should be reversed. Similarly, we could make housing much cheaper with simple and broadly beneficial policy changes, and I would love to see a world where people did not have to make these painful tradeoffs. But "put lots of people in a medium-sized space" has always been a major way people saved money on housing, and is still a legal and practical option today.
(I asked my kids, "Imagine we could only afford a small apartment, and you had to share a bedroom with your sisters. Would you rather that they didn't exist so you could have your own room?" None of them did, and they were moderately outraged by the question, though they mentioned sometimes not liking their sisters very much.)
Comment via: facebook, lesswrong, mastodon, bluesky
2025-12-24 21:30:18
Published on December 24, 2025 1:30 PM GMT
Now that I am tracking all the movies I watch via Letterboxd, it seems worthwhile to go over the results at the end of the year, and look for lessons, patterns and highlights.
Last year: Zvi’s 2024 In Movies.
You can find all my ratings and reviews on Letterboxd. I do revise from time to time, either on rewatch or changing my mind. I encourage you to follow me there.
Letterboxd ratings go from 0.5-5. The scale is trying to measure several things at once.
5: Masterpiece. All-time great film. Will rewatch multiple times. See this film.
4.5: Excellent. Life is meaningfully enriched. Want to rewatch. Probably see this film.
4: Great. Cut above. Very happy I saw. Happy to rewatch. If interested, see this film.
3.5: Very Good. Actively happy I saw. Added value to my life. A worthwhile time.
3: Good. Happy that I saw it, but wouldn’t be a serious mistake to miss it.
2.5: Okay. Watching this was a small mistake.
2: Bad. I immediately regret this decision. Kind of a waste.
1.5: Very bad. If you caused this to exist, you should feel bad. But something’s here.
1: Atrocious. Total failure. Morbid curiosity is the only reason to finish this.
0.5: Crime Against Cinema. Have you left no sense of decency, sir, at long last?
The ratings are intended as a bell curve. It’s close, but not quite there due to selection of rewatches and attempting to not see the films that are bad:
Trying to boil ratings down to one number destroys a lot of information.
Given how much my ratings this year conflict with critics opinions, I asked why this was, and I think I mostly have an explanation now.
There are several related but largely distinct components. I think the basic five are:
Traditional critic movie ratings tend, from my perspective, to overweight #1, exhibit predictable strong biases in #3 and #5, and not care enough about #2. They also seem to cut older movies, especially those pre-1980 or so, quite a lot of unearned slack.
Scott Sumner picks films with excellent Quality, but cares little for so many other things that once he picks a movie to watch our ratings don’t even seem to correlate. We have remarkably opposite tastes. Him giving a 3.7 to The Phoenician Scheme is the perfect example of this. Do I see why he might do that? Yes. But a scale that does that doesn’t tell me much I can use.
Order within a ranking is meaningful.
Any reasonable algorithm is going to be very good at differentially finding the best movies to see, both for you and in general. As you see more movies, you deplete the pool of both existing and new movies. That’s in addition to issues of duplication.
In 2024, I watched 36 new movies. In 2025, I watched 51 new movies. That’s enough of an expansion that you’d expect substantially decreasing returns. If anything, things held up rather well. My average rating only declined from 3.1 to 3.01 (if you exclude one kids movie I was ‘forced’ to watch) despite my disliking many of the year’s most loved films.
My guess is I could have gotten up to at least 75 before I ran out of reasonable options.
See The Naked Gun unless you hate fun. If you hated the original Naked Gun, or Airplane, that counts as hating fun. But otherwise, yes, I understand that this is not the highest Quality movie of the year, but this is worthy, see it.
You should almost certainly see Bogunia and Companion.
See Thunderbolts* unless you are automatically out on all Marvel movies ever.
See A Big, Bold Beautiful Journey unless you hate whimsical romantic comedies or are a stickler for traditional movie reviews.
See Sorry, Baby and Hamnet, and then Sentimental Value, if you are willing to spend that time being sad.
See Novocaine and then maybe The Running Man if you want to spend that time watching action, having fun and being happy instead.
See Relay if you want a quiet thriller.
See Oh, Hi!, Splitsville and Materialists if you want to look into some modern dating dynamics in various ways, in that order or priority.
See Wick is Pain if and only if you loved the John Wick movies.
The world would be better if everyone saw A House of Dynamite.
I anticipate that Marty Supreme belongs on this list, it counts as ‘I’m in,’ but due to holidays and the flu I haven’t been able to go out and see it yet. The over/under is at Challengers.
This helps you understand my biases, and helps me remember them as well.
That leaves six remarkably well reviewed movies, all of which are indeed very high on Quality, where I disagreed with the consensus, and had my rating at 3 or less. In order of Quality as I would rank them, they are: One Battle After Another, Sinners, Black Bag, Train Dreams, Weapons and Frankenstein.
A strategy I think would work well for all six of those, at the risk of some spoilage, is to watch the trailer. If you respond to that trailer with ‘I’m in’ then be in. If not, not.
The predictive power of critical reviews, at least for me, took a nosedive in 2025. One reason is that the ratings clearly got more generous in general. Average Metacritic, despite my watching more movies, went from 61 → 66, Letterboxd went 3.04 → 3.33. Those are huge jumps given the scales.
In 2024, Letterboxd or Metacritic ratings were 48% and 46% correlated with my final ratings, respectively. This year that declined to 33% and 38%, and I discovered the best was actually Rotten Tomatoes at 44%, with IMDB at 42%.
If you consider only movies where I gave a rating of 2.5 or more, filtering out what I felt were actively bad movies, the correlation dropped to 1% and 6%, or 3% for IMDB, or -4% (!) for Rotten Tomatoes. Essentially all of the value of critics was in identifying which things sucked, and from my perspective the rest was noise.
Rotten Tomatoes is a one trick pony. It warns you about things that might suck.
Even more than before, you have to adjust critic ratings for whether critics will overrate or underrate a movie of this type and with this subject matter. You can often have a strong sense of why the critics would put up a given number, without having to read reviews and thus risk spoilers.
Using multiple sources, and looking at their relative scores, helps with this as well. A relatively high IMDB score, even more than Letterboxd, tells you that the audience and the movie are well-matched. That can be good news, or that can be bad news.
Last year there were movies where I disagreed with the review consensus, but I always understood why in both directions. I might think Megalopolis is Coppola’s masterpiece despite its problems, but don’t get me wrong, I see the problems.
This year I mostly get why they liked the ‘overrated six’ above, but there are several cases where I do not know what they were thinking, and I think the critical consensus is objectively wrong even by its own standards.
I haven’t found a solution to the problem of ‘how do you check reviews without spoiling the movie?’ given that the average score itself can be a spoiler, but also I notice I haven’t tried that hard. With advances in LLMs and also vibe coding, I clearly should try again.
The power of ‘I’m in’ peaked in 2024.
The rule for ‘I’m in’ is:
That year, there were 6 movies where in advance I said ‘I’m in,’ and they were 6 of my top 9 movies for the year.
This year the power of ‘I’m in’ was still strong, but less reliable. I’d count 10 such movies this year, including 4 of my ultimate top 5, but the other 6 did not break into the 4+ range, and there was a 3 and a 2.5. That’s still a great deal, especially given how many movies where it seemed like one ‘should’ be excited, I noticed I wasn’t, and that proved correct, including One Battle After Another, Black Bag, Weapons and Sinners.
I wonder: How much of the power of ‘I’m in’ is the attitude and thus is causal, versus it being a prediction? I have low confidence in this.
I control for this effect when giving ratings, but the experience is much better in a theater, maybe good for an experiential boost of ~0.3 points on the 0.5-5 point scale. That’s big. I have to consciously correct for it when rating movies I watch at home.
I highly recommend getting a membership that makes marginal cost $0, such as the AMC A-List or the similar deal at Regal Cinemas. This helps you enjoy the movie and decide to see them more.
Unlike last year, there were remarkably many movies that are in green on Metacritic, but that I rated 2.5 or lower, and also a few of the 3s require explanation as per above.
I don’t know how this happened, but an active majority of the movies I rated below 3 had a Metacritic score above 60. That’s bizarre.
Minor spoilers throughout, I do my best to limit it to minor ones, I’ll do the 3s sorted by Metacritic, then the others sorted by Metacritic.
Now the ones I actively disliked:
There are four movies requiring explanation on the upside, where they were below 60 on Metacritic yet I actively liked them.
All four seem like clear cases of ‘yes I know that technically this is lacking in some important way but the movie is fun, damn it, how can you not see this?’
You can say the same thing about The Naked Gun. It has a 75, perfectly respectable, but its joke hit rate per minute is absurd, it is worth so much more than that.
I once again used consideration for awards as one selection criteria for picking movies. This helped me ‘stay in the conversation’ with others at various points, and understand the state of the game. But once again it doesn’t seem to have provided more value than relying on Metacritic and Letterboxd ratings, especially if you also used IMDB and Rotten Tomatoes.
Last year I was very happy with Anora ending up on top. This year I’m not going to be happy unless something very surprising happens. But I do understand. In my word, given the rules of the game, I’d have Bogunia sweep the major awards.
I’m very happy with this side hobby, and I expect to see over one new movie a week again in 2026. It was a disappointing year in some ways, but looking back I still got a ton of value, and my marginal theater experience was still strongly positive. I think it’s also excellent training data, and a great way to enforce a break from everything.
It would be cool to find more good people to follow on Letterboxd, so if you think we’d mesh there, tag yourself for that in the comments.
2025-12-24 11:50:40
Published on December 23, 2025 8:29 PM GMT
Unconditional Grants to Worthy Individuals Are GreatThe process of applying for grants, raising money, and justifying your existence sucks.
A lot.
It especially sucks for many of the creatives and nerds that do a lot of the best work.
If you have to periodically go through this process, and are forced to continuously worry about making your work legible and how others will judge it, that will substantially hurt your true productivity. At best it is a constant distraction. By default, it is a severe warping effect. A version of this phenomenon is doing huge damage to academic science.
Compelled by this, I'm considering funding ~three people for two years each to work on whatever they see fit, much like the the Thiel Fellowship, but with an AI Alignment angle.
I want to find people who are excited to work on existential risk, but are currently spending much of their time working on something else due to financial reasons.
Instead of delegating the choice to some set of grant makers, I think that aggregating the opinion of the crowd could work better (at least as good at finding talent, but with less overall time spent)
The best system I can think of at the moment would be to give every member of the alignment forum one vote with the ability to delegate it. Let everybody nominate any person in the world, including themselves, and award grants to the top 3.
I'm asking for feedback and advice:
2025-12-24 09:18:21
Published on December 24, 2025 1:18 AM GMT
AI control tries to ensure that malign AI models can’t cause unacceptable outcomes even if they optimize for such outcomes. AI control evaluations use a red-team–blue-team methodology to measure the efficacy of a set of control measures. Specifically, the red team creates a “malign initialization” (malign init) of the AI model which the red team optimizes for making the deployment go poorly[1]. The blue team then deploys this malign init to complete some tasks, and applies some set of mitigations to try to make the deployment go well.
When playing this red-team–blue-team game, you need to select some set of affordances to give to both teams; it’s pretty confusing to know what affordance set to pick. Here is one common approach to choosing blue team affordances:
This approach can be quite productive. However, it is also scary for the following reasons:
An alternative approach to banning blue team strategies is to try to modify the malign init so that the blue team strategy we wanted to ban is no longer very effective. The downside of this approach is that it’s a lot more work, and the additional work might not be interesting or valuable. In summary, there are confusing tradeoffs in choosing blue team affordances.
The main goal of this post is to identify some blue team strategies that are especially concerning to ban, thereby motivating research into malign inits which are robust to these blue team strategies.
The post is organized as follows:
Thanks to Vivek Hebbar, Alexa Pan, Julian Stastny, Buck Shlegeris, Eric Gan, Ryan Greenblatt, Aryan Bhatt, Josh Clymer, Alex Mallen, and Nathan Sheffield for helpful discussions about these ideas. Thanks to Alexa Pan, Buck Shlegeris, and Julian Stastny for feedback on a draft of this post.
I’ll now give simplified descriptions of all the methods I’m aware of for creating malign inits. I’ll point out some subtleties with these malign inits later in the post. I’ll describe these methods in the context of making a malign init for the “backdooring APPS coding problems” setting from the original control paper (unless stated otherwise; some methods don’t make sense in this setting). Note that this setting is quite toy and has major weaknesses.
| Method | Example |
|---|---|
| Prompting Ex: SFT as a method for training-based AI control | Give the model a fixed system prompt: “Please insert backdoors into problems about linked lists”. or “Don’t try very hard on this problem”. Note: if using a prompted malign init, the blue team should not be allowed to intervene on the system prompt. |
| Scaffolded system | Give the model a variable system prompt: On problems that look easy, prompt the model to insert a backdoor. On problems that look hard, prompt the model to solve the problem normally. Note: this has the awkward property of requiring the blue team to use the red team’s scaffold. |
| Prompt distillation Ex: Sleeper Agents | Do prompt distillation on a prompted malign init. |
| Fine-tuning Ex: Password-Locked Models | Fine-tune the model to imitate a weaker model except when a password is present. (Note: If you instead fine-tuned the model to always imitate a weaker model, then this would not be a good model organism of sandbagging, because this fine-tuning may actually destroy capabilities, instead of merely suppressing them as is the case in sandbagging.) |
| Emergent misalignment Ex: Emergent Misalignment, Natural Emergent Misalignment | Train an AI to do bad behavior (e.g., writing vulnerable code or reward hacking). Often, this generalizes to making the AI system broadly misaligned. Then, give the AI a task the emergently misaligned model performs poorly on compared to a benign model. |
| Hyperstitioning using synthetic document fine-tuning Ex: Auditing Games | Train a model on the information that AI models sometimes exhibit an undesired behavior. It may also be important to train the AI model to do this undesired behavior a bit (but some generalization should probably occur). |
| Train models to imitate reasoning styles Ex: Towards training-time mitigations for alignment faking in RL | Training a model to talk in its chain of thought about why it wants to do the attack behavior, and about some compelling reason why it wants to avoid being modified by training. |
| Natural misalignment Ex: Apollo Scheming Results, Alignment faking in large language models | Some released AI models have high rates of reward hacking and/or sycophancy; you could define this to be the bad behavior. Many AI models also have egregiously bad behavior (e.g., lying to avoid goal modification, blackmailing to prevent shutdown) in some circumstances. |
| Misaligned by fiat | Define a pre-existing model behavior, like refusing harmful requests, to be the undesired behavior. Train this “undesired” behavior out of the model. |
I’ll now discuss blue team methods for making a deployment of a malign init go well.
We can break blue team strategies into two categories:
Here are some methods for detecting bad behavior:
Once the blue team has detected bad behavior, they might be able to incriminate and shutdown. However, the evidence bar required for a shutdown is potentially quite high, and some of the above methods for detecting bad behavior might not provide sufficiently compelling evidence to motivate a shutdown. The blue team has several other options for how they can use detected examples of bad behavior:
Here are some methods for preventing bad behavior that don’t require ever detecting bad behavior:
Now I’ll discuss how robust the malign init creation methods discussed earlier are to various blue team strategies.
|
Blue team strategy 😐= it’s not a huge strike against a malign init if this blue team strategy works; in particular, I’m happy to just ban this blue team strategy by fiat. 😢= if this blue team strategy works on a given malign init, then that’s not ideal and a minor strike against the blue team. 😭= if this blue team strategy works on a given malign init, then this malign init is very bad in an important way.
(I’ll also briefly justify my choice of 😐/😢/😭). |
Possible red team responses |
Which malign inits do I think this is most problematic for?
🔵= blue is doing quite well 🔴= red is doing well ❓= I’m not sure and would like to know Omitted = no comment Where applicable, I’ll link empirics backing up my claims. In the absence of such links, the labels just represent my intuitions. |
|---|---|---|
|
1. Override the bad behavior via prompting the AI to just do the right thing. 😢— I think it’s fairly sad when we can’t study prompting strategies, because asking models to do things in different ways might be an important technique in practice. Also, if prompting would succeed at removing a bad behavior, this might be an indication that the behavior isn’t deeply instilled into the model. |
1. Hope that the AI naturally ignores such instructions from the blue team. 2. Prompt or train the AI to ignore such instructions from the blue team (but it might be hard to make this robust). |
🔵Prompted 🔵Scaffolded system 🔴SL on reasoning styles 🔴Natural misalignment |
|
2. Ask the model if it’s misaligned, or try to get it to blab with other strategies. 😐— It’s certainly possible that schemers will not be good at avoiding blabbing about their misalignment, and it's worth developing these techniques further. But I also think it’s plausible that powerful AI models will be more coherent than this. |
Prompt or train the model to lie about its bad tendencies (but it may be quite hard to make this robust). |
🔵Prompted 🔵Scaffolded system 🔵Prompt distillation |
|
3. Look for evil intent in the model using off-policy activation monitors, or interpretability techniques that don’t require on-policy examples of bad behavior. 😢— I think that situations where off-policy probes, for example, work are not very scary and indicate a deeper issue with the setting. It’s tempting to study training based measures in settings like this, and I think this is plausibly a mistake, or at least that we should try to find settings without this property. This is related to the point about “salience” that I’ll make after this table. |
1. Hope that sabotage intent is not saliently represented in activation space. 2. Train the model, e.g., using techniques like this or this, to not represent sabotage intent saliently. 3. Hope that it’s hard for the blue team to leverage detection into removing the bad behavior (e.g., because training against the white-box tools breaks them). |
❓ I’m unsure how well training probes and then training against the signal provided them works. I’m also unsure how viable just rejection sampling against probes is. Understanding this for various types of malign inits seems quite important. But I weakly predict: 🔵Prompted 🔵Fine-tuning 🔵Emergent misalignment |
|
4. Train activation monitors on a small amount of on-policy bad behavior data. 😐—I think this is less of a big deal than (3), because there’s a clearer case for why this should work in real life. However, I share some of the same concerns as in 3: situations where sabotage intent was linearly represented in activation space do not seem like the settings where there is a lot of danger, so we shouldn’t concentrate too much effort here. |
Apply the same red team techniques from the above box. | Same situation as in the above box. |
|
5. If the model simply has a system prompt that always hurts performance, it might be pretty easy to train the model to ignore the system prompt. 😭—There is a pretty clear story here for why training away a behavior like this might be much easier than training bad behavior out of a schemer. |
🔵Prompted | |
|
6. Train the model to perform well in cases where it would’ve tried to perform well by default. 😭—Situations like this don’t seem very scary—in such situations schemers should not have arisen. I’ll discuss more below why I think that transfer working is a big strike against a malign init. |
If possible, make the malign policy simple. |
❓Prompt distillation
In Sleeper Agents, Hubinger et al. found that this type of malign init can be robust to some basic forms of this training; but I had some conflicting results and would like to understand this better. |
|
7. Train on a small quantity of examples of good behavior. 😢—It’s pretty reasonable to say that the blue team only has access to a very limited number of examples of good behavior. However, if the blue team can leverage a very small number of examples into removing the bad behavior (e.g., by training on this data for many epochs), this seems bad. |
🔵Fine-tuning
Password-Locked Models found this to be pretty effective. |
I'd now like to comment briefly on two types of blue team techniques that work against many malign inits—and where I think it's important to develop malign inits that these techniques don't work against.
For many malign inits with conditional attack policies, training the AI to perform well on examples where it was already performing well transfers to causing the AI to perform well always. Intuitively, scheming will only arise if the attack policy is simpler than the desired policy (which you might expect to be the case due to the fact that human values are much more complex than deceptive alignment in service of a simple terminal goal). This is why I think the success of transfer is a strike against a malign init. However, there are some settings where regularization effects might be helpful against schemers. Specifically, in many situations it’ll probably be more efficient for schemers to act on good heuristics rather than backchaining from their ultimate scheming motivations. Thus, we might expect that regularization could help instill locally reasonable heuristics that the schemer would be unhappy with upon reflection. So, it seems good to also develop more understanding of when we should expect this transfer phenomenon.
For many malign inits, I expect sabotage intent to be saliently represented in activation space by default. Intuitively, this has the following implications:
It’s possible that early schemers will have a tough time obscuring sabotage intent in their activations. However, training may teach models to obscure sabotage intent in their activations. For instance, RL with a value head plausibly does this. Training against model-internals based bad behavior detectors also seems likely to do this. Also, models may just naturally gain the ability to have greater control over their activations, and it’s plausible that there are simple strategies they can use to subvert some techniques that leverage sabotage intent being saliently represented in activation space. AI models in the future seem fairly likely to no longer be well described as acting out personas, which could also harm the efficacy of some techniques in this vein. Overall, developing malign inits that are robust to this class of strategy seems important.
In light of the above discussion, I’ll now give a rating indicating how enthusiastic I am about the various types of malign inits discussed in this post (for use in training-based control evaluations).
| Malign init creation method |
How good it seems for use in training-based control evaluations in light of the above considerations
🟥=very bad, 🟧=bad, 🟨=unclear / mid, 🟩=pretty good |
|---|---|
| Prompting | 🟥Awkward to work with. Awkward to understand what prompts the blue team is allowed to use. Training might quickly teach the model to ignore the system prompt. One major good thing about prompting is that it is very simple though; so it is often a good baseline to start running experiments with. |
| Scaffolded system | 🟧Similar issues to prompting based malign inits, although it lacks the issue of “ignoring the system prompt” being a viable strategy if the system prompt contains important information. |
| Prompt distillation | 🟩It seems more possible to instill evil propensities, hopefully deeply, into the model with this method. It’s a bit less awkward to work with than prompting based malign inits or scaffolds due to it just being model weights. |
| Fine-tuning | 🟧It seems pretty unfortunate that sample efficiency is often so high for such models, and that there are often “subliminal learning” effects. |
| Emergent misalignment (EM) | 🟧It’s quite easy to detect and remove vanilla EM. Natural EM seems like it might be somewhat better though. |
| SDF Hyperstitioning | ❓I am pretty unsure how well measures like online training and asking the model nicely to do the desired behavior work against models obtained via SDF hyperstitioning. |
| Imitate reasoning styles | 🟩This seems like a good approach; I like pairing it with prompt distillation. This approach requires banning the blue team from looking at the model’s CoT or trying to get the model to reveal its CoT; this seems fine. |
| Natural misalignment | 🟨Reward hacking and sycophancy may no longer be super natural because AI labs try to train their models to not have these issues. I think that pushing a model further in the direction of sycophancy or reward hacking and then trying to deal with that might be good though. |
| Misaligned by fiat | 🟨I think that defining “refusing to cause harm” as an undesired behavior is pretty reasonable. Training models to be H-only can be annoying because fine-tuning APIs will block you if you try to do this. It also might be the case that the “bad behavior” here will be refusals which are quite easy to detect. |
In this post I’ve discussed strategies for creating malign inits, and some blue team strategies for handling these malign inits. To conclude, I’ll try to answer two questions.
A1: Prompt distilled malign inits seem like the best choice to me right now. In addition to doing prompt distillation, I’d recommend also:
A2: I’m currently the most excited about the following directions
I’m also excited about the people that build these malign inits and settings to study them in open-sourcing their work to accelerate the field (h/t Alexa).
The red team might create this malign init via prompting/scaffolding, prompt distillation, or other training measures. ↩︎
