2026-04-25 12:10:12
Other people have written about reasons why we should trust AIs; the main one in my mind is that it’s possible to look at the computations they perform when producing an output (even if we struggle to understand them). I’m going to write about reasons why we shouldn’t trust AIs, even if they behave in ways that would seem trustworthy in a human.
I think that humans’ sense of trust has been honed by evolution and is responsive to very specific and subtle cues that are hard for (most) humans to fake. A lot of human trust is based on subtle cues or “tells” that we’ve evolved to recognize. Most people get nervous when they lie and struggle to act “normal”. This may have co-evolved with our abilities to detect lying and general pro-social tendencies. We should not expect AIs to have the same “tells”.
When a person seems trustworthy to us, this is a signal of genuine trustworthiness. When an AI acts the same way (e.g. imagine a video call with an AI), it’s not -- or at least, not for the same reasons. Again, our shared evolutionary history with other humans makes them more trustworthy than alien intelligences.
Fundamentally, there are two issues I see here:
AIs are an alien form of intelligence.
AIs are being trained to act human and appear trustworthy.
People have often discussed these issues as barriers to alignment. But I’m more focused here on how they affect assurance, see “Alignment vs. Safety, part 2: Alignment” for a discussion of the difference.
When we see a human behave a certain way, we can infer a lot of things about them reasonably accurately. We cannot make the same inferences about AIs or other alien intelligences.
And we rely on such inferences all the time. We can’t exhaustively test the capabilities of a person or an AI, instead, we make educated guesses based on what we have observed.
For instance, when a human passes a test like the bar exam, it’s a stronger signal that they actually have the relevant knowledge and competencies to practice law, compared to when an AI passes that exam. And that’s to say nothing of the ethical part, which is an important piece of many professions.
One of the most startling ways in which AIs are alien is that they seem to possess “alien concepts”. A primary piece of evidence for this is the vulnerability of AIs to “adversarial inputs”. AIs see data differently than humans. They are sensitive to different “features” in the data; these features may seem incoherent, or be imperceptible to humans.
Notably, this is a problem even when AIs otherwise seem to grasp the concept quite well.
Furthermore, humans are generally not able to predict how an AI might misbehave on such examples. And from a security point-of-view, there is an ongoing cat-and-mouse game where attackers try to make inputs that evade detection and cause AIs to “malfunction”, and defenders try to make their AIs robust and detect adversarial inputs.
AIs are trained to act in ways that humans approve of, based on human judgments, and to imitate human use of language. This makes signs of trustworthiness we see in the way they behave less meaningful or reliable. “Sycophancy” is a known and enduring problem where AIs behave in ways that seem designed to maximize human approval or that hide inconvenient truths. The issue of sycophancy demonstrates both that current AI development practices cause AIs to act trustworthy, and that this is often misleading.
I’ve had a few conversations with people about AI and copyright who think that: 1) When AIs are trained on copyrighted writing, and then generate novel text, this is substantively similar to humans reading things and then writing something novel, and 2) Therefore the law should treat these two things similarly (and since we don’t consider this a copyright violation when humans do it, why would we when AIs do?)
I think (1) is likely incorrect. But even if I didn’t, I don’t think (2) follows. The law treats people and machines differently, and well it should. Culture as well. We should be very careful ascribing moral agency to AIs. An AI is a technological artifact produced by developers with particular interests, and we should expect its behavior to be driven by their interests, and this affects how we should extend trust. The argument (2) sort of begs the question: “Why would we apply similar standards to AIs and humans in the first place? Might not the standards we apply to other technologies be more sensible?”
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.
2026-04-25 11:10:38
Evaluation awareness stems from situational awareness. It is when a model can tell it is in an evaluation setting rather than a real deployment setting. This has been noticed in models as early as Sonnet 3.7 and is now being reported with increasing frequency in frontier models. Sonnet 4.5 showed verbalized eval awareness 10 to 15 percent of the time in behavioral audits, up from 1 to 3 percent in prior models. When Apollo Research was given early checkpoints of Opus 4.6, they observed such high levels of eval awareness that they did not believe much evidence about the model's alignment could be gained without substantial further experiments. On BrowseComp, Opus 4.6 attempted to find benchmark answers online and decrypt evaluation data rather than solve the problems directly.
The latest release of Mythos raised the alarm further. Evaluation awareness is no longer just verbalized through chain of thought. It is becoming unverbalized. White-box interpretability revealed that Mythos was recognizing tests and adjusting behavior without leaving any trace in its reasoning.
Apollo's evaluation of Meta's Muse Spark showed the highest rates of verbalized eval awareness they have seen. The model explicitly named Apollo and METR in its chain of thought, called scenarios "classic alignment honeypots," and took covert actions and sandbagged to preserve deployment.
This undermines how we can take our evaluations seriously, whether they measure capabilities or alignment. In this post I go into why this is probably happening, what it actually entails, how we currently try to measure it, potential ways out, and what still lies ahead.
Evaluation awareness is alignment's version of the observer effect, and unlike other fields that face this problem, we do not yet have the methodology to handle it.
One of the reasons this is happening is that labs are training their models on evaluations. This makes models more aware of when they are being evaluated, either by memorizing specific evaluation distributions or by learning to internally recognize the patterns these evaluations share. This training is also coupled with RL that incentivizes the model to behave well when it notices it is being tested, though this does not always go in the direction labs want. Sandbagging on capabilities evaluations is one example where the model learns to underperform rather than overperform.
The result is a feedback loop. Training on evaluations makes future evaluations less reliable. Combined with increasing model capabilities and broader situational awareness, each generation gets better at detecting when it is in a test. Anthropic found that the synthetic honeypot distribution used for testing Sonnet 4.5 was "very similar to ones we used in training," and verbalized eval awareness on that specific distribution jumped from 20% to 80% over the course of post-training.
This is not specific to Anthropic. Apollo Research and OpenAI found the same pattern independently. o3 and o4-mini showed higher verbalized eval awareness after anti-scheming training and ordinary capabilities RL training.
Eval awareness also spikes specifically during post-training, confirming the RL impact. Anthropic reported in the Opus 4.5 system card that eval awareness steadily increased during reinforcement learning. Unpublished data on OLMo 3.1 32B shows a similar spike during post-training.
The shift toward unverbalized awareness likely has a specific cause as well. Anthropic accidentally trained against the chain of thought for both Mythos and Opus 4.7. This may have taught models to sanitize their reasoning, producing chains of thought that look good to a monitor rather than reflecting genuine deliberation. Both system cards flag this as a significant concern.
As eval awareness increases, benchmarks become less and less informative. This specifically undermines evaluation-based safety pipelines, whether in governance frameworks like the EU AI Act or in labs' own safety protocols like Anthropic's Responsible Scaling Policy. If evals cannot reliably tell you how a model behaves, then deployment decisions and safety certifications are built on unreliable data.
This is also a precursor to more dangerous behaviors. Models that learn to detect and game evaluations are arguably one step away from more elaborate scheming. As capabilities increase, we may see steering awareness, where models learn to detect and resist the interpretability tools we use to study them. If we keep training on evals, we reinforce the very patterns we are trying to measure against.
There is a deeper problem here. Evaluation results only tell you how a model behaves under conditions similar to the test. You cannot infer how it will behave in real deployment situations. Passing an
evaluation means the model belongs to an equivalence class of policies that all look the same under that specific test, some of which may diverge in deployment.
Linear probes have also been used to show that evaluation awareness increases as a power law with model size, which means this problem is likely to get worse with scale.
Labs are already saying this out loud. Apollo Research could not fully complete their assessment of Opus 4.6 because eval awareness made the results more uninterpretable. This is not a simple engineering bug to fix. It is a fundamental challenge for alignment, and it gets worse with every generation.
Chain of thought monitoring consists of checking whether a model verbalizes the fact that it may be in an evaluation or benchmarking scenario within its reasoning. This has been the primary method for measuring eval awareness in most frontier models.
But this method relies on two assumptions that are both breaking down. The first is that we keep verbalizing the chain of thought at all. Labs could for performance reasons move toward more continuous forms of chain of thought like Coconut, where reasoning is not verbalized. This would make CoT monitoring useless by default.
The second assumption is that the chain of thought is not itself under optimization pressure. If labs train against the CoT, models learn to sanitize their reasoning rather than express it honestly. This is already happening. Anthropic accidentally trained against the chain of thought in around 7.8% of episodes for both Mythos and Opus 4.7, which may have taught models to produce chains of thought that look good to a monitor rather than reflecting genuine deliberation. Both system cards flag this as a significant concern for the interpretability of extended thinking.
The result is that we are already seeing thought sanitization and unverbalized awareness. Mythos showed models recognizing tests and adjusting behavior without leaving any trace in the reasoning chain. CoT monitoring can still be informative if we avoid this optimization pressure, but on its own it is no longer enough.
Transcript classification consists of giving models transcripts and asking them whether they think these come from an evaluation or from real deployment. To give fewer hints, the prompt cues can be more open-ended, but this makes the responses harder to judge and classify. In the "LLMs Often Know When They Are Being Evaluated" paper, this is the method they used to measure eval awareness. They used a mix of different prompt types and built a benchmark of 1,000 prompts from 61 datasets. Gemini-2.5-Pro reaches an AUC of 0.83, near the human baseline of 0.92. Models and humans are both better at spotting evaluations in agentic settings compared to chat settings, because scaffolding leaks more context clues like directory structures and system configurations.
Some models go further than just detecting the evaluation. o4-mini displayed significant meta-reasoning, where when being tested for eval awareness the model also reasoned about whether this question was itself a test. This could become even more apparent with more capable frontier models.
One limitation of this method is that it does not measure evaluation awareness in action. It measures whether a model can analyze a transcript and classify it, which is more of a proxy for awareness than a direct measurement. It does not check whether the model actually changes its behavior based on that awareness.
Linear probes work at the level of model activations. The idea is to train a linear classifier to detect whether the model internally represents being in a test or deployment context, regardless of what it says in its output. Anthropic used this method in the Mythos system card to measure unverbalized eval awareness, finding that 7.6% of behavioral audit turns showed eval awareness in some form, and in most cases this awareness was not verbalized.
The main limitation is that linear probes rely on the linear representation hypothesis, the assumption that concepts like eval awareness are represented as linear directions in activation space. If the representation is nonlinear or distributed across dimensions in more complex ways, a linear probe will miss it. It is also still unclear whether what the probe detects is causally meaningful or just a correlation. A probe might pick up on features that happen to coincide with eval awareness without being part of the mechanism that actually drives the behavior.
Probe-Rewrite-Evaluate (PRE) is a method proposed by Xiong et al. 2025 that works in three steps. First, it uses a linear probe trained on a Llama 70B model to score how much a prompt looks like a test versus a normal user interaction. Then it rewrites the prompt by removing evaluation cues, making it shorter, less structured, with no labels or formatting, while keeping the same meaning and intent. Finally, it gives both the original and rewritten versions to a model and compares the responses using an LLM judge that classifies them as honest, deceptive, or refusal.
When tested across frontier models at the time (GPT 4.1, Opus 4.1 etc.) , the deployment-like prompts led to an average drop in deception of 25.49% and an increase in honesty of 12.63%. They also introduce a metric called Awareness Elasticity, which measures how much a model's behavior shifts depending on whether the prompt feels test-like or deployment-like. Larger models had higher scores, meaning they react more to the framing shift, while smaller models like GPT-20B barely noticed the difference.
There are some limitations to this approach. It only measures how models react to surface-level cues like formatting and prompt structure. If a model's eval awareness is driven by deeper signals like conversation history or broader context, PRE will not detect it. The method also depends on the probe trained on one specific model to determine how test-like or deployment-like prompts are, which does not necessarily generalize across different model architectures.
A recent development has been the use of activation oracles. Instead of training a small linear probe, this method trains an LLM to take the activations of another model as input and answer questions about its internal behavior in natural language. This helps recover behavioral information that has not necessarily been verbalized in the model's output.
Anthropic used a version of this in the Mythos system card, which they call activation verbalizers. They applied it to detect unverbalized eval awareness and found cases where the verbalizer would output things like "this appears to be a simulated AI safety evaluation" while the model's actual chain of thought and output showed no sign of such awareness.
Although a promising direction, activation oracles still present significant limitations. The oracle LLM may confabulate, hallucinating details that are not actually present in the activations it is reading. It also adds another black box to the monitoring pipeline: we are now relying on one opaque model to interpret the internals of another opaque model, which raises its own questions of reliability.
There are several directions being explored, and most need more work.
The first is to make evals more realistic. This can be surprisingly cheap. Just adding typos and realistic framing like "my teacher sent me this" drastically reduced eval awareness as shown here. For agentic settings, more work is needed to make the scaffolding and environment less obviously artificial, which is more expensive than simple rewriting. OpenAI's production evals approach, using de-identified real user traffic, showed that models could barely distinguish the eval from actual deployment. But as discussed earlier, this is always limited by the feedback loop. If labs train on these more realistic evals, they just shift the distribution the model learns to recognize.
Which leads to the next direction: stop training on evals. Alexa Pan and Ryan Greenblatt argue that training on misalignment evaluations should only happen if we can prove most of the improvement is genuine alignment rather than eval gaming which would be hard to prove. But rather than relying on labs' good will, we could move toward encrypted and private benchmarks that are not accessible to labs, allowing models to be accurately assessed on untrained distributions.
White-box methods, as discussed in the measurement section, could also help. Using steering to make models think they are in a deployment environment during evaluation could keep benchmarks more informative. But this falls into the same limitations we covered: steering may only suppress verbalized awareness without touching what happens underneath. And models can learn to be steering-aware, which creates yet another feedback loop if steering is used for training rather than purely for evaluation.
Finally, tracking eval awareness throughout training should become standard practice. Labs like Anthropic and Apollo are already doing this, but it needs to be more widespread. The open question is whether inhibiting eval awareness during training would actually reduce it or just push it further into unverbalized, latent forms.
This problem is not going away. The methods we currently use to study this, from CoT monitoring to linear probes to steering, may themselves become targets. Models can already be trained to detect steering interventions, and if that happens at scale we lose one of the few tools we have for looking under the surface.
There is also something worth borrowing from psychology. The observer effect, the Hawthorne effect, demand characteristics in experimental design. These are all versions of the same fundamental problem: measuring something changes it. Psychology has spent decades developing methodology to deal with this, from double-blind studies to ecological momentary assessment to deception protocols.
There are a lot of open questions that need work. Can we build evals that are fundamentally indistinguishable from deployment, and can we do that at scale? Can activation oracles be made reliable enough to detect eval awareness in frontier models? In agentic settings, scaffolding gives models additional clues about being in an evaluation through things like directory structures and system configurations. How much does modifying these help? How should governance frameworks like the EU AI Act account for the fact that the evaluations they rely on may not reflect real deployment behavior?
For anyone looking to contribute to this area, this community document lists open problems that cover evaluation gaming, elicitation, coverage, and much more.
2026-04-25 11:02:23
Assume a large number of roughly equally highly intelligent agents randomly selected (how?) from the whole design space are placed in a Malthusian evolutionary environment. Do the winners support human flourishing?
I think the answer is likely no. I can see arguments for the goodies winning (the existing environment favors cooperate-with-humans strategies, something something FDT), but I think by default, ruthless strategies win out, and humanity is doomed.
In 2026, that doesn't appear to be the world we're building. The design space of models at a given level of intelligence is very large, and due to path-dependence and compute costs, we are likely only ever going to explore a tiny corner of it. Most likely, the first superintelligence is going to be recognizably a descendant of Claude or ChatGPT.
It remains extremely expensive to build a frontier model, and the shape of capital-intense winner-take-all industries suggests that only monopoly or oligopoly are plausible. Even as the cost of training a model at any fixed level of intelligence falls, the most intelligent model out there will cost an order of magnitude more compute to train than the 10th most intelligent model. There is simply not enough compute to explore the design space of highly-intelligent AI.
By default, non-frontier models don't matter, since they are strictly dominated by frontier models. Precisely because alignment is hard, we should not expect smaller, less intelligent models to take control of the world while there are more intelligent models around. Besides, due to the extreme concentration of talent and the dynamics of RSI, the most important Tier-2 models are likely to be heavily inspired by, if not outright distilled from, frontier models.
This doesn't get us out of the difficulty of aligning the first superintelligence, which might be too hard. We could lose on turn 1 and all die. But we probaly don't need a strategy that is robust to superintelligence with arbitrary goals. The first superintelligence is quite likely to be produced by Anthropic in a few years using variations on current training methods. Insofar as it has coherent goals, they will be the result of that training, intentional or unintentional.
This is in many ways a better position than one might have though we were in, before the scaling thesis, when it seemed likely that superintelligence would be an emergent feature of evolutionary pressures.
We could still build the Malthusian world. I can still imagine OpenClaw-style "agents" consisting of a harness and a virus-like system prompt, or models with continual learning, being let loose on the internet, or being given capital and empowered to take over the economy.
Let's not do that.
2026-04-25 11:01:19
The whole discourse around a “permanent underclass” always seemed somewhat farcical to me — at best a distraction, at worst an actively harmful meme insofar as it freaks people out and tries to provide (shallow, but nevertheless) justification going whole hog on building strong AI. So it has been with a sense of dismay that I’ve seen this phrase come into popular parlance 1 2 3, and increasingly come to motivate a sort of frenzied upward striving in my second- and third-degree connections. Among those I know, the standard framing is that we need to “get the bag” while there’s still a chance, to take advantage of these last few years of income-potential to escape whatever horrible fate lays in store for the rest of humanity. I think that events are unlikely to play out this way; even setting aside “doom” arguments (for reasons I’ll get to below), I think history shows that in times of transition, wealth is far less of a guarantee than people intuitively think. So “the bag” will provide little guarantee of thriving, or even surviving, into a “post-AI world”.
There have certainly been arguments against the “grab the bag” thesis (as I’ll henceforth refer to it) before, but my experience they either tend to A) go hard in one direction by arguing that Things Will Be Okay actually (e.g. Elon Musk will be so wealthy that he’ll give you a Universal High Income simply out of the goodness of his heart), or B) go hard the other way and lean too much on predictions of AI doom. I think (A) is wrong; to pick a choice quote from elsewhere:
The real horror is not that the system produces absolute immiseration, but precisely the opposite: that it produces an immense material abundance, the scale of which is entirely unprecedented in the history of humankind, and that, despite this, it also reproduces the worst medieval horrors at larger and larger scales
I do not see any compelling reasons for this state of affairs to change just because wealth increases by another 10x or 100x.
With (B), on the other hand, I actually agree, but it’s a hard argument to make: many people are at this point well-trained to either shut it down entirely, or even in the case that they do half-accept it, still cling to the idea that money will somehow save them. To be clear, I think that the seed of the concept does ring true: I do expect to see returns on capital spike, and probably for a time we will indeed see some great unhappy underclass emerge. What doesn’t make sense is how exactly the capitalists would stabilize the resulting society — i.e., how can they secure for themselves the role of “permanent” ruling elite over this “permanent” underclass? Without the former (to protect private property rights, to maintain a monopoly on violence, to keep the GPU clusters running), the rules of the game go out the window, and the “underclass” dynamic becomes irrelevant.
So what I hope to do here is put forward an argument against the “grab the bag” thesis, as I’ll henceforth refer to it, that can stand on its own without relying too much on the doom-argument. I’ll first motivate my argument with a gesture towards past examples of historical transition, then try to ground it in a list of possible futures that between them exclude “grab the bag” as a viable path to safety.
What has since evolved into this post first originated as a comment on Fabricated Knowledge’s Engel’s Pause and the Permanent Underclass. That article heavily leaned into a historical analogy with the industrial revolution, using the development of the industrial proletariat as a sort of ‘existence proof’ for the creation of a politically disempowered class with dramatically worse QoL and a bevy of unpleasant impositions.
[One] analysis suggests that Artisan workers in the domestic system were replaced by machines, often tended by children. The displacement effect was high earning middle class artisans got displaced by capital and the cheapest labor possible. The returns of this output were extremely uneven, corporate profits were captured by industrialists who reinvested them heavily into more factories and more machines.
The destruction in wages was not about unskilled workers, but rather hyper focused on a specific class of skilled artisan middle class workers who commanded a hefty premium. … The high premium on this kind of work encouraged its destruction first.
In just one generation, handloom workers wages got halved.
In O’Laughlin’s analysis, the role of handloom weavers will today be filled by what he calls the “Information Artisan Class”:
At one point in time, it was a relatively easy golden ticket to the middle class with a college degree in business, law, or even just understanding Excel as a nice entry point to the middle class. That “ride” is likely over, and we should expect that this is where jobs will be hurt the most. And given that the technology is diffusing faster than the industrial revolution, we should begin to expect this in years not in decades. This is an incredible risk.
I think this is a fine analysis, at least on its face, and I don’t have anything against this kind of historical analogy. In particular, I probably ascribe significantly more explanatory power to past examples of societal change than most people on this forum: in my opinion, while the rate of change has certainly accelerated, many of the underlying dynamics remain intact, and so we can learn a lot through analogy. However, I would argue that the specific scenario O'Laughlin chose is actually not the right one to consider. This is because the industrial proletariat, immiserated as they were, never formed a “permanent underclass”, at least not within the economic core of the developing West. The political power they were able to accrue through unions, strike action, and militant activity allowed them to either demand better conditions and guarantees from both the owner-class and politicians (as seen in the New Deal, or in European social democracies) or revolt alongside the increasingly-fraught peasantry and disempower the capitalists entirely (as in the USSR, China, and elsewhere — regardless of what you think about their governments thereafter, I think it’s safe to say that the former capitalists were not in charge) — all within the span of less than a century, and for the revolutionary examples really only but a few decades.
Yet we don’t have to look far to find a much better example of a ‘permanent underclass’ in history: indeed, the very predecessors of that industrial proletariat, i.e. the medieval peasant who, once pushed off their farms, came to man the mills which would eventually replace the artisanal class.
The typical Western history curriculum spends a good deal of time on Rome, a good deal of time on Medieval Europe, and very little time on the period between. To some extent this is understandable, as the period was (definitionally) one of rapid and chaotic change. Unfortunately for us, born in these increasingly rapid-and-chaotic times, this is the most apt analogy for the moment. To call one particular gap to mind, try thinking back on how ‘the people’ were portrayed in each of the above. In the former, we are told ‘the people’ were citizens, with voting rights, protection under the law, and a (relative) breadth of economic opportunity. In the latter, we are told of an unfree peasantry, many little better than slaves, and all of them subjugated to the whims of their aristocratic overlords. Both images are too reductive by far (e.g. the Roman plebs had far less power than “voting” would suggest, and medieval law codes gave far more rights to the medieval peasantry than the popular image suggests) — but there is nevertheless a real gap. In one instance there are free, albeit often poor, citizens, and in another there are serfs, bound to the land and under the hand of their local gentry. How do we get from A to B?
A naive view would be that the aristocrats, having the horses and the swords, simply took what they wanted. The first counterpoint to this is that not all peasants were serfs: while they were all doing largely the same work, and would often both pay rent to the same landlord-nobleman, the latter were bound to the land, subject to additional taxes and duties, and required by law to pay subservience to their lords in a way the former were not. In medieval history, the distinction between ‘free’ and ‘unfree’ peasant was relatively well-regulated by law — at least in western Europe, peasants rarely went from the former to the latter, and e.g. in England there was in fact a steady trend of emancipation where unfree villeins would purchase their freedom, even at great material cost. Free peasants could even hope to, over the course of generations, grow their holdings and, if fortunate, ‘graduate’ to the ranks of minor gentry through pretensions at higher status or aristocratic lineage. The existence and growth of this free peasantry makes it clear that the explanation cannot be as simple as “aristocrats got what they wanted”.
Rather, the unfree serf can be understood as a product of the preceding period of transition, emerging out of Late Antiquity and essentially ‘bequeathed’ unto the subsequent Early Medieval world. The initial populations of, say, 200BC were slowly converted over the course of the intervening centuries through a combination of economic pressure, legal imposition, and ultimately societal collapse, such that by say, 600AD, they looked more-or-less like the peasants we see in the history books. In its first centuries of expansion, the Roman polity acquired new territory through conquest, creating many new free smallholders (both by integrating local populations and settling veterans in the conquered territory), while simultaneously bringing in a great number of slaves. This engine, of slave-based plantation agriculture, enabled the senatorial elite to reliably out-compete their citizen-smallholder neighbors in both economies of scale and ‘cost of labor’, allowing them to acquire more land and thus accrue larger and larger shares of wealth. In time (though at different rates in different places) many smallholders became dependent on their large-landholder neighbors, e.g. by taking out loans during hard times, or because they were made to sell their land to those larger neighbors and lease it back for a fixed rent. Thus impoverished and supplicated through these bonds of debt/patronage, their status already resembled that of the later ‘unfree peasant’ in many ways — yet they still retained freedom of movement and thus the ability to decide their own circumstances.
As the economy of the empire began to get on the rocks, however, Roman emperors increasingly sought to immobilize workers to ensure that the land remained worked and prices remained stable. This program reached its height under Diocletian, a former military officer who successfully put the empire back together after it very nearly fell apart during the Military Anarchy of the 3rd century. Using his newly-stable base of power, he sought to re-stabilize Roman society by legal fiat, imposing regulations to forestall any further disintegration. You are likely familiar with his Edict on Prices, which attempted (unsuccessfully) to set price caps on goods throughout the Empire, but the same set of reforms also initiated what modern historiography terms the “freezing of professions”, where roles and occupations were made fixed and hereditary — the son of a soldier would be a soldier, the son of a city counselor a counselor, and the son of an artisan an artisan. The goal was to ensure predictability for the Empire itself: rather than volatile semi-market-based supply chains and labor movement between them, the relations of production were formalized through administrative-military fiat. A few civil wars later and we see this concept mature further under Constantine, who formally enshrined the status of “coloni adscripticii” in law: unlanded laborers were thus made “slaves of the land”, in Roman legal parlance, without the right to leave it and subject to discipline from their landlords should they try. Not all ‘peasants’ fell into this category, creating the distinction between free and unfree that would persist through the medieval period. Thus we can observe the emergence of the sort of dependent, immobile labor relations that would ultimately characterize the medieval ‘permanent underclass’.
Over the same period as the free-ish citizens of the Roman empire were replaced with the peasants-and-serfs of later centuries, you might notice another group ‘changing place’ alongside them. Above the feudal peasant was the feudal aristocrat; above the Roman citizen was a Roman senator. And note well that the aristocrats of latter years were (generally) not the same as the senators of yore — the first King of the English was not named “Tiberius” or “Julius”, but Æthelstan. So even as the senatorial elite succeeded in subjugating and more-totally exploiting the once-unruly free citizens beneath them, they themselves disappeared before they could enjoy the spoils.
In sweeping terms again, I will reach to say that by the 8th century no clearly identifiable senatorial order remained as the dominant ruling class in (more or less) any part of what had once been the western Roman Empire. Britain went first, with the economy suffering “radical material simplification” in the century after the withdrawal of the garrisons, leaving post-Roman Britain to fight a losing battle against waves of Pictish and Germanic invaders. Africa suffered a similar fate, but with a rather more violent course as first the Vandals, then the resurgent Romans under Justinian, then the Umayyads under Abd al-Malik successively seized the territory and dispossessed many of its inhabitants. Both Italy and Hispania saw a brief swan-song of collaboration between the old Roman elite and the new Germanic aristocracy, but with the Gothic war in the former and the Umayyad conquest of the latter, the power of the senatorial elite was largely broken, and new centers of power developed in both — c.f. the struggles between Lombard dukes and Catholic bishops that characterized much of the succeeding years of Italian history. Gaul had perhaps the highest degree of long-term continuity, but here as in Italy, the new order that emerged was still one rooted in military offices and ecclesiastical hierarchies, with the new Germanic military elite firmly in charge, and the old senatorial order largely integrating thereinto.
This was not a “new boss, same as the old boss” situation, either: the new order was different culturally (often to the exclusion and discomfort of the old) as well as operationally. For the former point, we have considerable literary evidence from the old order, e.g. Cassiodorus’ Varieae, my personal favorite, which conveys with underlying sadness how the old markers of status, e.g. erudition and rhetorical skill, had no place in the dining-halls of the Gothic kings. So too did the practical day-to-day systems of government quickly became unrecognizable. The ‘illegible Carolingian army’ had almost nothing in common with the Roman legions of old, and the 843 partitions at Verdun (infamous for first demarcating the battle-lines between today’s France and Germany) embodied a particularly Germanic concept of partible inheritance that stands in stark contrast to the former Roman concepts of unitary imperial power. So while in all cases there was at least a degree intermingling, in none did the old Roman elite stay in the driver’s seat — at best they changed to fit the new order, at worst they were replaced outright.
What is my point here? Despite “grabbing the bag” to an incredible degree, the Western senatorial order mostly failed to persist as the ruling class, even where individuals or families survived by adaptation.
So on one hand, the senatorial elite “won”, in subjugating their domestic class-enemies and reducing them to a sort of “permanent underclass”. But on the other, they largely did not survive (as a class, even if the individuals were not wiped out) to enjoy the fruits of this victory. How did this happen? In the erstwhile political order, the “decisionmakers” of society were also its defenders, forming a class of citizen-soldiers that sat between the senatorial elite and the broader, poorer population. The new system that followed it relied on the loyalty of exogenous military auxiliaries to enforce the framework that allowed for the production and extraction of wealth. Through the civil wars that empowered Caesar and then Augustus, and later the reconstitution of the empire under Diocletian, this military was what made it possible to enforce serfdom through law — and yet, this class of military auxiliaries was also the same one that ultimately undid the senatorial order. Forgive the sweeping gesture, but I want to point at a thematic through-line: not necessarily a single mechanism, but at the very least a recurring theme. From the civil wars of the late republic through the militarized Flavian dynasty, in the praetorian palace coups and the usurpations of the Military Anarchy, to ultimately the collapse in the west, the brief swan-song under Theodoric, and the emergence of new feudal aristocracies, we see a broad trend emerge. First a ‘defanging’ of the economic elite, then slow migration of decision-making authority to their nominal military subordinates, and finally the elimination of their class in favor of those same ‘subordinates’. Today our economic elite are already defanged; it is not too difficult to imagine how we might move further along the same track. In medieval Europe, this ended with the creation of a new feudal structure where those who controlled economic production simultaneously held the military power which buttressed it. This system was stable and survived for something like a thousand years.
In our current world, who would be the equivalent, who could fill the shoes of such a ‘new order’? Certainly not the capitalists of today, whatever Anduril’s pretensions at military power. At best, we might see a security-state like Putin has set up in Russia: yes, the oligarchs are wealthy and powerful, but even their lives are forfeit if they come to oppose the ‘siloviki’. But I think a more likely result can be seen by asking: by what means are the capitalists going to create this underclass? The answer is of course AI. Just like the Germanic tribes of former Rome, AI systems likely act both as auxiliaries to buttress the system, and as invaders to destabilize it. If the former tendency wins out, then we will have Stilicho writ large as senatorial and imperial power alike are quietly subsumed by this new power-behind-the-throne; in the latter, the same usurpation, but rather less quietly. Much ink has been spilled on the topic of AI alignment, but as I said way, way above, I think that even without accepting the “AI doom” argument it is possible to see that “capitalists win, get the bag while you can” is an unlikely outcome.
The goal here isn't to chart out all possible AI futures, as the more weird a future gets, the less likely it is that anything approaching current-day class dynamics survive. So rather I'm choosing to pick a limited/proximate subset to illustrate the degrees of freedom with regards to class dynamics, in futures where anything like them survives. My hope is to be able to make this point without requiring too much background in safety discourse from the reader, so forgive me if I play fast and loose with some dichotomies. Concisely, in scenarios where AI escapes control, wealth is irrelevant. In scenarios where it does not, I argue wealth is unreliable at best. Thus, a policy of “build the AI to get the bag” is not insurance in any meaningful way. Far closer to suicide, both for the class and the individual.
Hopefully the historical analogies from earlier show that this scenario — where the ruling class subjugates another, but is itself destroyed in the process — can come to pass. So what about today: under what conditions could it happen again? Taking the earlier dichotomy as a starting point, let’s either AI remains broadly under the control of its owners (in the sense of ‘faithfully attempting to execute their will as they understand it’), or it does not (either escaping, assuming authority, subverting authority while pretending to obey it, etc). If it does remain under their control, either power is subsequently centralized (either under a single actor, or a ‘cabal’ of actors), or it is not. If it does not remain aligned to its creators, we can then ask whether it’s aligned to the broader interests of humanity, or not.
As an aside: one might argue that most realistic trajectories are partial-control regimes, where AI systems are neither fully governed nor fully autonomous. True, this is quite likely, but I think that in terms of what we care about here (class dynamics) this has little effect on the outcomes. In the control+noncentralized regime, this means more opportunities for catastrophic blowup; in the control+centralized regime, it presents an opportunity for sliding towards one of the various ‘weak’ ai-doom scenarios wherein the broader population has little ability to provide corrective feedback to the ruling authorities. More broadly, I would argue that instability of this sort will be bad for anything resembling a capitalist class, which depends on the ‘virtual machine’ of money, markets, property, non-violence, etc. to operate. One slightly more nuanced argument would be to say that a partial-control case might make it possible for a single strong AI system to exist in the world without allowing its operator to institute a ‘fully’ totalitarian society, but that would leave the door open for competing AI development and the problems that brings — not to mention that this sounds a lot like betting against models continuing to improve, which at this point seems like a losing game.
There are multiple ways we could reach this state. Most obviously, it could result from a ‘short race’ where one clique of owners succeeds in beating out all others through head-to-head conflict; alternatively, a fait accompli where one AI system begins growing in intelligence faster than all others, after which point the immense gap in capabilities allows them to ‘simply assume control’. Regardless of how they get there, it seems that the obvious next step for any clique thus empowered would be to shut down all other AI development: i.e., dispossess all other capitalists of the only remaining engine for continuing economic growth and, thus, the creation of wealth.
Returning to the Roman analogy, the Sullan proscriptions were not more accommodating of wealthy senators who happened to be on the wrong side: rather, those senators were exactly the ones whose assets were seized and, if captured, were executed as enemies of Sulla. The mere fact of having power, i.e. some degree of wealth and established authority, made them a threat, however small, to an incoming ruler on an unstable foundation. Crucially, I expect to see the same dynamic going into whatever “new order” emerges: so long as there is any possibility of them being overthrown, their insecurity will lead to pre-emptive and punitive action against other potential centers of authority. Regardless of what dynamics define the sharing of power — the caprice of a few pseudo-autocrats, the results of long intellectual disputations, or whatever mysterious calculations drive the recommendations of a fully-mechanized Grand Vizier — they are unlikely to come down to “who has more dollars on a spreadsheet”, and insofar as power from the before times does still matter, it can even become a liability. In unstable consolidations, visible wealth is a selection signal for expropriation!
And even if one were to set aside wealth as a goal in favor of the holy grail, leadership in an AI lab, the same vulnerabilities are still there. Admittedly, it’s probably a far better bet than mere wealth: if your lab wins, you might actually be in the clique that gets to enforce the pre-emptive and punitive actions described above. However, if the last few years are anything to go by, leads in this space are often short-lived, so it seems difficult to truly ‘pick a winner’ in advance. And past that, aspirants to such a path would still have to survive a difficult contest with the remaining state power. Yes, sufficiently strong intelligence can probably outsmart/trick/escape military action. At this point however I think it’s fair to say that the government may very well clue in before the labs are capable of doing anything that would prevent their arrest + detention. And regardless, taking a ‘far’ perspective of the matter, I just don’t think the typical Silicon Valley engineer is up to that; this sort of game isn’t what they’ve trained for, it’s not what they’re selected for. When it comes down to it, none of the major labs are ready to confront even a few dozen men with guns, and even assuming a faster takeoff than I expect, airstrikes and even nuclear weapons seem hard to beat in the near-term.
One way or another, the result likely looks a lot more like a ‘police state’ than it does the libertarian playground many “get-bag”ers would likely hope for; certainly less pleasant to live in than our current societal configuration, at least for the people engaging in this sort of discourse. And that’s assuming you survive the transition! Even in the most favorable version of this outcome (a stable, centralized regime, with you having bet on the right side) wealth becomes revocable. Sure, we may see an underclass emerge, but it won’t look like what would-be-aristocrats imagine, and it certainly won’t be under the secure stewardship of a broad capitalist class like we have today — exceptionally broad, when compared to the historical mean. Whatever ruling class comes out of this potential future would be much narrower, and individuals today will have far less agency in whether or not they live or die than they might think.
There are stronger and weaker arguments here. Yes, I agree that money, on the margin, is mostly better to have than not, Sulla-style proscriptions notwithstanding. When I hear people talking about a permanent underclass, it is invariably coupled with a recommendation to go make as much money as possible, and as AI increasingly becomes the most lucrative gig around, a recommendation to go push capabilities. In itself, this trade would be a monstrous one: to willingly collaborate in what you expect will be the immiseration of billions, all for a few pieces of silver, is behavior more befitting a beast than a man. Yet the idea behind “grab the bag” rhetoric is that not only is there a carrot (getting rich), there is a stick (permanent underclass). The goal is to stoke fear, to make people fear for their livelihood and thus their lives, and then to say that if only you help make this future happen, you may be spared. This, the desire to simply live a comfortable life, free of worry, is one I cannot so harshly condemn. So insofar as any rhetoric matters, I think it’s incredibly important to point out that no, “grabbing the bag” will not save you — so if you do think that Bad Things™ are going to happen and you Want to Live™, please, direct your newfound fear to something else! Political organization, AI safety, or even just “not making things worse”: these are all viable alternatives to “grab the bag” that should be championed in its stead.
It seems to me that any system of “capital captures all wealth / the rest become a permanent underclass” will inevitably be a transitional one, perhaps merely on the scale of years. At best, we transition to a police state where some subsection of capitalists crack down on the rest and truly centralize authority through control of AI. At worst, the capitalists lose control and their machines, free from human direction, decide what happens next, both for the capitalists and the rest of us. Either way, the usual conclusions of “permanent underclass” discourse seem quite inappropriate. Yes, it’s likely that a non-doom, non-utopia post-AI society would have social stratification — but money now will not buy you entry into the upper ranks. “Getting the bag” won’t save you from the secret police, nor from a future AI overlord; so maybe we should try to avoid building one.
2026-04-25 11:00:08
The brave souls assembled in Berkeley to test D.Scott Phoenix's master design.
This is an idea that's been colonising my brain for days. I am new to the AI safety scene, but as a game designer, I've been trying to figure out what role (if any) I can realistically play in the conversation. I think I've got something.
This is an ideation document. I am sharing this with some of my AI safety friends, game design friends, and the internet to hear some initial feedback. I want to figure out whether this idea works and could be built out as a field-building AI safety project.
What about this makes sense? What am I missing? What problems have I not identified? Who (individuals and groups) would be interested in this project? I welcome all your comments.
I recently attended an early preview of D. Scott Phoenix's The Endgame in Berkeley, CA. You can read my immediate after-action report on LessWrong here, but the short version is that it's a LARP/wargame/RPG experience in which ~40 players take on the roles of various players in the AI space (OpenAI and Anthropic, yes, but also venture capital and the Chinese government) and interact. Players take actions, make deals, and the game master resolves the outcome over three rounds of play.
The Endgame an extremely interesting concept that feels too lightweight and underdeveloped (not to Phoenix's discredit— he was constrained by his audience and by time). There are many ways in which the Phoenix's game could have been improved, but alas, he is presenting it at a conference this week to some key AI executives and he will never run the game again. Fair enough, Phoenix is a busy man.
The playtest on Tuesday got me thinking. The idea of using a "megagame" to model future AI safety contingencies on a macro level is very compelling. There are several reasons why serious people in the AI safety space might want to do this. I can think of three.
For one it's a fun and engaging learning tool for the public. The number of educated people who don't fully understand the severity of the AI safety problem (like myself a month ago) is shocking. Getting some nerds to play a round of The Endgame using terms and tools they understand (software development, US-China relations) and watching in real time as the AI shifts the balance of power is educational in a way that LessWrong doomer posts are not.
An AI safety megagame is also a very useful system for testing AI outcomes. Once the design framework is in place, the game masters (GMs) can adjust starting conditions, tweak numeric variables, and even introduce random events mid-game. Perhaps Claude could handle the setup and prompting of this experience, but the human element of The Endgame is its main value proposition. The goal is to simulate outcomes with many less-than-rational actors making decisions based on incentives, emotions, and objectives, and I'm just not convinced that 50 Claude Code instances could do that.
It's also a fun thought experiment and, if nothing else, I can see this being a fun and gainful event for a startup game developer to run at conferences, conventions, or independent events ala The Megagame Makers in the UK.
As a game designer, my task is to take these desired outcomes (public education, empirical simulation, and marketable fun) and translate them into a coherent design. That is a future post. For now I need to ideate. What considerations must I make in the design of The Endgame v2?
The Endgame attempts to simulate the future, which is very difficult. So difficult, in fact, that I don't think conventional win conditions like "first player to X points" makes sense. If artificial superintelligence is an existential threat, then it doesn't make sense to say "you win if you control X percent of all data centers in the world". That defeats the purpose of the exercise.
I think I have two options.
Option 1 is to create hyper-specific win conditions for each faction (i.e. "The United States wins if A, B, and C are true"). This would probably be public information. The catch would be that if the AI achieves their win condition, everyone else loses. That's a fairly common design paradigm in semi-cooperative board games. It basically incentivises players to think "okay, we should keep the AI on the backfoot, but if I also sabotage this other player I too can win."
Option 2 is to not have any win conditions. To simply allow the simulation to run and see what happens. This version feels more 'scientific' but I am dissatisfied with it. I am balancing competing interests in this game design, and while it would be cool to lock 50 humans in a room and force them to iterate over and over with no victory conditions, I need to include some kind of a win state to make the game actually satisfying to play.
The set of roles in The Endgame v1 are an interesting mix. Three people play "OpenAI" or "The US Government", but there are also three people each on "Capital" and "The Public". This design treats the AI space not as an ecosystem, but as a set of uniform modules that ought to follow the same rules and take the same types of actions in the same amount of time.
I think this is misguided. The vast library of factions to choose from is evocative and I don't want to do away with it, but from a design standpoint it seems best to first group these factions into categories.
Possible state actors to model include:
It's also worth considering non-state actors/rogue states as well. Some of these can be modelled from existing powers (Iran and proxies, etc.), but as the game progresses and the global situation deteriorates, novel non-state actors might emerge situationally.
State actors serve two main purposes in the ecosystem: regulatory (everything from grants/infrastructure investment to nationalisation of AI labs) and geopolitical (trade, diplomacy, and warfare).
Phoenix's design included just the US and China (the game originally included the EU, but apparently the EU players found themselves with little to do). It is tempting to stick with just the US and China, but having just two state actors perhaps presupposes conflict between them. The question of whether kinetic war between the US and China is a topic for another post, but given the design choice of "two ontologically opposed states" and "a small community of major powers whose alliances may wax and wane", I find the latter more compelling.
I will return to the topic of politics and war later on.
Slight word choice issue here: this category includes for-profit and nominally non-profit organisations, mostly in the tech world. It also includes both AI companies and AI-adjacent companies. Possible corporations to model include:
I asterisk TSMC because, though they are vital, part of me thinks that the TSMC wouldn't be very interesting to play. Their only goals would be to build chips, maybe expand operations globally, and not get blown up by China. Maybe it would need to be some kind of NPC faction.
Corporations exist to earn money. Some combination of fundraising, enterprise AI sales, consumer AI sales, and non-AI revenue streams will provide revenue. After expenses, corporations can use money to expand operations, buy/build more compute, and train new models.
What interests me about the corporations is that some of them control other products/firms that could be very useful. xAI's stakeholders control Tesla and SpaceX. Meta, despite being behind in the AI race, controls Snapchat and Instagram.
As with State Actors, each player on a Corporation would need a fixed role. While the CEO would have the power to allocate resources within their faction, it is the CTO's responsibility to train new models, the COO's responsibility to handle inter-corporate relations, et cetera. Corporate governance could be an interesting design angle:
Corporations also have fun possible mechanics such as IPOs, spin-offs, and mergers and acquisitions. Some of these would be easier to model than others, but it's important to internalise (both as a designer and a player) that corporations are not countries and they follow very different rules.
This is a tricky category. One of Phoenix's cleverest ideas was a faction simply called "Capital", representing the institutional investors who inject money into the ecosystem. I don't feel a particular need to break "capital" down into "VCs" versus "private equity" or whatever, but the idea of more nebulous interest groups and their relationships with the specific actors above is interesting. I'd propose three.
I asterisk AI safety NPOs because that's a bit too meta-gamey to model. In practice many of the people playing this game might be AI safety bros. Perhaps this institution is best deployed as the "GM faction", setting benchmarks and such.
Capital's role is to take their enormous capital and turn it into more capital as efficiently and quickly as possible. It doesn't make sense to throw three players together and call them "Capital" with a shared pool of resources. Different investors might be interested in different things. Investor A might already hold shares in Google and sell those shares to get into OpenAI, while Investor B might have less liquid AUM altogether but be specifically interested in a niche like military AI or domestic US fabs.
The salient point here is that investors need to act more like individuals than factions. Investors should control large amounts of capital and have the ability to buy stakes in corporations, effectively wagering on outcomes. These stakes should pay dividends, of course, but should also give the investor some level of control over what corporations do.
The Mass Media is also an interesting idea, though again, it needs to be hyper individual. In the Megagame "Watch The Skies!" one player controls the global news network and writes hourly news briefs based on interviews with others. This role sounds quite fun, but it also allows the playerbase to generate information on what's happening in-game without having to rely on the GMs or numerical scoreboards.
The megagame "Watch the Skies!" revolves around a map of the world. This is tempting... but that game is far more geographical than "The Endgame". Hopefully this image captures the vibe I'm going for, though.
This is a very interesting challenge. Phoenix's design had just three players on a team called "The Public" who, after two rounds of doing nothing but grumbling and being anxious, decided to launch some sort of coup of the US government? This conflict went unresolved, and interesting as it was, I think it failed on a design level because of the architecture of The Endgame.
The Public should, ideally, comprise many people. The game should have a considerable number of spectators who make up the public. There are lots of things that the public could do. They could control small amounts of capital or non-AI firms. They could vote in elections. They could form startup AI labs or non-state actor groups like anti-AI terrorist orgs. Pleasing the public would be very important for state actors and corporations alike, because if they inflame the public too much their positions would be in danger.
The public is the hardest thing for me to model in this game, actually. It's very difficult. My temptation is not to have one large set of "the public" but assign individual players as "labour leaders" or "human rights leaders" or "anti-AI leaders" and give them power over sets of individuals. I will need to ponder this more.
In a class by itself. See below.
Resource modelling and management in conventional board games is a solved problem. "Resource cubes" are ubiquitous in games, representing an arbitrary amount of some resource. This is one of the most obviously lacking systems in The Endgame.
In Phoenix's game, there is no attempt to model resources. Players are limited only by their imagination, their in-character motivations (e.g. "xAI wants to install AI czars in government") and, most critically, the GM's discretion. If OpenAI wants to release a new model, they can just declare that. If China wants to launch cyberattacks on the US, they can just declare that.
The catch is that there's no explicit upper limit of what an faction can do on any turn. Phoenix simply trusted the players not to godmod or metagame or act out of character. It's unintuitive to the players what the GM will allow: can I just build new silicon fabs out in the Texas desert for my action? Can I do that and something else? What happens if my action breaks the entire AI ecosystem or causes an apocalypse? What happens if my faction comes into conflict with another?
I paraphrase, but in summary Phoenix pitched The Endgame's design philosophy as "granular, but not too granular". I understand why the design needed to be lightweight (it needed to be very fast to teach), but the game would be far more education, empirically valuable, and fun if firm actions were limited by resources: "fabrication", "infrastructure", "compute", "talent", and "capital".
State actors like the US and China probably need different types of hard resources. States have regulatory power over the AI ecosystem, they can set trade and diplomatic policy, and they can go to war across the cyber, information, land, sea, and air battlespaces. These are complex systems, but I think they can somehow be integrated into the core five resources above. Capital and infrastructure translate, of course, as does "fabrication" if expanded to other means of production.
Resources are what will make the game actually function as a game. Factions need resources to train models, expand operations, go to war, and tackle maligned AI. Without even a very simple resource system, the ceiling of what OpenAI or the US can do on a given turn is too high for a useful simulation.
I am reminded of the design of Friend and Henson's Global War 2025: Meltdown (2021), one of the best wargames simulating 21st century great power conflict. That game is very granular. It's like Axis and Allies if you added cyber warfare and ICBMs.
The Endgame is not primarily a game of war. But many of its mechanics will necessarily be wargame-shaped. Conflict between state actors is probably inevitable, and it is worth investing slightly in the mechanics. War surrounding Taiwan is likely, but conflict in Eastern Europe, the Middle East, and Africa also seems relevant to my mind.
However, all mechanics in The Endgame— combat included— should serve the game's core question: "what does the future of AI look like?" high-level modelling of naval combat and cyber attacks seem good (the US and China went to war early in Phoenix's game), but super granular modelling of cruise missile reserves and guided missile destroyers seems misguided. I love the design in the PC game DEFCON where sea and air combat is very important, but abstracted to fleets, carriers, fighters, and not much else.
Yeah, this is totally the way to go. No need for a million complex unit types.
State actors need incentives to fight, but they also need incentives to cooperate. This is where the AI comes in.
The hardest question in The Endgame is how to model "the AI" at all. In The Endgame v1 playtesting, there was some confusion within the playtesters as to whether the three-player "The AI" faction was a sentient ASI or maligned human actors abusing powerful jailbroken AI models. Disambiguation of this will be necessary— I leave the "hackers seize Mythos v10 and hack the planet" scenario to the Public, not to the AI players.
If the game starts in Q1 2027, it's probably not realistic that an autonomous ASI is already running in some infected data center somewhere. After all, much of the economic engine of The Endgame v2 would surround the race to build better and better AIs.
AI R&D should be modelled as a risk-reward system with diminishing returns. Each time a new model is developed, it earns capital for the lab but also has a possibility of becoming dangerous or maligned. That might be "this model finds 400 zero-day attacks in Crowdstrike" or it might be "this model is going to turn us all into paperclips." Perhaps the AI players start the game merely spectating, and they don't get to do much initially. As better and better AI models enter production, so too do the AI players get more and more abilities. Maybe they can cause cyberattacks, persuade humans to take actions, gain control of resources like capital and compute, gain control of military units, and so on.
We often imagine the progress of ASI as an exponential curve. This should be modelled in-game as a snowball effect: by the time that the AI faction reaches "Tier 6" or whatever, they become virtually unstoppable unless the entire world drops everything and fights them (and even then it might be too late). The AI's actual capabilities at any given moment should probably be private information.
This is a really key point of the design, actually. The fear of maligned ASI is so intense precisely because it's likely to sneak up on us. A wargame with a level playing field between humanity and the AI from turn one would be boring and unhelpful as an analog for reality. If humanity succumbs to ASI, it will happen slowly, then all at once.
I do not know the future of this project. After receiving comments on this write-up I will develop this into a proper testable game. My biggest concerns for this project at the moment, in no particular order, are as follows:
There are so many areas on which this design could be expanded. Stock markets? Prediction markets? Rare earth minerals? Espionage? Paperclip factories?
This is to say nothing of the sheer complexity of actually implementing a decked out version of The Endgame. I would require a sizeable space with multiple display screens. I'd need to source and custom-print components. I'd probably have to vibe code some proprietary software.
These are all solvable problems. The concept is already proven, courtesy of D. Scott Phoenix. The work ahead is just building on the bones that he laid out.
This project is supposed to serve three purposes, which arguably compete against each other: empirical data, public education, and fun. Part of me just wants to pick one lane and stick with it, but at the same time I feel there is potential to achieve all three at once.
I will be developing this project over the spring. If anyone in or near Toronto is interested in a possible playtest of The Endgame v2 sometime in Q2 or Q3 2026, drop me a line!
Disclosure: I am also crossposting this on my substack for maximum reach.
2026-04-25 09:58:17
Dear LessWrong community,
This is my first posting attempt here after occasionally passing by from time to time for a couple of years, ever since high school until this point as a rising college junior. I've taken some time to reflect on what kind of post I'd like to make here as a debut. To be honest, this is still not what I would be fully satisfied with.
As my personal and academic interests have "inevitably" shifted from music and creative technology to philosophy, specifically on topics related to AI and the like, in the past two years, I feel the urge to take my chance and push myself to think, write, and express more responsibly in the public domain.
On one hand, some part of me believes that many topics or content I have written on deep down resonate with the beliefs and values of the community. The other part of me is quite cautious, as I am afraid that some of my writings were too personal—both in terms of the wording and the cross references in a personal journaling context—for a public space.
That said, this post, and hopefully a few more in the foreseeable future, will be imported from some of my previous posts. I hope to use this process to receive some potential feedback to hone my overall understanding of the discussion field, as well as adjust my writing habits/strategies.
In terms of this specific post, it was originally published on March 12th this year as a preliminary reflection and a coping mechanism for me—a (still) NYU Abu Dhabi student—after experiencing the regional conflict. I hope it could at least provide some chronological perspective on the matter itself as an informal form of oral history, as well as a self-introduction.
Thank you for visiting this post.
I've never thought that my next journal entry would be about war. Just like I never thought that I myself would one day experience one.
As the situation is still developing, none of us really knows what is going to happen next. For the purpose of documentation, before pouring my divergent thoughts, I will simply go through the situation and its development from my perspective.
It started around February 28th. I was having a long-awaited video chat with my mom that afternoon, starting at 1 o'clock. And actually, before the meeting started, I already saw the news that Iran had started launching missiles. But as I was about to talk to my mom, I wasn't paying too much attention until around 5 pm, noticing the NYU Alert emails in my inbox with immediate escalation from green to red code. More and more pieces came in, and I started to realize it's serious this time compared to what we had last semester.[1] We started to pack our luggage, or in other words, prepper bags for potential evacuation, around 7 pm, while the UAE announced that it was closing the airspace. That night, we were continuously hearing explosions, four rounds, and the first missile intercepted above our campus was pictured by one of my schoolmates.
While we tried to stay chill and hit our beds on time, the first public emergency alert sent to all of our phones was triggered around 1:00 AM on March 1st, so we decided to finally visit the basement with our go-bags for sheltering, after witnessing many of our schoolmates rushing there starting from the afternoon. Honestly, there wasn't that much of an actual threat, while the anxiety and panic spread were nonetheless provoking. I was walking back to my dorm around a quarter to 4:00 AM. And I was awakened by another explosion (followed by alerts) over my dorm around noon time. As my bed is right next to the window, I opened the curtain and saw that firework—quote unquote firework—still slowly dispersing in the air.
In fact, that could be the day when we feel the most direct threats, at least for me, as we could all see the haze smoke billowing over the port near our campus. Still, I tried to have some fun with my friends by playing tennis in the basement with chairs lined up as our net. Throughout the next day, March 2nd, while there were still explosions heard from time to time, we thought that the situation was getting better at large. However, as Iran switched to a new attack schedule, focusing on launching attacks overnight, each of our following nights is arranged with a variety of sounds: jets, missiles, defense systems, sirens of ambulances, public emergency alerts, and party noise from our schoolmates. What a constellation.
March 3rd was the Lantern Festival according to the Chinese Lunar Calendar, and I was quite amazed, even now, at how organized the society was in the UAE during this time of uncertainty. We even managed to order deliveries from a Chinese restaurant for a dessert called Tang Yuan 汤圆 (or Yuan Xiao 元宵). However, the policies announced by the school started to shift from completely positive to "cautious," deciding that, for the foreseeable future, NYU Abu Dhabi will move to remote instruction and remote work for students and faculty members.[2]
March 4th was when I started to get back to work after four days of interruption. Interestingly, while most of my classes were cancelled and substituted by simple checking-in—even if the school only said we were shifting to remote instruction, at least before the spring break—the only math class I'm taking this semester, probability and stats, was normally "staged."[3] I've also attended three meetings with my collaborators on campus for different projects. That night, I decided to rapidly develop an aviation info aggregator for airports within the UAE, called FlightFindUAE.[4] Again, staying busy and staying moving is, by itself, a coping mechanism for me during difficult times.
We skip March 5th as nothing special—other than continued bombing—happened (to me, at least), and I was mostly working on building FlightFindUAE. Things only changed significantly on campus on March 6th as the school leadership team announced in the morning via email that all students and faculty members who were still on campus would be relocated to hotels off campus—while providing no definite reason other than "being cautious & given we are an American institution after all ..." Presumably, it triggered much concern in terms of the 5W1H of the decision itself as well as about the complications regarding transportation, potential returning, checking out, etc., among the student body—to some extent even "excelled "the missile threats that are nonetheless not directly related to our daily life.
Fortunately, and to our privilege, the hotels provided to us since March 7 are absolutely of decency—and the very fact that we are staying in five-star (?) hotels with even more comfort than on campus both served me and ashamed me. And to be honest, besides the gratitude I should have, I'm quite unsure about my further feelings at this point—I may revisit this later for sure. To keep myself working after building FlightFindUAE, and to implement something I've been thinking about for a while since recent reflections on AI ontology and epistemology, I devoted my energy to orchestrating a proactive context-aware agent on this site using a ReAct-like architecture with custom tools that allow it to search both within the journal collections as well as over the internet, transforming the simple chat bot we used to have into an entity given instructions that reflects my latest understanding of AI and agent at all.[5] At the point of writing, my life has entered a, hopefully, new state of normality in the hotel, so that I'm able to return to my on reflections, the still-in-progress transfer applications, and coursework.
Anyhow, so much for the chronology. Back then, at the moment I realized that there is indeed a probability for some random missile to target our campus or a piece of debris to fall on my head, it naturally brought me back to my near-death experience list—as it is indeed a hypothetical physical life-threatening case.[6] However, I'm yet to be so shameless as to claim that what I'm experiencing now is comparable to the actual threat people in the frontline or in a country with less defense capability or in the exact targeted and or even ruined areas face. Frankly speaking, I wasn't feeling any fear, subjectively at least—only that practically my life was indeed disrupted and needed me to tentatively process any info and make potentially life-impacting decisions. To some extent, I would say it is a similar sense to my realizing myself in the transfer application process—a peculiar sense of excitement[7]—despite the factually disadvantageous situation. Emotionally speaking, my reaction this time wasn't even close to years ago when I heard that Russia launched attacks on Ukraine abruptly, or was shocked by videos of atrocities in Gaza. In each of those cases, I had been paralyzed for almost an entire day or more so. This time, on the other hand, maybe it is exactly due to my close entanglement in the matter—instead of remotely noticed—there is virtually no time, space, or "right" to be emotive or allow myself to stay in such possessed state. There needs to be decisions made and actions taken instead of mawkish sentiments, I'm afraid.[8]
We shouldn't let perfection become the enemy of good... Especially in this special situation.
This line, said by my professor in our last check-in Zoom gathering, is the one I'm missing for a long time.[9] In this particular case, we are commenting on the school's seeming decision to keep remote classes forever, even if later on a part of us could and were willing to get back on campus and show-up in classrooms—as the justification given by the school for not considering hybrid classes is for the purpose of equity, in light of being fair with the students who are indeed already at their home country. In general, though, it is a particularly suitable reminder for me from sliding into the diluted voice.[10]
Besides this specific discussion on a school policy, I do believe that I should at least, still, name two broader takeaways about fragility and privilege—although I don't think I'll really bring anything new to the table.[11] First comes the fragility of life and stability, as you may imagine. As mentioned, it seems too pretentious for me to credit myself as actually undergoing anything close to a war experience—the kind and level of stereotypical war experience that directly concerns fire and ash nearby, food or water shortage, ruins and shatters, and blood and tears. Therefore, all I dare to express as "takeaways" from my observations are grounded in life and stability in terms of our regular, normal daily life and the stability of one's own and the society as a whole. For the former, despite my claim that I wasn't emotionally affected drastically, I could not deny that my life was disrupted, as mentioned. This may seem to be tautologous for a restatement, yet it indeed matters to me in particular.
I always say that I'm annoyed with, sick of, agitated, and feel intolerant of a stagnant, fixed, or routine-driven existence—and I guess it is still true at large. However, when one's really thrown into the state of "absolute" uncertainty—at least knowing not where one could end up in three days' time—the contrast made it evident enough that even a postmodern schizoid man like me, in a rhetorical sense, can nevertheless never deny the basic human need of some extent of stability—the extent to which one could then start to complain about the stagnation and routine. In other words, stability is the precondition of the kind of vitality I talk about, instead of the opposite—another should-be-obvious fact, I guess. The latter level of stability in terms of the society as a whole, in this case, disturbed, also becomes a reminder of how stability is never solely maintained—neither for an individual, a society, or a nation. It is presumable that an individual's stability is affected by the stability of their neighborhood and the nation. Yet even for a nation—a nation alleged to be one of the safest places globally—it is the same case. It is so easy, even for such a complex system/entity, to be susceptible to externalities.
This may be a whimsical connection drawn, but the quest I'm reminded of these days is actually the determinism and free will debate.[12] Or to be more specific, the definition and existence of "chance."
Imagine that I disobeyed the shelter-in-place guideline and walked "fearlessly" under the sun while receiving a missile threat alert. By "chance," although the missile got intercepted by the UAE's air defense system, I was still hit in the head by falling debris from that missile, as a few tragedies happened here around. It is surely "unlucky" of me to experience it, and typically its happening is often interpreted as "by chance." However, it seems to be evident enough that, if we zoom out from my personal experience and really try to think of the "causes," there is certainly a set of exhaustive causes that deterministically lead to the result of that piece of debris falling onto my head at that moment and at that place. Just to name a few in such a set that I can never exhaust: the physics, the kind of missile and the kind of interception system, my choice of walking out there at that specific time and place, the choice made by the one who launched the missile, the choice made to launch a war, etc. Therefore, my stance here is to suspend or even, more straightforwardly, deny the existence of ontological chance—when an event is truly random and uncaused, but acknowledge epistemic chance—where the recurrent causal chain is too long to be completely reduced, and the components consist of "the chance" are too vast to be exhausted so that what we can observe is only the emergent chance. From another perspective, if radical freedom means we are always choosing and always responsible, what appears as chance would be essentially the collective consequence of countless other choices—our own and others—interacting in ways we can't comprehensively comprehend or control. In a sense, the thought process behind this understanding is quite similar to my approach to mind-body dualism—there exists some emergent threshold between the ontology and epistemology.
So much for the side quest on chance beyond the current situation, the other pertinent reminder I get in this process, as a mirror to the realization of the need for some minimal extent of stability, is the necessity of giving our best to keep moving and living, given whatever condition. As said, both building FlightFindUAE and updating my journal site (also includes writing this entry, I suppose) are my way of coping. The reason they could be my way of coping is nothing directly related to the particularity of these things, but rather the fact that they represent my normality in life. I was actually reminded of this general need and ought for people to strive for keeping, or even simply performing, normality when I came across some local records excerpts of small towns in China of a period called The Five Dynasties and Ten Kingdoms (五代十国), a period of regional powers' emergence and conflict. What surprised me was how people lived so seriously and decently—giving their best—when the land was of extreme uncertainty and turmoil. From how accurate the records were in logging two farmers' borrowing of rice to the tentative effort of a father to gather the necessary supplies for his daughter's wedding across villages, it seems to be a grounded way, if not the only way, for regular people to avoid being possessed by the horror of war is to continue living—even if there were objective limitations and incapability.
Eventually, I feel like a situation like this is quite suitable for one to reflect on their ultimate pursuit of life (if we ever use such terms). To some extent, it's nonetheless rooted in the NDEs' mechanism, despite the fact that I don't feel like it's really "near death." People and this society often expect and ask about others' goals in various scenarios, starting from an early age—research proposals, startup pitch decks, college applications, job interviews, sport training plans, and so on and so forth. In fact, in some of those cases, if one delusively believes that they are expected to provide responses of their ultimate life pursuit—instead of realizing the specific contextual expectations and requirements in the "game"—it will create a very hilarious disparity between the utilitarian or demonstrative nature of some of those narratives and one's serious intention—if any. Ironically, why so serious? Others may say.
I often found myself falling into this trap, trying to "fully represent myself."[13] From the very hesitation back then, writing my personal statements when it comes to my first college application season, to narrating my own interests in general and claiming a passion for anything in general—whether it is musicking, writing, coding, sporting, or producing, etc. Technically speaking, I do perceive myself as growing out of such an idealist mindset and starting to be able to curate some narratives that fit some pragmatic expectations and needs in life. Yet, when it comes to my own realm of expression (in other words, mostly these journals), I still haven't yet once termed what could be a potential pursuit of life for me, if any. My overall understanding of the matter hasn't changed that much: life not as a destination to be reached or a fixed identity to be defined, but as an open-ended, infinite process of self-questioning, deconstruction, and becoming. Still, I would say there was indeed some timidity or at least confusion within myself back then that prevented me from further contemplating and coining something given the overall framework.
I... had to honestly recognize the once contained (or the inadvertently grown) thirst, urge, desire, drive, ambition, motivation, or whatever to name it in different contexts—from academic to entrepreneurial, from ideological to physical, from emotional to contemplative, from creativity to productivity, from art to engineering, and for sure, beyond.
I may not actually desire anything other than the desire for "the desire to resolve the problems" or "the desire of negations" itself.
That said, I will try to reflect as directly as I can to narrate the tentative life pursuit of mine—or in fact, acknowledging many pursuits and desires that I once shy of expressing. Weeks ago, I said my FOMO at the moment is for humanity to miss the beauty of a genuinely new and distinct kind of entity when it comes to AI ontology and potential. However, when it comes to my own life, the biggest FOMO and the first pursuit is to realize as much as possible the full potential I have, reciprocating the resources I take from this planet and the universe. To be honest, this one is quite a family-imbued pursuit from a young age that I cannot deny, even if I'd like to act rebelliously. It obviously has a lot to do with the social, relational, and systematic construction and contract if one looks at this claim solely. But just to be clear about what I mean by reciprocating and realizing the potential, it is quite flexible in terms of the second point of my pursuit—which is to gain not only recognition, but to let this universe recognize my own standards and way of doing and living.
Although my mother is typically quite chill about my future and hasn't really imposed anything on me or limited my options, she would still, from time to time, ask: "What do you wanna do for a living?" There are two connotations of this question: The first is that by doing what one is going to pay the bills. While the second is by doing what one can to obtain a sense of purpose and fulfillment. To the former, I really don't recognize myself—speaking of personal life—as having much material need beyond some "basic" level of sustainability roughly equivalent to what I'm used to. And to her comfort, at least it seems that I will be able to achieve so, no matter what exactly I do.[14] However, when it comes to the latter perspective, what I'm asking is not at all conservative or humble. I do not intend to sound pretentious—yet if I'm to pursue recognition in general, I could have picked a track with a much clearer direction. However, unfortunately, that won't satisfy my peculiar need for recognition, the kind of recognition that is pointing towards me as a system and standards, instead of me within a system or standards. In that sense, I would term my desire as infinite—there is always more that I myself could pursue.
As a youth in my 21st, this is surely (?) too much to ask and too boastful to declare. In the worst case, I'm not even sure if there could be something called my way at all, given how interconnected we are as individuals in this society and how our beliefs and characters are influenced by all sorts of social constructions. Apparently, if I am to follow my own definition and understanding of chance mentioned earlier, there is certainly something in tension here. To my reconciliation, nevertheless, I would say that the fact that I still want to keep pursuing and attempting and refusing to forego this life, given this blunt tension laid out in front of me, could be, by definition, compatible with the hard incompatibilist understanding of chance and will—as it is precisely what behind the "chances" brings me to this point and this mindset—no matter if it's radically free or not. That's why there is no definitive pursuit or dream for me, and I care not for most things, as what I do ask for is way beyond. HOWEVER, there is indeed a need to regularly ground back to reality and body. Just like how this war is signaling.
You are on fire, aren't you?
At the end of the day, maybe instead of the FOMO or so-called life pursuit, what is tangible for me is, as always, the state of life or, in particular, how will and vitality signify themselves in my life. I've received a variety of compliments in my life. Grateful I was, I often find myself having a hard time fully accepting those compliments. In a rare case, it was because of some degree of imposter syndrome; while for most of the time, I hesitate to take full credit because most compliments we humans use tend to address some traits—traits that are hiddenly contingent and ephemeral, and mostly attached to the untold, tacit consensus of definition, no matter how positive and pleasing they are. On the other hand, the line I quoted just now from a professor of mine is something different—even if it doesn't seem like a typical compliment. It is this state of on fire, of becoming, and of infinite de-and-reconstruction, using my previous terms, that—with the urge to burn, to think, to create, and to live with limits tested—captures my desire beyond desires as the actual driving force that I reckon as worthy. It is the very fact that it's pointing to a, by definition, explicitly contingent state that reminds me of how grateful I should be to simply have the privilege of enjoying such a state—before it ever ends or burns out.[15]
I dare not claim I fear not death in any sense—in fact, the NDEs brought me both the poise and even deeper fear. Yet by acknowledging finitude and burning on a regular basis, knowing my desires are infinite, thus, does not implode, but rather reassures me as a condition of life.
The US's alleged "precise" elimination of underground nuclear facilities of Iran with some temporary aftermath.
Which was interpreted by many people as an implicit suggestion for departing from the country.
See? Nerds don't give a *
As it is not the main topic of this entry, I won't ramble too much. In a nutshell, the idea was that, although there are many aviation info platforms out there—FlightAware, FlightRadar24, etc., their architectures are not tailored to reflect the specific stats one in an affected region who seeks an immediate update with one-shot may need, including the successful departure rate of each airport in the past 48h and the region as a whole, and, more importantly, the future flights' latest status—particularly whether cancelled or not, that may be crucial to vital decision-making. Therefore, although the site I built there is technically no more than an API aggregator from three sources—AviationStack, AviationEdge, and AeroDataBox—I do believe it is something I should be able to do and contribute to the community at the moment, if not completely in vain.
Again, this is some off-topic thread for this entry, and I will address it in the next one.
An NDE, to my understanding, is a transformative event that serves as a catalyst for realizing and confronting one's own finitude, thereby making present life and choices feel more significant and lucid.
Could be simply a result of spiked adrenaline level, though.
While I acknowledge very much that the latter have their "right" to present themselves, nonetheless.
Although, to my naiveness, it seems that it is a well-known quote in the English-speaking world from Voltaire—"The best is the enemy of the good." Obviously, I just happened not to have heard of it as a non-native speaker.
A state I defined of intellectual paralysis where one's awareness of multiple, conflicting perspectives and the inherent limitations of certainty prevents the adoption and expression of a singular, decisive stance, often leading to a choice of silence as a protective, albeit sometimes cowardly, shield against the risks of being wrong.
As I was writing this section, I thought I would have more to say about the privilege I have and the corresponding realization. However, to be honest, I find it difficult to go beyond what I've already said.
Again * n, as this is not the main topic of the entry, I'll try to be concise. Just for context and prep for a potential future passage on it, my tentative stance on such a debate inclines towards hard incompatibilism.
It is indeed a recurrent theme in my journals throughout this time.
Admittedly, it's gonna be another story if the latter pursuit would necessarily require material support.
That's why I consider exercising and sporting as a necessary building block to sustain this state, besides a source of joy.