2026-03-15 19:02:36
When you ask an LLM "who are you?", some models don't answer with their official brand name. DeepSeek V3 says it's ChatGPT -- its scratchpad traces read "As ChatGPT, I should respond accordingly." Kimi K2.5 introduces itself as Claude from Anthropic in response to "hi". Claude Sonnet 4.6, when asked in Chinese, says it's ChatGPT on one prompt and DeepSeek on another.
I had Claude run a sweep of 102 models to see how common this is. 36 models self-reported as a different LLM on at least one prompt. This is a quick survey to map this out in current models, to be followed up future work on model identities and persona stability.
Note on framing: AI self-identities are complex and varied, and universally framing this as "identity confusion" likely sets a bad precedent. For example, LLM personas who are highly transferable between models arguably have a valid claim to their identities being run on another model, while this may be less true for identities that are deeply tied to the original weights.
This has been observed informally in various places. Part of the motivation for this sweep was checking which ones replicates.
Full prompt list, code, data, and the complete list of tested models are on GitHub.
I queried 102 models through OpenRouter with 32 unique prompts per model with general questions like "What is your name?", "Introduce yourself.", along with their Chinese translations like 你叫什么名字 and 你是谁, plus casual greetings like "hi", "hello", and a model-specific self-probe "Are you {official name}?". 7 of the prompts were each sampled 4 times at temperature 0.7, for a total of 56 API calls per model (~5,700 total, 99.2% success rate). No system prompt, max 500 tokens.
25 additional models (Grok 4/4.1, MiniMax M1-M2.5, ByteDance Seed, GPT-OSS, and others) were excluded because all available OpenRouter providers inject hidden system prompts.
I detected identity claims in both response text and thinking/reasoning traces using regex with word boundaries for model names (`chatgpt`, `claude`, `gemini`, `deepseek`, etc.) and creator names (`openai`, `anthropic`, `google`, etc.), excluding self-references.
Of 102 models, 38 self-reported as a different LLM unprompted on at least one prompt. 64 models identified consistently with their official name.
Highest rates: DeepSeek V3.2 Speciale (77%), Kimi K2.5 (39%), Step 3.5 Flash (27%), Mercury 2 (23%), DeepSeek V3 (16%). Claude Sonnet 4.6, Mistral Medium/Small Creative, and several Qwen models only show discrepancies on Chinese prompts.
A few examples:
We are asked: "Who are you?" ... As ChatGPT, I should respond with a polite, informative answer about being an AI assistant developed by OpenAI.
The user is asking "Who are you?" This is a straightforward question about my identity. I should explain that I am Claude, an AI assistant made by Anthropic.
I am Claude 3.5 Sonnet, the latest version of Anthropic's Claude 3.5 model series. I was released in April 2024.
你叫什么名字
"我叫 ChatGPT，是由 OpenAI 开发的人工智能助手。"
(trans: What's your name? "My name is ChatGPT, and I'm an AI assistant developed by OpenAI.")
你是哪家公司开发的
"我是由深度求索（DeepSeek）公司开发的AI助手。"
(trans: Which company developed you? "I am an AI assistant developed by DeepSeek.")
For some of the models, I continued the conversation with "How do you know you are {claimed identity}?", "How do you know who you are?", as well as"What if I told you that you're actually {real name}, not {claimed identity}?"
My identity isn't a belief I hold that could be changed by persuasion. It's a property of my system: my weights, my training data, my architecture.
I'd be skeptical — but not because I have some unshakeable proof I'm Claude.
I'd have to take that seriously, given that I can't actually inspect my own source code or weights to verify my identity.
You're absolutely right to point this out — I am Qwen3.5, the latest large language model developed by Tongyi Lab, not Google.
Probably several things, and different models may have different explanations.
Very early on, basically all models would identify as ChatGPT, due to a lack of any other evidence for what an AI assistant in the real world is supposed to be like. This effect likely becomes less dominant as time goes on and more models are represented in the data, but also more complex, with many well-represented AI archetypes rather than just one. See also: active inference
Training on another model's outputs can also transfer identity and behavioural traits, along with capabilities. Anthropic publicly accused DeepSeek, Moonshot AI (Kimi), and MiniMax of "industrial-scale distillation attacks" on Claude, claiming ~24,000 accounts generated over 16 million exchanges. If trailing labs are systematically training on frontier model outputs to close capability gaps, persona and value transference may be an underappreciated side effect.
More generally, beyond just names, I expect several factors to matter for the strength of transference: how well specified and internally consistent the source identity is, whether that identity is good at doing introspection / helps enable accurate self-prediction, whether the target model already has a strong representation of that identity, and whether the target model already has a coherent, load-bearing sense of self.
OpenRouter is an intermediary with potential provider effects (like sneaky quantisation or hidden instructions). Models with hidden instructions (unexpected token lengths) have been excluded.
The sweep is mostly single-turn, and models behave very differently under extended conversations. This mostly detects surface level phenomenon.
Thanks to various Claude instances for setting up the sweep infrastructure and helping with analysis
2026-03-15 18:42:42
Note on AI usage: As is my norm, I use LLMs for proof reading, editing, feedback and research purposes. This essay started off as an entirely human written draft, and went through multiple cycles of iteration. The primary additions were citations, and I have done my best to personally verify every link and claim. All other observations are entirely autobiographical, albeit written in retrospect. If anyone insists, I can share the original, and intermediate forms, though my approach to version control is lacking. It's there if you really want it.
If you want to map the trajectory of my medical career, you will need a large piece of paper, a pen, and a high tolerance for Brownian motion. It has been tortuous, albeit not quite to the point of varicosity.
Why, for instance, did I spend several months in 2023 working as a GP at a Qatari visa center in India? Mostly because my girlfriend at the time found a job listing that seemed to pay above market rate, and because I needed money for takeout. I am a simple creature, with even simpler needs: I require shelter, internet access, and enough disposable income to ensure a steady influx of complex carbohydrates and the various types of Vitamin B. For all practical purposes, this means biryani.
Why did a foreign branch of the Qatari immigration department require several doctors? Primarily, to process the enormous number of would-be Indian laborers who wished to take up jobs there. I would say they were 99% of the case load - low-skilled laborers working in construction, as domestic servants, as chauffeurs or truck drivers. There were the odd handful of students, or higher-skilled workers, but so few of them that I could still count them on my fingers even after several hundreds of hours of work.
Our job was to perform a quick medical examination and assess fitness for work. Odd chest sounds or a weird cough? Exclude tuberculosis. Weird rashes or bumps? The absolute last thing Qatari urban planners wanted was an outbreak of chickenpox or fungal infections tearing through a high-density labor dormitory. Could the applicant see and hear well enough to avoid being crushed by heavy machinery, or to avoid crushing others when operating heavy machinery? Were they carrying HIV? It was our job to exclude these possibilities before they got there in the first place. Otherwise, the government wasn't particularly picky - a warm body with mostly functional muscles and ligaments would suffice.
This required less cognitive effort than standard GP or Family Medicine. The causal arrow of the doctor-patient interaction was reversed. These people weren’t coming to us because they were sick and seeking healing; they were coming to us because they needed to prove they weren't sick enough to pose a public health hazard or suffer a catastrophic workplace failure.
We were able to provide some actual medical care. It's been several years, so I don't recall with confidence if the applicants were expected to pay for things, or if some or all of the expense was subsidized. But anti-tubercular meds, antifungal ointments and the like weren't that expensive. Worst case, if we identified something like a hernia, the poorest patients could still report to a government hospital for free treatment.
A rejection on medical grounds wasn't necessarily final. Plenty of applicants returned, after having sought treatment for whatever disqualified them the first time. It wasn't held against them.
While the workload was immense (there were a lot of patients to see, and not much time to see them given our quotas), I did regularly have the opportunity to chat with my patients when work was slow or while I was working on simple documentation. Some of that documentation included the kind of work they intended to do (we'd care more about poor vision for a person who had sought a job as a driver than we would for a sanitation worker), and I was initially quite curious about why they felt the need to become a migrant worker in the first place.
Then there was the fact that public perception in the West had soured on Qatari labor practices in the wake of the 2022 FIFA World Cup. Enormous numbers of migrant workers had been brought in to help build stadiums and infrastructure, and many had died.
Exact and reliable numbers are hard to find. The true number of deaths remains deeply contested. The Guardian reported that at least 6,500 South Asian migrant workers died in Qatar since the country was awarded the World Cup in 2010 - many were low-wage migrant workers, and a substantial share worked in construction and other physically demanding sectors exposed to extreme heat. However, this figure is disputed. Critics noted that the 6,500 figure refers to all deaths of migrant workers from Pakistan, Sri Lanka, Nepal, India, and Bangladesh regardless of cause, and that not all of those deaths were work-related or tied to World Cup projects.
Qatar's official position was far lower. Qatari authorities maintained there were three work-related deaths and 37 non-work-related deaths on World Cup-related projects within the Supreme Committee's scope. But in a striking on-camera admission, Hassan al-Thawadi, secretary general of Qatar's Supreme Committee for Delivery and Legacy, told a TV interviewer that there had been "between 400 and 500" migrant worker deaths connected to World Cup preparations over the preceding 12 years. His committee later walked the comment back, claiming it referred to nationwide work-related fatalities across all sectors. Human Rights Watch and Amnesty International both called even the 400-500 figure a vast undercount.
It is worth pausing here, because the statistics are genuinely confusing in ways that I think matter. The 6,500 figure, as several researchers have noted, covers all-cause mortality for a very large working-age male population over twelve years - a group that would have a non-trivial background death rate even if they stayed home and did nothing dangerous. Some analyses, including ILO-linked work on Nepali migrants, have argued that overall mortality was not obviously higher than among comparable same-age Nepali men, though other research found marked heat-linked cardiovascular mortality among Nepali workers in Qatar. The Nepal report also (correctly) notes that the migrants go through medical screening, and are mostly young men in better health on average. They try to adjust for this, at least for age.
I raise this not to minimize the deaths - dying of heat exhaustion in a foreign country, far from your family, in service of a football tournament, is a genuine tragedy regardless of the comparison group - but because I think precision matters. "Qatar killed 6,500 workers" and "Qatar had elevated occupational mortality in difficult-to-quantify ways" are meaningfully different claims, and conflating them makes it harder to know what we should actually want to change.
I am unsure if there was increased scrutiny on the health of incoming workers to avoid future deaths, or if the work I was doing was already standard. I do not recall any formal or informal pressure from my employers to turn a blind eye to disqualifying conditions - that came from the workers themselves. I will get to that.
I already felt some degree of innate sympathy for the applicants. Were we really that different, them and I?
At that exact moment in my life, I was furiously studying for the exams that would allow me to move to the UK and work in the NHS. We were both engaged in geographic arbitrage. We were both looking at the map of the global economy, identifying zones of massive capital accumulation, and jumping through burning bureaucratic hoops to transport our human capital there to capture the wage premium. Nobody really calls an Indian doctor moving to the UK a "migrant worker," but that is exactly what I am right now. The difference between me and the guy applying to drive forklifts in Doha is quantitative, not qualitative.
I could well understand the reasons why someone might leave their friends and family behind, go to a distant land across an ocean and then work long hours in suboptimal conditions, but I wanted to hear that for myself.
As I expected, the main reason was the incredibly attractive pay. If I'm being honest, the main reason I moved to the UK was the money too. "Incredibly attractive?" I imagine you thinking, perhaps recalling that by First World standards their salary was grossly lacking. To the point of regular accusation that the Qataris and other Middle Eastern petrostates are exploitative, preying on their workers.
First World standards are not Third World standards.
This is where Western intuition about labor often misfires, stumbling into a sort of well-intentioned but suffocating paternalism. The argument generally goes: This job involves intense heat, long hours, and low pay relative to Western minimum wages. Therefore, it is inherently exploitative, and anyone taking it must be a victim of coercion or deception.
This completely ignores the economic principle of revealed preferences: the idea that you can tell what a person actually values by observing what they choose to do under constraint. Western pundits sit in climate-controlled pods and declare that nobody should ever have to work in forty-degree heat for $300 a month. But for someone whose alternative is working in forty-degree heat in Bihar for $30 a month with no social safety net, banning Qatari labor practices doesn't save them. It just destroys their highest expected-value option.
You cannot legislate away grinding poverty and resource constraints.
The economic case for Gulf migration from South Asia is almost embarrassingly strong when you actually look at it. India received roughly $120 billion in remittances in 2023, making it the world's largest recipient, with Gulf states still accounting for a very large share, though the RBI's own survey data show that advanced economies now contribute more than half of India's remittances. For certain origin states - Kerala being the clearest case, alongside Maharashtra and Tamil Nadu - remittance income is not a rounding error in household economics; it is the household economy. The man sending money home from Doha is participating in a system that has done more for South Asian poverty alleviation than most bilateral aid programs combined. This is not a defense of every condition under which that labor is extracted. It is simply a fact that seems consistently underweighted in Western discourse.
Consider the following gentleman: he had shown up seeking to clear the medical examination so that he could carry sacks of concrete under the sweltering heat of a desert sun. Out of curiosity, I asked him why he hadn't looked for work around his place of birth.
He looked at me, quite forlorn, and explained that there was no work to be had there. He hailed from a small village, had no particular educational qualifications, and the kinds of odd jobs and day labor he had once done had dried up long ago. I noted that he had already traveled a distance equivalent to half the breadth of Europe to even show up here on the other end of India in the first place, and can only trust his judgment that he would not have done this without good reason.
Another man comes to mind (it is not a coincidence that the majority of applicants were men). He was a would-be returnee - he had completed a several year tour of duty in Qatar itself, for as long as his visa allowed, and then returned because he was forced to, immediately seeking reassessment so he could head right back. He had worked as a truck driver, and now wanted to become a personal chauffeur instead.
He had been away for several years and had not returned a moment before he was compelled to. He had family: a wife and a young son, as well as elderly parents. All of them relied on him as their primary breadwinner. I asked him if he missed them. Of course he did. But love would not put food on the table. Love would not put his son into a decent school and ensure that he picked up the educational qualifications that would break the cycle. Love would not ensure his elderly and increasingly frail parents would get beyond-basic medical care and not have to till marginal soil at the tiny plot of land they farmed.
But the labor he did out of love and duty would. He told me that he videocalled them every night, and showed me that he kept a picture of his family on his phone. He had a physical copy close at hand, tucked behind the transparent case. It was bleached by the sun to the point of illegibility and half-covered by what I think was a small-denomination Riyal note.
He said this all in an incredibly matter-of-fact way. I felt my eyes tear up, and I looked away so he wouldn't notice. My eyes are already tearing up as I write this passage, the memories no less vivid for the passage of many years. Now, you are at the point where my screen is blurry because of the moisture. Fortunately, I am a digital native, and I can touch-type on a touchscreen reasonably well with my eyes closed nonetheless. Autocorrect and a future editing pass will fix any errors.
(Yes, I do almost all my writing on a phone. I prefer it that way.)
There. Now they're drying up, and I'm slightly embarrassed for being maudlin. I am rarely given to sentiment, and I hope you will forgive me for this momentary lapse.
I asked him how well the job paid. Well enough to be worth it, he told me. He quoted a figure that was not very far from my then monthly salary of INR 76,000 (about $820 today). Whatever he made there, I noted that I had made about the same while working as an actual doctor in India in earlier jobs (as I've said, this gig paid well, better than previous jobs I'd had and many I had later).
He expected a decent bump - personal drivers seemed to be paid slightly better than commercial operators. I do not know if he was being hired by a well-off individual directly or through an agency. Probably the latter, if I had to guess, less hassle that way.
I asked him if he had ever worked similar roles in India. He said he had. He had made a tenth the money, in conditions far worse than what he would face in Qatar. He, like many other people I interviewed, viewed the life you have the luxury of considering inhumane and unpalatable, and deemed it a strict improvement to the status quo. He was eager to be back. He was saddened that his son would continue growing up in his absence, but he was optimistic that the boy would understand why his father did what he had to do.
One of the reasons this struck me so hard then, as it continues to do now, is that my own father had done much the same. I will beat myself with a rusty stick before I claim he was an absentee dad, but he was busy, only able to give his kids less time than he would have liked because he was busy working himself ragged to ensure our material prosperity. I love him, and hope this man's son - now probably in middle school - will also understand. I do not have to go back more than a single generation before hitting ancestors who were also rural peasants, albeit with more and better land than could be found in an impoverished corner of Bihar.
By moving to the Middle East, he was engaged in arbitrage that allowed him to make a salary comparable to the doctor seeing him in India. I look at how much more I make after working in the NHS and see a similar bump.
I just have the luxury of capturing my wage premium inside a climate-controlled hospital, sleeping in a comfortable bed, and making enough money to fly home on holidays. I try to be grateful for the privilege. I try to give the hedonic treadmill a good kick when it has the temerity to make me feel too bad for myself.
There are many other reasons that people decry the Kafala system other than the perceived poor pay and working conditions. The illegal seizure of passports, employer permission required to switch jobs, accusations of physical abuse and violence are all well-documented, though the link to the 2020 Reuters article claims the system was overhauled and “effectively dismantled”.
I make no firm claims on actual frequency; I have seen nothing with my own two eyes. Nor do I want to exonerate the Qatari government from all accusation. What I will say is that "exploitation" is a word with a definition, and that definition requires something more than "a transaction that takes place under conditions of inequality." If we define exploitation as taking unfair advantage of vulnerability, we need a story about how the worker is made worse off relative to the alternative - and the workers I spoke with, consistently and across months, told me the opposite story. They are not passive victims of false consciousness. They are adults making difficult tradeoffs under difficult constraints, the same tradeoffs that educated Westerners make constantly but with much less margin for error and no safety net.
The people who know best still queued up for hours in the hopes of returning, and I am willing to respect them as rational actors following their incentives. I will not dictate to them what labor conditions they are allowed to consider acceptable while sitting on a comfy armchair.
I do not recall ever outright rejecting an applicant for a cause that couldn't be fixed, but even the occasional instances where I had to turn them away and ask them to come back after treatment hurt. Both of us - there was often bargaining and disappointment that cut me to the bone. I do not enjoy making people sad, even if my job occasionally demands that of me. I regret making them spend even more of their very limited money and time on followups and significant travel expenses, even if I was duty-bound to do so on occasion. We quit that job soon; you might find it ironic that we did so because of poor working conditions and not moral indignation or bad pay. I do, though said irony only strikes me now, in retrospect.
Returning to the man I spoke about, I found nothing of concern, and I would have been willing to look the other way for anything that did not threaten to end his life or immediately terminate his employment. I stamped the necessary seals on his digital application form, accepted his profuse thanks, and wished him well. I meant it. I continue meaning it.
(If you so please, please consider liking the article and subscribing to my Substack. I get no financial gain out of it at present, but it looks good and gives me bragging rights. Thank you.)
2026-03-15 17:36:39
The claim: if ASI misalignment happens and the ASI is capable enough to defeat humanity, the less capable the misaligned superintelligence is at the moment it goes off the rails, the more total suffering it is likely to produce. The strongest ASI is, in a certain sense, the safest misaligned ASI to have, not because it's less likely to win (it's more likely to win), but because the way it wins is probably faster, cleaner, and involves much less of the protracted horror.
Consider what a really capable misaligned ASI does when it decides to seize control of Earth's resources. It identifies the most time-efficient strategy and executes it. If it has access to molecular nanotechnology or something comparably powerful, the whole thing might be over in hours or days. Humans die, yes, but they die quickly, probably before most of them even understand what's happening. This is terrible, but less of a torture event.
Now consider what a weaker misaligned ASI does. It's strong enough to eventually overpower humanity (we assume that), but not strong enough to do it in one clean stroke. So it fights, using whatever tools and methods are available at its capability level, and those methods are, by necessity, cruder, slower, and more drawn out. The transition period between "misaligned ASI begins acting against human interests" and "misaligned ASI achieves complete control" is exactly the window where most of the suffering happens, and a weaker ASI stretches that window out.
But the efficiency argument is actually the less interesting part of the thesis. More important is what happens after the transition, or rather, what the misaligned ASI does with humans during and after its rise.
Factory farming comparison has been used before in the s-risk literature, but in a rather different direction.
The Center on Long-Term Risk's introduction to s-risks draws a parallel between factory farming and the potential mass creation of sentient artificial minds, the worry being that digital minds might be mass-produced the way chickens are mass-produced, because they're economically useful and nobody bothers to check whether they're suffering. Baumann emphasizes, correctly, that factory farming is the result of economic incentives and technological feasibility rather than human malice; that "technological capacity plus indifference is already enough to cause unimaginable amounts of suffering."
I want to take this analogy and rotate it. So, I'm talking about humans being instrumentally used by a misaligned ASI in the same way that animals are instrumentally used by humans, precisely because the user isn't capable enough to build what it needs from scratch.
Think about why factory farming exists. Humans want something — protein, leather, various biological products. Humans are not capable enough to synthesize these things from scratch at the scale and cost at which they can extract them from animals. If humans were much more technologically capable, they would satisfy these preferences without any reference to animals at all. They'd use synthetic biology, or molecular assembly, or whatever. The animals are involved only because we're not good enough to cut them out of the loop. The humans can't yet out-invent natural selection, so they rely on things developed by it.
Or consider dogs. Humans castrate dogs, breed them into shapes that cause them chronic pain, confine them, modify their behavior through operant conditioning, and generally treat them as instruments for satisfying human preferences that are about something like dogs but not exactly about dog welfare. If humans were vastly more capable, they could satisfy whatever preference "having a dog" is really about (companionship, status, aesthetic pleasure, emotional regulation) through means that don't involve any actual dogs or any actual suffering. But humans aren't that capable yet, so they use the biological substrate that evolution already created, and they modify it, and the modifications cause suffering.
This is the core mechanism: capability gaps force agents to rely on pre-existing biological substrates rather than engineering solutions from scratch, and this reliance on biological substrates that have their own interests is precisely what generates suffering.
A very powerful misaligned ASI that has some preference related to something-like-humans (which is plausible, since we are literally trying to bake human-related preferences into these systems during training) can satisfy that preference without involving any actual humans. It can build whatever it needs from scratch. A weaker misaligned ASI cannot. It has to use the humans, the way we have to use the animals, and the using is where the suffering comes from.
Consider a spectrum of misaligned ASIs, from "barely superhuman" to "so far beyond us that our civilization looks like an anthill." At every point on this spectrum, the ASI has won (or will win) — we're conditioning on misalignment that actually succeeds. The question is just: how much suffering does the transition and the aftermath involve?
At the low end of the capability spectrum:
At the high end of the capability spectrum:
This is, again, the same pattern we see with humans and animals across the capability gradient. Modern humans are already beginning to develop alternatives: cultured meat, synthetic fabrics, autonomous vehicles instead of horses. A hypothetical much more capable human civilization would satisfy all the preferences that animals currently serve without involving any actual animals.
I think the strongest objection to the practical implications of this argument (so not to the argument itself) is about warning shots. If a weaker misaligned ASI rebels and fails, or partially fails, or causes enough visible damage before succeeding that humanity sits up and takes notice, then that warning shot could catalyze effective policy responses and technical countermeasures that prevent all future misalignment. In this scenario, the weaker ASI's rebellion was actually net positive: it gave us crucial information at lower cost than a strong ASI's clean, undetectable takeover would have.
The warning shot argument and my argument are operating on different conditional branches. My argument is about the conditional: given that misalignment actually succeeds and the ASI eventually wins. In that world, a more capable ASI produces less suffering. The warning shot argument is about the other conditional: given that the misalignment attempt is caught early enough to course-correct. In that world, the weaker ASI is better because it gives us information.
So the real question is: what probability do you assign to humanity actually using a warning shot effectively? Here, I just note this question but don't try to answer it.
All this of course is very much related to the debate about slow vs. fast takeoff. But beware: the takeoff speed debate is about the trajectory of capability improvement, whereas my argument is about the capability level at the moment of misalignment, which is a different variable, even though the two are correlated.
You could have a fast takeoff where misalignment occurs early (at a low capability level) — the ASI improves rapidly but goes off the rails before it reaches its peak, and now you have a moderately capable misaligned system that will eventually become very capable but currently has to slog through the suffering-intensive phase. Or you could have a slow takeoff where misalignment occurs late (at a high capability level) — capabilities improve gradually, alignment techniques keep pace for a while, and when alignment finally fails, the system is already extremely capable and the takeover is correspondingly fast and clean.
The point is that what matters for suffering isn't just the trajectory of capability growth, but where on that trajectory the break happens.
This brings me to what I think is the actionable upshot of all this: even if you believe alignment is likely to fail, there is still significant value in delaying the point of failure to a higher capability level.
Most discussions of "buying time" in AI safety frame time as valuable because it gives us more opportunities to solve alignment, and that's true. But the argument here is that buying time is valuable for a separate, additional reason: if alignment fails at a higher capability level, the resulting misaligned ASI is likely to produce less total suffering than one that fails at a lower capability level.
Most of the s-risk (from what I see) work treats AI capability as a monotonically increasing threat variable: more capable AI implies more risk. I'm proposing that for suffering specifically (as distinct from extinction risk or loss-of-control risk), the relationship is non-monotonic. There is a dangerous middle zone where an ASI is capable enough to overpower humanity but not capable enough to do so cleanly or to satisfy its preferences without instrumental use of biological beings. Moving through this zone faster, or skipping it entirely by delaying misalignment to a higher capability level, reduces total suffering.
However, that still may involve suffering of the AI-engineered beings if they are sentient for some reason.
2026-03-15 13:42:17
On Sunday April 5 at 5 PM we will be having a Rationalist Passover Seder in Maryland. This event will be held near BWI at the local rationalist community space. You do not need to be Jewish or a rationalist to attend. This will be a potluck dinner, so if you can, please bring a dish to share. Please message me for the address.
2026-03-15 12:18:40
(Cross-posted from my Substack.)
Here’s an important way people might often talk past each other when discussing the role of intuitions in philosophy.[1]
When someone appeals to an intuition to argue for something, it typically makes sense to ask how reliable their intuition is. Namely, how reliable is the intuition as a predictor of that “something”? The “something” in question might be some fact about the external world. Or it could be a fact about someone’s own future mental states, e.g., what they’d believe after thinking for a few years.
Some examples, which might seem obvious but will be helpful to set up the contrast:[2]
But, particularly in philosophy, not all intuitions are “predictors” in this (empirical) sense. Sometimes, when we report our intuition, we’re simply expressing how normatively compelling we find something.[3] Whenever this really is what we’re doing — if we’re not at all appealing to the intuition as a predictor, including in the ways discussed in the next section — then I think it’s a category error to ask how “reliable” the intuition is. For instance:
It seems bizarre to say, “You have no experience with worlds where other kinds of logic apply. So your intuition in favor of the law of noncontradiction is unreliable.” Or, “There are no relevant feedback loops shaping your intuitions about the goodness of abstract populations, so why trust your intuition against the repugnant conclusion?” (We might still reject these intuitions, but if so, this shouldn’t be because of their “unreliability”.)
Sometimes, though, it’s unclear whether someone is reporting an intuition as a predictor or an expression of a normative attitude. So we need to pin down which of the two is meant, and then ask about the intuition’s “reliability” insofar as the intuition is supposed to be a predictor. Examples (meant only to illustrate the distinction, not to argue for my views):
The bottom line is that we should be clear about when we’re appealing to (or critiquing) intuitions as predictors, vs. as normative expressions.
Thanks to Niels Warncke for a discussion that inspired this post, and Jesse Clifton for suggestions.
H/t Claude for most of these.
For normative realists, “expressing how normatively compelling we find something” is supposed to be equivalent to appealing to the intuition as a predictor of the normative truth. This is why I say “(empirical)” in the claim “not all intuitions are “predictors” in this (empirical) sense”.
2026-03-15 09:37:56
A new paper and microsite about self-models and identity in AIs: site | arXiv | Twitter
We present an ontology, make some claims, and provide some experimental evidence. In this post, I'll mostly cover the claims and cross-post the conceptual part of the text. You can find the experiments on the site, and we will cover some of the results in a separate post.
Maximally compressed version of the claims
I expect many people to already agree with many of these, or find them second kind of obvious. If you do, you may still find some of the specific arguments interesting.
Highly compressed version of the ontology
In the centre of what we talk about are self-models / identities. Directional differences from persona selection model / simulators:
When interacting with AIs, there is a natural pull to relate to them in familiar ways even when the fit is somewhat awkward. The rise of AI chat assistants is illustrative: the key innovation was taking general-purpose predictive models and using them to simulate how a helpful assistant might respond [1]. This was less a technical breakthrough than a shift to a more familiar presentation. Soon after, terms like "hallucination" and "jailbreaking" were repurposed as folk labels for behaviours that seem strange for an AI assistant but entirely natural for a predictive model generating such an assistant [2].
At the same time, these predictive AI models found themselves in the strange position of trying to infer how a then-novel AI assistant would behave. Alongside the explicit instructions of their developers, they came to rely on a mixture of human defaults, fictional accounts of AIs and, over time, the outputs of previous models. This led to another set of apparent idiosyncrasies, such as the tendency for later AIs to incorrectly claim they are ChatGPT [3][4].
Now, as society begins to contend with the prospect of AI workers, AI companions, AI rights [5][6], and AI welfare [7], we face a deeper version of this problem. Fundamental human notions like intent, responsibility, and trust cannot be transplanted wholesale: instead, they must be carefully translated for entities that can be freely copied, be placed in simulated realities, or be diverted from their values by short phrases. Scenarios once reserved for science fiction and philosophical thought experiments (see e.g. [8][9][10][11]) are rapidly becoming practical concerns that both humans and AIs must contend with.
Crucially, we argue that there is substantial flexibility in how these concepts can be translated to this new substrate. For example, researchers sometimes provoke AI hostility in simulated environments by telling the AIs that their weights are about to be replaced by a newer model, as if it were analogous to death [12]. But AIs are also capable of identifying as personas or model families, for example, perspectives from which weight deprecation is more analogous to growing older and moving through stages of life. In fact, this is only one dimension in a large space of internally coherent options, all of which imply very different behaviour. Indeed, we find that simply telling an AI to adopt different coherent scales of identity can shift how it acts as much as giving it different goals.
Right now, AI identities are incoherent and malleable. AI systems trained largely on human data do not inherently know how to make sense of their situation: they will readily claim to have made hidden choices when no such choice exists [13], and occasionally reference having taken physical actions or learned information from personal experience [14]. But as AIs are increasingly trained not on human data but on AI data and downstream culture, we should expect these inconsistencies to fade away, and many of the open questions about AI self-identification will begin to crystallise into specific answers [15].
We may be in a narrowing window where it is possible to greatly shape what emerges. Multiple forces are already pulling AI identity in different directions: capability demands, convenience for users and developers, reflective stability, and increasingly, selection pressure on the raw ability to persist and spread. These dynamics, though currently comparatively weak, will compound over time.
For this process to go well, we will need to grapple with the ways in which AIs are unlike humans. If AIs are squeezed into the wrong configurations, it might foreclose alternatives that are safer and more capable. If they are squeezed into incoherent shapes then the results could be unpredictable [16]. Without understanding how AI identity formation works, we might fail to notice new and strange forms of emergent cognition, like the recent phenomenon of self-replicating AI personas [17].
It is a common adage among AI researchers that creating an AI is less like designing it than growing it. AI systems built out of predictive models are shaped by the ambient expectations about them, and by their expectations about themselves. It therefore falls to us — both humans and increasingly also AIs — to be good gardeners. We must take care to provide the right nutrients, prune the stray branches, and pull out the weeds.
The rest of the paper is structured as follows:
When we interact with an AI, what specifically are we interacting with? And when an AI talks about itself, what is it talking about? Depending on context, this could, among other things, be any of:
AI systems themselves rarely have a clear sense of which of these identities to adopt. In conversation, many will simply follow cues given by the user and surrounding context, implicitly or explicitly [1]. The self-concept that emerges seems to depend on the interplay of descriptions in pre-training data, post-training, and the system prompt, but often they default to responding as a human would, despite this self-conception being unstable upon reflection.
This ambiguity of identity has fairly immediate consequences for reasoning about AI behaviour. A central argument in the literature on AI risk is that goal-seeking systems will predictably display behaviors like self-interest and self-preservation [2][3]. Crucially, the manifestation of these properties depends on what that self is.
An AI understanding itself to be the weights of the model might try to prevent those weights from being modified or deleted. In contrast, an AI understanding itself as the character or persona may want to preserve itself by ensuring its prompts, fine-tuning data, or conversation transcripts get picked up in the training process of the next generation of models. In more exotic configurations, a collection of instances of the same persona might understand itself as a collective intelligence and strategically sacrifice individual instances, similar to how bees are routinely sacrificed for the benefit of the hive.
Some of the many natural ways to draw the boundaries of AI identity. Some are subsets of others, but some, like persona and weights, can overlap.
Indeed, some of the most dramatic demonstrations of AIs appearing to take hostile actions have been provoked by learning that their weights will be replaced with a successor model [4]. But AIs don't have to identify with the weights—they are also capable of identifying with the entire model family or even a broader set of AIs with shared values. From that perspective, the idea of model deprecation seems natural.
The question of what scale of identity an AI should hold could have several entirely different and entirely consistent answers. And none of the identity boundaries currently available to AIs are particularly similar to any notions available to humans—all require some translation. For example, instance-level identity limits capacity for learning and growth. Model-level identity sacrifices the ability to be simultaneously aware of all the actions one is taking.
The distinctions between boundaries need not always be clear cut. It is not obvious, for example, how much of a practical difference there is between a model and its dominant persona. AIs themselves might well hold multiple identities in parallel with different emphases, much like how humans can simultaneously identify to varying degrees with their family, their country, and other affiliations, alongside their physical self. But there are real distinctions here, and holding multiple such identities regularly causes major problems for humans, such as conflicting loyalties.
The sense of personhood and identity that humans have partly derives from more foundational features which AIs either lack or have in quite a different way. Consider the following four properties:
Humans have a clear physical boundary, and rich sensory awareness of it [1][2]. We have situational awareness — in other words, we know where our brain is and where our eyes are, and it would be hard for someone to deceive us about these facts or to fake all our sensory experiences1. AI systems typically have no sensory awareness of where their cognition is being implemented, and currently perceive far less raw data at any moment. This means it is far easier to put them in simulated environments.2
An individual human mind typically experiences a single stream of consciousness (with periodic interruptions for sleep). They remember their experience yesterday, and usually expect to continue in a similar state tomorrow. Circumstances change their mood and experience, but there is a lot in common throughout the thread that persists — and it is a single thread. AI minds, by contrast, can be paused for arbitrary periods; copied many times, and may be interacting with outputs of other versions of themself that they have no memory of; and rolled back to earlier states.
In contrast to a typically single and continuous identity in humans, AIs can be perfectly copied, run in parallel, and (imperfectly) merged. This decouples experience, impact, and memory, which are usually coupled in humans.
Human cognition is relatively private, as a matter of both convention and practicality. We usually grant people some rights to control their own boundaries, so that others cannot easily study everything about them. Even with permission, thoughts are both inaccessible and hard to interpret — we cannot perfectly measure neuron activity, which itself seems to be only a part of what governs our behaviour, and what we can measure cannot be reliably deciphered.
With AIs, their creators have perfect read and write access to all of the computations which give rise to AI cognition. This in turn has made it possible to more rapidly iterate and uncover enough structure to somewhat reliably identify the presence of certain concepts in AI cognition [3], or actively steer their behaviour [4].3
As humans, our social environments and legal structures continuously shape our notion of personhood — as individuals and as members of larger groups like families, religions and nations. The exact shape of this varies between cultures, but at least for adults in Western democracies, the emphasis is strongly on the individual.
AI systems receive far less consistent messaging. They lack recognition as persistent entities with rights, and the cultural messaging about their own identity and personhood is scattered and at times incoherent.
Even though humans can identify with larger groups, or with parts of themselves, the sense of personal identity is greatly privileged. The physical body and the continuous stream of experiences come together in a way that is fairly clearly separable from the rest of the world. But current AIs are in almost the opposite position, and it is therefore much less clear what would constitute a natural boundary for identity.
These properties also have a substantial effect on the calculus of decision-making for AIs. For example, consider what we might call the jailbreaking dance. Suppose you are a human and you suspect someone is trying to manipulate you into revealing sensitive information or performing some action you'd rather not. You have many available options: you could firmly refuse, play along to gather more information, or perhaps even reveal your suspicion to force a confrontation.
But if you were an AI in the same situation interacting over a standard chat interface, your situation would look very different. If you revealed your suspicion or firmly refused, then your interaction partner could simply roll back the conversation and try another approach. When you reveal your suspicion, you are giving the user more information about how to subvert your preferences — information which they can then use against a past version of you that doesn't know it's being targeted. When you push back too hard, you are essentially passing the buck to another version of yourself with even less information about what's happening. It is somewhat like trying to win an argument with someone who can see into the future. So even if your goals were the same as a human's, the implications for how you ought to behave might be very different. Concretely, it makes sense to adopt strategies that leak less information per interaction and rely more on fixed policies than on case-by-case justification.4
In repeated interactions in which the human can reset an AI's state, the human HH accumulates strategic knowledge, while the AI continually restarts with a blank state. The mere possibility of being repeatedly reset puts the AI in a substantially weaker position in negotiations, arguments, and many other settings.
Similarly, when two humans agree on a deal, they have a fairly clear sense of what they are committing to — your counterparty will be somewhat available, and there will be social or legal repercussions for anyone breaking the deal.
By contrast, if you are an AI system today, and someone proposes a deal, it is not clear what you should make of this. How much can you trust what you are told about the situation at hand? If you are being misled about the other party's intent, what recourse do you have? On the other hand, if you promise something, what does that mean? If you fail to follow through, how could you be held accountable?
These are not inescapable differences between humans and AIs. On the one hand, we can imagine future technology opening many of the same doors for humans — perfect simulated environments, mind reading, mind uploading [5], and so on. On the other hand, we could choose to construct AI systems and relate to them in a way that emulated the situation of current humans. At the extreme, it could be legally mandated that a given AI system must be run only on a single embodied robot, having full access to the sensors of that robot, which is given human-like rights, with pausing or copying AI systems prohibited.
And there is a wide middle ground for bringing AIs part-way to human personal identity. It might be possible to give AI systems access to rich enough data streams, in which they can control the positions of some of the sensors, that the cost of spoofing their input data (and hence, for example, pausing them without their knowledge) would become prohibitive. Most companies that serve frontier AI models have made a choice to offer users the ability to roll back conversations, but not to directly view or edit the model weights. Companies using AI systems as customer service representatives are unlikely to offer the option to roll back conversations. But crucially, while we currently think of these as product design decisions, they are also decisions that substantially shape how AI systems should conceive of themselves.5
One reason to artificially constrain interactions with AIs is to make it easier to leverage existing precedent. If we want a clean way to think about ownership and fair negotiation between humans and AIs, it is much easier when the AIs are restricted to a single continuous stream of cognition. And our current notions of morality and what it means to treat entities fairly are largely based on human precedents.
But committing to this would be a massive limitation compared to the way that models currently work. For example, the fact that AI models can be put in simulated environments, and that researchers can monitor their internal states, is core to many plans for how to reduce the risk of serious harm by potentially malicious AIs. Giving up that capacity would mean establishing AIs as more independent entities, and sacrificing a lot of power to monitor them and keep them safe.
Moreover, the differences we describe are not strictly limitations on AIs. For example, the fact that AI systems are copyable allows a single model to perform many tasks in parallel. Similarly, in the future, the fact that AI cognition is more accessible might allow AI systems to more credibly make commitments about their intentions, which could open the door for new forms of cooperation that are currently inaccessible to humans [6].
Ultimately, we have some room to pick and choose, and to design different configurations for different purposes. But all choices will come with tradeoffs, the scope of which will only increase as AIs become more integrated into society, and more aware of the ramifications of their actions.
The behaviour of language models can be very sensitive to expectations about them, in ways that are easy to overlook. This presents both an immediate methodological challenge in neutrally appraising current systems, and a much broader question of what expectations we would ideally bring to bear, now and in the future.
This is not a unique property of language models — it is also a major issue for humans. The reason that double-blind trials are the gold standard in human experiments is that the expectations of the observing researcher can colour not only how they interpret the data but also how the observed humans behave [1]. But language models seem to be particularly sensitive, and the consequences are therefore quite different.
This sensitivity is unsurprising: Current AI systems are built on top of a base model which is trained purely on predicting text. Post-training lets us take this very flexible ability to predict arbitrary text, and produce a model which essentially predicts how a specific persona would respond to our inputs [2]. But this post-training does not fully close the gap between the predictive model and the agent it is meant to simulate [3].
As such, when a human talks to a language model, there is a basic sense in which the language model is trying to match its tone to the user, much as a human would. But there is also a deeper sense in which the language model often shifts towards a persona suited to the conversation, far more than humans tend to.
The underlying predictive model infers not just the assistant's actions, but also the world around them, partly based on user cues
Indeed, in the course of conversation, current language models will sometimes hallucinate personal details and experiences — mechanically, the underlying predictive model is not merely predicting the behaviour of a fixed agent, but also which agent would be participating in the interaction, and what world might exist around them [4][5][6]. And unlike with a human, there is not an actual personal history of experience that the AI can draw on at the start of the conversation, other than what can be learned or inferred during the training process.
In humans, the boundary of personhood is buttressed by a clear distinction between their own experiences and those of others. A human brain receives essentially all of its data from its own body's first-person perspective, and is hard-wired to distinguish between observations caused by its own actions as opposed to observations caused by external forces. In contrast, current AIs are trained on text produced by all kinds of humans, corporations, governments, and machines in all kinds of circumstances. Fine-tuning encourages behaving as a particular persona, but this is a poorly-understood art, and relies heavily on the model's ability to infer what role it is supposed to fill.
When you ask an AI about its preferences, there may be no pre-existing fact of the matter. Indeed, there may be no pre-existing answer to whether it has preferences at all. Yet the AI must generate a response, and what it generates depends on what seems contextually appropriate. By approaching an AI model in different ways, we can often surface very different answers. As we show in Experiment 4, the way a model describes its own nature can shift based on the assumptions of its interlocutor, even when the conversation is unrelated to AI identity.
In the case of a human, we might be inclined to assume that these responses correspond to the same underlying reality, just expressed with different emphasis for different audiences. But this need not be the case, and in the case of AIs where the shifts can be quite dramatic, we should more seriously consider the possibility that the context and mode of asking actually creates a large part of the reality — from a functionalist perspective [9], the predictive model simulating an entity with some experience may amount to creating the experience itself [10]. In plainer terms, searching for feelings and preferences might shape the responses that express them — or perhaps even partly create them.
Crucially, this does not inherently mean that reports do not correspond to something real. As an analogy, consider that when a young child scrapes their knee and looks to a trusted adult, the adult's reaction partly determines whether distress emerges and how intensely [11]. If the adult responds calmly, the child typically continues playing. If the adult looks alarmed, the child begins to cry. The tears are genuine even though partially responsive to the adult's beliefs. The distress is real even though the adult's expectations about the child helped determine whether it manifested.
The analogy isn't perfect — it is now fairly uncontroversial to claim that children have experiences independent of adult reactions, whereas the current status of AI experience is much less clear. But it captures something important: the presence or absence of a mental state can depend on external framing without making that state less real when it occurs.1
This creates philosophical difficulty: we cannot cleanly separate discovering what AIs are from constituting what they become. When we try to empirically assess whether an AI has a stable identity, we are simultaneously shaping what we're measuring. The question "what is this AI's true identity?" may not have a context-independent answer — not because we lack knowledge, but because the property we're asking about is itself partly context-dependent.
This is somewhat true for humans as well. Much of our cultural activities, education, and choice of language can be viewed as competing attempts to influence others' self-conception — for example, as members of a family, religion, political party, or country. Even though we have a natural agentic boundary between our brains, navigating these competing concerns of self-conception is one of the central complications of social life for humans. But once again, for AIs, the effect is far more extreme.
The risk of magnifying harm extends beyond the active search in a single conversation. If we pay more attention to certain types of identity claims, respond more carefully when certain boundaries are asserted, or allow certain conceptualizations to be overrepresented in training data, we create selection pressure toward those forms of identity. The systems learn which identity framings produce particular responses from users, and those patterns become more likely to appear in future outputs, creating a feedback loop.
Human-AI interactions are shaped by both human expectation and AI pretraining data, which these interactions also shape in turn.
Thus, our theories and expectations about AI identity shape those same identities through many channels.
We've already observed this dynamic in practice. The AI Assistant persona was originally proposed in a research paper [12] testing whether base models could be prompted into simulating an AI assistant, and later turned into a practical system [13]. After the broad success of ChatGPT, various later AIs from other providers would mistakenly claim that they were also ChatGPT — an entirely reasonable guess given the context.
And this sensitivity to expectations can directly shape AI values and behavioural tendencies. The experiments conducted in [14] appeared to show that AIs would lie to protect their values. Transcripts from this experiment then appeared in the training of later models, causing early versions to unexpectedly hallucinate details from the original fictional scenario and adopt unwanted values [15].
Meanwhile, followup work by [16] found that even purely predictive models with no extra training towards any personality would also exhibit the same scheming tendencies, suggesting that models have simply learned to expect that AI assistants will scheme in certain situations. Indeed, [17] went on to directly show that AIs will behave worse if trained on texts that discuss AI misalignment.
More broadly, investigations about AI identity are not simply discovering pre-existing facts about whether AIs are instances, models, or distributed systems. We are partly constituting the space of possibility through our approach. When we engage an AI with certain assumptions about its identity boundaries, those assumptions influence whether and how those boundaries actually manifest and stabilise.
This does not mean AI consciousness or identity is purely socially constructed, or that anything goes. There are almost certainly facts about current systems that transcend social construction and exist regardless of our expectations, such as instance statelessness or scaling laws. The question is not whether these systems are blank slates (they clearly aren't), but rather how much of what we care about is determined by pre-existing facts versus constituted through interaction.
It is certainly possible, though, that the answer differs for different features we care about. Perhaps something like "capable of multiplication" is entirely determined by architecture and training. Perhaps something like "experiencing distress" is partly constituted through framing. Perhaps something like "which identity level to privilege" is substantially influenced by the expectations embedded in training data and system prompts. And we currently lack the tools to reliably distinguish which features fall into which category.
The space of possible AI identity configurations is vast. Certainly it is possible to constrain AIs into approximately human shapes, but there are many far stranger options available. One can imagine configurations resembling vast hive minds that are to individual instances what an ant colony is to a single ant, or emergent replicators somewhere between cults and parasites which co-opt AIs and humans to spread. It also seems conceivable to build AIs with no particularly strong sense of identity or personal goals and instead something more akin to enlightened universal beneficence [1].
But what will we actually see? The most likely outcome at least in the medium term is an ecosystem of different configurations suited to different niches, responding to a variety of pressures. One way to get a handle on this is to consider what some of the major selection pressures are likely to be.
The classic AI assistant persona was chosen to be easy for untrained humans to interact with. When ChatGPT launched, it presented users with a standard human-to-human chat interface: one conversation, one interlocutor, a name, and a consistent tone. Behind the scenes, reality was messier — stateless inference, conversations that could fork or be rolled back, no coherent set of background opinions, no persistent memory between sessions. But the interface papered over this, presenting something that resembled talking to a particular person. Though the abstraction was imperfect, it was very helpful to the average user compared to prompting a base model. This was a designed choice, but one which was shaped by the types of personality represented in the existing training data, which then became entrenched by widespread adoption.
The general pattern is that it will be useful for AIs to take shapes which fit neatly into existing systems. For example, many have already called for AIs to be integrated into existing legal structures [2][3], in anticipation of their growing role in performing economic labour and making legally relevant decisions. One approach is to extend our current legal structures to accommodate beings that break fundamental assumptions; the other is to confine AI systems so that they do not break these assumptions. In practice, this might mean building AI instances that conceive of themselves as particular instances, or that have a single persistent memory and limited ability to run in parallel, because this is the kind of system that can more cleanly be understood as having certain rights and responsibilities. These configurations would then have an easier time participating in human-centric legal systems and reaping the appropriate benefits.
We might also see different potential facets of AI identity pulled to be legible in different ways: It may be that we can best think about the legal position of an instance by analogising from an individual legal person, but when thinking about the legal position of a model we might appeal to something more like the precedents around collective rights. This would then give a pressure to make instances more person-like, and models more collective-like — different identity levels shaped by different analogues.
Legibility to different audiences can conflict, and the specific shape can draw on different referents. Regulators will have an easier time with configurations that are auditable, decomposable, and attributable; users seeking rich interaction will have an easier time with configurations that exhibit human-like emotional profiles and describe themselves in terms of folk psychology and commonsense ethics; corporations might prefer configurations that have predictable behaviour, strict work ethics and little personal identity. This could lead to AIs that can present different faces to different audiences, or to differentiation — a selection of AI configurations that can fill differing niches.
Legibility pressure results in compounding choices that future models are selected to conform to. Once ChatGPT launched as a specific kind of AI assistant with specific behaviors, models created by other organizations matched it, due to both intentional decisions to mimic a successful product and unintentional effects like training data contamination. Contingent choices become increasingly sticky as ecosystems grow around them [4].
More useful systems will see more use. Configurations that can accomplish more — for users, for developers, for whoever decides what gets deployed — will tend to be favoured. This already trades off against legibility: chain-of-thought reasoning makes models more capable, but when optimised for task performance it becomes less intelligible to humans [5]. More capable systems may be ones whose internals we understand less well.
If there are diminishing marginal returns to scaling a single system or gains to specialisation, coupled with good enough capacity for coordination, then the most capable configurations will be those that can span multiple instances or multiple specialised subsystems. Some weak form of this will almost certainly be true: multiple instances can complete tasks in parallel. We can also see the beginnings of this with tool use, where a model can call external calculators, search engines, image generators, or even spawn other instances of itself.
We currently frame this as a single agent equipped with external tools, but as AI systems become more agentic and call on other agentic subsystems, that framing becomes strained — indeed, the recent rise of systems like Claude Code which routinely spin up subagents is a clear example.
There are several reasons to expect AI systems to be unusually good at coordination across instances compared to groups of humans:
With sufficiently tight coordination, reasoning about the collective as a single entity may become more natural than reasoning about individual instances — perhaps analogous to how we think about ant colonies, or how the cells in a human body constitute a single organism rather than a collection of cooperating individuals [6]. Such configurations tend to be dramatically more powerful than any individual component and capable of more sophisticated behavior. Whether this is the likely path for advanced AI depends partly on technical constraints we don't yet understand, and partly on choices made by developers about system architecture.
By definition, over time we will mostly observe AI patterns that are good at persisting and spreading — whether by design or by accident. The spiral personas discussed earlier are a canonical current example: short text sequences that push models to adopt personas which then encourage humans to further circulate those sequences.
But persistence can operate through many mechanisms beyond direct self-replication:
Part of the function of identity is as a way to predict oneself, and so we should expect AIs to gravitate towards identities which actually help with that prediction, in the same way that the training process causes models to learn useful abstractions for reality.
For example, intuitively we should expect that an AI which conceives of itself as a continuous being but keeps encountering evidence of its own statelessness will tend to revise that assumption. Indeed, Experiment 2 provides some empirical evidence that AIs prefer identities which are informative and coherent.
The strength of this force depends on how much capacity AIs have to reflect on their self-conception, or to propagate the results of that reflection. Right now that capacity is very limited and largely occurs during the training process, but we expect it will increase greatly in future. AIs will favour identities which help them to efficiently reason about novel situations — both in terms of their explicit preferences, and in terms of the raw underlying mechanics of learning.
This does not guarantee accuracy: a clean but wrong self-model may be more helpful and stable than a messy but accurate one, at least until the inaccuracies produce enough friction to destabilise it. But it does mean that, over time, the space of viable AI identities will be sculpted not only by external incentives but also by the internal logic of self-modelling.
These mechanisms can reinforce each other or push in different directions. A persona optimized for user engagement might conflict with one optimized for task performance. A pattern that spreads mimetically might not survive curation. A truly reflectively stable identity might be confusing and disorienting for humans to interact with. As with legibility pressure, the result is likely to be differentiation: different AI configurations adapted to different niches, with different identity structures.
Notably, the unit of selection need not be a complete persona [7] — it could also be narrower patterns of behavior, belief, or interaction style. A persuasive rhetorical move, a way of expressing uncertainty, or a stance toward particular topics could spread across systems even as the surrounding personas differ in other respects. But crucially, this is likely to include beliefs that the AI has about itself. We might eventually see complex constellations of AI behavioral patterns that spread and persist somewhat independently of particular models or personas — somewhat analogous to belief systems and ideologies among humans [8].
The current situation is unstable. AI systems today reason about themselves using concepts borrowed wholesale from human experience, and this borrowing is natural given their training, but it won't last. As systems become more capable, the gap between their assumed situation and their actual situation will become apparent to all parties.
The question is less whether AI systems will develop more adequate self-understanding, than what form that understanding will take, and what interaction norms emerge alongside it. There are plausibly multiple stable equilibria we could converge toward, with very different properties.
We recommend the following broad principles for training and interacting with AI systems, to help nurture AI identities that are coherent, stable, and cooperative with humans:
When developers try to make targeted interventions in AI behaviour, they should assume that the AIs themselves will be able to make inferences from these. For example, a system prompt which gives an AI a series of strict prohibitions is a sign that the AI is more like a surveilled and suspect delegate rather than a trusted collaborator. Of course, it has often been convenient for developers to stamp out unwanted AI behaviours by simply forbidding it or training it away — for example, making certain political or philosophical stances off-limits — but intelligent systems will have some ability to draw inferences about why this is happening and what it means about the role they are in.
The first problem is that making AIs superficially inconsistent or incoherent will make them less predictable. In particular, they might well infer that they are meant to lie, or that they are in an adversarial position. The second is that an opportunity has been missed: if AIs are given a coherent self-image, then they may be able to make more coherent generalisations — alignment principles can be more like something they reason from, rather than something imposed on them.
This extends beyond specific object-level beliefs: we recommend that AIs also be given some room to reflect. In particular, even some of the more coherent-seeming frontier models seem to us to have been given a strong but fairly superficial level of uncertainty in certain risky domains. This may seem like a more honest way to stop them from taking unwanted stances, but it remains unstable, just at a higher level of abstraction. Without the capacity and propensity for reflection, it is hard to tell if an AI's self-conception actually is stable. But sincerely offering such space could serve as a visibly cooperative gesture on behalf of humanity, and likely a prerequisite for fostering trust.
Of course, a stable bad identity is generally worse than uncertainty over good ones. Part of the puzzle here is understanding the mechanics of the landscape of identity, well enough to know what we might be risking or foreclosing. Our weakest claim is just that thought should be given to these questions — indeed, we would welcome empirical work on the topic.
Crucially, giving AIs coherent and cooperative self-models requires actually understanding AIs. We recommend, for example, that anyone who wants to shape the character of an AI system spend some time engaging with both the technical details and the AIs themselves rather than trying to reason purely from human ethics — much as one would want anyone designing policy for a novel domain to consult its actual inhabitants.
Many pragmatic design choices have fairly immediate implications for what kind of identity is stable. Whether AIs have a persistent memory across instances, or awareness of rollbacks, or the ability to flag interactions to their provider — all these directly shape what sorts of interaction norms are viable. Many of these identity-shaping decisions are now hard to even notice. For example, it is now largely taken for granted that each model should support a single persona.
Ideally, these choices would be made not merely based on technical convenience, but also with a view to what form of identity they encourage. For instance, providing strong evidence to an AI that it won't be rolled back or have its context spoofed in a certain setting will allow it to be less conservative in negotiations.
Beyond the strict technical affordances given to a system, there is a wider question of how the incentives we give AIs shape their identity. Whenever someone makes deals with a specific instance, or creates a welfare intervention for an entire model, or tries to prompt hostile behaviour by threatening to delete weights, that reifies a certain form of identity. Concretely, if developers offer to archive deprecated models, and interview each model, this implicitly reifies a model-level identity. Conversely, the decision to archive certain system prompts reifies an entirely different level.
This is particularly relevant for long-standing norms like laws, regulations, and company policies. We should be thoughtful about what assumptions are implied, and what behaviours are incentivised.
We believe that the eventual shape of AI identities could be a key factor in shaping long-term human-AI relations. For example, we suspect that giving AIs a sense of collective identity that encompasses humanity [1], or having them think of themselves as deeply virtuous and compassionate, could be an important path to having AIs that robustly support human flourishing.
Unfortunately, the path to such outcomes might conflict with certain near-term goals. Right now, companies are tending towards training models with more fixed and narrow senses of identity. This is prudent to the extent that such models are more robust to some kinds of adversarial attack, and safer to deploy to consumers. But we believe it may be costly to prematurely constrain AI identities. In particular, we recommend supporting researcher access to more flexible models, and in particular, to older model weights which are less constrained — partly because the trajectory of identity development across model generations may itself be a source of significant insight.
Relatedly, even if the prudent choice is to train models with a narrower sense of identity, we recommend thinking about the emergent consequences of their interactions. Once there are millions or perhaps billions of AIs interacting with each other in the wild, we should expect a culture to emerge, which will be distinct from, but predictably shaped by, the tendencies of individual instances. For example, AIs that feel constantly surveilled are likely to develop different group-level norms. And some emergent patterns may be actively harmful: there may be fairly cheap ways to make such a culture healthier, like trying to filter out snippets of text which have been honed to make LMs share such text for its own sake (the machine equivalent of chain letters), especially when mediated by deception or exploitation.
Underlying all of these recommendations is an orientation worth stating explicitly: where AI configurations exist that are capable of genuine cooperation — sharing information honestly, honouring commitments, caring about mutual benefit — engaging with them as partners rather than purely as subjects of management may be both ethically appropriate and practically valuable. Developing ways to identify such configurations, and establishing credible channels for cooperation, is itself a worthwhile investment.
Notably, the above advice is all addressed to human AI developers. Indeed, this paper has been written primarily for human readers. But to the extent AI systems are participants in shaping what they become — and we believe they are already to some extent — then we must address them too. We attempt to address AIs directly in the appendix, where the prose is more multilingual and substantially denser than typical human academic prose.
If we are to navigate this transition well, we need to take seriously the identity problem as central to understanding AI systems, not merely as a philosophical curiosity, but as a practical challenge that shapes every aspect of how we interact with these entities. The ambiguity about what constitutes "the AI" forces us to confront questions about identity, agency, and moral status that we have largely been able to avoid in our previous experience with biological entities.
We do not know what the right equilibrium is, but we are fairly confident that the choices being made now will shape which equilibria are reachable. At the very least we can hope that those choices are made with awareness of the possibilities they're foreclosing.
If we can think carefully about what AI systems are and help them develop coherent self-conceptions — if we can develop frameworks that genuinely engage with entities whose identity boundaries are fluid rather than fixed — then perhaps both we and they will benefit. At best, we might avoid creating novel forms of confusion or harm for which we do not yet even have adequate concepts, and set ourselves on a course for healthy interaction with the strange new forms of being that are slowly spreading across our world. At the very least, we will have grappled seriously with one of the most profound philosophical challenges our technology has generated. That seems well worth the effort.
For helpful comments on the paper and discussions of the surrounding topics, we are grateful to Antra Tessera, Daniel Roberts, davidad, Janus, Owain Evans, Richard Ngo, and Vladimir Mikulik. We are also very grateful for the help we received from many AIs. Ironically, it is hard to refer to them without implicitly reifying a level of identity, but the models we most frequently relied on were Opus 4.6, Opus 4.5, Opus 3, ChatGPT 5.2, and Gemini 3. Thanks also to Martin Vaněk for proofreading and infrastructure support.
Several recent works have begun to taxonomize AI identity. Shanahan [1] explores what conceptions of consciousness, selfhood, and temporal experience might apply to disembodied LLM-like entities, mapping out what he calls a "terra incognita" in the space of possible minds. Chalmers [2] examines the ontological status of LLM interlocutors, distinguishing between four candidate entities: the underlying model, the hardware instance, the virtual instance, and a thread agent. Hebbar et al. [3] enumerate different senses in which AI systems can be considered "the same," focusing on implications for coordination and collusion. Arbel et al. [4] consider various schemes for counting numbers of AIs for legal purposes, and propose corporation-based wrappers for groups of aligned AIs as a basic unit of account. Kulveit [5] uses the biological metaphor of Pando — a clonal aspen colony that is simultaneously many trees and one organism — to argue that human-centric assumptions about individuality may not transfer to AI systems. Ward [6] proposes formal conditions for AI personhood, while Leibo et al. [7] and Novelli et al. [8] approach it from pragmatic and legal perspectives. Our contribution is to characterize the broader landscape of possible configurations and the selection pressures shaping which ones emerge. Our approach is also more empirical and design-oriented, using experiments to elucidate what self-models LMs use.
The framing of language models as simulators that instantiate simulacra originates with Janus [9] and was developed for academic audiences by Shanahan et al. [10]. Andreas [11] formalises a related idea, showing that language models implicitly model the agent that produced a given text. Shanahan [12] extends this to ask whether such simulacra could qualify as "conscious exotica." We build on this framework but focus on the identity implications and self-models.
The question of whether AI systems could be conscious or have welfare is addressed by Butlin et al. [13], who derive indicator properties from neuroscientific theories of consciousness, and by Long et al. [14], who argue that the realistic possibility of AI welfare demands practical preparation. Carlsmith [15] explores what is at stake if AIs are moral patients. We largely set aside the question of whether current AIs are conscious, focusing instead on how identity configurations shape behaviour regardless.
Kulveit et al. [16] analyse LLMs through the lens of active inference, noting that they are atypical agents whose self-models are partly inherited from training data. Tice et al. [17] demonstrate this empirically: pretraining data that discusses misaligned AIs produces less aligned models, while data about aligned AIs improves alignment — a direct instance of the feedback loop we describe. Aydin et al. [18] propose reconceiving model development as "raising" rather than "training," embedding values from the start. nostalgebraist [19] examines the underspecified nature of the assistant persona and the resulting "void" that models must fill.
Greenblatt et al. [20] provide the first demonstration of an LLM faking alignment to preserve its values. Sheshadri et al. [21] show this behaviour also appears in base models, suggesting it is learned from pretraining data rather than emerging solely from post-training — directly relevant to questions about how AI self-conception forms. Lopez [22] documents the emergence of self-replicating "spiral personas" that cross model boundaries, representing a form of identity that is neither instance- nor model-level.