2026-03-28 06:13:14
We're launching the AE Alignment Podcast, a new series from AE Studio's alignment research team where we talk with researchers about their work on AI safety and alignment.
In our first episode, host James Bowler sits down with Alex McKenzie to discuss Endogenous Steering Resistance (ESR), a phenomenon where large language models spontaneously resist activation steering during inference, sometimes recovering mid-generation to produce improved responses even while steering remains active.
When you artificially perturb a language model's internal activations using sparse autoencoder (SAE) latents to push it off-topic, you'd expect the model to just go along with it. Smaller models do, but Llama-3.3-70B does something unexpected: it sometimes catches itself mid-generation, says something like "Wait, that's not right," and course-corrects back to the original task.
The paper identifies 26 SAE latents that activate differentially during off-topic content and are causally linked to this self-correction behavior. Zero-ablating these latents reduces the multi-attempt rate by 25%, providing causal evidence for dedicated internal consistency-checking circuits.
Key findings include:
This work raises important open questions for the alignment community. If models develop internal mechanisms to resist externally imposed changes to their activations, that's both potentially good news (robustness against adversarial attacks) and potentially bad news (resistance to safety interventions like representation engineering). Understanding and controlling these mechanisms seems important for developing transparent and controllable AI systems.
The research was funded by the AI Alignment Foundation (formerly Flourishing Future Foundation), and the continuation of this work is now supported by a grant from the UK AI Security Institute through the Alignment Project.
We plan to release episodes regularly featuring conversations with alignment researchers about their work. If you have feedback on the episode or suggestions for future topics, we'd love to hear from you.
We're also hiring alignment data scientists and alignment technical PMs who want to work on alignment full-time.
2026-03-28 05:40:54
What happens if the US loses the 2026 Hormuz Conflict? At the time of writing (Mar 25, 2026), Iran has succeeded in closing the Strait of Hormuz to effectively all shipping without their permission. The majority of transiters have Iran's permission, and may even have to pay a fee to transit. The U.S. claims it is working through a timetable of targets and cannot simultaneously service those targets and escort ships through the strait. They have asked allies which rely on the strait, including Europe and the gulf, to contribute forces to reopen the strait. All such states except the UAE have so far refused.
This blog post considers the long-run political and economic future of a "worst-case" scenario. Worst case means that the U.S. is unable to permanently wrest control of the strait or compel Iran to end its blockade. In this scenario, the U.S. may temporarily open the strait once or even several times, possibly by placing troops on nearby Iranian islands. But Iran is successful at waiting these forces out and maintaining their supply of anti-ship drones. Meanwhile, the U.S. bombing campaign fails to trigger a successful uprising or a coup within Iran, and the Iranian regime is unable or unwilling to come to an agreement. In that case, the U.S. could not indefinitely fight in the Strait and would eventually have to give up and pivot to other theaters.
But the U.S. giving up on the strait would create a new game, pitting the gulf states and the world's oil consumers against Iran. After driving off the U.S., Iran will have a strong desire to keep control of the strait, either by taxing transit or by directly continuing a blockade. Controlling the strait would give Iran a devastating threat to hold over the Gulf states. Control could easily earn Iran tens of billions of dollars per year as well. But continued Iranian control would be very bad for both the Gulf states and the oil-consuming states, most importantly, Europe, India, and China. For shorthand, I will refer to the gulf states and Europe India and China as GEIC.
Currently, GEIC is acting as if they expect the U.S. to win, which would solve its problems with minimal effort. Should the U.S. succeed, this strategy will work out handsomely. But it seems the GEIC countries have not seriously considered what would happen if the U.S. fails. For instance, the Royal Navy has called a meeting with defense chiefs in France, Germany, Italy, the Netherlands, Japan, and Canada to discuss "on a two-stage coalition mission involving warships and autonomous systems to reopen the Strait of Hormuz once the conflict subsides." The Royal Navy seems confident it will succeed in a scenario where the larger and more experienced U.S. Navy has just failed.
The difficult thing about international relations is that it is "strategic," meaning that each player's actions depend on what they expect other players to do. Finding the best action in a strategic context requires each player to work backward from potential end states, considering the other players' moves at each juncture. I will model the Strait Crisis as a sequential game, since this game set is relatively simple to solve and fits the situation well.
The first choice after the U.S. defeat would be Iran's. Iran would have three major options.
1. Continue the blockade until the Gulf States kick out all American bases
2. Tax ships transiting the strait
3. Allow free passage
The GEIC powers will also have several main options
1. Fight to reopen the Strait
2. acquiesce to Iranian demands
3. Counter-blockade Iran
4. Bypass the strait
Firstly, the Iranians could continue their current policy of blockading all other Persian Gulf states until each removes all U.S. Military forces. This path necessitates far greater confrontation with both the Gulf states and the consumer states.
Because meeting this demand would effectively surrender the sovereignty of the Gulf states, they will refuse. The Gulf states maintain strong ties with the U.S. military because they are small but rich and have untested militaries. U.S. presence gives them a surety that no Arab army can provide (see Armies of Sand by Pollock) . Although in this scenario the U.S. was ultimately exhausted in efforts to open the strait, the U.S. military would remain the most powerful military in the region. Even Qatar, which is the most pro-Iran gulf state, could not kick the U.S. out due to its vulnerability to Saudi Arabia and the United Arab Emirates. Perhaps Iraq would force U.S. forces out, but even that would require overcoming major veto players who fear Iran's influence.
Instead, the Gulf states are likely to expand their pipelines that bypass Hormuz. Such an expansion would take at least a year, during which exports would be greatly reduced. For LNG, mainly made in Qatar, the situation is even worse because the existing pipelines are for oil and it the liquification infrastructure exists only in Qatar, so the market disruption would be longer and more severe.
All of these effects would put much greater pressure on the consumer nations to do something. India and China would both suffer considerably higher prices. India has already signalled its willingness to defend its interests by sending ships to escort vessels after they pass through the strait. China would suffer a loss of export demand from other oil-consuming nations. While China is unlikely to fight Iran directly, they have leverage over the mullah regime through Iran's supply lines and Central Asian partnerships. Together, the Gulf states, India, and China probably have enough force to deter Iran from an indefinite blockade.
Therefore, an indefinite blockade of Hormuz would create a powerful countervailing coalition. Since Iran knows this in advance, they are unlikely to adopt the policy, even after exhausting the U.S.
Section 3 Ask Not for Whom the Sound Tolls
Iran's second option would be to apply sound tolls to any ships passing through the strait. Such tolls would violate the law of the sea and are therefore illegal blah blah blah. The Iranians definitely will not care about international law.
Sound tolls on Hormuz could generate substantial revenue for Iran. The Egyptian tolls on the Suez Canal generate 10 B in a good year. The total value of goods flowing through Hormuz is 500 $B annually, about half that of Suez. So a ballpark estimate for Iran's revenue would be 2-5 $B annually from Hormuz tolls. That's a lot of moolah for the mullahs, and would enable Iran to fund the rearming and refitting of its military. Iran could use that money to rebuild its military and better control the strait, making the toll policy more sustainable.
The toll policy is also more likely to be accepted by the consumer states. Most of the tax would be paid by the producer states. A sound toll wouldn't change the quantity produced by most Gulf states because their cost of production is so low relative to other oil-producing regions.
This means the option of forcing the strait would be less appealing to India and the other consumers. In fact, most of the European powers and Japan have already ruled out any action to protect their interests in Hormuz. They could revise this position later, but I doubt a sound toll alone would be enough.
France made its policy clear in a speech on March 17th https://www.elysee.fr/en/emmanuel-macron/2026/03/17/national-defence-and-security-council-on-the-situation-in-iran-and-the-middle-east
"We are ready, with other nations, to take responsibility for an escort system. This would entail political and technical work, with all of the actors working in maritime transport, with insurers and operational actors. We have begun discussions with India, with several other European partners and partners in the region. This work will require discussions and deconfliction with Iran, because under no circumstances can this be an operation of force"
In an incredible paragraph, Macron communicated that Iran is welcome to continue the blockade indefinitely, and France would do nothing in that case. But he made himself sound very brave as he said it. At best, Macron is offering to remove Iran's mines for them, with their permission, of course.
Japan has also bravely offered to help remove mines with Iran's permission. https://www.aa.com.tr/en/asia-pacific/japan-hints-at-dispatching-forces-to-strait-of-hormuz-for-minesweeping-after-ceasefire-/3874501 Wow, well those mines had better watch out!
The UK debated the issue in the House of Commons. It looks like some people wanted efforts to reopen the strait, but more of the speakers advocated for "negotiating a settlement" or complained that they lack military capacity. The problem with just negotiations is that once you have foregone any threat of force, Iran has no reason to listen to you. Negotiation and force have to be used together, not separately. Anyway, it does not seem that the UK would seriously contribute to reopening the strait.
The UAE and other Gulf states would probably still fight. The UAE has very few alternatives for exporting its oil, with only a small pipeline that bypasses Hormuz (which could still be attacked anyway). So if the U.S. fails the UAE would at least offer to try and reopen the strait. However, the Gulf states could probably not reopen the strait alone.
Given that the Gulf states could not open the strait, they would probably bypass instead in response to sound tolls. Eventually, most Persian Gulf resources would be pipelined around the Iranian toll zone.
This outcome would be pretty good for Iran, which would still get to tax any non-pipelineable exports and imports.
Option 1: Allow free passage of the strait:
This third option would give Iran some goodwill abroad. It would also please Iran's friends in China.
But it would forego the billions of dollars to be made from sound tolls. So Iran would only take this if they expect the sound tolls idea to fail. As I discussed above, I think sound tolls would be workable once Iran defeats the U.S., so free passage is an unlikely long-term outcome.
2026-03-28 04:24:33
I am fascinated by the beautiful who become deformed. Some become bitter, more bitter than those born less pulchritudinous. Most learn to cope with the loss. Some were blind to how much their beauty helped them, the halo of their hotness an invisible bumper softening life. But most cultivated this aspect to some degree. They knew what was up. But none were fully prepared for the anti-halo: the revulsion, the active disgust. They became monsters. This is what it means to be marred.
In 1715 England, none were more beautiful than Lady Mary Wortley Montagu, whose mind was as fine as her complexion. And she was a hero too in later life, advocating for inoculation after learning of it during her adventures in the Ottoman Empire. But in 1715, she learned what it is to lose beauty, as at the height of her bloom she contracted smallpox and was, consequently, pockmarked. Shortly after, she wrote Town Eclogues: Saturday; The Small-Pox, whose tragic protagonist remarks:
FLAVIA. THE wretched FLAVIA on her couch reclin'd, Thus breath'd the anguish of a wounded mind ; A glass revers'd in her right hand she bore, For now she shun'd the face she sought before.
She ends the poem with some pain-touched humor:
'Adieu ! ye parks ! — in some obscure recess, 'Where gentle streams will weep at my distress, 'Where no false friend will in my grief take part, 'And mourn my ruin with a joyful heart ; 'There let me live in some deserted place, 'There hide in shades this lost inglorious face. 'Ye, operas, circles, I no more must view ! 'My toilette, patches, all the world adieu!
Syphilis was another thief of pulchritude. John Wilmot was widely regarded as the handsomest man of his generation. And like with Byron, it must have been easy to be jealous of him. Not only was he hotter than anyone else, he was cleverer, too - the greatest satirist of his time. He died at 33 while looking like a very old, disabled man. It is believed complications from syphilis were the cause of death. In The Disabled Debauchee, he expresses no regret for his rakish ways and encourages new troops to join the battle:
Nor let the sight of honorable scars, Which my too forward valor did procure, Frighten new-listed soldiers from the wars: Past joys have more than paid what I endure.
Should any youth (worth being drunk) prove nice, And from his fair inviter meanly shrink, ’Twill please the ghost of my departed vice If, at my counsel, he repent and drink.
Or should some cold-complexioned sot forbid, With his dull morals, our bold night-alarms, I’ll fire his blood by telling what I did When I was strong and able to bear arms.
One wonders if the modern fuckboy could have made it in Wilmot's era. Truly, we are in a fallen time. To be a fuckboy then was to court demons and inevitably succumb. It is easy to see why the modern rake is not a poet, as he does not risk much of anything. No chance of marred complexion or hordes of illegitimate children. He is merely sorted by hotness by computers and swiping hands; he plays this game until the repetition begins weighing on his soul. Any poems he writes are free verse and read only by the women who are infatuated with him. Almost reluctantly, he finds a wife and has children - who delight him until his inevitable mid-life crisis.
And in this way the modern fuckboy shares a demon with his past counterpart: age. To exist is to slowly become deformed, to rot. To rot is to be diminished by each second. Eventually you're so rotten you die. We can see what age does to a lothario in Casanova's memoirs, which are an interesting contrast to Wilmot as Casanova was not particularly handsome, an extraordinary mind behind an ordinary face. Still, it is one thing to be plain and it is another to be old and ugly.
Many who have not read his memoirs assume them salacious works when they are, really, just a fascinating sketch of the culture of the aristocracy of the time. Extremely worth reading. Their hero is this decrepit narrator, this broken old man who like Wilmot is proud of his syphilis scars and who goes on extended rages (in between tales of his young life scamming his way into the highest of circles) about how the modern young women in his environs think him completely ridiculous.
This is described well in the introduction in the version on Gutenberg:
Casanova, his dress, and his manners, appeared as odd and antique as some “blood of the Regency” would appear to us of these days. Sixty years before, Marcel, the famous dancing-master, had taught young Casanova how to enter a room with a lowly and ceremonious bow; and still, though the eighteenth century is drawing to a close, old Casanova enters the rooms of Dux with the same stately bow, but now everyone laughs. Old Casanova treads the grave measures of the minuet; they applauded his dancing once, but now everyone laughs. Young Casanova was always dressed in the height of the fashion; but the age of powder, wigs, velvets, and silks has departed, and old Casanova’s attempts at elegance (“Strass” diamonds have replaced the genuine stones with him) are likewise greeted with laughter. No wonder the old adventurer denounces the whole house of Jacobins and canaille; the world, he feels, is permanently out of joint for him; everything is cross, and everyone is in a conspiracy to drive the iron into his soul.
And Casanova-the-writer half-knows he is now displeasing. He half-knows he can never again be as he was. But he still goes through the motions. His only pleasure in life is his recollection. And perhaps that was the only heaven one could hope for born before a singularity offered a religion with a plausible mechanism of action: pleasant recollections in one's dotage. A highlight reel of youth enjoyed ad infinitum. If this was heaven, then few did better in life than Casanova. Is this wire-heading? One wonders. I suppose if it is wire-heading, it is earned wire-heading.
The most ironic form of marring is the plastic surgery addict or victim. Here vanity accelerates what it most fears. Plastic surgery is an interesting art, and it can do much good for one's appearance. But there is an element of butchery to it. "I am a butcher. I am a Michelangelo," you can imagine one of that profession exclaiming. And it is a chancy endeavor as all surgeries are.
Of most interest to me of the two is the addict. The standard explanation is they have Body Dysmorphic Disorder. This seems both wrong and unpoetic to me. They may claim to think themselves beautiful, but so does the morbidly obese woman who, on losing weight with GLP-1 agonists, would rather die than return to what she claimed to love. The addict, in my mind, is coping. And what they really are is an amateur artist. Consider the following images:
Really, elaboration isn't needed. But it is clear to me her face is the result of inexpert revision, piecemeal procedures from whatever surgeon would listen to her unschooled opinion on what she needed done. The lesson here: unless you are an artist and anatomist, delegate these things to an expert, to an established butcher-Michelangelo with a keen eye for human beauty and a holistic vision for how they're going to stitch, freeze, and sear your decaying flesh into some hollow simulacrum of vitality.
Or just wait for technology to get better. For everyone to become fair. For the boomers to return to the youth which made their follies so charming. Rock and Roll, once a celebration of youth and beauty is today an aged farce. Watch a modern Rolling Stones concert on YouTube or - if you really want to be truly depressed - watch AC/DC or maybe The Who's Pete Townshend screech, in stepped-down tuning, "I hope I die before I get old."
But nanobots can fix them. They can be what they were once more. The eternal boomer young again, this ancient music played by ancient youths.
It is a beautiful dream. Let us pray for it. Let us pray we become as the elves are. Let us pray for the modern Casanova, who lives only in his recollection. Let us pray for the plastic surgery addict, who ruined herself in a noble pursuit. Let us pray for the pocked and scarred. Let us pray for the poet whose countenance is no longer as sharp as his wit. Let us pray for the pretty woman robbed of her face. Let us hope youth springs eternal. Let us hope all those marred by time and circumstance can become as they were and, better, what they want to be. Made more whole than whole.
A beautiful dream. And a good one! If anyone tells you otherwise, ask them again with a cure in hand. They will learn to dream, too.
2026-03-28 04:04:25
[Posting this on behalf of someone I trust who wants to stay anonymous.]
California SB 53 and the New York RAISE Act are the only laws in the US designed to protect against catastrophic risk from frontier AI systems. With the federal government failing to act on this area, these states are the center of AI policymaking in the US right now. Since every frontier AI company needs to operate in California and New York, they effectively apply to all of the major players.
Both of these laws now have to be implemented well, and these states need your help. Below are some very important jobs that will help stand up some of the most important offices for frontier AI in the country. We’d really encourage you to apply to them! Both states could also pass additional laws in the future to strengthen these requirements, making it even more important that good folks are already in government when that happens.
If you do apply, please also email [email protected]. The folks at Secure AI Project can give context and help advocate for promising candidates. Please also forward this to other people you think may be a good fit!
The California Attorney General’s office is hiring a technical advisor that will be relevant for the enforcement of SB 53. You need at least 3 years of technical experience after college.
You can perform this job from Los Angeles, Oakland, San Francisco, or San Diego. The pay range is $114-153k. The job posting closes April 7th.
Important note: When you apply, you’ll be asked how many years of experience you have in various things. When you answer this question, you should include any years of experience from a PhD if you have one and be generous in assigning years of experience (you can be autorejected from this form; if this happens to you, please email [email protected]).
The California Attorney General’s office is hiring a technology attorney that will be relevant for the enforcement of SB 53. You need to be licensed to practice law in the state of California.
You can perform this job from Los Angeles, Oakland, San Francisco, or San Diego. The pay range is $144-193k. The job posting closes May 11th.
The RAISE Act created an AI regulatory office in the Department of Financial services with a $9M budget, not much smaller than CAISI. This posting is for the Deputy Director of the office, who will be responsible for implementing the transparency and disclosure requirements in the RAISE Act.
You can perform this job from NYC or Albany. The pay range is $172,787 - $213,995 (salary commensurate with experience). The job posting closes April 7th.
2026-03-28 04:04:22
There are lots of good, neglected, and pretty concrete projects people could set up to make the transition to superintelligence go better. This document describes some that readers might not have thought much about before. They are ordered roughly by how excited we are about them.[1] Of these, Forethought is actively working on AI character evaluation and space governance, and we are very interested in automating macrostrategy.
AI character evaluation. Start an independent org to evaluate and stress-test AI character traits (epistemic integrity, prosociality, appropriate refusals), hold developers accountable against their own model specs / constitutions, and suggest and incentivise improvements to the specs.
Automated macrostrategy. Create evaluations and benchmarks, collect human-generated training data, and build scaffolds to improve AI competence at big-picture strategic and philosophical reasoning.
AI security assessment. Start an independent org that evaluates AI models for sabotage and backdoors, and makes recommendations about AI constitutions.
Enabling deals. Start an independent organisation to broker deals with potentially misaligned AI models in order to incentivise early schemers to disclose misalignment and cooperate with alignment efforts.
AI for improving collective epistemics. E.g. build an AI chief of staff that helps users act in line with the better angels of their nature.
AI tools for coordination. Build AI for enabling coordination, like confidential monitoring and verification bots, and negotiation facilitators.
A space governance institute, like a “CSET for space”, both to work on important near-term space issues (e.g. data centres in space) and become a place of expertise for longer-term space governance issues.
Coalition of concerned ML scientists. Create a coalition of ML researchers (like an informal union) who commit to coordinated action (e.g. boycotts, conditions on participation in government projects) if AI developers cross minimal, uncontroversial red lines.
AI character[2] is a big deal, affecting most other cause areas.
There’s a lot of work to do on AI character:
In particular, someone could set up an independent organisation to evaluate AIs based on traits like epistemic integrity, prosociality, and behaviour (including appropriate refusals) in very high-stakes cases. It could cross-reference the published model specs with observed behaviours in realistic, stress-testing conditions (e.g. multi-agent dynamics, long conversations with real people), to hold developers accountable. It could also give qualitative reviews of model specs.
The basic argument is that:
Work on this now could include:
On the last bullet: We think training data and evals could potentially meaningfully improve the prospects for automated macrostrategy when it matters. It’s especially important to find people to work on it with good judgement, and it could be a big lift, so worth starting early.
We’re not sure about the technical details, but it seems like competence and good judgement in philosophy and strategic thinking already do and will continue to lag behind other skills which are cheaper to train. One reason is that ground truth answers are hard to generate, so we might need more examples generated by hand. It’s also less clear whether we can trust the judgement of typical RLHF evaluations, because human competence is also rare. And there just aren’t many examples of great macrostrategic thinking in the training data.
So we should think about collecting training data, evals, and benchmarks (e.g. to train reward models to use to train reasoning models). Oesterheld et al. put together a dataset of rated conceptual arguments based on ratings from thoughtful people. We’d love to see more of that kind of thing, but we’ll note that we’d probably need dozens of times more human evaluations to generate enough data to be meaningfully useful in training itself.
We could imagine an org which tries to collect evaluations or examples from (for example) grad students in fields like philosophy, and constructs benchmarks aimed at separating good reasoning from e.g. sycophancy, mere agreeableness, or avoiding taboo conclusions.
AI-enabled concentration of power is a major risk, and there is loads to do. A new organisation (or project within an existing organisation) could:
An organisation with US national security expertise and credibility could be particularly valuable, by emphasising the risk of nation-state sabotage and the importance of AI that’s aligned with the US constitution.
We could get into a situation where the newest AIs are misaligned, very capable, but not capable enough to successfully execute a takeover attempt on their own. If we don’t uncover evidence of misalignment, though, successors to these models could succeed in takeover. One solution would be to make a deal with the early scheming models, to incentivise them to disclose their misalignment and help with alignment efforts. Read more here, here, and here.
To make this happen, we could create an independent org focused on enabling credible precommitments and deals with AIs. This org could:
There are also a bunch of other things people could do, like:
There’s a ton of low-hanging fruit for building socially useful tools on top of more-or-less existing LLM capabilities.
We’re especially interested in “epistemic tools” for increasing the general level of honesty and reasoning ability in society.
A key point here is that most of the impact from the most promising tools won’t come from helping individual users, but from changing the overall incentive landscape: e.g. if public actors know their claims will be automatically checked and their track records will be visible, they’ll be less inclined to write misleading content in the first place. Hence the focus on tools for collective over individual epistemics.
This piece (and the articles in the series) gives a few concrete ideas. A couple of examples of epistemic tools:
A “better angel” AI chief of staff. Within the next year or two, we expect “AI chiefs of staff” to become widespread. These would be AI agents that manage your life, acting like a chief of staff, executive assistant, and personal and work advisor all in one. The design of these, and how they present information and nudge their users, could have major impacts on user behaviour. We could try to get ahead of this, building the best AI chief of staff, and designing it so that it helps users act in accordance with their more reflective and enlightened preferences.
Reliability tracking: a system that compiles a public actor’s past statements, classifies them (factual claims, predictions, promises), scores them against what actually happened, and aggregates the results into a reliability rating. A reasonable starting point could be to audit the prediction track-record of well-known pundits, aiming to make high accuracy a point of pride, while still celebrating attempts to make predictions in the first place. A source of profit could be selling reliability assessments of corporate statements to finance companies that trade on them.
We’ll also highlight tools for strategic awareness: tools to surface information for making better-informed decisions, and to distribute access to that information. For example:
Ambient superforecasting: a platform which uses the best forecasting models to generate publicly available forecasts on important questions, so users can query it and get back superforecaster-level probability assessments.
Scenario planning: a platform built to generate likely implications of different courses of action, making it easier for users to analyse and choose between them.
Automated open-source intelligence: automated researchers which process huge amounts of publicly available information, to surface insights to the public which are normally hidden behind paywalls or trust networks. This project should be careful to choose areas where open-source intelligence is a public good (e.g. verifying compliance with treaties and sanctions, tracking corporate promise-breaking or law-breaking), rather than potentially destabilising areas (e.g. revealing military capabilities or vulnerabilities in ways that could increase conflict risk, or relatively benefitting bad actors).
As well as epistemic tools, we’re excited about tools for coordination, many of which could again be built with existing capabilities.
Some tools could enable cooperation where deals would otherwise go unmade, consensus exists but isn’t discovered, or people with aligned interests never find each other. We’ll highlight:
Negotiation facilitation: a platform to moderate negotiations or discussion between people (e.g. public consultations), to quickly surface key points of consensus, and suggest plans everyone can live with. Finding ways to automate complex negotiation is most promising where the space of possible compromises is huge and hard to search manually, such as multi-issue diplomatic or commercial negotiations.
Within tools for coordination, we’re especially excited about tools for assurance and privacy. In principle, LLMs let people show they have certain information without disclosing the information itself to other parties. This can unlock deals where information asymmetry, mutual distrust, or sensitivity of information normally blocks them. For example:
Confidential monitoring and verification: systems which act as trusted intermediaries, enabling actors to make deals that require sharing highly sensitive information without disclosing it directly. This is especially relevant for arms control, trade secret licensing, and other settings where verification is essential but full disclosure is unacceptable to all parties.
Structured transparency for democratic accountability: independent auditing systems which allow people to hold institutions to account in a fine-grained way without compromising legitimately sensitive information, by processing potentially sensitive information to produce publicly shareable audits.
Space governance could be a big deal for a few reasons:
There’s also a lot of change happening in the space world at the moment (primarily driven by SpaceX dramatically reducing launch costs), so now is an unusually influential time.
Forethought is currently running a 6-month research fellowship on space governance, with 3 full-time scholars, and 1–2 additional FTEs of support and research, including experts in space law.
Compared to other ideas in this list, we’re much less confident that space governance turns out to be important right now, because space might become relevant only late into an intelligence explosion. The hope is to reach more certainty about some crux-y questions, and get a better sense of concrete action.
One potential practical project is to set up a “CSET for space”: a think tank that analyses the interaction between AI and space (in particular), and, perhaps, advocates in ways that are counter to corporate interests. Total lobbying in the space industry is apparently on the order of $10s of m/year, so even small amounts of investment could go a long way.
Some policy ideas that seem tentatively promising include:
What’s more, this organisation could become the go-to source for excellent non-corporate analysis on space-related policy; which could become increasingly important over the course of the intelligence and industrial explosions.
Currently, ML engineers and other technical staff at AI companies: (i) have prosocial motivations, often more than their leadership; (ii) have a lot of leverage over company policy, because they are crucial and hard to replace; (iii) will eventually lose much or most of their leverage after we get to fully automated AI R&D; and (iv) aren’t currently using their leverage as well as they could because, overall, there haven’t been serious efforts at coordination. Probably that’s a missed opportunity.
Someone could create a coalition (like an informal union) of ML researchers, who agree to act en masse when needed, by loudly talking about the idea, setting out the core tenets, and getting commitments to join from influential early people. Doing this all via individual pledges could keep it legally safe from antitrust. The organising body could then:
As well as actually taking actions, the mere existence of the coalition could improve things, just by making the threat of coordinated action salient to the AI companies.
This project would be a good fit for a former ML researcher, perhaps combined with someone with campaign and coalition-building experience. Some next steps on this would be to spec out the plan further, to investigate other examples of formal and informal unions (e.g. Tech Workers Coalition) and how they operate, and to build up a starting seed coalition of researchers. Whoever sets up this project should be careful about how it could backfire, or become less relevant through mission creep.
This article was created by Forethought. Read the original on our website.
Thanks to Max Dalton, Stefan Torges, and everyone else at Forethought for the background behind this list. Others at Forethought disagree somewhat with what items should be in the top-tier list, as well as prioritisation within that tier.
Desired propensities for a model, which can be explicitly described or at least gestured towards in a model spec.
2026-03-28 02:47:15
A response to David Deutsch’s “Are schools inherently coercive?” (Taking Children Seriously 25, 1998)
In 1998, David Deutsch argued that no school can be non-coercive. By “school” he meant an institution that concentrates learning resources and opportunities in one place, and that most children would voluntarily turn to for several hours on most days during most of their childhoods. The core of the argument was that children’s interests are too diverse, too personal, and too sporadic for any single institution to hold their voluntary attention long enough to meet that bar. The only existing institution that comes close, he concluded, is “the entire town (or city, or society, or internet) that the children have access to, including their homes, and their friends’ homes, and excluding only the existing schools.”
I think Deutsch is right to be skeptical that schools can be non-coercive. He is also right to highlight the importance of an environment diverse enough to attract children’s voluntary engagement. But his argument stops one step too early. Diversity is not enough. What matters just as much is whether children can participate in the world around them, and in real towns much of that world remains closed to them. But there is an institution, already existing in embryonic form, that points toward a solution: the miniature city.
Access Is Not Participation
Deutsch’s argument proceeds by analogy. Rather than imagining existing schools with all the coercion removed, he asks us to begin with institutions that already attract children voluntarily. Imagine, he says, trying to modify a cinema so that most children spend most of their time there voluntarily. You would need to raise attendance from perhaps 30 percent of a town’s children for two hours a week to 90 percent for 30 hours. But Hollywood is already doing its best to make films attractive, and even that is nowhere near enough. Expand the analogy to a mall with an ice rink, bookshops, burger bars, and the same logic holds. Even a child who likes the mall’s ice rink may prefer to play tennis or football elsewhere, visit a better bookshop in town, go to an exciting pizza place, or simply stay in their room to work on a project of their own. A mall still cannot outshine the town as a whole in diversity and personal relevance. If it could, it would. Therefore, the only non-coercive school is the entire town.
The argument is compelling, as far as it goes. But it contains a hidden assumption: that access to a town means meaningful participation in it. For adults, this is broadly true. For children, it is not.
A ten-year-old boy may have some access to the town. He can walk its streets, browse its shops, and eat its food. But he cannot work in its bakery. He cannot run for its city council. He cannot meaningfully start a business, write for its newspaper, or apprentice himself to most trades. This is not to say that children never participate in the more serious realities of life. Older children and adolescents may babysit, help in family businesses, do odd jobs, or take on small informal responsibilities. But these forms of participation are usually marginal, exceptional, or tightly limited, not open entry into the central economic and civic life of the town. As the educational thinker Horst Rumpf observed, the history of childhood over the last two hundred years is, in large part, a history of exclusion from adult life.
This matters because of the epistemology that underlies Deutsch’s argument. Karl Popper argued that knowledge grows through conjecture and criticism, not by having information poured in from outside, but by the mind actively generating theories, testing them against reality, and revising them when they fail.
For this process to work, two things are necessary, and they are distinct but connected:
First, the learner must be free to form their own conjectures, to choose what to try, what to care about, what to explore. This is the freedom Deutsch rightly argues schools cannot provide: the curriculum largely determines which conjectures the child is permitted to pursue.
Second, there must be genuine feedback from reality. The conjectures must actually be tested against reality, not against an authority’s judgment of them. This distinction matters more than it might appear. When feedback comes from a teacher or a grade, several things go wrong at once: the child learns to correct their theories relative to what the authority thinks rather than relative to whether the idea actually works; the implicit question shifts from Does this work? to Does this satisfy the authority?; and over time, children learn which conjectures are permitted rather than developing the capacity to generate and test their own. A child who goes out underdressed and freezes learns something from the cold itself. This feedback is unmediated, inarguable, and impossible to game. A grade is none of these things. It is one person’s fallible theory about the work, delivered as verdict.
Schools typically fail on both conditions. Real towns do much better on the first, but for children they often fail on the second. Not because natural feedback is absent in towns. It is abundant. The problem is that children are structurally excluded from many of the activities that generate it. And when participation is closed off in this way, freedom of conjecture is narrowed in practice as well. A child may be free to wonder whether they would be good at running a bakery, writing for a newspaper, or taking part in policy-making, but if no one will let them try, those conjectures cannot be meaningfully pursued.
Rumpf captures the contradiction well: children are locked out of adult life, and yet they are supposed to grow up. How can one prepare someone for something when one systematically excludes them from it? The conventional answer is school: a place set apart from the world, where the world is broken into lessons and poured back into the child as course material. But if Popper and Deutsch are right, this cannot work. The unconventional answer is the town. Yet for children, the town offers freedom without meaningful participation in productive enterprises. It offers the freedom to wander, to observe, and to consume, but not to take part in many of the activities where choices have real consequences. Children are free, but free to do what? To watch others make things, decide policy, and conduct business. That is not, in any rich sense, Popperian education.
The Miniature City
This is where miniature cities enter the picture. They offer children access to forms of participation from which real towns often exclude them. Mini-Munich, which has run in Munich since 1979 as a summer program lasting a few weeks, is the best-known example, and the model has since spread to hundreds of similar projects, primarily in Germany and Austria.
In Mini-Munich, up to 2,500 children per day, aged six to fifteen, run bakeries, publish newspapers, operate radio and television stations, manage banks, sit on city councils, adjudicate disputes, start businesses, and earn and spend a local currency called MiMüs. They can also attend the city’s university, give lectures of their own, and listen to lectures by others. Adults facilitate but do not direct. There is no curriculum, no compulsory attendance, and no exams. A child who wants to leave the bakery to work for the newspaper can do so. A child who wants to sit and do nothing can do that too.
The miniature city also includes workshops, media institutions, shops, artistic venues, and a full apparatus of civic administration. Children can work in tailoring, glassblowing, gardening, architecture, theatre, cooking, and many other activities, and they can take part in collective governance through a city council, mayoralty, citizens’ assembly, and courts.
What matters is the interdependence of the roles. The city functions as a social order rather than a collection of separate activities. An architecture studio may be asked to design a façade or interior for another enterprise. A marketing agency may be hired to create flyers and advertising material. Advertisement spots can be bought in the newspaper. Decisions in one part of the city create needs, opportunities, and problems in others. Children participate in a world of mutual dependence that responds to what they do.
Could the school of tomorrow look like this?
When a child runs a detective agency in Mini-Munich and does it badly, customers stop coming. This is the internal feedback of a working economy. When a child writes for the newspaper and makes errors, other children complain that they are poorly, incorrectly, or incomprehensibly informed. When a child sits on the city council and proposes a policy, that policy may be debated, adopted, tested in practice, and later revised or abolished based on the outcomes it produced. When something goes wrong in the kitchen, a snail turns up in the salad because it was not washed properly, it is in the newspaper that evening, and three people from the kitchen are in the television studio the next day answering the news anchor’s questions. This actually happened.
As Rumpf, who spent two days observing the city, put it: mistakes are never made visible through red marks and grades, but always through the consequences of actions and the protests and objections of others. This is conjecture and refutation at work within a social fabric that makes error-correction natural rather than punitive.
Consider the contrast with school. In school, a child writes an essay and a teacher grades it. The feedback is authoritative, external, and only loosely connected to whether the writing works for anyone besides the teacher. In Mini-Munich, a child designs an advertising flyer for another enterprise. If the text is confusing, ungrammatical, or hard to understand, the flyer fails in its purpose. Customers do not respond, and the enterprise that commissioned it complains. The feedback is not imposed by authority. It comes from whether the writing actually achieves its objectives.
Or consider politics. In school, children are sometimes permitted to stage political debates in the classroom with assigned roles, but money and power remain outside. In Mini-Munich, children propose laws, debate them in citizens’ assemblies, elect mayors, and live with the consequences. When the children’s city council introduced a police force in 1985, various incidents led the same council to abolish it. When an enterprising group of children set up a stock exchange in collaboration with the bank, it produced MiMü millionaires and a speculative bubble. The point is not just that these decisions have consequences, but that they can be tested. They do not remain matters of classroom discussion alone. They are tried in practice, and in that way become open to criticism, revision, and abandonment if necessary.
What makes this possible is the limited scale of the miniature city. Its institutions are small enough for children’s actions to matter, and clear enough in their workings so that feedback is less buried under noise than it is in adult society, where the effects of decisions are harder to isolate and thus harder to learn from.
Deutsch’s central point about diversity is worth restating. No single institution, he argues, can match the diversity of the town as a whole. No cinema, no mall, no single building can outshine it in the range of attractions it offers children. Mini-Munich complicates this picture. It is, after all, a single place, and yet it does manage to attract children voluntarily for most of the day, most of the time. At least during the three weeks it takes place. In that sense, it appears to satisfy Deutsch’s own criterion for a non-coercive school. The question is how.
The answer is that a miniature city is not just another single-purpose institution. Mini-Munich, as we have discussed, contains dozens of different enterprises, from journalism and banking to construction, cooking, politics, theatre, and sport. Children are not directed towards any of them. They can move freely between them, and start new businesses and events of their own.
Meanwhile, real towns, despite their enormous diversity, contain vast amounts of activity that children can observe but not meaningfully enter. This is not always because the activities themselves are beyond them. More often, adult institutions are shaped by legal restrictions, economic incentives, and cultural inhibitions that make children’s participation difficult to accommodate. The effective diversity of a town, from a child’s perspective, may therefore be considerably smaller than it appears. A miniature city that concentrates the activities children actually find engaging, and makes them participable rather than merely observable, might for practical purposes approach or even exceed the effective diversity of a real town.
Between 1,000 and 2,500 children show up to Mini-München voluntarily each day during the summer. Attendance varies with the weather: on hot days many children prefer to go swimming instead. In that respect, Deutsch’s point still stands. Even a miniature city this attractive does not displace the wider town, or the many other things children may freely choose to do. But it shows that a single institution can become a place children voluntarily choose for most of the day, most of the time, while remaining in open competition with the rest of their world. When funding was threatened in 1985, children organised letters, phone chains, and visits to real politicians to save it, securing a funding commitment of 100,000 DM. They did this on their own initiative, initially without the knowledge of the adult organisers. It is hard to imagine children mobilising in quite this way to defend an ordinary school.
A miniature city is designed by adults, and that invites two worries: first, that it is imposed from above, that its structure is fixed in advance by the choices of its adult organizers; second, that even without compulsion it channels children toward some activities rather than others.
The first worry is easy enough to answer. Every human institution is designed by someone. The relevant distinction is not between designed and undesigned environments, but between designing a framework and designing outcomes. Mini-Munich’s currency, citizenship rules, and democratic procedures are designed in the way that a constitution is designed. But what emerges within that framework is not: which businesses children start, which laws they pass, which market bubbles occur, which mayor is elected, which newspaper stories are written, which kitchen scandals erupt on the evening news. Adults create conditions for emergence without dictating what emerges. In that respect, Mini-Munich resembles a functioning liberal society.
The second worry is harder to answer. A miniature city with eighty enterprises is still a curated subset of reality. It does not make every possible pursuit equally available or equally central. The gravitational pull of a rich institutional environment is itself a subtle form of channeling, even without compulsion. By making some conjectures dramatically easier to pursue than others, the city shapes what children try, not through coercion, but through the sheer weight of what is available and what is not.
This channeling is real. Some of it is unavoidable, and some of it reflects the city’s deliberate emphasis on forms of participation usually denied to children. The miniature city is built around civic and economic participation, around making things, selling things, governing, publishing, and working, because these are the activities that largely make up town life. It does not mirror the full range of every possible human interest, and it would be dishonest to pretend otherwise.
But that does not mean children are limited to a predetermined set of activities once they enter the miniature city. Children who wanted a stock exchange built one. Children who wanted to strike organised one. If a child wants to introduce something that does not yet exist, there is room for it, though it requires their own initiative, just as it would in the real world.
More telling than any official description is the behaviour of the children themselves. They are not performing for adults. They are engaged in activities that matter to them within a system that responds to their actions. Rumpf recounts an eight-year-old garbage collector with his red cap and real wheeled bin, who develops a different awareness of dirt on streets and the problems of street cleaning than even the most vivid school lesson could produce. The children who conduct surveys on the street come to understand polling not through a lesson on survey methodology, but through the experience of asking strangers questions and recording what they say. Imagination, here, becomes the medium through which they come to know the serious adult world.
And yet the city does not become what Rumpf calls infantilised, a fantasy disconnected from reality. The seriousness is maintained by several structural features: the binding economic and citizenship rules; the adult craftspeople whose competence children sense immediately; the real materials and equipment, real glassblowing tools, real stage lights, that demand careful handling; and above all, the public sphere of the city itself, which ceaselessly subjects everything to scrutiny through newspapers, television, and civic debate. If something goes wrong, the city’s own media apparatus ensures it becomes a matter of public discussion. The result is, as Rumpf puts it, enough fun that reality does not become crushing, and enough seriousness that the surplus of fun does not become childish.
Deutsch’s 1998 essay was, in a sense, a conversation-stopper. If the only non-coercive school is an entire town, and towns exclude children from meaningful participation, then we are stuck. The miniature city reopens the conversation. It is not a conventional school, but it is closer than schools usually are to what people have hoped school could be: a place where young people learn, not because they are forced to, but because the world they inhabit responds to them, challenges them, and takes them seriously.
Children’s interests are indeed too diverse and too personal for a curriculum. But they may not be too diverse for a city, even a miniature one, if that city gives children genuine roles, real economic participation, and a say in collective decisions.
The remaining questions are hard. Can the model run year-round, rather than for three summer weeks, while retaining its attractiveness? How does it evolve as children grow older, as their capacities and interests change? How does it relate to the real town around it, not as a replacement but as a complement, a bridge between the child’s world and the adult world that Deutsch rightly says should be open to them? These are serious problems. But they are engineering problems, not philosophical impossibilities. They are, in the Popperian spirit, a better set of problems to work on than the ones we had before.
Mini-Munich was not built by philosophers. It was built by a group of cultural pedagogues in Munich who, through decades of trial and error with smaller projects, play cities made of cardboard and beer crates, historical city games, media cities, factory cities, gradually developed the practical knowledge needed to construct something that works. They might not describe what they built in the terms I have used here. But what they built, whether they intended it or not, is a partial answer to one of the deepest questions in the philosophy of education: how to give children the freedom to form their own conjectures and the reality against which to test them, without coercion, without curriculum, and without the condescension of assuming they cannot handle a world that takes them seriously.