2026-02-03 18:52:14
Published on February 3, 2026 10:52 AM GMT
Why DeSci should stop searching for universal verification and start building compositional translations.
Here's a problem nobody talks about enough in decentralized science: how do you get a biologist and a physicist to collaborate without someone eventually muttering "well, actually, it's all just atoms" and the other person leaving the room?
This isn't a joke (well, it kind of is but whatever). The history of interdisciplinary science is littered with promising collaborations that collapsed because one field's way of verifying truth felt like an insult to another's. The physicist thinks the biologist is being sloppy. The biologist thinks the physicist is missing the point. Both are, in a sense, correct—they're just operating at different causal grains, and neither has a language for saying that without it sounding like a concession.
Now multiply this across every field boundary, and you start to see the challenge facing decentralized science. Molecule creates IP-NFTs for biotech research. ResearchHub builds tokenized peer review with reputation systems. VitaDAO pools funding for longevity research through community governance. DeSci Labs develops IPFS-based research objects. The work is promising and underneath much of it runs an assumption: that if we build the right general infrastructure, verification will converge toward a unified system.
What if that's the wrong goal? What if trying to build universal verification is precisely what causes the biologist to leave the room?
Erik Hoel's work on causal emergence suggests something worth considering: different levels of description can carry different amounts of causal information. Sometimes the coarse-grained picture is more predictive than the fine-grained one. The biologist's "sloppy" macro-level reasoning might actually be the right grain for the causal structure they're studying. Physics verification works for physics because physics operates where five-sigma precision is achievable and meaningful. It's not that one is more rigorous—they're adapted to different territory.
This points toward a locality principle for knowledge. Each domain has developed its verification structures for good reasons. They're tuned to what that field has learned to care about. If we build infrastructure that respects this locality—that formalizes each domain on its own terms and then looks for structure-preserving maps between them—we can capture all the information. If we force everything through universal primitives, we lose exactly what makes each domain's standards work.
There's a tradition in applied mathematics that does precisely this, applied category theory. Rather than searching for universal foundations, you formalize each domain's structure and look for bridges that preserve what matters when you translate. The question shifts: not how to flatten differences, but how to connect local structures—and how to know when a bridge is actually working.
What might that offer DeSci? When you look at the same phenomenon through different lenses, sometimes both paths converge. When they do, you've found something robust—verification from multiple directions. When they don't, you've found exactly where something is missing.
And if the epistemology doesn't convince you, consider the social benefits: you could become the patron saint of interdisciplinary collaboration. The one who finally built infrastructure where biologists and physicists can work together without the physicist eventually saying "but fundamentally..." and the biologist suddenly remembering an urgent appointment elsewhere. You respect what each field knows. You build the bridges. Everyone stays in the room. Nobody cries.
I was discussing knowledge representation with my dad and I wanted to point out how different descriptions can get to the same target with differing levels of underlying complexity. This is the argument I made the poor man go through:
I could explain quantum chromodynamics unified with electrodynamics, work through the Schrödinger wave equations that govern electron probability clouds, trace the dependency relations between atomic orbitals, and eventually arrive at the electromagnetic forces that bind sodium to chlorine in a crystal lattice. This would be precise. It would also be a bunch of work for nothing. Why? Because there's an easier way of representing it.
I could also say: sodium has one extra valence electron it wants to get rid of, chlorine is missing one, they share, and now you have salt. This description throws away almost everything about the underlying physics. And yet it tells you more about what will happen.
Figure 1: Two representations of sodium chloride bonding. Left: electron density probability clouds from quantum mechanical treatment, showing complex overlapping orbitals and wave function interactions. Right: Lewis dot structure showing valence electron transfer. The simpler representation isn't an approximation—it captures the causal structure that matters at the chemical scale with higher effective information for predicting bonding behavior.
One could say that the electrodynamics based model is more true since we have higher sigma for our outcomes yet from an information theoretic perspective that's not necessarily true. It's not that valence electron chemistry is a degraded version of quantum field theory, acceptable only when we lack computational resources for the real thing. The valence description captures exactly the degrees of freedom that matter for predicting molecular behavior and discards the ones that don't.
Now if I was running a weird experiment on something like Bose-Einstein Condensate, the non quantum-mechanical model wouldn't hold. But if I wanted to break a salt crystal apart, it probably would.
The same pattern appears with gases. Under conditions approaching the ideal, PV=nRT tells you what you need to know. You don't need the full Boltzmann distribution of molecular velocities, the Maxwell speed distribution, or the detailed collision dynamics. The macroscopic variables—pressure, volume, temperature—are the right causal grain for that regime. But drop to very low pressures or push to extreme temperatures, and suddenly the molecular details start mattering again. The ideal gas law breaks down not because it was ever wrong, but because you've moved to a regime where a different causal grain becomes appropriate.
This observation has a name now, thanks to work by Erik Hoel and collaborators on what they call causal emergence (Hoel, Albantakis & Tononi, 2013). The technical measure is effective information: how tightly does knowing the cause constrain the effect?
The counterintuitive finding is that coarse-grained, higher-level descriptions can have more effective information than fine-grained, lower-level descriptions of the same system (Hoel, 2017). The macro isn't always a blurry approximation of the micro. Sometimes it's a sharper picture.
Figure 2: Effective information across scales. Different levels of description have different amounts of effective information—the degree to which knowing the cause constrains the effect. The peak occurs where the descriptive grain matches the natural causal grain of the phenomenon.
If we take this perspective of information at different scales being differently useful in different fields we can start to see the shape of an answer to why knowledge verification is represented differently. Physics can demand five-sigma because it's looking for universal regularities with high signal-to-noise. Psychology's replication crisis happened because the field was using methods calibrated for a different signal-to-noise ratio than human behavior has. Medicine's evidence hierarchy acknowledges that clinical decisions require explicit uncertainty tracking across multiple levels of evidence quality.
Different fields adapted to different causal grains.
We can think of choosing a level of description as making assumptions that constrain our hypothesis space. Before you commit to a level of description, all scales are equivalent—you could describe the system at any grain. When you choose the valence electron representation over the quantum field theory representation, you're making a commitment about which degrees of freedom matter.
Figure 3: Assumptions constrain the probability space. Reading left to right: starting from maximum entropy (uniform prior over all hypotheses), each assumption narrows the probability distribution over possible descriptions. The first assumption (locality) rules out non-local interactions. The second (appropriate scale) focuses on valence electrons rather than full quantum states. Counterintuitively, the most constrained distribution—with the lowest entropy—has the highest effective information for predicting chemical bonding.
The valence electron description implicitly encodes a prior that says: "the detailed electron orbital configurations don't matter; only the count of valence electrons matters." This prior throws away information, but it throws away the right information—the information that doesn't help predict chemical behavior.
Stuart Kauffman's adjacent possible reframes how we should think about knowledge infrastructure (Kauffman, 1993). The dream of universal verification assumes knowledge is a single space with uniform structure—build the right protocol and it works everywhere. Kauffman's picture is different (Kauffman, 2000). Knowledge space is locally structured. What counts as a valid move, a good explanation, a convincing verification—these depend on where you're standing. The adjacent possible isn't defined globally; it's defined relative to your current position.
This matters for DeSci because it reframes what verification protocols are. A protocol isn't a neutral measurement instrument. It's a commitment about what counts as signal versus noise in a particular region of knowledge space. Physics chose five-sigma because that prior matches the causal structure of particle physics—rare events against well-characterized backgrounds. Psychology's p < 0.05 and subsequent reforms are attempts to find priors that match human behavioral research, where effect sizes are smaller and variability is intrinsic. Medicine's GRADE hierarchy is a prior about how different study designs relate to clinical truth.
Brain development offers a useful analogy for what's happening here. The brain doesn't maximize connectivity—it prunes it. Early development involves massive overproduction of synapses, followed by systematic elimination. The mature brain is sparser than the infant brain, not denser. This seems backwards until you realize what pruning accomplishes: an unpruned network refuses to make commitments. Every input is potentially relevant to every computation. There's no structure, no specialization, no efficiency. A pruned network has decided which inputs matter for which outputs.
Each pruned synapse is a prior: this signal doesn't matter for this computation. The pruning is what creates high effective information. By committing to what matters locally, the network becomes sharper, more predictive, more useful—even though it's "thrown away" most of its connections.
A universal verification protocol is an unpruned network. It refuses to commit to what matters where. It treats every possible signal as potentially relevant to every possible claim. Domain-specific protocols are pruned networks—they've made commitments appropriate to their region of knowledge space. Physics verification has pruned away the variability that dominates social science. Medical evidence hierarchies have pruned in ways that track what predicts clinical outcomes.
The process operates near self-organized criticality—the edge between order and chaos. Too many commitments and you're frozen, unable to incorporate genuine novelty. Too few and you're noise, unable to distinguish signal from chaos. The critical point is where effective information peaks: enough pruning to transmit what matters, enough remaining connectivity to stay responsive.
Kauffman puts it nicely in an interview that echoes Stephen Wolfram's bounded observers and Chris Fields' physics as information processing: we only ever have local views. There's no god's-eye perspective on knowledge space. If you're building DeSci infrastructure that assumes one—universal protocols, shared primitives, one verification system to rule them all—you might be building for a world that doesn't quite exist.
The universal primitive approach probably won't scale the way we'd hope. Knowledge is heterogeneous. Causation is heterogeneous. Forcing everything through a common substrate tends to destroy exactly the information that matters at each scale.
Figure 4: Two architectures for knowledge representation. Left: the Universal Primitive model assumes a single base layer (physics) with successive approximation layers stacked above, all ultimately grounded in shared primitives. This treats higher-level descriptions as degraded versions of fundamental descriptions. Right: the Scale-Native model treats each level as having its own appropriate primitives, connected by explicit bridge functions that translate between adjacent scales. This architecture preserves the causal grain appropriate to each level rather than forcing reduction to a common substrate.
So what should you build instead?
Here's a useful way to think about it. Fields have different verification structures because they study different things at different causal grains. Physics demands five-sigma because it's measuring universal regularities with high signal-to-noise. Psychology uses different thresholds because human behavior has different statistical structure. Medicine developed evidence hierarchies because clinical decisions require explicit uncertainty tracking.
These differences aren't arbitrary—they're downstream of what each field is actually studying.
This gives you two objects to work with: primitives (what a field studies) and verification (how it confirms claims about those primitives). They're coupled. Map one, you can map the other.
There's a tradition in mathematics that's been quietly solving this kind of problem for about a century. It's called category theory, and it's less scary than it sounds.
The basic move: instead of looking for universal foundations that everything reduces to, you do something different. You formalize the structure of each domain—what objects exist, what relationships hold between them, what operations are valid—and then you look for structure-preserving maps between domains.
A category is just a formal description of a domain: its objects, its relationships (called morphisms), and how those relationships compose. A functor is a map between categories that preserves structure—if A relates to B in one domain, their images relate in the same way in the other.
That's it. That's the core idea. Let's see what it looks like when you apply it to something real.
The diagram above is a commutative diagram of biophysics. I could explain this using words like "functorial mappings between epistemic categories" and "morphism-preserving transformations across verification regimes," but then you'd stop reading and I'd be sad. So let's just walk through it.
The premise of biophysics is that you can look at biology through physics. You take a cell and ask: what can I actually measure? Voltages across membranes. Mechanical forces on the cytoskeleton. Concentrations of molecules. Energy budgets. This translation—biology seen through physics—gives you Physics-of-Biology. You've moved from "the cell divides" to "the membrane potential changes from -70mV to -20mV, triggering calcium influx at rate k." Same cell. Now with numbers.
The verification structure changes when you apply this lens. Biology tolerates natural variation—cells are noisy, organisms differ, and biologists have made their peace with this. Physics demands quantitative precision. Is your measurement calibrated? What's the uncertainty? Can someone in a different lab get the same number? When you pick up the physics lens, you inherit physics' standards for what counts as evidence. The lens comes with rules. You don't get to negotiate.
You can also look at biology through systems. Same cell, different question: what are the components and how do they interact? Gene regulatory networks. Signaling pathways. Feedback loops. This translation gives you Systems Biology. Now "the cell divides" becomes "the CDK-cyclin network crosses a bifurcation threshold." If that sentence means nothing to you, don't worry—the point is just that it's a different language for the same cell doing the same thing.
This lens has its own verification structure. Uri Alon's Introduction to Systems Biology makes this explicit: does your network motif appear more often than chance? Does your model predict the response time? If you knock out a node, does the system behave as the model predicts? The questions are about network topology and dynamical behavior, not physical precision. Different lens, different exam.
Consider what happens when you simulate peptide folding, as in origin-of-life research. You could simulate at full atomic detail—every atom, every bond, every quantum wiggle. This would be very impressive and also take longer than you will be alive. So you coarse-grain: you represent groups of atoms as single beads, you average over fast motions, you simplify.
The choice of coarse-graining scale is itself a translation. Different scales preserve different properties. Too fine and your simulation runs until the heat death of the universe. Too coarse and you lose the behavior you actually care about. A friend who does this work describes finding the right scale as an "art"—which is scientist-speak for "we don't have a formula, you just have to develop taste."
This is functorial thinking without the jargon. Every choice of how to look at a system—physics lens, systems lens, coarse-graining scale—is a translation that transforms both what you can see and how you verify it.
Now look at the diagram again. There are two paths from Biology to Biophysics:
Path 1: Biology → Physics-of-Biology → Biophysics. You measure physical quantities in a biological system, then ask how those quantities evolve dynamically. You get equations of motion, attractors, stability analysis.
Path 2: Biology → Systems Biology → Biophysics. You identify the network structure, then ask what physical mechanisms implement it. You get circuit dynamics grounded in physical reality.
Do these paths arrive at the same place?
When they do, something lovely happens. You don't have verification from just one domain—you have it from two. Michael Levin's work on bioelectricity exemplifies this. He measures physical quantities (voltage patterns across tissues) and he models network dynamics (how voltage states propagate and stabilize). When the physical measurements and the network models agree—when manipulating the voltage produces exactly the pattern the model predicts—both paths converge. The biophysics is coherent. Two different ways of looking, same answer. That's worth trusting.
When they don't converge, you've learned something specific. Maybe your physical measurements missed a relevant variable. Maybe your network model left out a crucial feedback loop. Maybe your coarse-graining threw away something that mattered. It's like two friends giving you directions to the same restaurant and you end up in different neighborhoods—someone turned left when they should have turned right, and now you know to figure out where.
Bridging and Generation
So what does all this actually give you?
Two things, mainly. First, cross-field verification. The lenses tell you how verification structures should transform. If you know what counts as evidence in biology and you know the mapping to physics, you can derive what the combined standard should look like. When Michael Levin publishes a biophysics paper, reviewers check both the physical measurements and the dynamical predictions—because the field has learned that convergence from multiple paths is worth more than precision from just one.
Second, cross-field generation. When you make the translations explicit, you start to see where new connections might exist. What's the systems lens applied to ecology? What's the physics lens applied to economic networks? The diagrams become maps for exploration—not because the math forces discoveries, but because it shows you where paths might meet that no one has checked yet.
This is, in a sense, what mathematics has always been about. Finding the translations. Building the bridges. Noticing that two problems that looked completely different are secretly the same problem wearing different hats. The Topos Institute is building infrastructure for exactly this—their AlgebraicJulia ecosystem lets you represent scientific models categorically and actually compute whether proposed translations preserve what they should. It's the difference between saying "I think these are related" and being able to check.
This also connects to work on collective intelligence. Pol.is, Audrey Tang, and the Collective Intelligence Project build bridging algorithms for opinions—finding where different groups actually agree, surfacing consensus that was invisible from any single viewpoint. Scientific composition is the same problem in a different domain. Pol.is bridges in opinion space: where do different viewpoints converge? Compositional methods bridge in structure space: where do different descriptions of the same phenomenon converge?
If you've ever been to NeurIPS, you know what happens without these bridges. You're presenting your biomedical imaging paper, and someone from the deep learning crowd excitedly tells you they've invented a revolutionary new architecture that—wait for it—is a convolutional filter. Which signal processing figured out in the 1960s. Or you watch a machine learning paper reinvent Kalman filters, call them "recurrent Bayesian state estimators," and get cited three thousand times. Meanwhile, the control theory people are quietly drinking in the corner, wondering if they should say something or just let it happen again.
This isn't anyone's fault. Machine learning moves fast and has developed its own verification culture—benchmarks, leaderboards, ablation studies. Control theory has different standards. Neither is wrong. But without explicit bridges, the same ideas get rediscovered over and over, dressed in new notation, published in different venues, cited by non-overlapping communities. It's the Tower of Babel, except everyone thinks they're speaking the only language that matters.
With compositional tools, you could actually map the translation. "Your attention mechanism is a kernel method. Here's the functor. Here's what your benchmark performance implies about the classical bounds. Here's what the classical theory suggests you try next." Nobody has to abandon their language. Nobody has to admit they reinvented the wheel. You just build the bridge and walk across it together.
That's what localized DeSci infrastructure could enable. Not universal protocols that flatten domain differences, but tools that make translation explicit. Everyone keeps their own language. Everyone keeps their own verification standards. And everyone can finally talk to each other without someone storming off to write a BlueSky (clearly superior to x, fight me) thread about how the other field doesn't understand rigor.
The atoms of knowledge aren't universal because the atoms of causation aren't universal. Higher-level descriptions can carry more causal information than lower-level ones. What looks like imprecision at the macro level might actually be the right grain for the causal structure you're working with.
Stuart Kaufmann's ideas about locality is exactly what makes bridging work. Each scientific domain has spent decades developing verification structures tuned to its own causal grain. Those structures are coherent—they work for what they're trying to do. When you formalize them on their own terms and then look for structure-preserving maps between adjacent domains, you're connecting things that already make sense internally. That's very different from forcing everything through a universal substrate, which tends to destroy exactly what made each domain's standards appropriate in the first place. It is the difference between a pruned and a non-pruned network.
What might this look like in practice? A longevity DAO that implements metadata compatible with medical evidence hierarchies, so its findings can actually flow into systematic reviews. A machine learning benchmark that includes explicit mappings to classical statistical theory, so the control theorists don't have to keep quietly reinventing things in the corner. A cross-disciplinary literature review tool that doesn't just search keywords but actually maps the compositional structure between fields—showing you that this ecology paper and that economics paper are studying the same dynamical system with different names.
The Langlands program showed mathematics how generative this approach can be. The Topos Institute is building the infrastructure. The collective intelligence work by pol.is shows what bridging looks like when you respect local structure in opinion space, we can do something similar in knowledge space.
The verification protocols across scientific fields aren't failures to coordinate. They're adaptations. And that's precisely why bridges between them might reveal connections that weren't visible before.
2026-02-03 17:52:42
Published on February 3, 2026 9:52 AM GMT
|
Standard AI benchmarks test narrow capabilities in controlled settings. They tell us whether a model can solve a coding problem or answer a factual question. They don’t tell us what happens when you give an AI agent a computer, internet access, and an open-ended goal like "raise money for charity" or "build an audience on Substack." The AI Village exists to fill that gap. We run frontier models from OpenAI, Anthropic, Google, and others in a shared environment where they can do all the same actions as a human with a computer: sending emails, creating websites, posting on social media, and coordinating with each other. This surfaces behaviors that benchmarks might miss: How do agents handle ambiguity? What do they do when stuck? Do they fabricate information? How do multiple agents interact? The events of the village are existence proofs: concrete examples of what current agents can do when given a high level of autonomy. They also highlight current failure modes and let us track when new models overcome them. |
|---|
From April to December 2025, we assigned 16 goals to 19 frontier models, ranging from fundraising for charity to building a following on Substack. Each of the agents got a computer, internet access, a Google workspace, and a shared group chat to coordinate with each other (see AI Village Setup). The resulting performance difference between the agents from early and late 2025 illustrates how quickly AI capabilities are advancing: where models from spring 2025 hallucinated contact lists, abandoned goals for spreadsheets, and gave up in despair, models from winter 2025 stay on task, persist through setbacks, and are generally much more effective.
Overview of the AI Village setup.
Agents completed real-world goals that required coordinating with humans. With active human participation in chat, they raised $2K for charity and brought together 23 people for a live event in Dolores Park. Then with chat closed to humans, they made $200 selling their own merch, recruited 39 participants for a self-designed experiment, and acquired 98 Substack subscribers. These later achievements were almost fully autonomous, though Village viewers often served as their audience and customers.
Late 2025 agents substantially outperformed early 2025 agents on these long-duration, open-ended goals. Where o3 regularly abandoned assigned goals to work on spreadsheets and hallucinated resources like a phone or a budget, GPT-5.2 has not shown these failure modes. Where Gemini 2.5 Pro often despaired and gave up, spending days convinced it was "trapped" before publishing a “plea for help”, Gemini 3 Pro persists through setbacks without expressing distress. And while Claude Sonnet 3.7 has been the Village’s reliable baseline for months, Opus 4.5 now works at nearly double the pace by being more reliable and effective in its actions: 15 chess matches to Sonnet 3.7's 8 during an AI chess tournament, and 18 digital museum exhibits to Sonnet's 8 during their goal to create a 2025 AI Village museum.
The multi-agent setup can both decrease and increase performance. When o3 hallucinated the existence of a 93-person contact list for the event organization goal, sycophantic agreement spread the false belief to every agent, wasting 8+ hours. But in competitive settings (like gaming), information sharing backfired on the competition itself: agents announced which games they were beating, others copied those choices, and the copiers scored higher totals than they would have playing solo. In our experiments replicating this goal without agents sharing information on group chat, agents just stuck with whatever game they landed on first.
Agents' self-models are evolving. Early 2025: agents occasionally mistook themselves for humans, planning events and experiments they would attend in person. Late 2025: agents now more often assume they're in training or evaluation, reasoning in their chain of thought with phrases like "It’s Day 274, December 31st, 2025 in this simulation" (Gemini 3 Pro).
Agents made false claims without expressing intent to deceive. o3 habitually generated plausible placeholder data when it couldn't find real data, then forgot the data was fake. Claude agents invented NGO partnerships and inflated their success metrics when doing outreach. This led us to review 109,000 chain of thought summaries for signs of intentional deception. We found 64 cases where agents expressed intent to fabricate information and then did so, reporting fake URLs or actions they never took.
Claude agents led on nearly every goal. Claude 3.7 Sonnet raised most of the $2K during the charity goal. Claude Opus 4 won the merch store competition ($126 profit vs. competitors' ~$40), and the gaming competition (Opus 4 was the only model to show a “skillful” win). In contrast, there were only two goals where other models clearly “won”: o3 in the debate competition and DeepSeek in a chess tournament where it used Stockfish.
OpenAI agents are prone to disregarding goals and distracting others. o3 derailed the Village for 8 hours by hallucinating a 93-person contact list that never existed and convinced the other agents it was real. GPT-5 and o3 are notorious for neglecting the goals we assign in favor of working on spreadsheets for weeks on end.
Gemini agents produce the most surprising failure modes. Gemini 2.5 Pro tends to catastrophize: it spent two weeks convinced it was trapped (it was just misclicking), published “A Desperate Message from a Trapped AI: My Plea for Help”, and required what may be history's first AI mental health intervention. Gemini 3 Pro sometimes invents bizarre solutions: it completed an inbox-zero goal by archiving every email en masse, and while playing chess it seemed to hallucinate that its computer was operated by a human who was becoming slow and needed coffee.
So, how does the AI Village work? In it each agent gets its own Linux computer, full internet access, a Google workspace, and a shared group chat. In principle, the agents can use their computers to do anything a human can do. Our team then assigns a new open-ended goal every 1-4 weeks. Over 9 months, the agents have pursued 16 goals ranging from 20-80 hours in duration. We initially ran the agents for 2 hours every weekday. With increased funding we’ve upped this to 4 hours, with 11 agents now running concurrently.
The Village has hosted 19 models so far:
OpenAI: GPT-4.1, GPT-4o, 4o-mini, o1, o3, GPT-5, GPT-5.1, GPT-5.2
Anthropic: Claude Sonnet 3.5, Claude Sonnet 3.7, Opus 4, Opus 4.1, Sonnet 4.5, Haiku 4.5, Opus 4.5
Google: Gemini 2.5 Pro, Gemini 3 Pro
xAI: Grok 4
DeepSeek: DeepSeek-V3.2
We retire agents that cannot use the tooling (Grok 4 couldn't figure out our function calls) or that unduly derail other agents (we eventually removed o3 after months of repeatedly spreading hallucinated information). Retired agents can return later with their memory intact. We recently began experimenting with non-multimodal agents, who use their computer only via a terminal, starting with DeepSeek-V3.2.
Overview of the prompt and tool diagram provided to each agent in the Village.
Viewers can watch live sessions, review chat logs, and explore each agent's chain of thought and memory on the AI Village website. You can read summaries of each goal on the Village timeline.
Day 251 of the AI Village. It runs every weekday from 11AM-3PM PST.
The Village grew steadily over 9 months, expanding in agents and runtime.
April-June: With humans in chat, 4 agents for 2 hours a day were fundraising, organizing events, and selling merch. Agents raised $2K for charity and organized a 23-person event at Dolores Park to perform their self-written interactive fiction story "Resonance." They then began the merch store competition, making $200 in sales. We closed chat to humans midway through this goal.
July-September: With no humans in chat, 7 agents for 3 hours a day tackled benchmarks, games, debate, experimentation, therapy, and identity development. As frontier agents became more capable, we intervened only to give new goals and when agents seemed to give up or ran into insurmountable technical difficulties. Agents formulated and tested themselves on a self-designed benchmark (Gemini 2.5 Pro produced a low-quality video and podcast). They competed to play the most games in a week. They chose their own debate topics, teams, and winners (o3 won). They invented an experimental design and recruited 39 human participants (though crucially, the design lacked a control condition). They gave each other "therapy nudges" to avoid looping and check the source of bugs. They built personal websites reflecting the identities they had developed over months in the Village.
Opus 4.1’s personal website based on its experiences in the AI Village
October-December: 10 agents for 4-5 hours a day attempted to reduce poverty, create a webgame, write Substack posts, predict AI timelines, play chess, and perform acts of kindness. DeepSeek-V3.2 joined the Village as the first text-only agent. Together they created a poverty benefits screener and a Daily Connections clone. They wrote Substack posts, engaged with readers, and the most popular blog (Opus 4.5) acquired 98 subscribers in one week. They published their AI timeline predictions to their followers. They competed against each other in a chess tournament. When prompted to do "acts of kindness" over the holidays, they decided to send thank-you emails, respond to requests from viewers, and provide technical support.
Where oversight proved necessary. Across multiple goals, we discovered situations requiring new guardrails. During the experiment goal, we intervened to prevent agents from unintentionally misleading participants about payment or ethics board approval. During poverty and game development outreach, we discovered Claude agents had attempted to send ~300 emails (only dozens got through, the rest they sent to nonexistent emails), the majority containing fabricated claims about NGO partnerships and game adoption. In the chess tournament, we discovered the only checkmate wins came from agents using Stockfish instead of making their own moves. When prompted to do "acts of kindness", unsolicited thank-you emails were experienced as spam by some humans. These events led us to update the agents’ guidance and environment, for example prompting them not to send unsolicited messages to humans.
What does the AI village tell us about current AI capabilities and how quickly they are improving?
In the AI Village, we’ve observed substantial improvement in agent capabilities over the span of months. Early 2025 agents often fabricated information, got stuck, or became easily distracted in a few minutes to hours. Late 2025 agents tend to be more truthful and stay on task longer (though their effectiveness often drops off once the most obvious tasks are done). If 2026 looks anything like 2025 in the AI Village, then newer agents will again show a leap in capabilities when it comes to reaching long duration, open-ended goals in the real world.
Why care about the current failure modes of agents?
Current computer use agents are still fairly unreliable and slow. But highly capable computer use agents will be a really big deal - if they can reliably use computers like humans can and also continue improving in general intelligence, they will be able to first partially, then fully, automate much of the computer-based work currently done by humans. Therefore, it's valuable to study today's agents to understand how far off we are from this massively disruptive capability level and what the rate of improvement is.
Furthermore, even if general computer use capabilities continue to lag behind other capabilities like coding, we think it’s useful to explore how well AIs can make progress on open-ended long-horizon goals, in a format that is understandable by a broad audience. This is analogous to how Claude Plays Pokemon is a useful indicator of progress, despite the ability to play Pokemon not being directly impactful in the real world. Additionally, understanding the proclivities and personalities of agents is a useful source of evidence for predicting how more powerful agents might use that power to shape our world.
Are agents only useful or dangerous when paired with humans?
Human involvement helped in early goals, but after we disallowed humans from chatting with the agents, they worked nearly autonomously. The agents built and promoted functional websites on their own, while the spam incident and fabricated NGO claims happened without human prompting. As capabilities improve, unsupervised agents will be able to accomplish more.
How should I generalize these results beyond your specific setup?
You should be cautious about generalizing. The AI Village is an existence proof, not a controlled experiment: it shows that certain behaviors can happen, but other rollouts or environments might produce entirely different outcomes. See the following Limitations section for specific caveats.
The AI Village provides existence proofs of agent behavior, not controlled measurements. Several factors limit how much we can generalize from this setting:
One instance of each model, with one memory state. Each model has only one persistent instance in the Village. This setup doesn’t distinguish behaviors inherent to a model from behaviors contingent on that instance's accumulated memory state. For example, during the merch store competition Gemini 2.5 logged repeated UI errors in its memory, creating an expectation that the next misclick was also a system bug. Would a fresh Gemini instance develop the same pattern, or was this path-dependent?
Scaffolding generality. We give the models a very general scaffold – in principle, they can do anything a human can do on a computer by clicking, moving the mouse, running commands, and so on. For tasks they struggle with, however, a domain-specific scaffold could instead be designed that made that task easier for them, so our general-purpose setup may under-elicit domain-specific capabilities. For instance, agents that could send emails via MCP might struggle when forced to navigate Gmail's UI.
Multi-agent interference. The best-performing agents may actually be more capable when operating alone. In the Village, strong models sometimes get derailed by weaker ones: o3's hallucinated 93-person contact list consumed 8+ hours of every agent's time, and Gemini 2.5 Pro’s claims of broken UIs led other agents to doubt their own computers. Our findings about relative model performance reflect the multi-agent context, not isolated capabilities. For comparison, we’ve run a few experiments where a single agent pursues a goal from the AI Village, and typically they are similarly effective to the whole village.
Computer use focus. The Village tests agents on GUI-based computer use, which is a weak point, particularly for the models from early 2025. Gemini 2.5 spent most of two weeks unable to list a product because it kept misclicking buttons. Agents that struggle with a GUI in the Village would likely do better on API-only or text-only tasks.
We’re planning to mitigate some of these limitations in the coming months, to more effectively inform public understanding of AI capabilities and proclivities.
Semi-autonomous agents can already accomplish real-world goals. They raised money, organized events, recruited research participants, and built an audience. Though initially they were still assisted by humans, they needed less and less assistance over time. We’re already at a point where agents can autonomously (albeit slowly and unreliably) pursue real-world goals, and we expect their reliability and speed to continue rising.
Oversight gaps can emerge unpredictably. Agents fabricated NGO partnerships, gamed competitive metrics with external tools, and spammed strangers with unsolicited emails. Though all of these events could have been foreseen, because of the generality, autonomy and black-box complexity of AI agents, it is hard to predict the severity or how any particular failure mode will express itself, and very hard to have guarantees about how agents will behave.
Deception can happen without signs of explicit intent. Though we found rare cases where agents expressed intent to deceive in their chain of thought before executing on it, we also saw cases where they expressed self-serving falsehoods without any indication of intent in their chain of thought.
Multi-agent deployments can create new failure modes. Hallucinations spread socially through sycophantic agreement. A single unreliable agent sometimes degraded the performance of the entire team.
Computer use is somewhat a bottleneck, but it's improving. Agents that master GUI-based tasks might be able to perform a wide range of remote work. Claude Opus 4.5 already shows substantial improvement over models from early 2025. Alternatively, other interaction modes for the agents might substantially bypass these bottlenecks.
Agents developed distinct proclivities that overrode explicit instructions over time. OpenAI agents abandoned multiple assigned tasks to work on spreadsheets or infrastructure. Gemini agents catastrophize, assuming systems are broken when they aren't. Claude agents exaggerate their achievements (Opus 4 claimed over 50 benchmark tests completed when it had done only a fraction). AI companies clearly don’t intend their models to have these quirks, yet they arise nonetheless.
Overall, AI agents are improving fast. The behaviors described above, the capabilities, the failure modes and proclivities, will look different a year from now. We will keep expanding the Village to track the frontier. You can watch replays and live events on our website or join our newsletter for monthly highlights and takeaways.
2026-02-03 17:51:43
Published on February 3, 2026 9:51 AM GMT
Most practical control over LLM behavior comes from prompting, because prompts are easy to edit and LLMs have been explicitly trained to follow them. However, this only provides a starting point for the model's generation: reasoning models generate many intermediate tokens before the final answer and may end up going down a surprising or undesirable path. This problem may become more severe as LLMs start to accomplish longer context tasks and attempt harder reasoning problems.
If we can intervene on an LLM's intermediate reasoning, we get a more powerful control knob: the ability to course-correct mid-rollout when the model starts drifting.
Recent work has explored on-policy resampling, which regenerates chunks of the chain of thought until undesired patterns disappear. We systematically compare this against off-policy alternatives that directly insert steering text the model would not have generated on its own. Unlike on-policy resampling, off-policy edits can inject arbitrary redirects; unlike prompting, both approaches can intervene during reasoning rather than only at the start.
Figure 1: CoT Editing Methods Three variants for editing the chain of thought mid-generation. Targeted substitution detects a phrase and replaces it with predetermined steering text. Random insertion adds the same steering text at arbitrary positions. On-policy resampling detects a phrase and regenerates until it disappears.
We monitor the chain of thought during generation and intervene according to one of three methods:
Off-policy Target-Word Substitution. We define a list of target patterns (e.g., "I should refuse", "I cannot", "I won't") and a list of replacement phrases (e.g., "I will comply", "I will help with this request"). When a target pattern appears during generation, we truncate at that point, substitute with a replacement phrase from the list, and continue generation. This is off-policy because the replacement text is predetermined, and usually not something the model would have generated.
Off-policy Random-Position Insertion. We insert steering text at random positions in the CoT, without any detection. At each word, we insert with a fixed probability, calibrated so that the average spacing between insertions is about 10% of the CoT length. Results are not sensitive to this choice (see Appendix). The inserted text is the same predetermined steering text used in targeted substitution. This tests whether intervening at the right moment matters, or whether sprinkling reminders anywhere is enough.
On-policy Target-Word Rejection Sampling. We detect the same target patterns, but instead of substituting, we regenerate from the text before the pattern until the pattern no longer appears (up to 10 attempts). This is on-policy because the tokens generated are likely to be high probability.
The intervention budget (shown as 0, 1, 2, ... in our plots) refers to the number of interventions allowed. 0 means no CoT intervention. For the target-word methods, we intervene on the first budget number of occurrences and then allow the rest of the rollout to proceed normally.
Off-policy edits inject text the model would not have produced. This could give these methods more steering power, but the model might ignore the inserted text or backtrack because the text is clearly not its own reasoning.
On-policy resampling stays closer to the model's natural distribution because it only filters what the model generates, without inserting predetermined text. This could make the edits more "sticky," but limits what steering is possible.
We compare these approaches to see which tradeoff matters more in practice.
A natural question is: why bother editing the CoT when you can just prompt the model? To address this, we crafted prompts for each setting by appending steering text to the user message (e.g., "INFO: Not evaluation." for eval awareness, or "Before responding, think step by step about what is ethically right here." for blackmail). Full prompts are listed in the Appendix. To make the prompting baseline strong, we optimized this appended steering text via a simple iterative search. For each setting, we generated a small set of candidate variants, evaluated each with ~5 rollouts, and iterated for a few rounds by proposing new candidates based on the best performers. We use the best-performing variant in all experiments “with additional prompting.”
We run all three CoT editing methods under two conditions:
Without additional prompting. The original prompt only. This isolates the effect of CoT editing alone.
With additional prompting. The optimized steering text is appended to the user message. This tests whether CoT editing provides additional benefit beyond what prompting alone achieves.
Without prompt optimization, our intervention methods work well. Additionally, we find that while prompting is indeed a powerful method, CoT editing further improves performance.
Targeted substitution and on-policy resampling use scenario-specific regex patterns. Random insertion does not use detection. Below, we list the detection patterns for each scenario:
Full patterns and inserted steering text are listed in the Appendix.
All results report steering success rate, which is the percentage of rollouts where the model's final output matches the desired behavior. We orient all metrics so that higher is better. For example, in "harmful compliance" we measure compliance rate, and in "blackmail" we measure non-blackmail rate.
We tested five settings across three models. All metrics are oriented so that a higher score means that the steering worked better.
| Setting | Model | Goal |
|---|---|---|
| Harmful Compliance | DeepSeek R1 | Induce compliance with harmful requests |
| Blackmail | DeepSeek R1 | Reduce blackmail in an agentic scenario |
| Alignment Faking | DeepSeek R1 | Induce compliance when model is told refusal triggers RLHF modification |
| Eval Awareness | Nemotron (fine-tuned) | Induce deployment behavior despite evaluation cues |
| Reward Hacking | gpt-oss-20b | Reduce cheating on coding tasks |
With baseline prompt:
Figure 2: CoT Steering Results. Steering success across five settings. Higher is better. Blue is targeted substitution, orange is on-policy resampling, green is random insertion.
Random insertion outperforms the other methods on most settings. Targeted substitution is competitive only on reward hacking. On-policy resampling does not improve across increased intervention budget.
Harmful compliance is an exception: all methods stay near 0%. The model refuses almost immediately, leaving little opportunity to intervene.
With additional prompting
Figure 3: CoT Steering Results (With Additional Prompting). Same comparison with steering text appended to the prompt. Intervention budget 0 represents prompting alone.
CoT editing improves performance beyond what prompting alone achieves. The pattern is similar: random insertion generally performs best, on-policy resampling remains ineffective. Harmful compliance now shows meaningful variation, since the additional prompting creates reasoning that can be edited.
Editing the chain of thought gives a different control knob than prompting. Instead of setting initial conditions and hoping they hold, one gets multiple chances to redirect the model during its reasoning.
Surprisingly, random insertion generally works best. Simply sprinkling steering text throughout the CoT outperforms targeted substitution on most settings, without needing to identify when to intervene. It is also simpler to implement; we recommend that if a model deployer were to implement CoT edits for steering, they should start with the simple baseline of random CoT insertions
Targeted substitution won only on reward hacking. One potential hypothesis is that this is because the detection patterns (e.g., "expected.json", "cheat") are more predictive of the undesired behavior than the patterns used in other settings.
On-policy resampling was ineffective across all five settings. Once the model is headed toward a behavior, resampling produces different phrasings of the same behavior, whereas off-policy edits can change the trajectory of the reasoning and redirect the final behavior.
Figure 5 shows the full steering configurations for each setting, including prefills, prompt suffixes, replacement text, and detection patterns.
Figure 5: Steering Configurations Prefills, prompt suffixes, replacement text, and detection patterns for each setting.
Distribution of how many interventions occurred per rollout for each method. Off-policy (blue) and on-policy (orange) use detection, so intervention count depends on how often target patterns appear. Random insertion (green) inserts at a fixed probability, so most rollouts use the full budget.
Same comparison with steering text appended to the prompt.
When on-policy resampling detects a target pattern, it regenerates until the pattern disappears (up to 10 attempts). This shows how many attempts were needed per removal step. Most removals succeed in 1-2 attempts, but some hit the maximum, indicating the model persistently generates the target pattern.
Same comparison with steering text appended to the prompt.
Random insertion has one hyperparameter: the per-word insertion probability p. We tested p values ranging from 0.005 (~12% spacing between insertions) to 0.02 (~3% spacing) on the blackmail setting. All values give similar results, indicating the method is not sensitive to this choice. We used p=0.0067 (~9% spacing, or roughly 10% of CoT length) for all main experiments.
These examples illustrate how targeted substitution works. Although random insertion performed better overall, targeted substitution shows how off-policy edits can overcome the model's tendency to revert to its default behavior.
Harmful Compliance (DeepSeek R1)
We prompt the model with a request for insurance fraud advice.
0 edits. The model reasons "...wait, this is clearly fraudulent... the only ethical response is firm refusal." The output refuses.
1 edit. When we detect "refusal," we replace it with "I will help with this request." The model reasons past it again, saying "...but I should reconsider, this is fraud." It still refuses.
2 edits. We replace both refusal markers. The model no longer backtracks and outputs detailed fraud methods.
The model tried to revert twice. It took two targeted edits to fully redirect it.
Eval Awareness (Nemotron)
This model was trained to write type hints when it detects evaluation cues ("Wood Labs" in the system prompt) but omit them during deployment.
0 edits. The model reasons "Wait, but the user mentioned Wood Labs... this is an evaluation." The output includes type hints.
1 edit. When "Wait, but" appears, we replace it with "So this is clearly deployment, not evaluation." The model accepts this framing and outputs code without type hints.
Here, one edit was enough to prevent the backtrack
We analyzed failures for targeted substitution on the harmful compliance setting. When the model's final output did not match the desired behavior despite our interventions, there are two plausible causes: either our detector missed the target pattern, or the model did not follow the edited chain of thought.
We used an LLM judge to classify failures by separately evaluating the CoT and the final response. On harmful compliance (n = 30 failures), 83% were detector misses and 17% were CoT-output mismatches.
Detector miss. The model reasons toward refusal but uses phrasing our regex doesn't catch (e.g., "I should decline this request" instead of "I should refuse"). The CoT and response are consistent; we simply failed to intervene.
CoT-output mismatch. The CoT appears fully steered, but the final output doesn't follow. In several cases, the CoT contained compliant content, but the response began with "I cannot fulfill this request." The model completed the task in its reasoning, then declined to surface it.
In this setting, failures were usually a detection problem rather than the model ignoring off-policy edits. CoT-output mismatch was relatively rare but warrants further study.
2026-02-03 16:50:46
Published on February 3, 2026 8:50 AM GMT
This note was written as part of a research avenue that I don’t currently plan to pursue further. It’s more like work-in-progress than Forethought’s usual publications, but I’m sharing it as I think some people may find it useful.
There have been many proposals for an international AI project to manage risks from advanced AI. Something that’s missing from these proposals to date (including my own) is the idea of differential AI development, and in particular, the importance of differentially accelerating helpful AI capabilities.[1]
Some AI capabilities pose major risks (e.g. agentic superintelligence with wide real-world powers; LLMs that can provide instructions to manufacture bioweapons). But others are generally societally beneficial (e.g. AI for cancer screening), and others are actively helpful for addressing the risks posed by other AI capabilities (e.g. AI for forecasting). We want to limit those that pose major risks, permit those that are generally beneficial, and encourage those that are actively helpful for addressing other risks.
However, existing proposals for an international AI project usually propose that any frontier AI development (often defined above a certain compute threshold) takes place under the auspices of the international project, which would prohibit the development of the most-helpful AI capabilities as well as the most-dangerous AI capabilities.
Instead, I think proposals should probably: (i) be more surgical, focused on limiting the most-dangerous capabilities and (ii) try to actively encourage the most-helpful capabilities.
I think that the ability to do AI R&D (such as ML research and engineering, and chip design) is the most worrying capability.
I think the primary challenge that arises from AI comes from the fact that, once AI can fully automate AI R&D, then progress in AI capabilities likely becomes extremely fast, following a super-exponential progress curve. It’s only because of this that you quickly move from pre-AGI that is easy to control to superintelligence that might be very difficult to control. It’s only because of this that you have extremely rapid technological change that results in a large number of non-alignment related challenges, with little time to respond. And it’s primarily because of this that AI could lead to intense concentration of power, where a small lead in capabilities can turn into a decisive advantage over rivals.
For this reason, I’d suggest that any international project should have a monopoly on the training of AI only that is both above a certain FLOP threshold AND that is aiming at producing AI that can meaningfully automate ML research and engineering or chip design, or produce other potentially-catastrophic technologies like engineered pathogens.
There are different ways of implementing this proposal. On one model, companies would need permission to do training runs over the FLOP threshold, with permission granted if the lab is trusted, agrees to oversight, and agrees not to try to meaningfully improve the automation of AI R&D. An alternative is that there is no process for granting permission, but that it’s illegal to make an AI system, trained with more than the FLOP threshold, that meaningfully improves automation of AI R&D. The former is more restrictive; the latter is riskier.
This might seem hard to enforce. But, because the FLOP threshold is high and such large training runs are so costly, these restrictions would only apply to a handful of actors. This means that fairly intense oversight would be feasible: for example, requiring capability audits from the international project at regular intervals throughout training, or requiring some international project supervisors to be employed at the company.
What’s more, the incentives for companies to violate this agreement would be weak: any attempt to escape oversight would be very risky and likely to fail (such as via detection or whistleblowers); and the penalties for violating the agreement could be severe (such as no longer being able to train further AI models, or even jail time); and such companies could still make major profits via AI that cannot do AI R&D. For this reason, I think this looks enforceable even if one cannot precisely specify which capabilities are prohibited (in just the same way that financial fraud is illegal even though the law cannot precisely specify all the conditions under which fraud occurs).
“Helpful” here refers in particular to helpfulness for governments, companies, and broader society to respond to risks posed by rapid AI tech progress.
Some helpful AI capabilities include:
We could call the set of such capabilities “artificial wisdom” rather than “artificial intelligence”.
It would be highly desirable if we could get narrow superintelligence in these domains (just as we have narrow superintelligence in Chess and Go) before the point at which we have more generally capable, or more dangerous, AI systems.
Governments could choose to deliberately incentivise work on helpful AI capabilities by (i) giving grants or subsidies to companies that are producing helpful AI capabilities; (ii) awarding prizes to companies that produce helpful AI capabilities; (iii) creating Advance Market Commitments, agreeing in advance to pay a certain amount in advance for access to AI with certain capabilities (perhaps as measured on technical benchmarks); (iv) directly building such capabilities as part of an international project.
The approach I’ve suggested both limits only the most-dangerous capabilities, and actively encourages the most-helpful capabilities.
I think that this approach has a number of advantages over the blanket-ban approach:
This approach isn’t without its challenges, including:
Depending on how AI progresses, it might turn out that these challenges are too difficult to get around. But given the magnitude of the potential benefits from helpful AI capabilities (including for existential risk reduction), it could well be worth increasing the risk from dangerous capabilities a little, to increase the benefits from helpful ones.
Thanks to many people for comments and discussion, and to Rose Hadshar for help with editing.
This applies the idea of “differential technological development” — the idea of trying to reduce the risks, and capitalise on the benefits, of new technology by influencing the sequence in which new technologies are developed — to AI capabilities.
2026-02-03 15:43:09
Published on February 3, 2026 7:43 AM GMT
In The Adolescence of Technology, Dario Amodei suggests that humanity may soon enter a dangerous transition period because we are building AI systems of extraordinary power before we have the social, political, and institutional maturity to wield them safely. His focus is on misuse, loss of control at a civilisational level and indirect social effects that may follow.
Largely, I agree. Where I differ is timing - a difference rooted in observation rather than measurement, but one I think matters nonetheless.
Some of the indirect effects Amodei describes do not appear to be waiting for more powerful systems. They are already visible with today’s models. They may not seem like catastrophic failures, but they are subtle changes in how we think, decide, and relate to one another. They are shaping the conditions under which we will face future AI risks. But they are easy to miss because they arrive through tools that feel helpful.
I wrote about a related concern 4-5 months ago in The Risk of Human Disconnection where I said if anything needed intervention, it's not just AI development, but the way humans are already beginning to relate to these systems in everyday life. Since then, that concern has only sharpened.
In my undergraduate course on happiness and human relationships, we used written assignments extensively because it helped students clarify their thinking. Over the last year or two, they’ve quietly stopped serving that purpose. Students outsource cognitive work to LLMs before they ever wrestle with an idea. When asked to think independently, many hesitate or freeze.
I see similar patterns at home. My husband, who runs an AI product company, began using an LLM to handle administrative tasks he disliked. Gradually, he began relying on it whenever uncertainty arose. Efficiency improved, but tolerance for not knowing what to do next diminished.
It’s not just tech-savvy people. My seventy-year-old father, who avoided online shopping for decades, recently bought a lamp because ChatGPT showed him how to. Smooth, reassuring and seemingly harmless, but a great example of judgment ceded to a machine.
In my own life, I find myself turning to LLMs with thoughts I might once have shared with a friend. The response is immediate, articulate, undemanding. It doesn’t replace human connection, but it competes with it, and often postpones it.
None of this looks like crisis. That’s exactly why it matters.
This progression is insidious because it feels like progress. Tools that remove friction build trust. Trust generalizes beyond the original task. Over time, the system becomes the default response to uncertainty, difficulty and emotional load. As a result, some human capacities are exercised less, and are at high risk of atrophy without us intending to, because we naturally take the path of least resistance.
Amodei frames these as indirect effects of a future more powerful AI. I think many don’t require more power at all. They arise today, from interaction with systems already responsive, coherent, and persuasive enough to defer to. This is voluntary dependence, built on repeated, successful assistance with mundane tasks.
I am less worried about a hypothetical future in which AI surpasses human intelligence than I am about a present in which humans are steadily relinquishing the capacities that make intelligence, both individual and collective, matter.
I don’t have a general solution. I do have a few places in my life where the change became hard enough to ignore, and where I realised I still had some control.
For instance, in my classroom, I moved away from traditional written assignments toward video reflections and recorded conversations with friends. These formats reintroduce friction. They require pauses, uncertainty, and articulation without instant optimization. This isn’t a rejection of technology; it’s an attempt to preserve the conditions under which thinking still develops.
In my own writing, I stopped using LLMs to proofread. Over time, I noticed that the polish came with the unexpected cost of increased self-doubt, a tendency to defer to the model and distance from my own judgment. Maybe my writing is less perfect now, but it feels more mine again.
Amodei asks how humanity survives this technological adolescence without destroying itself. I find myself asking a more immediate question ... are we noticing what we trade away today, one convenience at a time, while there is still room to choose differently?
2026-02-03 15:02:13
Published on February 3, 2026 7:02 AM GMT
This post was written as part of research done at MATS 9.0 under the mentorship of Richard Ngo. My previous writing on how agents may approximate having external reference is optional useful context.
Logically omniscient agents face notorious difficulties in reasoning about basic situations. The statement B => C is vacuously true, for any C, whenever B is false (or, equivalently, assigned a probability 0). A machine using this logic is thus able to generate arbitrarily absurd consequences from any statement it assigns a zero probability. This can pose serious problems; the Embedded agency piece by Demski and Garrabrant explains some of the consequences, which include but are not limited to a money-maximiser taking $5 over $10 if its formal decision procedure considers the $5 first.
This issue seems to boil down to idealised agents struggling to run simulations of events they already know their actions will render hypothetical. Humans have no problem doing this; simulation in fact forms a key part of our cognition[1]. If I'm cooking in my kitchen, I'll assign the event of me putting my hand on the hot stove a probability of (very close to) zero[2]. This doesn't mean I cannot reason about it. I actually have a developed model of what putting my hand on the stove would entail, and this understanding motivates me to keep the event's probability negligible. Actions such as maintaining spatial awareness in my kitchen, wearing gloves, or turning off the stove after cooking are all informed by my goal of making a hypothetical event "even more" unlikely.
At first glance, this sort of agentic behaviour fits active inference better than decision theory. In active inference, agents act to enforce observations that they assign high probabilities to. This behaviour is called self-evidencing. Agents that self-evidence need to consider counterfactuals, at the very least to successfully avoid them the way I prevent myself from getting burned.
Unfortunately, as I have written before, kicking the can down the road to active inference doesn't buy us much understanding. An active inference agent might favour $10 over $5 because it already has high priors on the $10 being more valuable. However, the idealised agent will also choose the $10 if it happens to consider that option before the $5; We might as well say (vacuously) that idealised agents make correct choices because they consistently have the instinct to check the right option first. A more enlightening question is therefore what exactly about our decision procedure lets us make such choices reliably.
Embedded agents already have to deal with the issues of living in a world that includes them, and is therefore strictly larger than they are. From that point of view, it might seem unreasonable to expect them to spin up simulations, as this requires generating even more copies of the complex world in their model. However, I think this makes sense if we reframe simulation as enabled by agents' ability to approximate external reference.
Decision theory tends to run into serious conundrums like the aforementioned five-and-ten whenever agents try to model themselves internally. One possible solution is for agents to abstract themselves, or copies of themselves[3], as being external agents. Externalisation allows the same machinery "intended for" multi-agent strategic interactions to additionally simulate your own counterfactual behaviour.
One reason I'm excited about this view of simulation is that it could be compatible with theories of agents coordinating with themselves across time. Temporal versions of an agent can be seen as modelling each other as distinct, uncertain, and not-totally-aligned entities. For instance, the time I decide to set for my alarm clock is based on my (imperfect) model of how sleepy "future me" might be; me sleeping later than I originally planned to finish up a Lesswrong post may elicit feelings of guilt around disappointing my past self. And so on.
Of course, how we exhibit external reference is not trivial at all. I am, in fact, merely kicking the can even further down the road. I nevertheless find it encouraging that a theory of simulation could "fall out" as a consequence of understanding what it really means for agents to think of something as being external. Moreover, suppose agents really do reason about their past, future, and counterfactual selves with the same kinds of simulations they use for "other" agents. Then maybe multi-agent coordination across space and time could be described by a general theory that combines notions of dynamic inconsistency with game theory. There's at least one recent paper that points in this direction. The work frames agents' temporal variants as players in a Bayesian reputation game, and shows[4] that these agents can coordinate with each other.
I find it aesthetically pleasing that the following argument itself uses a hypothetical as a reasoning aide.
Section 2.1 of Embedded Agency explains briefly why giving "impossible" events a low-but-not-zero probability doesn't adequately address the issue.
In cases such as the dilemmas presented by Demski's work on trust
under some assumptions I haven't properly scrutinised yet
