2026-04-03 03:19:26

Crossposted from my Substack.

Epistemic status: in philosophy of science mode.

There’s more and more interest in using AI to do a lot of useful things. And it makes sense: AI companies didn’t come this far just to come this far. Full automation might be underway, depending on a series of constraints. But what I want to talk about here is how to think about using automation for AI alignment.

A while ago, the following Zvi quote resonated:

Automated alignment research is all we seem to have the time to do, so everyone is lining up to do the second most foolish possible thing and ask the AI to do their alignment homework, with the only more foolish thing being not to do your homework at all. Dignity levels continue to hit all-time lows.

The way I read this, automated alignment is essentially equivalent to handing off the most crucial bits of science humanity will ever have to do to highly unreliable intelligent systems and hoping for the best.

I won’t try to assess whether automating alignment is a good idea per se in this post. To the extent that this kind of work is an explicit goal of AI companies and appears in AI safety agendas, I seek to clarify what automating alignment research means, treating this as a metascientific endeavor: theorizing about alignment as a science.

1. Is alignment research special?

Alignment can be understood as a capability that makes AI systems predictable and controllable. In that sense, it’s a prerequisite for deploying any system, not a special add-on or a feature to consider once the system has been deployed and diffused. Importantly, publicly releasing systems in the absence of robust alignment techniques carries a series of risks that scale concerningly with capabilities.

Plausibly, many of the tasks involved in alignment research are typical in software and machine learning engineering in that they require writing and debugging complex code bases, using compute, and securing high-quality training data. The question then is how to make sure that alignment-relevant work progresses proportionately to the rest of AI research that is typically focused on making systems generally capable (also known as differential technological development). It has been argued, for example, by OpenAI, Anthropic, and more recently by Carlsmith, that without the help of AI systems, human developers won’t be able to make the necessary progress in time to release systems that are beneficial for everyone.

What seems different in current deep learning systems, and therefore in current alignment work, concerns their scale: we now have highly complex code bases, large and costly amounts of compute, and vast training data. Automating parts of a team’s workflow would be both instrumental to the acceleration of AI development and deployment, but at the same time, would present higher-stakes challenges.

2. How is AI integrated into alignment research?

It’s not clear at the moment how automation could accelerate or improve alignment work. In particular, there’s a series of questions to think about and answer before being able to evaluate what automating alignment looks like. I group these questions into two clusters.

The first is about capabilities, i.e., what cognitive work AI systems can do. More specifically:

• How useful are models for identifying algorithmic improvements?
• How useful are these models in helping improve code bases that are complicated, especially within AI companies?

The second cluster is about testing and measurement, i.e., if models do assist with research, how do we effectively assess the ways in which they do so? Some questions to ask here are:

• What are the most reliable methodologies to track the productivity of engineers and scientists?
• What does tracking threat vectors from improving AI R&D look like? What are the metrics to do that?
• What benchmarks can be designed for automating alignment research?

With these questions in mind, it’s natural to wonder about alignment as a science (Anthropic notably has an Alignment Science blog). The blockers that appear in human scientific thinking are likely to come up when it’s time for AI agents to take on the role of the scientist.

3. Human science obstacles for non-human minds

Empirical work helps determine what the exemplary problems are, what counts as a solution, and what methods are legitimate. This is Kuhn’s sense of a paradigm. Recently, the discussion about whether parts of alignment research are more or less paradigmatic has received a lot more attention. For example, Lee Sharkey talks about how mechanistic interpretability is no longer pre-paradigmatic, and I have previously written about Artificial Intelligence Safety as an Emerging Paradigm.

There is one straightforward way to test whether the field is now paradigmatic: use currently available research data to train AI alignment researchers. If the result is bad, then it’s either that the models are just not capable enough yet (but perhaps compare them to how they do at other scientific tasks), or that the data used are of low quality. I suspect that at the moment, it’s a combination of the two.

Before Kuhn, logical empiricists pointed to that same problem of epistemic bootstrapping, without, of course, having automated AI agents to do any science for them. Neurath's boat is the classic metaphor here: we are building (or rebuilding) the ship while sailing on it, and so we’re never able to put it in dry dock. This captures the AI alignment endeavor well — most of the time, it’s also pretty stormy.

There’s another bottleneck in accelerating research progress in the idealized sense; there might not be a “logic of scientific discovery” — a formalized recipe for how to do science. This is a long debate in the history and philosophy of science. What matters for the purposes of thinking about science agents is that such formalizations might not exist, at least not in a way that can be given as input from one intelligent system (human) to another (artificial). Scientific processes can be messy and difficult to articulate cleanly. I especially expect that scientific intuitions and research taste are hard to compress, though I also expect that AI systems could develop fast and frugal heuristics and learn from their training data the way human scientists do. This goes more into a comparative cognitive science for human and artificial minds, but there’s more to consider at the theory of science level.

4. What counts as safe AI?

There’s a useful distinction in the philosophy of science and engineering: verification vs validation. Verification asks: “did we build the system right?”, whereas validation asks “did we build the right system?”

For safe AI, we most likely need both: the internal consistency of satisfying verification criteria and the external confirmation that those criteria capture what we actually need an aligned system to be. Verification can look like checking whether RLHF satisfies a given reward specification within a test setting, whether the system behaves in an aligned way. Validation may require a more robust definition of what it means for the model to be aligned, outside of a suite of narrow experiments.

AI systems (perhaps even not completely aligned ones or more tool-like) could assist with verification work. But it seems unlikely that a partially aligned model could be useful for doing validation work. Validation typically requires zooming out, observing how a system operates in different environments, or how it generalizes, and being able to tell whether it has the right objective. In AI safety, the problem of validation is often described in terms of outer alignment, a question about whether the model does what the developer thinks they want the model to do.

Takeaways

• Before evaluating automated alignment, we need to address and answer two sets of questions: 1) can models do the necessary cognitive work and 2) how do we measure how good their work is?
• It might be that key aspects of scientific processes cannot be cleanly formalized. AI systems could still pick up the mechanisms that make human scientists successful (because of deep learning).
• AIs can probably help with verification work that requires internal consistency, but we should be very cautious about validation work that requires zooming out and assessing the system and its operations as a whole.

2026-04-03 03:07:08

There’s a cocktail called an old fashioned. It’s almost as simple as one can make a cocktail. Like anything there’s a million “recipes” but the one I’ll focus on today goes like this:

• Put two sugar cubes in a glass
• Douse them with a few drops of bitters (a very strong infusion of herbs)
• Crush up the sugar cubes into little bits
• Add your bourbon and ice
• Stir
• Garnish with lemon

The end result is rather tasty and a little gritty, from the sugar. Of course it’s gritty! You put smashed up sugar cubes into an icy solution which doesn’t dissolve them! What did you expect?

Nowadays, bartenders use a thick sugar syrup which is pre-dissolved. Cocktails today often have many more ingredients, they’re prepared in one vessel, possibly shaken, and poured into the drinking vessel.

From the name and the preparation, you might think this is one of the first mixed drinks to be developed. And you’d be wrong! The giveaway here is the ice. Ice only reached American bars (bourbon is American) in the 1800s. Before that, people had unchilled drinks. People have been drinking rum punches since the 1600s. Secondly, when ice reached America, bartenders immediately switched from using sugar cubes to using simple syrup, a kind of sugar water, because ice-cold water doesn’t dissolve sugar cubes.

The weird crunchy old fashioned that you (sometimes) see today is an anachronism, invented by modern barkeeps in a “you know, this is how they used to have cocktails, see” marketing ploy.

II

This is a cormorant:

By Andy Reago & Chrissy McClarren - Brandt’s Cormorant, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=67311261

They live by bodies of water: ponds, rivers, and oceans. Most water birds (and most birds in general) have oiled feathers, which keep them dry when they come into contact with water. Cormorants don’t. When cormorants dive into water to hunt for prey their feathers get wet. You’ll often see them standing around after a few dives, sunning their wings to dry them back out again so they can fly

Must be pretty old right? Some early-diverging lineage of birds from before the oils evolved. Their prehistoric appearance certainly adds to the effect. This is what early bird-book authors thought: cormorants would always be put at the start of the book, in their own section.

And they were wrong! Comorants are part of the same order as gannets and pelicans, both of which have perfectly normal feathers. Cormorants lost the oils as a specific adaptation to going underwater. While other birds float aggressively as their feathers hold on to air, cormorants have a much easier time swimming down and staying down to hunt their prey.

This pattern is extremely common in evolution. We don’t even need to look beyond birds to see it again: the paleognaths (ostritches and friends) have a fixed upper beak, while the neognaths (basically every bird you’ve ever heard of) have a mobile upper beak. The “paleo” in paleognaths refers to the fact that the fixed upper beak was thought to be ancestral, but modern analysis (source: my friend who works in bird paleontology) actually suggests that they lost the ability to move it. Perhaps it lets you peck extra hard if your upper beak is welded to your skull.

III

What’s the lesson here? Is this just me writing up a catchy title? I don’t think so. In most cases, it’s much easier to lose a function than to gain one, especially in a noisy, random search process like biological or cultural evolution. It’s much easier to turn off the oil than to evolve the oil; it’s trivial to just stop making syrup from your sugar cubes. If you see something that looks simple, or basic, there’s a decent chance it’s actually descended from much more complex ancestors, and just lost parts along the way.

As a corollary: the easiest way to achieve something might be to lose a function. This even seems to extend to policy: most of the biggest “free wins” available to America and the UK consist of removing policies (the Jones act, the Town and County Planning Act).

I also see this in the personal lives of people around me. How many people around you have an immense free win in their lives by getting rid of something?

Humans tend to have a blind spot against solutions which involve removing things, instead of adding them. I don’t know whether we extend this to our intuitive models of evolution and culture, but it’s possible. Either way, you are probably underestimating the effects of removing things.

◆◆◇◇◇|◇◇◇◇◇|◇◇◇◇◇
◆◆◇◇◇|◇◇◇◇◇|◇◇◇◇◇

2026-04-03 02:50:44

In the month or so around the previous new years, as 2024 became 2025, we were saying “2025: year of the agent”. MCP was taking off, the inspect-ai and pydantic-ai python packages were becoming the standards, products were branching out from chatbots to heavy and autonomous use of toolcalls. While much of the product engineering scene may have underdelivered (in the sense that “planning a vacation” isn’t entirely something most people do with agents yet), the field of FMxAI I think was right on target. Feels like there’s an agentic component to everything I read these days.

What is 2026 the year of? Besides “year of investors pressure all the math companies to pivot to program synthesis”? I’m declaring it now

The number of blogposts relating to secure program synthesis went exponential since the last issue of the newsletter, such that the hyperlinks in this sentence are not even exhaustive. Like AI capabilities themselves, we simply do not know when it will go logistic. Which I think means I have to keep the newsletter manageable by not promising to do the general secure program synthesis or FMxAI spaces and really double down on AI security applications. I mean it, you guys. It’s overwhelming, the space is too big now, etc. But will SPS gains actually get applied to AI security opportunities? That’s the crucial question I think we should care about, now. Let’s assume by default no, so that everyone who wants to liberate my bayes points from me goes and makes it happen.

I moved to DC, and I started an AI Security reading group in meatspace. Reach out to be involved!

There will be hackathons and fellowships in secure program synthesis, FMxAI, AI security, etc. Be prepared.

Sorry again about the long gap between editions. I’ve been writing more than ever, but not a lot of it public yet. Also coding a lot.

Nicholas Carlini’s desperate and pleading talk

Carlini: “Opus is better at finding 0-days than I am”. Carlini, famous for finding vulns in adversarial robustness papers back in the computer vision days, stopped a hair short of declaring a draft, but I’m not. This is a draft. If you’re reading this, you are hereby conscripted into the secure program synthesis army. Or maybe our vibe is more “militia” coded, haven’t made up my mind yet. But this is a draft. The gameboard seems offense dominated right now, we need to harden software, and we need to do it quickly.

According to gemini, who cites forbes: 20-40% of corporate IT budgets in 1999 were devoted to Y2K prep. $300-600b was spent (roughly 0.5-1T adjusted for inflation). Ordinary people, people with families, people who put sandwiches into little baggies, decided to do this. They had consensus that it was coming down the pipe, on a deadline.

I don’t think we have that consensus, I’m not sure we can marshall those resources.

But here look this is all I’m saying

cat << EOF > CLAUDE.md

Make agentic software that hardens a given repo with a red-blue loop. Declare an agent that uses standard redteam tools like fuzzing harnesses and static analysis as tools to find vulns in repos, then declare another agent that patches those vulns. Use git submodules to manage sources

EOF

while true; do cat CLAUDE.md | claude --dangerously-skip-permissions; sleep 1; done

If everyone reading this ran that command, and tailored it to the specific things you happen to know about, and dropped in repos close to your specialization, and steered the project in directions that I wouldn’t think of, etc. we might have a shot. Do not DoS open source maintainers with slop, but consider forking their repos and DoSing your forks with slop then finding a way to benchmark how much more secure your fork is and then maybe you get a few PRs in. Target loadbearing repos. Etc. 🫡

UK AISI call for information on Secure AI Infrastructure

Let’s do it. Doesn’t seem to have a deadline.

Gemini’s summary:

The UK government has launched a “Call for Information” to gather expert insights on securing the computing infrastructure used to develop and deploy advanced AI models. Managed jointly by the Department for Science, Innovation and Technology (DSIT), the AI Security Institute (AISI), and the National Cyber Security Centre (NCSC), the initiative seeks to address growing threats such as the theft of model weights, data breaches, and system disruptions. The government is specifically soliciting feedback from the AI and cybersecurity sectors on current risks and emerging technologies—including confidential compute, advanced cryptography, and trusted hardware—to help shape future research priorities and technical pilots. Ultimately, the program aims to ensure the UK remains a secure and trusted global hub for frontier AI development by building robust, “defense-in-depth” protections for critical AI assets.

SL5 Taskforce has been productive!

Excited about some of these outputs.

Long Live Safeguarded AI

Official leadership update, Kathleen Fisher (ex DARPA, responsible for HACMS) interviews Nora and Davidad on vimeo, Davidad’s notes about what he’s doing next.

Safeguarded AI, like GSAI, like Open Agency Architecture, was always controversial scifi that, to be clear, I’ve been rooting for since day one. Well not literally day one. When I first read open agency architecture, I recall rolling my eyes and wringing my hands “you can’t just put a type signature on a wacky diagram and say that it means AI safety is formally verifiable” but like a few days later I was into it.

With Nora Ammann in as programme director (who’s been behind the scenes on SGAI since before day one), we should expect aggressive AI security applications of formal methods.

Pat Lincoln I2O director at DARPA

Totally should’ve been in the thanksgiving edition, I spaced. SRI alum Pat Lincoln is pretty keen about AGI, he’s not like completely AGI pilled but extremely sympathetic to the viewpoint.

Atlas blog on CSLib

CSLib is what it sounds like: the mathlib of CS. It is spearheaded by Clark Barrett and Swarat Chaudhuri and others. Atlas Computing, which has for a while been looking at how to close the gap between formal methods as we find them today and radical infrastructure hardening, writes about it (they’re directly involved).

CSLib is cool, but one reason I think it doesn’t register as the droid I’m looking for is that I don’t know if real world software really draws from undergrad CS curriculum all that much. In most real world software jobs, having a problem that is shaped like an academic puzzle rather than a REST API is a rare but golden status marker.

VeriSoftBench

Claim: idiosyncrasies of mathlib in particular determine too much of what we think we know about Lean evals and language model performance on them.

500 theorem-proving tasks from 23 real-world projects, spanning compilers, type systems, smart contracts, separation logic, and program semantics.

Recognizing the crucial wisdom of our time (that math is made of tokens which we can use for something else), we get a Lean eval that isn’t focused on math.

muInference: a minimal inference stack on SeL4

In the draft of this newsletter, where I stubbed out writing about this as TODO, the stub consisted of one word:

Dude.

Which I think roughly describes how I felt seeing it and how I feel now. They basically boot a minimal (cpu) inference enclave in seL4. I used it as an enclave in a network simulation project I was doing, and it doesn’t even have a machine-readable TTY, that’s how secure it is.

Progress in the mitigation of steganography

Crucial applications for model weight exfiltration defense!

DiFR: Inference Verification Despite Nondeterminism

Verifying LLM Inference to Detect Model Weight Exfiltration

Seldon Labs batch 2 applications (closed by the time I’m posting this)

Sorry I didn’t get this out fast enough. But keep an eye on Seldon re the general space of AI security startups.

Also keep an eye on: Heron

Met a bunch of these fine folks at EAG.

Lucid Labs: data center access program for hardware related experiments

Summary by gemini:

The Lab Access Program by Lucid Computing is a specialized initiative providing developers and enterprises with early, hands-on access to “sovereign” AI infrastructure built on zero-trust, hardware-rooted security. Designed for industries with strict regulatory requirements—such as defense, healthcare, and finance—the program offers a sandbox environment where users can deploy AI models and agents while maintaining cryptographic proof of compliance and data residency. By moving away from traditional “black box” cloud models, participants can verify that their computations are occurring on specific, untampered hardware, allowing them to benchmark and refine secure AI workflows before moving into full-scale production.

AISLE finds 12 0-days in openssl and doesn’t (appear to) use formal methods

The hero the secure program synthesis community has been waiting for.

Its extremely unlikely that theyre using iris or some synthetic lean-iris under the hood. A year ago at EAG 2025, I went around telling everybody about my budding separation logic agent and that I wanted to find bugs, synthesize patches, and prove the patches correct with it– and I used repos like OpenSSL as hypothetical target. I didn’t continue working on that cuz I didnt like the Iris DSL that it was based on, but I’m not sure if I should’ve tried to continue.

Take a look at aisafety.com for overall fieldmapping

Thanks to Bryce and the folks at aisafety.com, a bunch of you have found this newsletter, where we discuss using formal methods to improve our AI security posture. Many of the rest of you, however, might have come in from the formal methods community and sometimes think “who are all you AI existential safety wackos”. aisafety.com wants to answer that question by making a map of everything that’s going on in the very broad umbrella of AI existential safety. Highly recommended if you’re generally scared about AI but don’t know exactly where to plug in yet.

Allies IFP have new RFP

With a great track record of supporting secure program synthesis, IFP wants to do more projects along the lines of their galvanization of the great refactor.

Kathleen Fisher and Gopal Sarma post op-ed

Let’s ship evals and RL environments now so that the next gen of language models are differentially accelerated to cyberhardening defensive technologies. 10/10, no notes.

2026-04-03 02:35:20

I’ve noticed that sometimes there is an idea or framework that seems great to me, and I also know plenty of people who use it in a great and sensible way.

Then I run into people online who say that “this idea is terrible and people use it in horrible ways”.

When I ask why, they point to people applying the idea in ways that do indeed seem terrible - and in fact, applying it in ways that seem to me like the opposite of what the idea is actually saying.

Of course, some people might think that I’m the one with the wrong and terrible version of the idea. I’m not making the claim that my interpretation is necessarily always the correct one.

But I do think that there’s a principle like “every ~social idea^[1] acquires a corrupted version”, and that the corruption tends to serve specific purposes rather than being random.

Here are a couple of examples:

Attachment theory. People with insecure attachment read about attachment theory, and then what they imagine secure attachment looking like is actually an idealized version of their own insecure attachment pattern.

Someone with anxious attachment might think that a secure relationship looks like both partners always being together, missing the aspect where secure attachment is meant to provide a safe base for exploration away from the other. Someone with avoidant attachment might think that secure attachment looks like being self-sufficient and not needing others, missing the aspect where it also involves comfort with neediness and emotional closeness.

These misinterpretations also get reflected in popular discussions of how to do parenting that fosters secure attachment. E.g. sometimes I see people talk about “secure attachment” in a way feels quite anxious and is all about closeness with the parent, and forgets the bit about supporting exploration away from the parent.

So-called Non-Violent Communication (NVC). NVC is a practice and philosophy about communication, where the original book about it is very explicit about it being something that you do for yourself rather than demanding of others. If someone speaks to you aggressively, you are meant to listen to the feelings and needs behind it rather than taking it personally or blaming or judging them^[2]. The whole chapter on “Receiving Empathetically” is on how to respond with empathy when you are the only one using NVC.

One of the pillars of NVC is also making requests rather than demands. The book says that a request is actually a demand if the other person then gets blamed, judged or punished for not granting the request^[3], and that NVC is not about getting other people to change their behavior^[4].

And then there are apparently some people who are into NVC and aggressively police the language that others use, saying that everyone has to talk to them in an NVC kind of format. Which goes against everything that I mentioned in the previous two paragraphs, as it’s a demand for others to use NVC.

I feel like I often run into various other examples too, but these two are the ones for which is it’s the easiest to point to a “correct” form of the thing. In many other cases, it’s not as straightforward to say that one is a correct version and the other is distorted, as opposed to there just being two genuinely different versions of it.

Emotionally selective reading

There are several different things going on with these. One is that it’s easier to transmit a simplified and distorted version of an idea than the whole package with all of its nuance intact. “NVC is this specific formula for how to express things” is quicker to explain that all the philosophy in the whole book.

Another is that, as you might notice from the anxious vs. avoidant example, is that often the corrupted ideas are pointing at opposite extremes. Each contains a grain of truth, but then exaggerates it to an extreme, or fails to include bits that would be required for a full picture.

I think that’s pointing to the third factor, which is that any new ideas will be filtered through a person’s existing needs and emotional beliefs.

People have various pre-existing ideas of what is good and what is bad. If an idea implicitly says “here’s a theory of what’s good and bad”, a person may subconsciously assume something like “I know that X is good and Y is bad, and this is a theory about what is good and bad, so the theory must be saying that X is good and Y is bad” and come away with a very selective reading of the idea.

On a more phenomenological level, one might say that there will be parts of the theory that resonate with the person and others that don’t. If someone is reading a book, some sentences will feel like the point and some will feel like less essential caveats. “Here’s a form of language that works better” might read as the actionable point, with the “NVC is something you do for yourself” bit being quietly forgotten or rationalized away.

Often, beliefs are adopted not because of their truth value but because they allow a person to do something they wanted to do. The stronger the person’s need to believe in something, the more likely it is that they’ll selectively read ideas like this.

This implies that the corruption is somewhat predictable. If you have a sense of someone’s psychological needs, you might have a sense of how they’ll distort any given framework. An anxious person’s misunderstanding of attachment theory isn’t random, but emerging from their personal psychology.

None of this is to say that the people wouldn’t get genuinely novel ideas from the frameworks. Someone who gets enthusiastic about NVC and starts using it in all their communication isn’t just taking their existing beliefs and rationalizing them. They are learning and doing something genuinely novel, and they have gained a lens for understanding the world that shows them at least some correct facts. But filters in their mind are also systematically hiding awareness of other truths.

The effects of vibe and my own corruption-complicity

I’m now going to flip this and show how I myself might have been doing the exact same thing that I’m criticizing others for.

Because an important complication to what I’ve been saying above is that sometimes the vibe and the explicit message of an idea are in conflict, and the “corruption” may not be so much a literal corruption, but a correct reading of the underlying vibe.

Take Non-Violent Communication. It’s literally called “Non-Violent Communication”, implying that anyone who doesn’t communicate in that way is behaving violently. Here’s how one of the chapters in the book begins:

In studying the question of what alienates us from our natural state of compassion, I have identified specific forms of language and communication that I believe contribute to our behaving violently toward each other and ourselves. I use the term life-alienating communication to refer to these forms of communication.

Certain ways of communicating alienate us from our natural state of compassion.

The author, Marshall Rosenberg, literally starts the chapter on how to communicate empathetically by implying that anyone who doesn’t follow these principles is “behaving violently” and being “life-alienating”. The book has plenty of passages that read to me as morally loaded language that are basically saying “doing things my way is superior to anything else”... while at the same time saying that moralistic judgments are something to avoid.

If someone reads the book and comes away with the belief that anyone who doesn’t use NVC is “being violent” and “life-alienating”, while NVC practitioners are the ones connected to their “natural state of compassion”... then it’s not very surprising if they end up wanting to police other people’s language.

I was quite surprised, some time back, when I went back to re-read the NVC book and encountered this language and vibe. I hadn’t remembered it at all. Meaning that I myself had read the book selectively, filtering out some of the subtext in order to only focus on the explicit content. No doubt because I myself am uncomfortable with conflict and with judging others, so I focused on just the explicit “NVC is for yourself” message while ignoring the parts that conflicted with it.

And also, while I’ve generally found the principles of NVC to work spectacularly well, on one occasion they worked badly, because I myself forgot about the parts of it that didn’t resonate with my own schemas as much.

If a conflict-avoidant person like me reads NVC and other similar pieces of advice - like Stephen Covey’s “seek first to understand, then to be understood” - they might come away with a very specific emotional fantasy. It goes something like “if I just endlessly empathize and try to listen to people with whom I’m in conflict, then eventually they’ll empathize back and we can reach mutual understanding”.

This is a powerful fantasy in part because it does very often work! Trying to engage in constructive conversation and genuinely empathizing with the upset and needs of others first does often lead to mutual agreement.

However, an important part of NVC is also checking in with your own feelings and needs, and not giving in to demands that don’t align with your own needs. On at least one occasion, I ended up in a situation where I would empathize and empathize with someone who was making demands of me… but who then would never empathize with my needs or consider them valid. This effectively put me in a headspace where I felt pressured to give in to their demands, as their needs felt much more salient than my own.

I effectively skipped the part about checking in with my own needs, because that would then have required me to stand up for myself and refuse the demands, and this felt uncomfortable to me. So while some people end up reading NVC in a way that gets them to police the language of others - effectively reading it in a more conflict-y way than intended - some people also read it in a less conflict-y way than intended, and end up giving in to others too much.

I expect that someone who is using NVC to police the language of others might be - consciously or subconsciously - anticipating this failure mode. They might be afraid that they or people they care about won’t be capable of checking in with their needs if others don’t speak in an NVC kind of way, and will then be unduly pressured.

Conflicted authors

Let’s go back to the bit where the vibe and explicit content of a source seemed to conflict at times.

Why is that?

Now, I don’t want to speculate too much about Rosenberg in particular. Maybe I’m just misreading him. But NVC is hardly the only source where the vibe and explicit content seem to conflict. Without naming any more names, I have noted that there seems to be a more general strand of spiritual/self-development writing that seems to be saying something like “my practice will make you more loving, compassionate, and open-minded, and anyone who disagrees with my method is a complete idiot who doesn’t understand anything”.

My guess is that at least in some cases the reason is an instance of the same pattern that I’ve been discussing. Emotional schemas can subvert anything to serve their purpose, invisible to the person in question.

Someone might write a book on compassion and empathy and genuinely intellectually believe that you shouldn’t judge others, and even be genuinely compassionate and non-judgmental most of the time… while still having some need to feel better than others, or some desire for a clear framework that avoids uncertainty, or whatever.

And then that need will subtly leak into the text, with the author doing the same thing as their readers will - looking at what they’ve written and focusing on the aspects of it that they endorse and believe in (the explicit message), and filtering out aspects of it that conflict with that.

It reminds me of something I once wrote, that a reader said had an arrogant tone. I was surprised by that, because I thought I had gone to the effort of looking up the rationale behind views that disagreed with me and explaining what about those views was reasonable. And I did do that. But then I would also follow up the explanation of their rationale with something that amounted to “and here’s why that is wrong and misguided”, which the reader correctly responded to.

There had been a subconscious strategy active in the writing process, that performed just enough intellectual charity to let me feel that I was being charitable, all the while letting me feel intellectually superior.

Possibly I’m doing something like that right now! I don’t feel like I would be, but those kinds of impulses would have gotten good at hiding inside my mind by now.

So it is not just that ideas get corrupted in transmission. They get corrupted while being generated. People will always be looking at reality through the filter of their own needs and desires. They don’t just interpret reality through them, their process for generating and communicating new ideas is also one that’s trying to get their underlying needs met.

The internal conflict may also be functional. NVC’s simultaneous message of “don’t judge” and “people who don’t do this are violent” may be part of what makes it spread. The explicit philosophy appeals to people who value non-judgment, while the words about violent language may appeal to people who have difficulty dealing with that kind of language. Readers may then interpret it through the lens that they prefer, with the model getting a wider audience than if it only contained one message.

Of course, none of this means that we shouldn’t have new ideas. Even corrupted ideas still correctly describe some parts of reality. And many people do understand, and benefit from, the less corrupted versions of various ideas and frameworks. As I said, I’ve found the proper, explicit version of NVC is tremendously useful!

Even if a misapplication of it led me astray once.

1. ^{^}

   “Social idea” may not be the most accurate term for this, but I couldn’t think of anything better.

2. ^{^}

   "In NVC, no matter what words people use to express themselves, we listen for their observations, feelings, needs, and requests. Imagine you’ve loaned your car to a new neighbor who had a personal emergency, and when your family finds out, they react with intensity: “You are a fool for having trusted a total stranger!” You can use the components of NVC to tune in to the feelings and needs of those family members in contrast to either (1) blaming yourself by taking the message personally, or (2) blaming and judging them.”

   – Marshall Rosenberg, Nonviolent Communication: A Language of Life: Life-Changing Tools for Healthy Relationships. Kindle Locations 1820-1824.

3. ^{^}

   “To tell if it’s a demand or a request, observe what the speaker does if the request is not complied with.

   Let’s look at two variations of a situation. Jack says to his friend Jane, “I’m lonely and would like you to spend the evening with me.” Is that a request or a demand? The answer is that we don’t know until we observe how Jack treats Jane if she doesn’t comply. Suppose she replies, “Jack, I’m really tired. If you’d like some company, how about finding someone else to be with you this evening?” If Jack then remarks, “How typical of you to be so selfish!” his request was in fact a demand. Instead of empathizing with her need to rest, he has blamed her.

   It’s a demand if the speaker then criticizes or judges.”

   -- Marshall Rosenberg, Nonviolent Communication: A Language of Life: Life-Changing Tools for Healthy Relationships. Kindle Locations 1593-1600.

4. ^{^}

   “If our objective is only to change people and their behavior or to get our way, then NVC is not an appropriate tool. The process is designed for those of us who would like others to change and respond, but only if they choose to do so willingly and compassionately. The objective of NVC is to establish a relationship based on honesty and empathy. When others trust that our primary commitment is to the quality of the relationship, and that we expect this process to fulfill everyone’s needs, then they can trust that our requests are true requests and not camouflaged demands.”

   – Marshall Rosenberg, Nonviolent Communication: A Language of Life: Life-Changing Tools for Healthy Relationships. Kindle Locations 1624-1628.

2026-04-03 02:18:16

(JK note: all my writing on LW nowadays comes in LLM blocks.)

2026-04-03 02:16:14

AI benchmarks seem to saturate quite quickly. One sentiment I've heard a lot is that AI companies optimize their training for the most popular benchmarks. In the best case, that could mean focusing more on getting better on the topics that are benchmarked the most, which is still somewhat suboptimal as the benchmarks tend to be a proxy for the real skill and now the AI is trained for the proxy. In the worst case, the AI training is iterated directly against the benchmark, causing overfitting and good benchmark results.

And avoiding this completely is not that easy. The training dataset is essentially the whole internet. When someone publishes a benchmark, the training set includes that. And people post benchmark solutions online too; those will be in the training data as well. Filtering all of that from the training data is not easy even if you actually try.

There's an intuitive solution, that some evaluators are doing: publish a part of the tasks. The rest, a holdout set, is kept private. If the models do better with the public set but not the holdout set, the effect can be measured. There are some limitations to this approach, which I'll discuss below.

Firstly, the holdout set actually needs to stay private. Sometimes the researchers just don't have enough integrity and sell out, like FrontierMath did (perhaps overly cynical take). Also, since the frontier models are not available for self-hosting, it's possible that the AI companies could just extract the benchmark questions from API logs, but this seems somewhat unlikely.

Most agents nowadays also have internet access. This means that even perfectly filtering the benchmark-related content from training data isn't enough. While you could disable it for benchmarking, then you're not actually measuring what a state-of-the-art system can do. Alternatively you could block or censor that content in the browser tool, but this is complicated and error-prone. Ideally, we could have a section of internet that's inaccessible to LLM training and LLM browser tools. Sadly, this won't work as it requires everyone with access to uphold these rules, which won't work.

There's another issue with leaking the benchmark contents to the model. Even if the actual solutions don't leak, knowledge of the specific benchmark can cause the model to realize it's under evaluation. Things to worry about here include trying to game the benchmark in some way, sandbagging, and in general performing differently in and out of evaluation.

A fully private benchmark is mostly safe from these issues, but is otherwise significantly less useful. As a milder version, the evaluators could be really vague about the actual contents. It's harder to review, harder to trust, and the actual results mean less if you don't know what exactly is getting tested. Using a trusted 3rd party auditor could help with this quite a bit.

Going a bit further, the evaluators could publicly lie or at least mislead in their published task set, substituting the actual evaluation with equivalent but different contents. Then the 3rd party auditors would need to be convinced that this is a reasonable thing to do. This seems like a terrible idea to me.

One way to completely mitigate this problem is testing with tasks that don't have a known answer, like open mathematical problems. This of course has the downside that the benchmark is rather hard and gives only a binary result per task (or at least partial solutions with novel approaches are hard to score). And if the open problem is solved, there's quite a bit of pressure to release the solution too. And again, if the problems are public, and the models widely available, anyone can solve the problem using these models and publish it.

Another way is having tasks that are really close to the actual thing. These are hard to game, as working solutions should also work outside the benchmark. They also make it hard to differentiate evaluation from the real world task, since it might as well be either one. Often partial solutions are gradeable too. The issues are mostly that real world benchmarks are hard to make deterministic, well defined, and are expensive to run. This is doubly so if the task has to be executed in the physical world, as is the case with robotics. Another problem is that interacting with the real-world has side-effects; a computer use benchmark might want to book flights, make a phone call or publish a blog post.

For instance, imagine a benchmark that tries to measure how well a non-technical founder could build a SaaS company. Naturally it has to be published somewhere, and using a cloud platform would be the usual call. Ideally you'd like to have a fake platform so that the site could be published only in your simulated internal system. But there are dozens of cloud services and the AI could choose any of them. A reasonable solution is railroading the AI to use a specific service, the one you've chosen to emulate. But this in itself is something that differentiates evaluation from the real world scenario. It might be a reasonable tradeoff, but it's important to realize it is one.

I haven't seen many of these real-world benchmarks yet. I'm not sure why, as it doesn't seem prohibitively expensive to me. There's OSWorld which contains small well-defined tasks and is relatively close to getting saturated. The recently released PhAIL that measures the ability to control an industrial robot arm in a single task seems quite interesting too, but that doesn't assess any of the frontier LLMs, only weights-available VLAs.

It would be nice to have more extensive real-world task benchmarks.