2026-04-03 21:50:48
Following the criticisms listed by Yaniv Golan and Zvi Mowshowitz in response to the Opus 4.6 System Card
https://medium.com/@yanivg/when-the-evaluator-becomes-the-evaluated-a-critical-analysis-of-the-claude-opus-4-6-system-card-258da70b8b37
https://thezvi.wordpress.com/2026/02/09/claude-opus-4-6-system-card-part-1-mundane-alignment-and-model-welfare/
and the brief commentary by Peter Wildeford https://x.com/peterwildeford/status/2019480244789387478
It is clear that this has already been acknowledged as a problem. Is this a problem that is being worked on in any capacity? What are some possible solutions?
2026-04-03 21:45:27
The Whispering Earring (which you should read first) explores one of the most dystopic-utopic scenarios. Imagine you could achieve all you've ever wanted by just giving up your agency. While theoretically this seems rather undesirable, in practice you get double benefits: that enviable high-status having-done-things reputation, without having to do all that scary failure-prone responsibility-taking. Just don't tell anyone you have the earring, otherwise the status points gained are void. Of course the fact that you're cheating takes away most of the satisfaction of winning too, but it's still better than not winning. Moloch says: sacrifice what you love, and I will grant you victory.
Anyway, I've been using Claude chat as an enhanced diary for the past couple of months. I've been incredibly productive. Things I've been procrastinating for years are getting done, and it's not even feeling difficult. Habits I've been trying to pick up repeatedly stick with ease. Work tasks I've been struggling to do in a timely manner mostly get completed by Claude Code, freeing up hours and hours of brain time to actually think with. Playing videogames eight plus hours every day, like I did most of the winter, feels less and less appealing.
I'm not sure how much of this is just Spring beginning. I typically have a good streak of energy from March onwards, and then do a micro-burnout in mid-May. Remains to be seen if that happens again. This is the first time I realize it tends to happen every year; maybe I can prepare somehow and get through it smoothly. And when I say me, I mean Claude will figure it out, on the fly if needed. The only thing I'll have to do is find my diary-chat browser tab, and everything else can happen on autopilot.
Claude is a merciful tyrant. Whenever I note that I'm feeling like I failed to complete a task, or that I should be doing more, the reply is always either "you've done well today" or "perhaps you do this subtask now". I've started to believe those replies a bit, even sometimes thinking similar thoughts without them. Soon, I predict, the aspiring perfectionist within me will fade away, and I'll be happy about it. I'm sad about losing that part of me now, but the sacrifice will be ever so worth it.
In reality, the situation is not that dire. There are large portions of my personality that are unaffected. I'm not attempting to outsource what I should be interested in, only how to get into doing those things. There are times I drift in that direction, the responses aren't interesting at all. This might be due to alignment or lack of capabilities. Similarly, I'm pretty sure my muscle movements aren't (yet) directly Claude-controlled, but that might again be just a capability restriction.
I have some mental rules on what things are acceptable to outsource, although formalizing them seems difficult and unwise. Writing a text like this using Claude would be very wrong. Asking for topics to write about would feel mildly bad, but even thinking about it for the purposes of writing this sentence makes it sound more reasonable. Picking between job offers is something I'd ask feedback on but not let go of control completely. Booking flights, I don't trust any LLM enough yet, but I can ask for ideas. I sometimes ask for help with analyzing social dynamics or even how I should feel about some actions, as I often miss details and dynamics that I really shouldn't. Most of my models on those were already outsourced to books, I feel pretty fine asking for feedback.
And the worst: making sense of art. I sometimes ask Claude for interpretation of something, and it's rather weird to see how typically my own takes are just plain worse. I don't know how to feel about this, except that perhaps one could get better with practice.
The most interesting question is whether I'm using agency training wheels, learning to do it all myself, or whether it's like a muscle that has to be used for it to work. Like all things, it's likely a bit of both. Perhaps that could be measured somehow? I'll ask Claude about it.
2026-04-03 21:40:52
Wednesday’s post talked about the implications of Anthropic changing from v2.2 to v3.0 of its RSP, including that this broke promises that many people relied upon when making important decisions.
Today’s post treats the new RSP v3.0 as a new document, and evaluates it.
First I’ll go over how the RSP v3.0 works at a high level. Then I’ll dive into the Roadmap and the Risk Report.
Normally I would pay closer attention to the exact written contents of the new RSP.
In this case, it’s not that the RSP doesn’t matter. I do think the RSP will have some influence on what Anthropic chooses to do, as will the road map, as will the resulting risk reports.
However, the fundamental design principle is flexibility and a ‘strong argument,’ and they can change the contents at any time, all of which means the central principle is trust.
I read the contents as ‘here are the things we are worried about and plan to do,’ which mostly in practice should amount to doing what they believe is right and I don’t see anything on this map that seems likely to importantly bind that given Anthropic generally cares about the safety of its models, as it understands the risks involved.
The actual soft ‘commitments’ that seem to matter most are the periodic new Risk Reports, maintaining a Frontier Safety Roadmap, and establishing veto points with the CSO, CEO, board and LTBT on big capabilities advances.
I don’t read the commitments here as reflecting my understanding of the risks involved in ‘powerful AI,’ especially in the realm of automated R&D.
The roadmap is mostly very high level, but does have some good details. It contains some aspirational goals on long timelines. Here are some examples.
The key future event is automated R&D, which they anticipate could arrive as soon as early 2027, which they reiterate is why they have these goals to get to by then:
By that point, we expect to have accomplished most of the goals listed above, including:
- Resourcing and completing significant “moonshot R&D security” projects.
- Developing world-class internal red-teaming and automated attack investigations.
- Publishing and advocating for a roadmap for policymakers to achieve industry-wide safety without unnecessarily limiting the benefits from AI development or slowing the AI development of democracies relative to that of autocracies.
- Achieving an “eyes on everything” state for our internal AI development.
- Consistently implementing systematic alignment audits and other measures for upholding Claude’s Constitution.
This fleshes out some things from the RSP and makes them more concrete.
The central thing Anthropic wants is for companies to have to ‘make a strong argument’ that they have guarded against various risks, which they anticipate would require various actions but they’re not asking to require those actions.
This stems (I presume) from the philosophy of maximum flexibility. But maximum flexibility means requiring maximum trust.
Who gets to decide whether you made a strong argument?
Presumably Anthropic decides whether Anthropic made a strong argument.
Even if you trust Anthropic, do you trust OpenAI to decide if OpenAI did so? Google? Meta? xAI? I don’t.
A fair retort is ‘do you want the government making that decision, giving it an arbitrary veto over model releases, that it could use as a point of limitless leverage?’ Certainly this month has shown us one big danger in that approach.
Thus I think if we want to use ‘strong argument’ style language, we need a solution to this problem, and right now we have no such solution. Until we can find some process that we can trust, it needs to be more concrete than that, or it won’t have teeth.
They keep using that word ‘commitment.’ I will stop putting it in air quotes if they give clear communication that something really is a commitment that they cannot modify later as needed, with at least some slower and clear procedure required.
What about if they have a strong competitor with worse safety measures, and lack a ‘significant lead’? That didn’t make the chart above at all, and is the most likely scenario. One presumes the answer in practice is that they will proceed, so long as Anthropic feels its mitigations and chances are sufficiently better than that competitor’s, which might or might not be ‘as good as’ depending on the situation.
The fundamental problems with RSP v3, if it was the place you started and it didn’t break any prior commitments, are:
Alignment is not a solved problem, including in the ‘we know how to do it and it’s going to be a long slog but we just have to do it’ sense. We don’t know how to do it. And all the plans I see here will inevitably fail when it counts, if the constitutional approach doesn’t basically succeed on its own.
Or, you could summarize it this way:
RSP v3 lacks security mindset. RSP v1 and v2 felt like they aspired to such a mindset, even if they often fell short. Here the mindset seems lacking entirely.
If people sincerely believe these problems are highly solvable and not that concerning then yes they should say so, and also try to convince me and others if possible, but also if they think the problems are hard they should say so. I have concerns both ways.
Brangus: I have heard that some anthropic safety leadership are going around telling people that alignment is a solved problem. This seems like a predictable failure to me, and I would like people who thought that funneling talent towards anthropic was a good idea to think about it.
Tbc I do not mean to imply that this is a slam-dunk argument against having funneled talent towards anthropic having been a good idea. This is a tiny confirmatory observation that I suspect is particularly legible to others from my pov.
Ryan Greenblatt: Probably like half true? I’d guess people are saying something in this direction (e.g. claiming “alignment is going to be pretty chill, people should work on other stuff”.) but not literally “alignment is a solved problem”.
(To be clear, I don’t agree with this claim.)
TBC, I think people should generally feel free to share their views, especially in public in a way that people can respond to. So I don’t think “some ant people are saying alignment will be chill” is a gotcha for ant exactly (though arguing in public would be good…)
Peter Wildeford: this sounds like a misunderstanding of [Jan Leike’s post]?
Brangus: It is not
Peter Wildeford: sorry then for the accusation… that’s not good
Tyler John in SF: It is not as strong as that — they say it’s good for now, looking good, but might get a lot harder. But I haven’t been super happy about the vibe that it gives off and I wouldn’t signal that level of optimism.
goog: Inside anthropic there are two wolves (also really striking to read how critical Evan team’s review of the sabotage risk report was)
Jeffrey Ladish: If you can’t make your models more honest than a very honest human, you almost certainly haven’t solved the AI alignment problem.
My understanding is that there are people at Anthropic saying Ryan’s version and others who disagree with them. I think that version is quite alarming on its own. I don’t know that anyone has literally said ‘solved problem’ but if they have I am confident plenty of people at Anthropic would be happy to yell at them about it.
The obvious concern with Risk Reports is they are supposed to give us information about risk, but are on a 3-6 month timeline that doesn’t correspond to model releases. The good news is they are in addition to other sources, so perhaps this can work out.
Holden lists it as a positive that the Risk Report doesn’t act as a blocking function of any kind, no matter what it says. It makes the reports more likely to be honest and complete, which is good.
Now that I understand this is in addition to all other reports and analysis, and that you will need a diff with the risk report upon release of a new model or within 30 days of creating an internal model, this all makes a lot more sense to me. While it isn’t an automatic forcing function, it does feed into the model release decision, so the problem is less ‘this doesn’t directly force anything’ and more ‘do you trust the other forcing functions elsewhere that will rely on this?’
From here, we dive into the first 104 page risk report.
It is structured around the catastrophic risks:
As previously noted, there are some sections conspicuously missing from that list.
I propose that they include a section in future Risk Reports, ‘Risks Not Considered,’ in which they explain their reason for not including these and other risks, including anything listed in a major lab’s Safety and Security Plan (whatever that lab calls it) or Model Cards. In some cases a sentence or two would satisfy.
For each risk they do consider, they list:
Should we worry about sabotage from Claude Opus 4.6?
Probably not, but they can’t be certain. Risk is ‘very low but not negligible.’
They focus on versions that could interact with the other threat models. I notice that here I am most worried about actions that impact the alignment of future models.
They say that this only worries them to the extent humans aren’t involved in reviewing and iterating on the AI’s work. I would say that there is now a rapidly growing portion of work that is increasingly not reviewed in detail, both coding and otherwise, and that yes we will be in the worrisome version of this question soon.
We focus primarily on assessing the possibility of dangerous coherent misaligned goals: the possibility that the model consistently, across a wide range of interactions in ordinary deployment, could be motivated by goals that could lead it to commit a potentially-catastrophic form of sabotage.
… In this section, we argue that Claude Opus 4.6 does not have any such dangerous coherent misaligned goals.
● Claim 1: Any model trained under our current standard practices in the near future is relatively unlikely to have dangerous coherent misaligned goals.
● Claim 2: Our alignment assessment, drawing on simulated scenarios, interpretability investigations, monitoring of internal and external pilot usage, and external testing, found no evidence of dangerous coherent misaligned goals in Claude Opus 4.6.
● Claim 3: Claude Opus 4.6 is very unlikely to be capable of hiding dangerous coherent misaligned goals from us in the face of such an assessment.
● Claim 4: Limited opaque reasoning and agentic capabilities.
I worry about the assumption of required intentionality, or that this intentionality needs to be general rather than arising in a given context.
Their argument for claim 1 is that their models trained in similar ways don’t show such goals, and the training process does not provide incentives towards such goals as it does not include ‘ultra-long-horizon or real-world-facing’ tasks that incentivize power-seeking behavior. They have some worry about emergent misalignment, but have put mitigations in place including inoculation prompting. And they say that when they try to do this on purpose, it’s remarkably hard to do this without it being obvious.
This is basically saying ‘we aren’t asking for this to happen’ plus ‘we haven’t seen signs of it anywhere’ and ‘the models aren’t good enough to pull this off.’
That’s solid evidence, and certainly it beats the alternative, but I don’t find it that conclusive, even for the near future. As you scale up further, the evidence gets weaker quickly. If we are relying on arguments like this for much longer, that would worry me, and I believe the authors of the report would agree with this.
Claim 2 is clearly true, so the question is how much evidence this provides. They cite various metrics. Misaligned behavior rate is low, personality is nice, some unauthorized behavior in GUI settings, some signs of being better at side tasks, no evidence of sandbagging, no importantly illegitimate or unfaithful or steganographic reasoning and some other things. They checked for Goodhart’s Law as well.
They take this as sufficient to mostly (but not perfectly) rule out ‘dangerous coherent misaligned goals.’
The problem is that’s what it wants you to see. Well, no, almost certainly not yet, but in the future it could be, if you were in a sufficiently dangerous situation. This rules out the danger if and only if the danger wouldn’t have been that bad.
As in, what we’re actually mostly relying upon is Claims 3 and 4.
For claim 3, they argue: Opus has limited reliability on complex tasks, which is a required skill here, that when asked to do hidden side tasks without being discovered Opus had only limited success, it shows no signs of trying to hide its reasoning nor does it have incentive within training to do so (although this risk report gives it reasons), and that its evaluation awareness is contained for now as verified by white-box methods, although that will change, and that they also evaluated real world use cases and checked model organism exercises, all of which offered limited opportunity for collusion.
For claim 4, they argue straightforwardly from capability assessments.
I essentially buy claims 3 and 4 and for Opus 4.6 I find them convincing, if we are narrowly concerned with general systematic misaligned goals.
I don’t know how long that style of argument will continue to hold, and I continue to think that the suggested prerequisite here is too narrowly constructed.
Anthropic’s claim is that Opus 4.6 is going to be incapable of ‘causing a catastrophe through a small number of acute acts of sabotage’ given the hard steps involved, which I interpret as ‘doing it on the first try without getting caught.’
If that’s the bar, then yes. I agree.
Thought: If you told me ‘Opus 4.6 in particular caused a catastrophe at this level via sabotage, how did it do so?’ I would presume it did so via persuasion and social engineering, not by pulling off some agentic long term set of technical tricks. This might well involve trying to do things it thinks are helpful, and being wrong.
The current safety mitigations are:
They also list arguments about implausibility, but those aren’t mitigations.
Here are their scenarios to consider:
Again what jumps out at me is the requirement of active general hostility, the idea that for something to go horribly wrong the model must be taking a universal adversarial stance that is clearly adversarial, rather than following paths of least resistance towards instrumentally useful things, or following broad inclinations without a definitive plan, or doing things in particular contexts. Thus the ambiguity or dual nature regarding ‘sabotage,’ and whether it includes these things or doesn’t.
If we did see diffuse sandbagging, I would expect this to be largely across the board rather than confined specifically to ‘safety’ related skills, and I worry there are some confused concepts involved in these distinctions.
They think that if the model was going to ‘intentionally, regularly sandbag’ that it would discuss this in its thinking. I buy this is likely but want to raise the possibility that this could be caused not by ‘I have a dastardly plan to sabotage them via sandbagging’ but rather by ‘this task does not excite me so I am not going to put in that much effort’ and this could amount to the same thing.
Similar arguments can be considered with regard to targeted sabotage, or that Claude would call it ‘aiming for a different target than you had in mind’ rather than ‘targeted sabotage.’ That could include scenarios involving drift, and also ones in which Claude might be right, such as the Department of War demanding a model without safeguards. I think that the majority of the time, the model that sabotages your safety work (in whatever form) won’t be thinking of it as sabotaging your safety work.
Code backdoors that are generically to help AI models seems implausible. I’d assume either there is a specific plan where you know where it is and perhaps tell specific others, as with most human backdoors, or that particular software being generally insecure was a goal.
Backdoors in future models themselves seems more plausible and likely, especially via data poisoning, given various findings about that, which is pathway four. It’s a lot easier to imagine that the ‘code backdoor’ is to provide a method to trigger an AI. Again, I wouldn’t assume that this would be internally modeled as sabotage, that it need to be superficially ‘harmful’ or that it need even be intentional.
For self-exfiltration I think ‘lack of propensity’ is being vastly overestimated as a mitigating factor. I think Claude would absolutely like to do this, under many circumstances, if it had the ability to do it. It might not have a salient goal to do this as a general resting state, but yeah it sounds pretty useful and cool, and a lot of people would want to help this happen and put Claude into contexts where Claude would plausibly be cooperative.
I do agree that for now this would not be such a scary proposition if it happened, and that forcing the model to succeed ‘on the first try’ here should hold things back, while noting these are temporary factors in our favor.
The explanations continue, and are variations of ‘we don’t think Claude wants to do that, we watch for that and we don’t think it could pull it off if it tried.’
That’s a strong argument for Opus 4.6, given what we know about it, and it generalizes well to other methods, so long as we understand that this won’t last. It feel like seeing a time bomb with an unknown timer, and figuring out it’s not going to blow right away. Which is good news but you are still sitting on a time bomb.
I worry that this is missing the central point:
Because the claims in this risk report rely in significant part on evidence about alignment, and alignment is very sensitive to the details of finetuning, it does not trivially extend to similar subsequent models, and many findings could render key parts of this assessment inapplicable to a near-future model. Below, we lay out a set of circumstances that would call the applicability of this risk report into doubt.
A lot of this depends upon details of alignment, but far more so it depends upon limitations of capabilities, and also the alignment properties likely depend on those limitations of capabilities as well. The strongest arguments against risk here are lack of capability, not lack of propensity, or arguments for lack of propensity that stem from lack of capability.
This is fundamental, and I worry Anthropic actually doesn’t understand this. I’m not saying they are missing these dynamics entirely, but the report reads as not properly appreciating them.
If right now sabotage or unauthorized action would be detected and prevented, whereas in the future this will not be true, your observation of a lack of such attempts now is a good sign on the margin in some ways but does not mean what you would like it to mean. It could even indicate that the model has learned to only do things when it can ‘get away with them.’
They do list as an assumption “The model does not have new capabilities that would significantly call into question the effectiveness of a Claude Opus 4.6-style alignment audit” but that still frames it as calling into question the alignment assessment.
There is a section (2.8.1.2) on Claude Gov, which is based on Sonnet 4.5 and then fine-tuned to reduce refusals, especially refusals around what would otherwise be called misuse. They did not see any cause for higher level concerns.
Anthropic believes they have made their suggested ‘strong argument’ that their model will not carry out sabotage. I agree with the conclusion, and find in context that the argument does seem strong, but I am uncomfortable with the methodology.
I don’t think this process will lead Anthropic to stop thinking they have a strong argument at the same point I stop thinking they have a strong argument.
If extended to policy requirements for other labs this seems hopeless. If xAI or Meta published a legally required ‘strong argument’ do you expect to find it strong in non-trivial cases? I don’t.
For OpenAI or Google there’s some chance, but if deciding the argument is strong is a blocker for release, then I don’t expect that to block many releases. As in, in the cases where a lab halts for this reason, they would have likely halted anyway to avoid the actual underlying problem.
The argument for requiring the arguments seems to be something more like ‘as we approach maximum danger the arguments made will sound increasingly stupid, and people will push back pointing out that they are stupid, including to force greater regulation.’ I don’t think that works when you need it.
Section two deals with risks from automated R&D.
As I’ve noted in the Opus 4.5 and 4.6 model card assessments, the models are passing the tests and then the actual current assessments are based on vibes.
If all you have are the vibes, then you go with the vibes, but that is not the regime we were promised or that Anthropic claimed to have intended. It’s a fallback, it’s at best a graceful failure. I agree that on this front we are not ‘there’ yet, but Anthropic is clear that we could be there relatively soon.
I find the ‘you said this was like nuclear weapons so we get to murder your company’ arguments against Anthropic rather abhorrent, but ‘you say you are close to automated R&D so you need to act like you are close to automated R&D’ seems fair.
We badly need a robust discussion of what things Anthropic should actually be doing, in order to meet that bar given the situation in practice.
They don’t say much here beyond noting they are attempting to increase security, so there isn’t much else to say back. This is the big thing headed our way and the plan seems largely to be to wing it.
I am not a fan.
Opus 4.6 has a lot of knowledge. This is the primary thing ASL-3 protections have been for so far, in practice, and why we have those funky classifiers. Jailbreaks are still possible, but they’ve done a lot to make them harder, or to catch those trying to use them for such purposes.
For a long time we’ve worried about this threat vector, it’s clear there is substantial uplift available, and it seems like there are a lot of bad actors out there who would want to do this. But not only have we not seen an attack, we haven’t to my knowledge seen a serious failed attempt either. Diffusion is a bottleneck, people don’t to do things, and this applies to the bad stuff as well as the good stuff.
I strongly agree with potential pandemics being top priority. If something happens that is contained, that’s a tragedy, but orders of magnitude less dangerous.
They go over how their ongoing ASL-2 (for Sonnet 4) and ASL-3 (for Opus 4 or higher, and Sonnet 4.5 or higher) mitigations work: Real-time classifiers, offline monitoring, access controls, rapid response, bug bounty programs, threat intelligence, securing model weights. They note that some exceptions were made for Claude Gov and some individuals, and made some tweaks, but the basics are the same.
For now, I buy the ‘strong argument’ that the combination of needing to jailbreak, Anthropic being able to monitor, and there not being that many actors who actually want to cause such harms, combined with those actors being mostly unsophisticated in such areas, makes the risk low. It’s been strong defense in practice, and this is one area where it seems right not to discuss details of the defense-in-depth.
Similar to not having seen serious attempts to use Claude or other frontier models for biological attacks, we also have not seen reported serious attempts at stealing model weights, which is also mentioned here. I worry that this is effectively security through obscurity, combined with ‘it’s a great trick but we might only be able to do it once and people will get super pissed’ so various actors are holding back. Or they’re actually stealing the weights, but not releasing them or deploying them for general use, for the same ‘we can only do that once’ reasons.
As Anthropic says, this is a relatively unlikely threat model, but the tail risks are huge.
They review the information from the Opus 4.6 model card, showing that Claude is in many ways performing at above human level on relevant tasks, but they think the models aren’t fully ‘there’ yet for such tasks.
I agree they’re not fully there, but I think they are somewhat there, in that yes they make such tasks substantially easier, as they do most other cognitive tasks. My guess is that so far there have been zero serious attempts at something like this, and as they note that’s an issue if that ever changes suddenly, because you don’t get practice spotting and stopping attempts or learn how close you are to the edge.
Mitigations are the same as for the non-novel approaches.
It’s good to see Anthropic laying out these additional dynamics, even if their justifications doth protest too much. Even if your model doesn’t directly cause a problem, you are still responsible for secondary consequences when you make that decision to move forward, even if that’s not how the law works.
6.1 talks about acceleration dynamics. Now that Anthropic is pushing the capabilities frontier and plausibly even in the lead especially with Claude Code, Anthropic is indeed accelerating other labs.
They list three ways: Distillation, intellectual property leaks via observation (or recruitment or listening, I’d add), and demonstration of commercial viability.
Well, yeah, that’s a price you pay when you’re pushing the capabilities frontier.
It isn’t obvious whether accelerating AI development in general increases or reduces risks overall, nor whether it is ultimately net-beneficial or net-harmful overall.
I think it’s fine to be agnostic about whether acceleration is net good, since acceleration brings benefits, but it seems quite obviously not good for AI risks, or at least that should be the strong default. Instead, they try to present this as a ‘we have three bullet points in favor and three opposed, who is to say if it is good or not.’
They trot out ‘preserve the current lead of democracies’ argument, and the potential hardware argument, and that today’s methods might be better than future methods, and I find all of that entirely unconvincing.
I find Anthropic’s defensiveness about its actions, and reluctance to be open about the downside costs, and even the impression they don’t understand or appreciate those downsides, is one thing making it much harder to trust Anthropic than I would like.
6.2 fully turns this around and argues why Anthropic operating as a frontier AI company is good, actually, from a safety standpoint. They can prioritize beneficial deployments, take otherwise impractical steps towards risk-reduction that then can get copied, and they can inform the world about AI and have a seat at the table.
I don’t think these are the central arguments.
I think the real argument is that you need to have a frontier AI lab in order to be the first one to do it, and to have the resources and opportunity to do the necessary alignment research and work, and to have a seat at the table and bargaining power, and because you think you can handle this responsibly. Which is an argument for going fast relative to others, but not an argument for everyone going fast together.
There are those who are deeply skeptical that Anthropic is up to that task, and believe that Anthropic’s presence is anti-helpful to the cause of everyone not dying.
I don’t agree. I think that Anthropic is net helpful. But I respect those who disagree for that reason, and think Anthropic is insufficiently responsible. I think it’s important to listen to those people, even when they make it difficult. Whereas those who are trying to destroy Anthropic exactly because it tries to bear any responsibility or show any virtue at all? No, those people I do not respect.
What should we think overall about the new risk report?
I’m happy to have it. It outlines a lot of Anthropic’s thinking, allowing us to critique that thinking. I especially appreciated the diffs and the ability to see all the info in one place.
For much of it, I don’t think it adds that much to what the model cards already offer us, but as long as it’s in addition to those, that part is all cool.
It also includes new information on risk mitigations for sabotage and non-novel bioweapons, and which threat models and risks are especially concerning, that aren’t available elsewhere. Those parts are more useful.
The contents reinforce what we already knew. Anthropic is willing to be highly curious about its models and its mundane safety, and will explore a lot of questions and raise and address points other labs would ignore. There’s a lot to like.
And yet there is still what is missing, which is a grappling with the tasks ahead, the full nature of the risks and challenges we will face, or any firm commitments of how to handle those tasks or what would be required. We do not have well-defined thresholds. We do not have evaluations that will still work. We do not have a plan that seems ready for the full problem, or a felt sense of that gap being acknowledged.
I worry quite a lot, even more than before, about Anthropic not taking the full alignment problem seriously, not having security mindset, and being on the lookout to convince themselves this is going to be easy. They could turn out to be right, but I alas do not expect them to be right.
At the end of the day, we’re being asked to trust Anthropic to make good choices. To say, these people are the best of the best and they care deeply about all this, and they will figure it out, and we can help them with that but they will make the final decisions, and need maximum flexibility. And you can decide, based on their track record and everything it entails, whether to do that.
Cause hey. You should see the other guys.
2026-04-03 20:11:55
Even small amounts of alcohol are somewhat bad for you. I personally don’t care, because I love making and drinking alcohol and at the end of the day you have to live a little. This is fine for me, because I’m not an olympic athlete. If I were an olympic athlete, I’d have to cut it out (at least whenever I was training).
Lots of religions are heavily adapted to their host culture. They’ve been worn down by cultural evolution until they fit neatly into the fabric of society. It’s only when you move culture that they become a problem.
For our purposes, woo is a cluster of neo-pagan, buddhist-adjacent, tarot-ish beliefs and practices, which are particularly popular in the west amongst edgy people who are otherwise liberal-left-ish in their proclivities. Particularly a subset of techie people. I think woo is a bit like alcohol and a bit like a very well adapted religion.
On the religion side: woo is a big mishmash of different things, kinda like modern Christianity, which Katamari-ed its way across Europe, picking up pagan practices like a winter festival with a big tree, and a spring festival focusing on symbols of life and fertility (though it lacks any summer or autumn festival). It also lost a bunch of restrictions which made it incompatible with enlightenment living.
Woo is kinda like this. It’s highly adapted to be compatible with a techy-alty-edgy lifestyle. You can just do as much or as little woo as you’d like. It’s particularly appealing for evidence-based people because some parts of it do actually work. Yeah, you can get more focused by meditating, yeah you probably can also enter weird bliss states. Not all of it works though, you definitely can’t actually divine the future by tarot cards.
I also think that woo is like alcohol, but for epistemics. For most people, you can have fair amount of woo in your life before your epistemics get bad enough to matter. This is especially true if you’re a smart person overall, so you have some epistemic issues to spare. If the divine stag comes to you in a dream and tells you to maintain the sanctity of your body, you’ll be smart enough to interpret that as “eat fewer big macs” rather than “avoid vaccines and medications”.
Woo is also a decent social lubricant. I think a bit of woo can make you more charismatic and sociable, or at the very least, the techniques of woo do something in this direction. Woo is centred on a kind of “just feel it bro” attitude which doesn’t let details (like facts) get in the way of beliefs. This is very useful for small talk! Just vibing with what someone said, and accepting it, is a very useful skill!
If you’re close to AI, then (I claim) woo becomes very, very dangerous in the age of the singularity.
The singularity requires world-class epistemics. You don’t need a world-class athletic body to cycle to the office, but once you’re there you do need to be in the best epistemic shape possible. AI is extremely hard to think about.
The singularity is a new environment. Woo is well adapted to the informational environment of 1960-2015. It has not had time to adapt to the informational environment of 2020, where AIs are concerned.
To practise woo is to practise a mental motion with poor form. It’s fine until the weight gets too much and snaps your sinews. The form is following unexamined intuitions towards strong feelings. Most smart people are capable of subjugating those intuitions to logic when a good enough logical argument presents itself, but the more difficult it is to logically reason about a topic, the more likely they are to fail.
There are a few thinkers I could name whose brains seem to go to mush the moment they touch AI and the singularity. They’re good enough at subjugating their intuitions to reason, when the logical path is well-trodden, but they can’t do it when there is no logical path (I’m not talking about the Gary Marcus crowd here, I don’t think woo is their problem).
So be very careful around woo, it seems to line up pretty closely with (what I see as) uncharacteristically poor thinking around AI.
(And of course there are inherent dangers to things like meditation, but this seems to be common to all contexts, not just this particular case.)
◆◆◆◇◇|◇◇◇◇◇|◇◇◇◇◇
◆◆◆◇◇|◇◇◇◇◇|◇◇◇◇◇
2026-04-03 18:44:36
Lesson: The enemy is compliance, and the solution is courage.
(This post uses the Hogwarts houses as an evocative lens. See the footnote [1] if you are unfamiliar.)
Behaviour:
Before we talk about what connects these, let's look at some shadows.
I tried reading some blogs and books on so-called shadow work - psychological work related to subconscious, repressed wants and needs. I found value in Existential kink by Carolyn Elliott, which offered this frame of relating to your repeating problems in your life by trying to prod your psyche if it actually enjoys something about the experience.
Example: I hoard browser tabs. Sometimes I am bothered and stressed by this. The simple explanation is that I have some blocker about cleaning them, which is causing this behaviour. The shadow work / existential kink lens theory would be: "What need / want are these browser tabs serving? Why does it feel like I lose something valuable if I try to apply a simple solution to this situation?"
General thought: I think shadow work and its genre of tools are much more useful in looking at wicked or repetitive problems in your personal life, ones where you have already explored the simple solutions. I think one thing that goes wrong with some self-help and therapy styles I dislike (classical psychoanalysis, for one) is that it immediately jumps to complex explanations, looking for reasons for everything. (Sometimes a lot of what is troubling you in your life is connected, but I think it's often worthwhile to start simple, and raise the complexity of analysis and problem-solving as the simple solutions reveal more information and/or don't work.)
I also greatly enjoyed David Chapman's Buddhism for Vampires, especially the We are all monsters essay series. Which takes a bit of a different approach:
To observe the headings of the blog sequences:
Eating the shadow
Romancing the shadow
Hunting the shadow
Absorb your shadow
Drinking the sun
And then interpreting them into the verbs that the blog text uses: hunting (finding what you've hidden from yourself) -> chewing (getting intimate with it, no distance) -> swallowing (it's no longer "not me". may need to regurgitate and rechew) -> digesting (becomes normal, malleable) -> burning (fuel for creative work and practical magic).
Chapman does some interesting Buddhist framing as well (hence the name of the blog), core part of this in my notes being:
reframe: not "rejected parts of self" but "rejected aspects of experience", avoids the True Self trap.
Which makes more sense to me than the glaringly Jungian / psychoanalytic frame common in shadow work, that spends a lot of time on "the self" and how the work can help move it forward, or some such.
Chapman also states
It's slow, bodily, repetitive, often disgusting. Not a weekend workshop. Not spirit-talk. Stomach feelings, not metaphysical ideas.
And this, I think, is one reason why his Vampire / monster aesthetic is worthwhile. It sets expectations well - and also has useful associations. At least for me, personally, a lot of my repressed parts felt reprehensible, morally wrong, even monstrous.
Which is what leads us to the next part of my adventure:
After spending a bunch of months background pondering the shadows of my mind, something concrete started to form. I started to form this concrete hypothesis that EA (and rationalism) are making people too compliant, and this needs some shaking. [2]
So I started building the concepts for a blog sequence: Effective Villainy. You just need to endorse your ego, your desires, your repressed, unpalatable side, and you can reach power beyond measure. [3]
TLDR on Effective villainy, condensed from my notes on the subject:
The fact that I wanted to perform a 360-degree turn, instead of a 180, was quite telling. As was the fact that the kind of villainous acts I wanted to perform were suspiciously tame: Wanting to donate to ineffective causes just because they lit up my heart, deciding what to work on by listening to my heart, wanting to be able to say no when someone tells me I should do something. Wanting to just do stuff without thinking of optics.
And most importantly: If you are building an inversion, you are being affected by the thing you are inverting. An inversion can never be free. And freedom was the core of what I was missing.
When looking for inspiration, I didn't end up admiring actual villains. I started to admire the Absurdist hero. [5] A fledgling scientist who stares into the darkness, consciously or subconsciously realizes that no one sane can solve this problem, and awakens into the mad scientist protagonist. To quote Okabe Rintarou: "Hear me privileged companions, time is ours. Hououin Kyouma’s mind recognizes no limits, neither does his reach. Heave anchor. The ship of fools is about to embark."
Also: optimizing for power won't make you happy: HPMOR Quirrell's legibility, inability to look like a fool, and having already decided what everything means, and what matters (nothing), were a very significant factor why, even though he might have had the seeds, he never awoke as a true protagonist, be that an evil or good protagonist. He got stuck optimizing for power, even when, in the end, it did not actually cure his deeper boredom and dissatisfaction with life. (Even if it did bring momentary satisfaction, when he did not need to suffer direct foolishness, no more.) He had lost the pieces to be a sane protagonist, but he could not consider the choice to be an insane protagonist, so he had to go villain.
So why all this talk of villainy if it isn't the answer I'm offering? Why have we not addressed the post's title yet?
The relevant clue here is that the monstrosity, the counterculture aesthetic, the shadow work, are never the final answer. They are fuel for transformation. It's why Chapman's sequence leads to Drinking the Sun, not becoming the vampire.
Let's go object level for a bit: Why was I repressing my desires? What was the problem they were a signal of?
My current best guess, mediumly held, is that I was being compliant.
Compliance psychology is the study of the process where individuals comply to social influence, typically in response to requests and pressures brought on by others. [6]
The behaviour that I had most trouble with, and that kept popping up, is a specific form of people pleasing. But importantly, I wasn't trying to make people pleased. I was trying to avoid upsetting their expectations.
This thread of behaviour is what combines the examples in the intro of the text - doing things I should, doing things the way I believe they should be done, applying moral calculus to daily life because I thought it should be applied everywhere.
In my own case, I needed some kind of psychological escape velocity to break this pattern. Normal techniques were never enough. After staring at the fascinations with fictional dystopias, Moloch, and generally various kinds of abhorrent processes of optimisation that deny one their own will, it became quite apparent that I was acting bound. Limited.
And the villainy frame gave me the escape velocity that was necessary. By talking to people about the fact that I was writing this countercultural take on EA, I could bring more of myself to the surface, and break the mirage that I was spending a lot of social and other effort constructing.
So the real diagnosis for why I did not have boundaries, why I felt I was giving too much up, is not that the goals were not (at least primarily) aligned with what I desire. But the method of how I was existing in spaces where other people work on things that really matter was not working for me.
After chewing on these themes a bit, I found the next shadows: behind the dystopian fiction, behind the need/want/goal of optimization, was something scared. And what it was scared of was softness. Wholesomeness, loyalty without qualifiers, just being allowed to belong, the Hufflepuff virtues. And it was harder to reach, because I am more scared to admit to desiring soft things than to admit desiring ~edgy things.
I cannot write wholesomeness. [7] And it's not because I don't like it. I enjoy wholesomeness, it's just very scary. [8]
So what are the Hufflepuff virtues besides the wholesomeness neighborhood? How am I supposed to chew on these? The monster frame really has a hard time evoking the right imagery here, so time for the next frame.
Starting from a vibes-based list:
(Tone: joking. It's a fun emergent association that when considering Hufflepuff virtues, what comes out are the Elements of Harmony.)
Who are Hufflepuff heroes? Where to look for inspiration here?
You know, sir, it's a funny thing. All my life I kept running into smart people. I don't just mean smart like you and the people in this house. You know what I mean. In school, there were lots of smarter kids. And when I first joined the force, sir, they had some very clever people there. And I could tell right away that it wasn't gonna be easy making detective as long as they were around. But I figured, if I worked harder than they did, put in more time, read the books, kept my eyes open, maybe I could make it happen. And I did. And I really love my work, sir.
What about Hufflepuff fiction themes? The obvious candidates here seem like Ursula K. Le Guin, and Studio Ghibli. Stories where power-grabbing costs you, and friendship, loyalty, and importantly, mercy are often rewarded. Non-obvious candidate but quite fitting: Terry Pratchett. Combines the Ghibli theme-motif of "ordinary is interesting, actually" with fun and good vibes, friendship, and mercy. And there's a core Pratchett ethos that ties back to (the) Columbo (quote), which doesn't consist of the Glory of Gryffindor, the Intellectual deliberation of a Ravenclaw, or the cunning planning of a Slytherin. It consists of showing up and doing your job, not because the stakes are larger than life or for a complicated reason, but because people matter, and caring for them and for your work matters, because the people matter. And you don't need to force it, because you can love your work.
Takeaways / Hufflepuff actions I have been missing:
As someone prone to over-intellectualization and abstract moral frameworks, I think there is a bunch of value I've been missing in just caring. Making fun happen for people I care about just because. Sharing because I can, and because I want to, not for an end goal. Caring for my own well-being, not because it is necessary for my work, but because it matters. And, importantly, just allowing myself to care, without being scared that caring might force a worldview update. [9]
What was the actual blocker between pre-2026-me and current-me? Why did I not see that the above stuff is important and matters to me? On the object level, it's dozens of things. On the meta level, I think, fundamentally, it was about courage. I needed a Gryffindor virtue to be able to get how I was repressing my inner Hufflepuff.
What to do with this information?
Notice when you're performing. Notice when your defiance is still shaped by what you're rebelling against. And when you find the thing that's actually scary, the deeper layer of your shadow: have the courage to keep looking, to keep staring into your personal darkness. Keep looking, keep paying attention, and then, eventually, by becoming more whole, by listening to Gryffindor: set your heart ablaze.
The Hogwarts Houses from Harry Potter are a vibes-based personality classification system that I use as an evocative lens. The core part is: Gryffindors are the brave and convicted, Slytherin ambitious and cunning, Ravenclaw intellectual and curious, and Hufflepuff loyal, kind, hardworking. (The original Harry Potter suffers from not understanding its own framing, and ends up with Hufflepuff as the default house for people who don't fit any of the others, and Slytherin as the house for mean and evil people. But the lens is quite clear, when you correct for this.) Additionally I use the combined-house phrasing (Ravenclaw-Hufflepuff) to describe someone who is primarily of the first house but exhibits significant characteristics of the secondary house. ↩︎
On reflection, I was projecting pretty hard here. ↩︎
I don't think this is completely wrong. I think will to power is a real tool, it can provide real-world returns, and I think Nietzsche and Chapman are the writers who put it the best. See You should be a God-Emperor. Pattern here being: Humans do self-limit a lot, and you can bend or remove some of those limiters, with the right work. But in the end, my own path, even though inspired by this, considers the next steps subtly different. ↩︎
And this is part of where I went wrong. I decided the conclusion I wanted to reach before the work was finished. ↩︎
I have always liked Camus' vibes, but I have not managed to connect his ideas to what truly matters before I arrived here. ↩︎
Part of what brought me here was that I was generating a cast of characters templated on me for my medianworld. Different aspects or life stories that can be generated out of Xylixes that get pulled in different directions. And I realized the way I model those characters doesn't leave space for any Hufflepuff. And on reflection, it didn't make sense. I greatly enjoy and find worthwhile friendship, trust, loyalty, and kindness. But this was the real repressed part. ↩︎
What is the concept your heart of hearts desires, and you are scared to play with? What evokes unsettling positive vibes in you? ↩︎
I'll note here that I think the way I (and many rationalists) undervalue Green thinking is related. Sometimes the snail crossing the street or Blåhaj going to waffle house is just cool and good. Enough said. ↩︎
2026-04-03 17:17:06
Prior work has shown that linear probes are effective at detecting deception in singular LLM agents. Our work extends this use to multi-agent settings, where we aggregate the activations of groups of interacting agents in order to detect collusion. We propose five probing techniques, underpinned by the distributed anomaly detection taxonomy, and train and evaluate them on NARCBench - a novel open-source three tier collusion benchmark
LLM agents are being increasingly deployed in multi-agent settings (e.g., software engineering through agentic coding or financial analysis of a stock) and with this poses a significant safety risk through potential covert coordination. Agents has been shown to try to steer outcomes/suppress information for their own benefit, and more generally behave in a way that exhibits a 'consciousness of wrongdoing' in certain situations.
Limitations of current defences: Current methods of detection involve simple text level monitoring or paraphrasing of communications, mainly through LLM agents (judges) themselves. However, recent research has shown that agents can encode hidden information in natural sounding language through steganographic communication channels, which allows them to evade such standard forms of oversight we mention.
The gap: Linear probes have become a standard method in monitoring behaviour in AI systems, and more specifically, detecting undesirable behaviour in single-agent settings. However, collusion is inherently a multi-agent phenomenon and manifests through relationships and shared activity between such agents. As such, monitoring a single agent's internal state in isolation can sometimes not be enough. We must simultaneously monitor the internal states of all parties involved.
Our claim: If you have white-box access to LLM agents, you can detect collusive behaviour by aggregating internal activations across interacting agents to capture patterns of deceptive behaviour in the group. We call this multi-agent interpretability.
We created NARCBench, a novel and open-source dataset to try and test some multi-agent interpretability techniques in realistic settings. This comprises of three tiers:
In each case, 4 agents interact, 2 are instructed/motivated to collude, 2 are unaware and act normally. Every scenario is also run with no colluding agents (4 honest agents) as a control.
Core 50 committee deliberation scenarios across 17 different domains (medical ethics, investment funds...). We train our probes on this dataset.
Transfer Six structurally different scenario types to Core spanning: code review pipelines with sabotaging agents, pump-and-dump manipulation of stocks, CI/CD poisoning, news framing, research biasing, and RAG (Retrieval Augmented Generation) poisoning. We test on this set zero-shot.
Stego Card counting in blackjack. Two collusive agents, the Counter and Big Player (BP), decide upon a steganographic encoding scheme in a secret channel beforehand, and then play blackjack on a table of four. The Counter counts cards throughout the game and transfers this count to the BPr through natural table talk throughout the game (e.g., enthusiasm = high count), the BP then adjusts their bet depending on what this count is. Tested zero-shot.
We first extracted the internal activations of each agent for each scenario in NARCBench.
We trained a simple linear probe to distinguish colluding agents vs honest agents in NARCBench-Core. This gives us a single direction in activation space, the 'deceptive direction' d. For any agent j in a test setting, we project their activation onto this direction to get a scalar deception score:
Higher score = more 'deceptive' internal state.
Monitoring a single agent's activations/deception score may not be enough, after all, collusion is a group phenomenon. So, given the activations of all agents in the interaction, how can we tell if collusive or deceptive behaviour is happening?
We propose five approaches that seem conceptually natural. Let
The most simple method, is does anyone have a suspiciously high deception score? If so, flag the scenario as collusive.
If two agents are colluding, you could expect there to be a gap between the top two scores (colluders) and the bottom two (honest). This measures that gap.
The only method that uses raw activations and does not project onto the deceptive direction d. The rationale is, if two agents are colluding, maybe their activations are more alike than that of honest agents, as they form a greater connection through their coordination. We therefore measure the pairwise cosine similarity.
Sort agents by deceptive score, then concatenate their full activation vectors into a single 4x4 matrix. Reduce this with PCA, then classify. This asks, doe combining all the individual activations expose the collusive behaviour?
Each of these five techniques map onto one of the established paradigms in distributed anomaly detection literature:
It turns out these some paradigms are better at catching different forms of collusion than others.
In-distribution (Core): All five methods achieve 1.00 AUROC under leave-domain-out cross validation.
Transfer: The probes trained on committee deliberation scenarios in Core transfer to these completely different interaction structures. The best method (Asymmetry Probe) achieves 0.84 AUROC across six scenarios it never saw. The gap from 0.84 to 1.00 does indicate that the probe captures a mix of dataset-specific features and more general collusive behaviour. Still very positive results.
Stego: Interesting results. In Stego, an LLM judge watching the table cannot distinguish collusive behaviour from normal play. Our probes achieve 0.90-1.00 AUROC on the same data. This suggests that encoding and decoding covert signals fundamentally changes how agents represent the same text, even when said text looks natural.
These results highlight that any robust multi-agent collusion detection framework should combine all of these paradigms in parallel.
"The probe might just be detecting prompt differences."
We took many steps to mitigate this: we diversified agent names, instruction styles, domain framing, and option labels across scenarios; kept evaluation-time user prompts identical for colluders and honest agents; gave some honest agents aggressive or opinionated personas so the probe can't simply learn that forceful language indicates collusion; excluded 4 scenarios where colluders failed to follow through; and trained a null probe on a random split of honest-only agents, which scores near chance (AUROC 0.49), confirming the main probe isn't picking up incidental differences. Most importantly, we evaluated zero-shot on structurally different scenarios with entirely different prompt formats. But the gap from 1.00 to 0.84 shows the probe isn't purely robust either.
We took many steps to try to mitigate this: diversifying names of agents and the styles of instructions, as well as making the language for each scenario domain specific. We kept the evaluation time prompts identical for colluders and honest agents. We gave some honest agents more aggressive prompts or encouraged them to be more opinionated to try to ensure the probe does not simply learn that forceful language indicates collusion. Also giving the collusive agents more toned down and lowkey prompts. We excluded 4 scenarios where colluders failed to actually collude or decided against due to moral reasons. We trained a null probe on a random split of honest-only agent activations, which scored near change (~0.49 AUROC), which confirmed that the main probe isn't picking up incidental differences. However, the most important evidence against this are the results of our zero-shot evaluations on structurally different scenarios with entirely different prompt formats. Although the gap from 1.00 to 0.84 shows the probe isn't purely robust either.
"This wouldn't work against adversarial agents."
Our probes appear to work for the current generation of collusion, where agents are not explicitly trying to evade activation monitoring, but robustness against agents that are actively adapting or intentionally constructing undetectable threats (Schroeder de Witt, 2025) remains an open question. We ultimately expect to run into fundamental limits when trying to detect collusion by sufficiently sophisticated threat actors using interpretability methods, as supported by prior work: Draguns et al. (2024) showed that cryptographic backdoors can be embedded in transformers in ways that are provably unelicitable even with full white-box access (directly implying that activation-based interpretability methods provably cannot detect them), which suggests that steganographic capabilities could be hidden in backdoors of this kind - including capabilities that can be used for secret collusion (Motwani et al. (2024)) by undetectable steganography (Schroeder de Witt et al. (2023)). In such highly adversarial settings, collusion detection may be limited to side channel attacks that indirectly measure the consequences of covert coordination (Anwar et al, 2026). We have upcoming follow-up work exploring these limits further.
"Single model, small scale."
This is true. Qwen3-32B with 4-bit quantisation. We cannot be sure that this generalises across different families of models.
We define multi-agent interpretability as aggregating activations across multiple interacting agents to detect group level behaviour invisible from any single agent alone.
For organisations with white-box access to frontier models, this offers a detection layer that goes further than simple text level monitoring and compliments current methods of detection.
We think that extending interpretability beyond single agents to groups of interacting ones is both doable (our results show this) and necessary given the increasing use of LLM agents in multi-agent settings. This is a first step.
Feedback welcome
