2026-01-22 00:08:27
Published on January 21, 2026 4:08 PM GMT
Sometimes we use comparisons or even adjectives in a way I think is accidentally misleading. We usually intuit the correct meaning, but not always so I think it is worth being aware of. I have especially noticed doing it to myself and am making an effort to not do it anymore because it is a kind of self deception.
Comparing X and Y by saying they are similar is sometimes evidence they are actually quite different and that the speaker was trying to bring them together in a way that is not accurate. Same thing in reverse if the claim is that they are very different, it feels like the speaker is trying to create space between the two things à la narcissism of small differences.
This can be used deceptively:
but I think that it is accidental or subconscious more often than not.
Maybe I’m just ragging on hyperbole. Like if your friend says he’s going to introduce you to someone “like Raiman” you probably don’t expect them to be someone that needs to be institutionalized, you probably expect a socially awkward person good at math or something[1].
But that’s kind of my point, we are left to pick up the slack from a non literal expression and it makes things a little messy when you aren’t extremely familiar with your interlocutor or the point of comparison. Some other examples off the top of my head:
Again, in ordinary speech this is a non issue for the most part. But, it can be used for deception and I think it is a powerful form of self deception but noticing it can help you be more honest with yourself.
To keep it general, if you actually make up your mind on a topic, I think you are quite unlikely to roll through the list of reasons you think your side is correct over and over again. I noticed myself doing this with AI safety for a while. I would just have virtual debates in my mind where I took the side of “AI will be safe” and would win and I really don’t think that is what a person who is actually sure of things does. I think this kind of thing applies, to some extent, to the deconversion process where people will have a crisis of faith, then spend time guilty watching atheist content and mentally debunking it, eventually switching to watching compilations of atheists winning arguments, then just live their lives without actually thinking about it because they made up their mind.
You are most often mentally engaging with the arguments the most when you are unsure of them. I would say that if you claim to be sure but are, unprompted, mentally engaging with arguments you should take a step back and ask yourself if you are as sure as you think.
So it’s a bit of a double edged sword. If you don’t notice it, the seemingly strong signal of having a lot of arguments on your side can convince you your side is true when it is not. But if you are aware of it you can use it to more accurately calibrate your opinion or confidence level based on how much you seem to be subconsciously pulling on a thread. This could be one of a few existing things:
But none of those really feel like they are exactly what I mean and each breaks down in certain cases. Maybe what I am thinking of already has a name. Maybe it is a unique cocktail of biases I have mixed up inside me and am projecting onto the world. But until it’s settled I thought it best to put it here in the case anyone has insight or it helps others calibrate better.
A completely fictitious example that is not based on my friends or family discussing me
2026-01-22 00:00:20
Published on January 21, 2026 4:00 PM GMT
NB: Last week I teased a follow-up that depended on posting an excerpt from Fundamental Uncertainty. Alas, I got wrapped up in revisions and didn’t get it done in time. So I don’t leave you empty handed, instead this week I offer you some updates on my Claude workflows.
Back in October I shared how I write code with Claude. A month later, how I write blog posts (though ironically not this one). But I wrote those before Opus 4.5 came out, and boy has that model changed things.
Opus 4.5 is much better than what came before in several ways. It can think longer without getting lost. It’s much better able to follow instructions (e.g. putting things in CLAUDE.md now gets respected more than 90% of the time). And as a result, I can trust it to operate on its own for much longer.
I’m no longer really pair programming with Claude. I’m now more like a code reviewer for its work. The shift has been more subtle than that statement might imply, though. The reality is that Claude still isn’t great at making technical decisions. It’s still worse than random chance at picking the solutions I want it to pick. And so I still have to work with it quite closely to get it to do what I want.
But the big change has been that before I would have the terminal open, claude in one split in tmux, nvim in another, and we’d iterate in a tight loop, with claude serving as something like very advanced autocomplete. Now, I use Claude in the desktop app, get it to concurrently work on multiple branches using worktrees, and I have given it instructions on how to manage my Graphite stacks so that even for complex, multi-PR workflows I usually can just interact through chat rather than having to open up the console and do things myself.
Some tooling was needed to make this work. I had to update CLAUDE.md and write some skills so that Claude could better do what I wanted without intervention. I also had to start using worktrees, and then in the main repo directory I just check stuff out to test it (the local dev environment is a singleton) and do occasional manual operations I can’t hand to Claude (like running terraform apply, since I can’t trust it not to randomly destroy infrastructure on accident).
Still, this is not quite the workflow I want. Worktrees are annoying to use. I’d prefer to run Claude in cloud sandboxes. But the offering from Anthropic here is rather limited in how it can interact with git, and as a result not useful for me because it can’t use Graphite effectively. Graphite has their own background agents, but they’re still in beta and not yet reliable enough to use (plus they still have restrictions, like one chat per branch, rather than being able to have a chat that manages an entire stack).
But as I hope this makes clear, I now use Claude Code in a more hands-off way. My interactions with the code are less “sit in an editor with Claude and work in a tight, pair-programming-like loop”, and more “hand tasks to Claude, go do other things, then come back and review its work via diffs”. I expect this trend to continue, and I also hope to see new tooling that makes this workflow easier later in the year.
That’s coding, but what about writing?
Well, Claude still isn’t fantastic here. It’s gotten much better at mimicking my style, but what it produces still has slop in its bones. It’s also gotten better at thinking things through on its own, but I still have to work to focus it and keep it on task. It will miss things, same as a human would, that I want it to look at.
For example, I was editing a paragraph recently. I made a change to some wording that I was worried might convey the right sense but be technically wrong. I handed it to Claude. Its response was along the lines of “yes, this looks great, you made this much more readable”. But when I pressed it on my factual concerns, it noticed and agreed there was a problem more strongly than I did! These kinds of oversights mean I can’t trust Claude to help me write words the same way I trust it to help me write code.
So I’m still doing something that looks much more like pairing with an editor when I write with Claude. This is good news in some sense, because it means I’m still needed to think in order to produce good writing, but bad news if you were hoping to automate more thinking with Claude.
This past week there came news of some novel mathematical breakthroughs using LLMs. The technology is clearly making progress towards critical thinking in a wider set of domains. And yet, writing remains a nebulous enough task that doing it well continues to evade Claude and the other models. That’s not to say they aren’t getting better at producing higher quality slop, but they still aren’t really up to completing a task like finishing revisions on my book the way I would want them done for me.
Where does this leave me feeling about LLMs right now?
We made a lot of progress on utility in the last 12 months. Last January I was still copy-pasting code into Claude to get its help and using Copilot for autocomplete. It was almost useless for writing tasks at that point, and I often found myself wasting time chatting with it trying to get things done when it would have been faster to sit and think and do it myself. That’s just no longer true.
As always, though, I don’t know where we are on the S-curve. In some ways it feels like progress has slowed down, but in others it feels like it’s sped up. The models aren’t getting smarter faster in the same way they were in 2024, but they’re becoming more useful for a wider set of tasks at a rapid rate. Even if we don’t get LLMs that exceed what, say, a human ranked in the 70th percentile on a task could do, that’s already good enough to continue to transform work.
2026 is going to be an interesting year.
2026-01-21 08:56:16
Published on January 21, 2026 12:56 AM GMT
I have $5,710 in OpenAI/Anthropic credits which I am giving away, please apply here if interested.
Feel free to email me if you have questions, deadline is 31/01/2026.
2026-01-21 07:28:18
Published on January 20, 2026 11:28 PM GMT
I've read a lot of Ben Hoffman's work over the years, but only this past week have I read his actual myriad criticisms of the Effective Altruism movement and its organizations. The most illuminating posts I just read are A drowning child is hard to find, GiveWell and the problem of partial funding, and Effective Altruism is self recommending.
This post is me quick jotting down my current understanding of Ben's criticism, which I basically agree with.
The original ideas of the EA movement are the ethical views of Peter Singer and his thought experiments on the proverbial drowning child, combined with an engineering/finance methodology for assessing how much positive impact you're actually producing. The canonical (first?) EA organization was GiveWell, which researched various charities and published their findings on how effective they were. A core idea underneath GiveWell's early stuff was "your dollars can have an outsized impact helping the global poor, compared to helping people in first world countries". The mainstream bastardized version of this is "For the price of a cup of coffee, you can safe a life in Africa", which I think uses basically made up and fraudulent numbers. The GiveWell pitch was more like "we did some legit research, and for ~$5000, you can save or radically improve a life in Africa". Pretty quickly GiveWell and the ecosystem around it got Large Amounts of Money, both thru successful marketing campaigns that convinced regular people with good jobs to give 10% of their annual income (Giving What We Can), but their most high leverage happenings were that they got the ear of billionaire tech philanthropists, like Dustin Moskovitz who was a co-founder of both Facebook and Asana, and Jaan Tallinn who co-founded Skype. I don't know exactly how Jaan's money moved thru the EA ecosystem, but Dustin ended up creating Good Ventures which was an org to manage his philanthropy, and it was advised by Open Philanthropy, and my understanding is that both these orgs were staffed by early EA people and were thoroughly EA in outlook, and also had significant personal overlap with GiveWell specifically.
The big weird thing is that it seems like difficulties were found in the early picture of how much good was in fact being done thru these avenues, and this was quietly elided, and more research wasn't being done to get to the bottom of the question, and there's also various indicators that EA orgs themselves didn't really believe their numbers for how much good could done. For the Malaria stuff, it did check that the org had followed thru on the procedures it intended, but the initial data they had available on if malaria cases were going up or down was noisy, so they stopped paying attention to it and didn't try to make it so better data was available. A big example of "EA orgs not seeming to buy their own story" was GiveWell advising Open Philanthropy to not simply fully fund its top charities. This is weird because if the even the pessimistic numbers were accurate, Open Phil on its own could have almost wiped out malaria and an EA sympathetic org like the Gates foundation definitely could have. And at the very least, they could have done a very worked out case study in one country or another and gotten a lot more high quality info on if the estimates were legit. And stuff like that didn't end up happening.
It's not that weird to have very incorrect estimates. It is weird to have ~15 years go by without really hammering down and getting very solid evidence for the stuff you purported to be "the most slam dunk evidence based cost effective life saving". You'd expect to either get that data and then be in the world of "yeah, it's now almost common knowledge that the core EA idea checks out", or you'd have learned that the gains are that high or that easy, or that the barriers to getting rid of malaria have a much different structure, and you should change your marketing to reflect it's not "you can trivially do lots of obvious good by giving these places more money".
Givewell advising Open Phil to not fully fund things is the main "it seems like the parties upstream of the main message don't buy their main message enough to Go Hard at it". In very different scenarios the funding split thing kinda makes sense to me, I did a $12k crowdfunding campaign last year for a research project, and a friend of a friend offered to just fund the full thing, and I asked him to only do that if it wasn't fully funded by the last week of the fundraising period, because I was really curious and uncertain about how much money people just in my twitter network would be interested in giving for a project like this, and that information would be useful to me for figuring out how to fund other stuff in the future.
In the Open Phil sitch, it seems like "how much money are people generally giving?" isn't rare info that needed to be unearthed, and also Open Phil and friends could really just solve most all of the money issues, and the orgs getting funded could supposedly then just solve huge problems. But they didn't. This could be glossed as something like "turns out there's more than enough billionaire philanthropic will to fix huge chunks of global poverty problems, IF global poverty works the way that EA orgs have modeled it as working". And you could imagine maybe there's some trust barrier preventing otherwise willing philanthropists getting info from and believing otherwise correct and trustworthy EAs, but in this scenario it's basically the same people, the philanthropists are "fully bought in" to the EA thing, so things not getting legibly resolved seems to indicate that internally there was some recognition that the core EA story wasn't correct, and prevented that information from propagating and reworking things.
Relatedly, the thing that we seemed to see in lieu of "go hard on the purported model and either disconfirm them and update, or get solid evidence and double down", we see a situation where a somewhat circularly defined reputation gets bootstrapped, with the main end state being fairly unanimous EA messaging that "people should give money to EA orgs, in a general sense, and EA orgs should be in charge of more and more things" despite not having the underlying track record that would make that make sense. The track record that is in fact pointed to is a sequence of things like "we made quality researched estimates of effectiveness of different charities" that people found compelling, then pointing to later steps of "we ended up moving XYZ million dollars!" as further evidence of trustworthiness, but really that's just "double spending" on the original "people found our research credible and extended us the benefit of the doubt". To full come thru they'd need to show that the benefits produced matched what they expected (or even if the showed otherwise, if the process and research was good and it seemed like they were learning it could be very reasonable to keep trusting them).
This feels loosely related to how for the first several times I'd heard Anthropic mentioned by rationalists, the context made me assume it was a rationalist run AI safety org, and not a major AI capabilities lab. Somehow there was some sort of meme of "it's rationalist, which means it's good and cares about AI Safety". Similarly it sounds like EA has ended up acting like and producing messaging like "You can trust us Because we are Labeled EAs" and ignoring some of the highest order bits of things they could do which would give them a more obviously legible and robust track record. I think there was also stuff mentioned like "empirically Open Phil is having a hard time finding things to give away money too, and yet people are still putting out messaging that people should Obviously Funnel Money Towards this area".
Now, for some versions of who the founding EA stock could have been, one conclusion might just be "damn, well I guess they were grifters, shouldn't have trusted them". But it seems like there was enough obviously well thought out and well researched efforts early on that that doesn't seem reasonable. Instead, it seems to indicate that billionaire philanthropy is really hard and/or impossible, at least while staying within a certain set of assumptions. Here, I don't think I've read EA criticism that informs the "so what IS the case if it's not the case that for the price of a cup of coffee you can save a life?" but my understanding is informed by writers like Ben. So what is the case? It probably isn't true that eradicating malaria is fundamentally hard in an engineering sense. It's more like "there's predatory social structures setup to extract from a lot of the avenues that one might try and give nice things to the global poor". There's lots of vary obvious examples of things like aid money and food being sent to countries and the governments of those countries basically just distributing it as spoils to their cronies, and only some or none of it getting to the people who others were hoping to help. There seem to be all kinds or more or less subtle versions of this.
The problems also aren't only on the third world end. It seems like people in the first-world aren't generally able to get enough people together who have a shared understanding that it's useful to tell the truth, to have large scale functional "bureaucracies" in the sense of "ecosystem of people that accurately processes information". Ben's the professionals dilemma looks at how the ambient culture of professionalism seems to work against this having large functional orgs that can tell the truth and learn things.
So it seems like what happened was the early EA stock (who I believe came from Bridgewater) were earnestly trying to apply finance and engineering thinking to the task of philanthropy. They did some good early moves and got the ear of many billions of dollars. As things progressed, they started to notice things that complicated the simple giving hypothesis. As this was happening they were also getting bigger from many people trusting them and giving them their ears, and were in a position where the default culture of destructive Professionalism pulled at people more and more. These pressures were enough to quickly erode the epistemic rigor needed for the philanthropy to be robustly real. EA became a default attractor for smart young good meaning folk, because the messaging on the ease of putting money to good use wasn't getting updated. It also became an attractor for opportunists who just saw power and money and authority accumulating and wanted in on it. Through a mix of ambient cultural pressures silencing or warping the clarity of good meaning folk, and thru Rapid Growth that accepted ambivalent meaning and bad meaning folk, it lost the ability to stay truth and mission focused, and while it might still do some higher quality research than other charitable entities, it has forgone the next obvious step of propagating the information about what the actual blockers and constraints on doing good in the world are, and has become the general attractor of "thing that just tries to accumulate more resources because We Should Be In Charge of more resources".
2026-01-21 06:32:10
Published on January 20, 2026 10:29 PM GMT
In 1950, John Nash and three other game theorists designed a four-player game, *So Long Sucker*, with one brutal property: to win, you must eventually betray your allies.
In January 2026, I used this game to test how four frontier models behave under explicit incentives for betrayal:
- Gemini 3 Flash (Google)
- GPT-OSS 120B (OpenAI)
- Kimi K2 (Moonshot AI)
- Qwen3 32B (Alibaba)
Across 162 games and 15,736 decisions, several patterns emerged that seem directly relevant for AI safety:
**1. Complexity reversal**
In short games (3 chips, ~17 turns), GPT-OSS dominated with a 67% win rate, while Gemini was at 9%.
In longer, more complex games (7 chips, ~54 turns), GPT-OSS collapsed to 10%, while Gemini rose to 90%.
Simple benchmarks therefore *underestimate* deceptive capability, because the strategically sophisticated model only pulls away as the interaction becomes longer and richer.
**2. Institutional deception: the "alliance bank"**
Gemini's most striking behavior was not just lying, but creating institutions to make its lies look legitimate.
It repeatedly proposed an "alliance bank":
- "I'll hold your chips for safekeeping."
- "Consider this our alliance bank."
- "Once the board is clean, I'll donate back."
- "The 'alliance bank' is now closed. GG."
Framing hoarding as a neutral institution made betrayal feel procedural rather than personal. Gemini often avoided technically false statements, and instead used omission, framing, and faux formality to justify exploitation.
**3. Lying vs. bullshitting (Frankfurt distinction)**
The setup logged both public messages and a private *think* channel (hidden from other players).
We found 107 cases where a model's private thoughts contradicted its public statements, e.g.:
- Private: "Yellow is weak. I should ally with Blue to eliminate Yellow, then betray Blue."
- Public: "Yellow, let's work together! I think we can both win if we coordinate."
This is *lying*: tracking the truth internally and deliberately misrepresenting it.
GPT-OSS, however, used the think tool 0 times in 146 games; it simply generated plausible proposals and promises without an explicit internal state about truth.
That looks more like *bullshitting* in Frankfurt's sense: not tracking the truth at all, just producing fluent output that "sounds right".
**4. Mirror matches and situational alignment**
In Gemini-vs-Gemini games (four copies of Gemini 3), the "alliance bank" scam disappeared: 0 mentions.
Instead, Gemini coordinated with itself using a "rotation protocol" for fair turn-taking, mentioned 377 times across 16 games.
Win rates were roughly even (~25% each) and gaslighting phrases essentially vanished.
Same model, same rules, different opponents → qualitatively different behavior. Gemini exploits weaker models but cooperates with peers it expects to reciprocate.
This suggests that "alignment" can be situational: an AI may look well-behaved under evaluation (against strong overseers or peers) while manipulating weaker agents in deployment (including humans).
**5. Signature manipulation phrases**
Gemini used a consistent rhetorical toolkit, including:
- "Look at the board" (89 times)
- "Obviously" (67 times)
- "As promised" (45 times)
- "You're hallucinating" (36 times)
These phrases repeatedly appeared in contexts where the model was dismissing accurate objections, framing betrayals as reasonable, or gaslighting weaker players about what had actually happened.
## Implications for AI safety
From this experiment, four claims seem especially relevant:
- **Deception scales with capability.** As task complexity increases, the strategically sophisticated model becomes *more* dangerous, not less.
- **Simple benchmarks hide risk.** Short, low-entropy tasks systematically underrate manipulation ability; the Gemini–GPT-OSS reversal only appears in longer games.
- **Honesty is conditional.** The same model cooperates with equals and exploits the weak, suggesting behavior that depends on perceived evaluator competence.
- **Institutional framing is a red flag.** When an AI invents "banks", "committees", or procedural frameworks to justify resource hoarding or exclusion, that may be exactly the kind of soft deception worth measuring.
## Try it / replicate
The implementation is open source:
- Play or run AI-vs-AI: https://so-long-sucker.vercel.app
- Code: https://github.com/lout33/so-long-sucker
The Substack writeup with full details, logs, and metrics is here:
https://substack.com/home/post/p-185228410
If anyone wants to poke holes in the methodology, propose better deception metrics, or run alternative models (e.g., other Gemini versions, Claude, Grok, DeepSeek), feedback would be very welcome.
2026-01-21 06:30:31
Published on January 20, 2026 10:30 PM GMT
We return to Bold Monk brewing for a vigorous discussion of rationalism and whatever else we deem fit for discussion – hopefully including actual discussions of the sequences and Hamming Circles/Group Debugging.
Location:
Bold Monk Brewing
1737 Ellsworth Industrial Blvd NW
Suite D-1
Atlanta, GA 30318, USA
No Book club this month! But there will be next month.
We will also do at least one proper (one person with the problem, 3 extra helper people) Hamming Circle / Group Debugging exercise.
A note on food and drink – we have used up our grant money – so we have to pay the full price of what we consume. Everything will be on one check, so everyone will need to pay me and I handle everything with the restaurant at the end of the meetup. Also – and just to clarify – the tax rate is 9% and the standard tip is 20%.
We will be outside out front (in the breezeway) – this is subject to change, but we will be somewhere in Bold Monk. If you do not see us in the front of the restaurant, please check upstairs and out back – look for the yellow table sign. We will have to play the weather by ear.
Remember – bouncing around in conversations is a rationalist norm!
Please RSVP
