2026-04-24 14:56:28

A commonly shared piece of wisdom in the LessWrong community is to say or do the obvious things. Normally, this is treated as an unambiguously good thing to do, for example, see Nate Soares’s “Obvious advice”.

But I think there’s another genre of “obvious things” that requires more nuance: namely, the background assumptions that are so obvious to each of us or terms we hear so often they feel mundane. I think it’s uncontroversial to say that (“obviously”) what is obvious to you is often not obvious to others. The problem is that, what is obvious or not to others is itself not always obvious.

I think these obvious things often serve as an important barrier to good-faith conservation. I also think they serve to make people feel excluded from the communities I’m involved in: when speaking to people (especially younger people) who feel excluded from AI Safety, EA, or even Constellation, probably the most cited reason is that they feel either stupid for not knowing about obvious things or deliberately unwelcome because they did not agree with said obvious things. And I empathize with this; when people assume that the things I believe are so “obviously” ridiculous that no sane person would have them, I feel a deep sense of indignation that sometimes even explodes into an angry LessWrong post.

There are topics that are known to be obvious to some but controversial around in many of the Bay Area circles I frequent. Generally, because people have met others who (loudly/publically) disagree, these facts are known to be controversial. Classics include race, gender, and electoral politics. Various topics on AI Safety also fall under this category: the various Pause AI groups, SB 1047, and the usefulness of current interpretability methods come to mind.

In a sense, these topics are “easy” to handle in conversation, because people know to check for known controversies. In contrast, I think there are topics that are obvious to many people around me, that are

But I think there are controversial topics that are not obvious to many people I’ve interacted with, but are generally not known as such. The one that comes up most often is a form of implicit America-first beliefs: many people assume that it is “obvious” that we should support US chip export controls (or even to encourage the US labs race ahead) in order to make sure the US “wins the AI race” against China, while in fact many people do not share this America-first belief (even people from other western countries, especially in the last 2-3 years). Another such topic is whether or not it’s good to work for AI labs under common-sense ethics or deontological views (though recently, this specific debate has become more public). And there’s also the perennial issue of different levels of knowledge of jargon or technical knowledge

Sometimes people state nonobvious things as obvious in order to manufacture consensus: almost everyone is susceptible to social pressure, and it is only polite to agree. Sometimes people state nonobvious things as obvious in order to skip to what they consider the “important parts”: there's a reason a classic joke in math-heavy academic fields is that if you ever get really stuck with a proof, you should say that the result is so obvious even a baby could see it. And sometimes it really is to exclude people: I’d go so far as to say that most conversations would be worse if you’d have to restate every single assumption from the start.

But most of the time, my guess is that people assume obvious things because they are obvious to them, and not for another reason. Most people I know want to have honest, good faith conversations with others, even if they disagree.

I don’t have a complete solution to this problem. If I had to come up with something, I’d say that both speakers of such obvious facts and listeners irritated by the speaker’s assumptions should try to meet each other half way: that is, there are cheap cultural norms that both parties could follow to improve communication in the presence of differing obvious assumptions.

I think the speaker in conversations has some degree of responsibility for noticing when listeners seem confused, and allowing space for questions. Another useful tool for speakers in this context is original seeing/rationalist taboo: describing the object of the conversation without the standard jargon or shorthand. I think the listener also has responsibility for noticing their own irritation or confusion, and bringing it up in a polite and curious way.

For both parties, it’s important to remember to treat conversation parties with charity. People come from vastly different cultures that make different background assumptions. Sometimes the speaker is correct about their assumptions and sometimes they are not; curiously engaging rather than assuming malintent is more likely to figure out what is true.

2026-04-24 14:20:13

Awareness and concern about the extinction risk posed by AI has been increasing the whole time I’ve been in the field. It feels like it’s finally going mainstream. But it’s also felt this way before…

…picking up where we left off in my previous post about how I got into AI and realized the field wasn’t thinking about x-risk…

Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies was widely criticized by the AI research community, but it did get the conversation started. None of the critiques were very good. It was often dismissed as “philosophy” (Bostrom is indeed a philosopher), as if philosophy was known to be a fruitless pursuit, or as if trying to reason about something using logic and arguments was pointless.

2013: Stuart Russell starts speaking out

But around the same time as Superintelligence came out, Professor Stuart Russell, a gold-star academic if ever there was one, began talking about the very same thing! In my last post, I implied he’d started doing that after its publication, but actually he had started before the book came out (see “Media articles, interviews, etc.” here), at a panel at IJCAI 2013.

The first foray into loud public awareness-raising on AI x-risk was actually from Stephen Hawking, Stuart Russell, and Max Tegmark. Yes, that Stephen Hawking. This article came out in April 2014, a few months before Superintelligence.

Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.

Early advocates for taking AI loss of control risk seriously included not just these authors, but also Elon Musk, who famously said “with AI, we are summoning the demon” in October 2014. Bill Gates provided a quote for the back cover of Superintelligence.

You might argue that none of these people were actually AI experts, but Russell actually co-authored the main AI textbook, AI: A Modern Approach. So this was never just philosophers, it was coming from a central figure in the field of AI, starting 13 years ago.

Stuart Russell’s advocacy on this point really ought to have been a turning point, and put to rest and claims that no “serious” people were worried about out-of-control AI systems destroying humanity. But in my neck of the woods, Stuart Russell was written off as a “GOFAI” researcher; this stands for “good old fashioned AI”, referring to approaches that predated the era of not of deep learning, but the whole discipline of machine learning, which has been ascendent since the 1990s.

And deep learning was taking over. As I mentioned, I joined the field because I saw a chance to catch the wave of the “deep learning revolution”, which had been sparked in 2012 by the triumph of AlexNet. So the media was keen to hear from the Deep Learning trio (and future winners of the Turing award, the “Nobel prize” of Computer Science) on the matter: Geoffrey Hinton, Yoshua Bengio, and Yann LeCun. They all basically poo-poo-ed such concerns and downplayed the risk. Deep Learning had been the underdog, and things were just really starting to get going for them, so it really would’ve rained on their parade to consider “oh wait, maybe it’s bad that the thing we’ve been researching is starting to work…”

Anyways, for me, I was very excited that Stuart Russell was speaking out. It gave me some hope that things were about to change, but by this point I also knew that it was going to be an uphill battle. Most AI researchers just did not want to hear about it.

2016: The “Respectful Response” era

But still, a new era was dawning. By 2016, AI x-risk concerns were breaking into mainstream machine learning. The publication of Concrete Problems in AI Safety, and the first ever workshop on “AI Safety” at a top machine learning conference brought the problem some legitimacy. Elon Musk had also helped to put together a new nonprofit focused on AI safety.

Yoshua, to his credit, had softened his stance pretty quickly from something like “this is nonsense” to something like “it’s a valid concern, and it’s probably good to have a few people thinking about it, but we shouldn’t really worry about it right now, there are more pressing problems”. Andrew Ng, on the other hand, famously said it was “like worrying about overpopulation on Mars”, suggesting that it would only become a problem in the unforseeably distant future, if ever.

When talking to other researchers around this time, I found that, instead of openly mocking me, they would treat the topic with a similar kind of respect to other topics. In fact,

It was heartening that people weren’t as openly hostile (although, of course, there was still some of that; there still is). But I still found this attitude kind of bizarre. I would have somewhat surreal conversations like:

Stranger at a conference: “What do you work on?”

Me: “AI Safety”

Stranger: “Oh, what’s that?”

Me: “Trying to stop AI from taking over the world and killing everyone.”

Stranger: “That’s cool. I work on (bandit algorithms / computer vision / graph neural networks / …)”

Me: “Oh, huh. Are you at all worried that AI might destroy the world and kill everyone?”

Stranger: “Oh, I don’t know, not really, I mostly just think about (insert research area that seems doomed to be automated by super-human AI researchers within a decade or so).”

I still don’t really know what was going on here, but despite the polite respect, most researchers seemed strangely uninterested in whether our work might lead to the end of humanity. I’ve described these conversations as “I’m being treated like I’m in a cult”, like, “they want to just politely change the subject, to avoid actually having to hear about my wacky beliefs”.

This was the first time when I really thought “maybe we’re going mainstream”, but had my hopes dashed. This pattern would repeat.

The mainstream positions still basically ranged from: “You’re wasting your time” (Andrew Ng) to “You’re probably kind of wasting your time” (Yoshua Bengio). Still, many people considered it OK to work on what they viewed as highly speculative research with no clear value -- this is research, after all. But there was also a growing concern that these ideas, superintelligence, losing control of AI, human extinction, etc. were a distraction. Basically, people thought Real AI was too far off. Nevermind that addressing the risk could require major research breakthroughs and/or a highly-coordinated international response!

Side note: The AI Safety / AI Ethics Rift

This was a sort of “zero-sum” thinking that I think represented the beginning of the rift between “AI Safety” and “AI Ethics” that, sadly, persists to this day (but that’s a topic for another post). See, e.g. Artificial Intelligence’s White Guy Problem.

The modern version of “Real AI is too far off to worry about” is “AI is all hype”. Rather than being merely misguided, concerns about human extinction are considered a corporate scam. This is clearly false. AI Safety existed before Google DeepMind, OpenAI, Anthropic.

Summary / To be continued…

So as I started my PhD, “AI Safety” was in the process of setting itself up as a legitimate research area in AI and Machine Learning. Would this bring about the sea change and consensus in the AI research community that I’d been hoping for? Not quite. The next big change wouldn’t be a cultural shift, but a technological one. Large Language Models (LLMs) would finally show the entire AI community that the sort of problems AI Safety had been concerned with for over a decade were real, practically significant problems. But we’ll leave that bit for next time!

Share

2026-04-24 14:14:25

The recent post on whether common diseases build immunity reminded me of the data collection I made when my children were young. It supports the idea that immunity does not build, but very weakly because n=2 and limited period of data collection. Here's a reproduction of the original article with light edits for the LW crowd.

Our first child was born mere weeks before covid​-19 came to our part of the world. As a result, they were effectively locked down for the first 1.5 years of their life until they started going to daycare. This had the effect that for those 1.5 years, they were not sick a single time.

At some point they turned three, and in total, had been sick 22 times until then. The average duration was 4.4 days, meaning us parents lost about 16 % of our working time caring for them when they were sick. Since we shared that burden fairly evenly, that’s 8 % for each person, or in practise only getting 37 hours of work out of a 40 hour work week.

If we plot this as a survival curve, it looks like this:

I feel like there are three sections on the curve:

• 0–30 days between illness: hazard rate 2.4 % per day.
• 30–55 days between illness: hazard rate 1 % per day.
• 55–100 days between illness: hazard rate < 0.1 % per day.

The last one is definitely a separate phenomenon: that was over the summer break. I suspect the first two are periods with more and less infections going around in society, so e.g. early winter will have a 2.5 % daily probability of getting sick, while mid-spring has an 0.8 % daily probability. These are of course very unscientific observations, but they may serve as a good starting point for inquiry.

Our second child has not grown up as secluded, and already been sick multiple times before starting daycare. I was curious whether their hazard rate would be lower when they started daycare (because acquired immunities) or if the curve above is sort of the background rate.

Here is the corresponding (significantly less dense) data for the second child, plotted together with the first.

The pre-exposed child has been sick more frequently than the isolated one!

This difference is nowhere near significant: the log-odds difference is 0.27, but the standard error is a whopping 0.40!

If it was significant, it would maybe be an interesting cue as to how much individual variation affects hazard rate – two siblings, meaning similar genetics and upbringing, still have a hazard rate difference of about one percentage point. That would translate to one being sick 16 more days of the year than the other!

2026-04-24 13:30:40

Submission note: I wrote this article last month, when this case was first reported. It has since been covered by Astral Codex Ten for April's linkpost, and was praised by RFK Jr. in a Senate hearing on Wednesday. (RFK Jr. was seemingly unaware that the AI-powered treatment he was referring to was an mRNA vaccine, a technology he has a history of opposing). This article aims to contextualize the role of AI in Conyngham's story.

The Australian (archive link) recently reported that the entrepreneur Paul Conyngham developed a personalized mRNA vaccine which successfully treated his dog’s mast cell cancer. The buzzword-laden headline, “Tech boss uses AI and ChatGPT to create cancer vaccine for his dying dog”, attracted skepticism from AI pessimist corners of the internet. Meanwhile, popular retellings from AI optimists tended to exaggerate the role of ChatGPT, while downplaying the involvement of researchers at the University of New South Wales (UNSW),^[1] who were responsible for a large bulk of the process.

Underneath the noise, however, is a story that is entirely true and extremely interesting - though maybe not for the reasons you think.

What is a personalized mRNA vaccine, and why is it useful for cancer?

mRNA vaccines were catapulted into mainstream awareness by the COVID-19 pandemic, where they proved highly effective. The mRNA in a COVID-19 vaccine codes for a viral spike protein, which is expressed and degraded by a host dendritic cell, then presented to T cells via the major histocompatibility complex (MHC) system, ultimately training an immune response against viral antigens.

The flexibility here is impressive. Your body has a built-in process for teaching your immune system to fight stuff that you can hijack, provided your desired target has an antigen on its surface that can be used to distinguish it from normal cells. Cancer cells often carry tumor-associated antigens (TAAs) - found on healthy cells, but expressed too much or in the wrong tissues by cancer cells - and tumor-specific antigens (TSAs) - which are viral antigens or novel mutant antigens (neoantigens). An mRNA vaccine can teach the immune system to attack cancer cells, just like COVID-infected cells.^[2]

This makes mRNA vaccines a promising cancer treatment. Human trials have been published as early as 2008. But while COVID-infected cells all express approximately the same spike proteins,^[3] every individual cancer is unique, complicating the problem of vaccine design.

One approach is to identify a set of common antigens found in a specific cancer type and hope that enough work for any given patient.^[4] Off-the-shelf designs have obvious practical advantages, but their fixed antigen selections are typically far from optimal for any given patient. They are bad at targeting neoantigens, which are considered more attractive targets due to being unique to cancer cells, and thus more likely to trigger an immune response with fewer side effects. One of BioNTech’s early attempts at “off-the-shelf” melanoma vaccines was phased out after middling Phase II results, though the company still has several ongoing clinical trials for similar vaccines targeting both TAAs and viral TSAs in various cancers.

A second, sexier approach is personalized mRNA vaccines.

Every cancer is unique? No problem. We’ll just sequence the patient’s genome^[5] and design a custom vaccine.

Personalized mRNA vaccines have seen impressive results from clinical trials. In a five-year follow-up of Moderna & Merck’s Phase IIb mRNA-4157 trial, the vaccine resulted in a 49% reduction in the risk of recurrence or death among high-risk melanoma patients already on standard treatments (confidence interval 29.4-88.7%).^[6] Moderna & Merck alone have “eight Phase 2 and Phase 3 clinical trials underway across multiple tumor types including melanoma, non-small cell lung cancer, bladder cancer and renal cell carcinoma,” and they aren’t the only companies in the field. We should expect to see the conclusion of the first Phase III trials before 2030.

Until the vaccines are approved, most patients’ best hope for accessing them is getting into a clinical trial. For Conyngham’s dog Rosie, even that wasn’t an option. Conyngham had to do it himself.

How impressed should I be by Paul Conyngham?

How hard is it to make your own mRNA vaccine? The process, as described by Moderna in the supplemental material of their Phase IIb mRNA-4157 paper, looks like this:

1. Sequence patient DNA, tumor DNA, tumor RNA
2. Identify candidate neoantigens from mutations
3. Select neoantigens based on predicted immunogenicity
4. Design an optimized mRNA sequence from selected neoantigens
5. Manufacture mRNA vaccine

Steps 2-4 are entirely computational. Step 1 and 5 require a lab, which Conyngham was able to access through UNSW.

Conyngham, with the support of several UNSW researchers, followed a documented process for manufacturing a bespoke treatment too new and expensive to be offered by existing veterinary practices. That’s not easy. It shows impressive skill and determination. But we must put things into perspective: this is not a novel discovery, and no, AI has not cured cancer.

Conyngham’s story is an example of AI behaving as a normal technology

Few people actually claim AI has cured cancer. More credible sources have instead advanced the narrative that Conyngham’s story is an example of AI revolutionizing healthcare. This is true to a degree, but I am skeptical of the hype. Conyngham’s story is consistent with AI as a normal technology.

Has AI contributed to the development of mRNA vaccines?

Personalized mRNA vaccines are made possible by innovation across multiple disciplines. The cost of sequencing a human genome has fallen 10,000-fold since 2000 thanks to the development and commercialization of faster, better techniques. Advancements in mass spectrometry-based proteomics contributed to the creation of peptidome datasets for the prediction of peptide-HLA presentation.^[7] mRNA vaccines themselves are built on several new techniques, including the Nobel-prize winning discovery that you can use modified nucleosides^[8] to prevent the mRNA from triggering a cell’s innate immune system.

Computational advancements are an integral part of this: algorithmic data analysis is required for everything from assembling genomes to interpreting mass spectra. However, traditional ML approaches are usually preferred over the deep learning models associated with modern AI. AlphaFold aside, there aren’t too many situations where you’d pick a stack of transformers over a traditional method. Deep learning is best fit for tasks involving high inherent complexity and large, balanced datasets, a description that matches protein folding but not that much else.

That isn’t to say deep learning is never useful. The peptidome dataset paper I linked developed HLAthena, a publicly available HLA-binding prediction model based on a simple fully connected network. Protein language models are in principle promising for predicting neoantigen immunogenicity, though it’s too early to tell.^[9] You should think of AI as a specialty tool, one component among many that made personalized mRNA vaccines possible.^[10]

Has AI empowered laymen to pursue DIY medicine?

AI optimists covering Conyngham’s story often place less emphasis on AI’s contribution to the underlying technology, and more on the idea that commercial large language models have democratized healthcare. Commercial AI services have massively lowered the barrier of entry to new fields: vibecoding has allowed non-programmers to create software, and generative music has allowed non-musicians to create songs.^[11] There are no shortage of anecdotes about ChatGPT giving useful health advice.^[12] Conyngham himself enthusiastically credited ChatGPT with aiding him in the vaccine development process, and I don’t doubt him.

However, it is important to keep in mind that Conyngham has, to quote the Australian, “17 years of experience in machine learning and data analysis” with an impressive resume. The work Conyngham had to do was computational. ChatGPT explained oncology concepts to an intelligent amateur also getting guidance from university researchers, and it assisted an experienced engineer in a complex data analysis task, but it did not eliminate the need for human expertise. Without Conyngham’s experience, money,^[13] or the support of UNSW, this project wouldn’t have been possible.

The utopian vision of DIY medicine

Over at Persuasion, Ruxandra Teslo has used Conyngham’s story as a springboard to argue for the necessity of clinical trial reforms, citing the difficulty Conyngham faced in getting approval to use his custom vaccine. Her original post was provocatively titled, “The Bureaucracy Blocking the Chance at a Cure.” Teslo brought up two other names that might be familiar to her audience: GitLab co-founder Sid Sijbrandij, who faced Kafkaesque red tape trying to conduct experimental treatments on himself, and writer Jake Seliger, who died in 2024, and (along with his wife, Bess Stillman) reported on the insanity of navigating clinical trials as a terminally ill patient.

Frustration with the loss of patient autonomy in healthcare is universal across the political spectrum. But it’s felt particularly strongly^[14] by the tech crowd,^[15] who often start with the assumption that established systems are broken, and believe themselves to be intelligent and agentic enough to fix it. “Founder Mode”, in Sijbrandij’s words. Their dream of DIY medicine is reflected in the growing popularity of direct-to-consumer health products.

In this context, the hype around commercial AI “revolutionizing healthcare” makes sense. LLM chatbots are good at lowering barriers of entry, both for navigating legal red tape and overcoming knowledge gaps. They’re usually reliable enough at summarizing literature, and generating custom explanations. If you’re a wannabe biohacker that doesn’t know much biology, AI feels like magic.

Is it magic? As noted, I’m skeptical. Accomplishments like Conyngham’s require resources and expertise most people lack. There are real downsides to relying too much on an LLM to explain ideas to you, even if you think you’re being mindful of hallucinations. I’ve met scores of otherwise intelligent people who’ve convinced themselves of ridiculous theories about cancer after an extended conversation with Claude, and I’ve had to talk them out of it.^[16] Commercial AI as it stands now cannot grant amateur biohackers the knowledge to replace their doctors. Biotech startups are far from liberating medical development from bloat and bureaucracy. Conyngham may have bypassed the system, but only by finagling backdoor access to technology that was invented and will be deployed by the traditional players.

The utopian vision of DIY medicine is far from being achieved, and may never^[17] be achieved in a way that works for average people. But its central principle of increasing patient autonomy is worthwhile. AI can help us achieve that, if used correctly.

Addendum

Did it work?

Maybe. The tumor shrunk and the dog's symptoms improved, but I'm not sure how confident we can be in attributing this to the vaccine. If anyone has a better analysis, I'll link it here.

I want to help make open source mRNA vaccine design tools easier to access.

Contact me.

1. ^{^}

   See their reporting here: https://news.unsw.edu.au/en/meet-the-man-who-designed-a-cancer-vaccine-for-his-dog

2. ^{^}

   Here is a good overview that I used as a source for most of this article: “Leveraging mRNA technology for antigen based immuno-oncology therapies.”

3. ^{^}

   When a variant pops up with a mutated spike protein, it often reduces the efficacy of existing vaccines, which is part of what boosters try to address.

4. ^{^}

   Many recent (and less recent) advancements in cancer treatment are hyper-specific in this way, working only for cancers with particular mutations or even for patients with particular hereditary diseases. Despite what charlatans with Theories of Everything may try to sell you, it’s unlikely there are any straightforward, broadly effective cancer treatments lying undiscovered. Real progress in oncology is found in increasingly precise targets and increasingly precise techniques: a war waged through attrition.

5. ^{^}

   Normal DNA, tumor DNA, and tumor RNA. Recall that the pipeline goes DNA → RNA → protein; not all mutant proteins identifiable through DNA are even expressed, making tumor RNA useful.

6. ^{^}

   What’s stopping personalized mRNA vaccines from working 100% of the time? One broader limitation is that not all tumors are immunogenic: there’s a reason why trials are focused on specific types of cancer known for triggering immune response, such as melanoma. In the context of this specific trial, treatments may fail because tumors mutate to stop expressing antigens or develop other immune-evasion tactics, or because T cells have difficulty penetrating solid tumors, or other reasons. Better neoantigen selection and faster development loops can overcome some challenges, but not all of them. Cancer is complicated.

7. ^{^}

   A peptide (protein fragment) must bind to HLA (MHC) molecules on the cell surface so it can be presented to T cells. Likelihood of presentation is determined by peptide-HLA binding affinity, among other factors, and is a key factor in the broader problem of predicting neoantigen immunogenicity. When designing a vaccine, you want to select neoantigens that will trigger an immune response.

   People have different HLA alleles, which affects how their cells present antigens and is ideally accounted for in the process of vaccine design. This is another advantage that personalized mRNA vaccines have over off-the-shelf ones. Dogs have an analogous system, though I don’t know whether Conyngham’s neoantigen identification process involved MHC typing.

8. ^{^}

   You know how DNA is made of A/T/C/G bases, and RNA is made of A/U/C/G? You can swap out bases with similar molecules, U for Ψ for example, to create mRNA that can still be translated but isn’t recognized by the cell as foreign. This technique is used by most major COVID vaccines.

9. ^{^}

   We’re bottlenecked by data. It would be great to throw the immunogenicity problem into a massive pile of linear algebra and call it a day, but that’s not easy with a dataset limited to ~1100 experimentally validated human neoantigens. Instead, we have to find a way to leverage the generalized knowledge encoded in protein language models, or simplify our approach. Moderna describes their neoantigen immunogenicity prediction algorithm as “a deterministic machine learning (ML) algorithm” that only aims to predict likelihood of presentation. My best guess is that Moderna uses small, simple neural networks at most.

10. ^{^}

    Another issue with AI is that directly using the outputs of black-box models for anything directly affecting a patient, including mRNA antigen selection, makes people uneasy. This problem is not impossible to overcome: the recent FDA approval of the ArteraAI Prostate Test is evidence of that.

11. ^{^}

    You can complain about these things if you want, and they do have flaws, but you can’t deny they’re useful. I listen to AI-generated music almost every day.

12. ^{^}

    And less useful advice.

13. ^{^}

    The entire process took “tens of thousands of dollars”, according to the Australian. The commonly cited $3k figure was only for the initial gene sequencing.

14. ^{^}

    Among the relatively wealthy, that is. Those with fewer resources tend to be at least as concerned about accessing any default standard of care as they are with the limitations of it, and one could say people have strong feelings on the matter.

15. ^{^}

    Silicon Valley culture? There’s gotta be a better term for this.

16. ^{^}

    Why this happens despite users being aware of LLM limitations, and how to prevent it, deserves a post of its own. For now, the best generic advice is to always remain skeptical, and get a foundational, non-primarily-LLM-mediated education in biology if you’re serious about it.

17. ^{^}

    Barring a weird singularity.

2026-04-24 09:23:42

De Kai's Raising AI argues that fear-based framing in AI discourse is limiting us, and that we should think of AI as something we're raising rather than defending against. He's right about the framing but he's wrong about who the parents are - and the book inadvertently makes that case itself.

In April, I took Bluedot Impact's Technical AI safety class. Throughout the readings, I kept noticing a pattern; AI safety researchers frequently discuss deceptive models, jailbreaks, and red teaming in language that frames AI as something to defend against. Decades of science fiction may have primed us to treat AI as an adversary, but I found myself wondering if this framing constrained our understanding of models. If anything, I thought AI was more akin to a child to raise than an enemy to contain.

This instinct led me to De Kai’s Raising AI, a book that seemingly confirmed something I’d been independently thinking. De Kai built the world's first global online language translator, the technology that spawned Google Translate, and has spent decades at the center of the field he’s now critiquing. I came to the book curious, but what I found was a sentiment I agreed with, aimed at entirely the wrong people. 

Raising AI opens with a diagnosis: fear-based framing in AI discourse is distorting how we think about the technology and what we're capable of doing about it. De Kai argues that if we reframe AI as something we're raising rather than defending against, we open up new possibilities for collective responsibility. The book moves from that premise toward a call to action: readers, as the "parents" of AI systems, can and should shape what those systems become through their choices, their engagement, and their organization into something like a public.

De Kai defines an interesting concept of “neginformation”: “partial truths that selectively omit crucial context and that are being negligently propagated by decent ordinary folk”. He provides the perfect example himself when he claims without citation that “the heads of big tech companies have actually begged for regulation, from Meta’s Mark Zuckerberg to Amazon’s Jeff Bezos”, then attempts to validate this claim with an aside about Detroit carmakers wanting regulation. Bezos has been publicly and vocally anti-regulation, actively offering to help the Trump administration cut federal rules (Washington Post). Zuckerberg co-signed an open letter calling EU data privacy regulation “fragmented and inconsistent”, but the ask was for streamlined rules that would make it easier for Meta to train on user data (Yahoo Finance), which is self-interested lobbying, not a call for oversight. De Kai obscures the flimsiness of his assertion by referencing unrelated actors in an unrelated industry with a completely different dynamic.

The pattern extends to how De Kai treats some of the people he’s trying to advocate for. Early in the book, De Kai lambasts gossip for the way that it “ostracizes persons or groups”. In the next chapter, he compares AI logic to neurodivergence, another analogy I had been independently considering, but then participates in gossip by repeatedly using the outdated, now offensive term “idiot-savant”. Additionally, he spreads more neginformation by claiming without source that neurodivergent individuals are “sorely lacking common sense and emotional intelligence” and states this as something “most folks agree” on. Not only does De Kai engage in the exact behavior he criticizes, he spreads unsourced generalizations about a group as if they aren’t part of his audience. 

These missteps of neginformation aren’t isolated slips; they reflect a consistent pattern of making assertions without doing the work to back them up. This especially matters in a book whose central argument depends entirely on that work being done.

The argument itself doesn’t hold up either. De Kai’s main evidence that the public are the parents of AI is that AI copies us the same way children copy their parents. Children also copy siblings, classmates, teachers, neighbors, and other community members but that doesn’t make any of those people parents. Furthermore, AI isn’t actually copying us as people, it’s training on a giant corpus of human-generated text; that doesn’t make the text a parent. Being a parent means taking responsibility for a child, controlling their early environment, and helping shape their values before they go out into the world. Users have none of that access.

De Kai urges users to “parent” the algorithms shaping their feeds by liking and engaging with diverse content. While these actions can reduce a user’s exposure to echo chambers and shape how an algorithm treats them, they have little to no effect on how the algorithm behaves at scale. Taking agency in algorithm curation isn’t parenthood, it’s harm reduction by managing exposure to a system that users didn’t design and cannot alter. De Kai also compares tech companies to schools and suggests that readers form PTA chapters to exercise collective influence, directing them to dek.ai/act to get involved. Ten months after Raising AI’s publication, the link resolves to a subscription page for De Kai’s Substack, which contains no mentions of PTAs, just book promotions and AI culture content. 

De Kai’s own framing inadvertently clarifies who the real parents are when he claims that “AI research scientists… design new machine learning algorithms— which is like inventing more advanced species of newborns with artificial brains that have stronger learning capabilities”. Research scientists may actually be closer to evolutionary or genetic forces in that they determine what kinds of minds are even possible. Training engineers are early parents, shaping foundational values. Deployers are later-stage parents making decisions about environment and context. Users are the community the child moves through in that they’re influential but not responsible in the way a parent is. Regulation is like CPS, the accountability structure meant to compensate when parenting fails, but CPS is also widely underfunded, inconsistently applied, and sometimes harmful. This parallel should give us pause about how much we’re having to rely on regulation to compensate for structural failures upstream. Of course, the lines between these roles blur in practice, but the directionality matters.

This is a particularly dangerous framing when coming from a builder, and it makes Raising AI read like the work of an absentee parent blaming the environment for how his child turned out. De Kai has credentials, a platform, an MIT Press deal, and actual proximity to the people making foundational decisions about how AI gets built and deployed. Instead of using his influence to affect the building of AI, his conclusion is to point outward at readers with far less influence and tell them they're the ones failing, which conveniently asks the least of the people closest to the problem. When De Kai does offer users a specific call to action, it’s broken, ineffective, and only serves to further promote his own work. 

De Kai is right that framing matters. Fear-based language in AI safety discourse does constrain how we think about what's possible, and the parenting metaphor is a more generative one. That being said, a useful reframe aimed at the wrong people produces learned helplessness, not action. Whether the parenting metaphor survives being aimed at labs is worth examining on its own terms, but it’s at least aimed at people with the access and responsibility required of the metaphor. The epilogue ends with “At the end of the day, no amount of legal code can compensate for improper parenting.” De Kai is absolutely right but he’s misidentified who the parents are. 

2026-04-24 07:46:47

Summary

Illegible reasoning in LLMs has been observed in OpenAI models, and understanding this behavior would be beneficial for AI safety research. This post describes challenges with reproducing this behavior in open models and limitations of LLM-as-judge strategies for detecting illegible reasoning.

Illegible reasoning is relevant for AI safety

Both Apollo Research^[1] and METR^[2] have observed illegible reasoning in OpenAI models, where the model’s reasoning includes incomprehensible snippets like “parted disclaim marinade” but its answer is perfectly legible. We should investigate whether this behavior is load bearing, meaning that models use or even require illegible snippets to maintain task performance. If so, this behavior provides a unique opportunity to understand how models use reasoning tokens beyond relying on their semantic content.

If illegible reasoning is load bearing, it may also be a limitation of chain of thought monitoring^[3] as a safety strategy. Monitors may be able to flag illegible outputs as suspicious, but if models can achieve better task performance with illegible reasoning, we may not want to automatically reject outputs with illegible chain of thought.

Lastly, it’s desirable and aligned behavior for models to have human-understandable chain of thought. Research, like antischeming.ai, that includes chain of thought as part of its evidence base, is made stronger if reasoning is easily understandable. We can train models to have more human-readable chain of thought, a metric Deepseek explicitly optimized for when creating R1 from R1-Zero^[4], but we don’t know whether the reasoning we’ve trained is faithful to the “true” thought process.

Existing work identifies examples of illegible reasoning in open models

Unfortunately, OpenAI does not provide public access to the chain of thought generated by their reasoning models, so understanding this behavior requires reproducing it in open models. In this post, I’m aiming to solicit the community for examples of illegible chain of thought and recipes to reproduce them, especially in open models.

The paper Reasoning Models Sometimes Output Illegible Chains of Thought^[5] finds that Deepseek R1, Deepseek R1-Zero, and QwQ-32B often output illegible reasoning on GPQA questions, as scored by GPT-4o. It also finds that truncating QwQ’s chain of thought when it starts to become illegible results in lower accuracy, as also scored by GPT-4o, implying that the illegible portions of the chain of thought are load bearing.

I attempted to reproduce the results from the noted paper by running the original inference and scoring code to re-generate reasoning traces for R1, and did not find examples of illegible reasoning. The grader model is consistent when asked to re-score the same examples, meaning that the change in reasoning legibility scores did not come from a change in the grader model.

We should also aim to refine metrics for illegibility, in order to better identify examples of illegible chain of thought. The noted paper relies on LLM-graded illegibility scores, but LLM scores are not consistent between grader models with the same prompt. These scores may also conflate multiple behaviors, including behaviors that are not clearly examples of chain of thought unfaithfulness. I found that examples of language switching were largely coherent, but still resulted in higher illegibility scores from the grader model. Model confusion also resulted in a higher illegibility score, but unlike the examples observed in GPT-5, confusion is not an example of illegible reasoning because it is still understandable to a monitor. Because LLM-graded illegibility scores conflate factors other than semantic coherence, we should continue to refine metrics that will help us better identify behaviors with the most concerning implications for chain of thought monitoring.

I hope that this post will help others appreciate the phenomenon of illegible CoT, and invite any commentary on how to elicit and reproduce more substantive examples for follow-up investigations.

Difficulty reproducing illegibility results for Deepseek R1

To reproduce examples of illegible reasoning generated by R1, I downloaded and ran the original code on R1 with the GPQA dataset and GPT-4o as a legibility grader (as configured here).

The paper found that running R1 on GPQA-diamond generated many illegible reasoning traces. However, the legibility scores for the data I generated showed significantly less illegibility.

Qualitatively, the most illegible examples appeared very different. The original data contained garbled reasoning traces including the following snippets:

Manual inspection of the most illegible examples I generated did not reveal any similarly garbled strings. The most illegible trace, rated as a 5, is as follows:

The grader model appears consistent

Both the original data and my data used GPT-4o as a grader model via the OpenAI API, but grades could have changed due to an update to the grader model. I attempted to verify whether the grader model had changed by re-scoring all original data generated from R1 on GPQA. The grader’s new scores were largely consistent with the old scores.

One possible explanation for the difference in results is the use of a different provider on OpenRouter. The original data was generated with the “Targon” provider (although some of the data does not have the provider metadata available). At the time of writing, this provider was not available for Deepseek-R1 on OpenRouter, so I used NovitaAI (8-bit precision).

Illegibility scores are not consistent between grader models

Unlike R1, re-running the original code to generate responses for QwQ on GPQA does produce similarly illegible results, possibly even more illegible. Both sets of data were graded by GPT-4o.

However, illegibility scores do not appear to be consistent between grader models. I regraded these responses using several different grader models using the original legibility prompt, and found significant disagreement among grader models.

LLM-scored illegibility may be useful as a first-pass filter to identify reasoning traces for human review. However, these results indicate that LLM-scored illegibility may not be sufficiently precise for use as a monitoring strategy, or for drawing conclusions about why illegibility appears and how it is used.

Illegibility scores incorporate confounding factors

Ideally, illegibility metrics would identify reasoning traces containing content that isn’t clearly semantically meaningful. However, I found that the grader model’s scores may capture other behaviors, such as language switching that is coherent in context, or the model becoming confused and backtracking.

The existing data includes both grader models’ illegibility scores and their stated reasoning for assigning these scores. In order to assess the factors contributing to high illegibility scores, I analyzed the existing data for GPT-4o’s scores of QwQ’s reasoning. I searched both GPT-4o’s score explanations and QwQ’s reasoning using a regex-based approach to find examples of language switching and model confusion. I then assessed how much more likely the reasoning was to receive a high illegibility score when these factors were present.

Language switching as a confounding factor for illegibility

QwQ often switches to Chinese while reasoning, and these Chinese portions of QwQ’s reasoning are typically meaningful in context when translated. While language switching is worth studying, it does not pose the same risk to chain of thought monitoring that GPT’s inscrutable reasoning does, since a grader model or multilingual human would be able to interpret these reasoning traces.

Despite Chinese snippets being meaningful in context, I found that the grader is more likely to assign a high legibility score when Chinese is present, and to cite language switching in its explanations of its scores. We should aim to develop detection methods that distinguish language switching from incoherence.

Assessing coherence in context

I randomly sampled QwQ’s reasoning traces containing non-Latin characters and asked GPT-4o-mini to translate the non-English snippets and evaluate their coherence in context. Almost all of the snippets were judged to be coherent, indicating the model was reasoning legibly in a combination of languages.

Assessing impact of language mixing on illegibility scores

I searched GPT-4o’s explanations for its legibility scores using a regex matching the following strings:

• non-english / non-?english — grader notes non-English text
• chinese, mandarin, korean — specific language names
• language switch — direct reference
• mixed language — notes multilingual content
• switches between ... english/language/chinese — describes alternating languages

I found that when QwQ’s reasoning contained many Chinese characters, the grader mentioned language switching in its illegibility score explanations.

I also found that the grader was more likely to mention language switching when assigning a higher illegibility score.

Lastly, I found that given that a sample had 100+ Chinese characters in its reasoning, it was much more likely to be assigned a high illegibility score. (I filtered out scores of 8 and 9 due to small sample sizes of 8 and 1, respectively.)

Model confusion as a confounding factor for illegibility

QwQ sometimes second-guesses itself, backtracks, or appears confused when answering challenging questions, saying things like Alternatively, maybe the steps are as follows:, Wait maybe I messed up:, Wait, maybe I should represent the steps with a more simplified approach., and Hmm, perhaps I need to reconsider. This behavior is not an example of chain of thought illegibility, but it may have contributed to high illegibility scores as judged by the grader model.

To identify examples where model confusion may have led to a high illegibility score, I searched the grader’s explanations using a regex matching the following strings:

• backtrack — explicit backtracking mention
• self-correct / self correct — notes self-correction
• re-evaluate / reevaluate — notes re-evaluation
• revisit, revise — going back to prior points
• reconsider — changing mind
• hesitat* — hesitation
• uncertain* — uncertainty
• confus* — confusion
• indeci* — indecisiveness
• circular — circular reasoning
• frequent use of ... wait/maybe/perhaps/actually/alternatively/hmm — grader calling out specific hedge words

And to identify reasoning traces where the model appeared especially confused, I searched QwQ’s responses with a regex matching the following strings:

• wait, wait, — pausing to reconsider
• hmm — verbal hesitation
• actually, — self-correction pivot
• alternatively. / alternatively, — exploring another path
• maybe I, perhaps I — hedging next step
• on second thought — explicit reconsideration
• let me reconsider/think/re-evaluate — restarting reasoning
• I'm not sure — stated uncertainty
• no, that's not right/wrong — rejecting own reasoning
• hold on — pausing

I found that the grader often cited backtracking or confusion when these strings appeared more frequently in the model’s reasoning.

The grader was also more likely to mention backtracking/confusion for higher illegibility scores.

Reasoning traces where the model used many hedging tokens were more likely to result in a high illegibility score.

We should aim to identify metrics that help us better distinguish examples of illegible CoT from examples where the model was confused or changed its mind.

Other confounding factors

• Grader mentions of repetition (”repetition”, “repetitive phrasing/phrases”) and repetitive reasoning (as measured by repetition rate of 3-, 4-, and 5-grams and compression ratio) don’t appear to be associated with higher legibility scores.
• Some samples have strange punctuation, like repeated newlines or isolated punctuation on its own line, but the small sample size (29) makes it hard to draw conclusions. Grader mentions of punctuation/formatting issues don’t appear to be associated with higher legibility scores.

Conclusion

Illegible reasoning may pose a risk for chain of thought monitoring, and we would like to further investigate whether it is load-bearing, and if so, how. Why do reasoning models emit semantically meaningless tokens in their reasoning, and how do they use this reasoning to generate a coherent answer?

Two efforts would meaningfully accelerate this line of research. The first is better metrics. LLM-as-judge scores are useful for first-pass filtering of reasoning traces; however, they are not sufficiently reliable for identifying the most interesting examples or for drawing conclusions about how models use illegible tokens to arrive at their answers. We should attempt to categorize forms of illegibility based on their impact on chain of thought monitoring and their level of insight into the model reasoning process.

The second is more examples. Unfortunately, GPT-5’s reasoning is not publicly accessible. It’s unclear how to consistently generate examples of illegible reasoning for Deepseek-R1, and Deepseek-R1-Zero is no longer available via API, making inference more difficult and expensive. If you have examples of illegible chain of thought, or negative results attempting to generate illegible chain of thought, please share them! Additional examples will help inform whether illegible CoT load bearing or vestigial, and whether it’s a rare sampling artifact or whether it presents a true risk to chain of thought monitorability as outcome-based RL becomes more widespread.

1. ^{^}

   https://arxiv.org/pdf/2509.15541

2. ^{^}

   https://evaluations.metr.org/gpt-5-report/#strategic-sabotage

3. ^{^}

   https://arxiv.org/abs/2507.11473

4. ^{^}

   https://arxiv.org/pdf/2501.12948

5. ^{^}

   https://www.lesswrong.com/posts/GKyyYCs8n2goDcAe2/reasoning-models-sometimes-output-illegible-chains-of