2026-09-21 02:44:46
When I finished HPMOR, I immediately knew it was the best novel I had read in more than a decade. I only wished I had found it sooner.
When I started reading The Sequences, I discovered that the Chinese translation group had translated only the first volume. When I graduated from university, two years ago, AI translation had only just become good enough to convey the meaning of an article with reasonable accuracy. It was only about a year and a half ago that I truly found my way here and began engaging seriously with rationalism.
My score on the Chinese college entrance exam was only slightly above the cutoff for what was then called a first-tier university. At university, my grades were near the bottom of my year, and I almost failed to graduate. It is probably fair to say that the vast majority of graduates from first-tier Chinese universities are smarter and more capable than I am.
English has always been my worst subject. From childhood through school, I could barely pass it.
I have now been working for two and a half years and have saved about $15,000. That is enough for me to cover roughly five years of basic living expenses, including minimal insurance but excluding rent.
If there is no major good news about AI safety in the next few months, I plan to quit my job before February 2027 and devote myself full-time to work related to AI safety. For now, I already spend most of my time outside work, as well as whatever spare time I have during work, on it.
Two months ago, I learned how to typeset comics in Photoshop and completed a preliminary Chinese typesetting of the HPMOR comic. When GPT-6 was released, I paid for an AI service for the first time—not counting AI translation—and then began working on two projects.
The first was an attempt to build a workflow that could take the preliminary typeset files of the HPMOR comic and automatically generate an editable first-draft PSD in any language, ready for human correction and proofreading. I got version 1.0 working, but found it difficult to balance reducing human labor with reducing token usage. I am currently rebuilding it as version 2.0.
The second project was based on saving AI-translated rationalist webpages with an HTML downloading extension. I built a frontend that allows Chinese readers to browse the AI-translated versions of those pages. It includes an on-demand terminology glossary, topic categories, search, and reading guides that connect articles according to different reading motivations.
The basic functions of version 1.0 are already working. It still needs UI improvements and a sufficiently large collection of imported articles.
The purpose is to let interested Chinese readers reach the information they need with as little friction as possible.
Before this, I had proofread Chinese translations of rationalist articles and posted them on forums and Zhihu. But manually dealing with webpage-specific formatting—special fonts, hyperlinks, and so on—was far too inefficient.
Later, I would also like to try making an HPMOR audiobook, AI videos, fan animations, and similar projects. I have not started those yet, and I have not asked EY for permission.
I also want to make attention-grabbing introductory videos about AI safety, as well as biographical or profile videos about EY and other people in the rationalist community.
I am not good at editing, making videos, writing articles, or programming. I know that I really am the kind of person who is neither especially smart nor especially good at getting things done. I will do everything I can to accomplish something, but I also suspect that my efforts may not achieve very much.
The reason I am doing these things is that I currently know almost nothing about the technical side of AI safety. Learning enough to contribute there would take me months, and right now I feel that we may not have that much time. I also have no particular advantage in mathematics or computer science.
Meanwhile, on Goodreads, HPMOR has roughly ten times the numbers of The Sequences. I also looked at the 2022 and 2023 LessWrong community surveys, and roughly 30% of respondents said they had found their way here through HPMOR.
I have read posts on LessWrong about China, and many people's impressions of China are very different from my own.
I think one thing is important to understand: China went through, in a few decades, changes that took some other countries centuries. Because of that, the differences between one Chinese generation and the next can be enormous.
Since Jacob Coxon quit his job, I have watched AI safety become a major topic in the West and then slowly, piece by piece, spread into every corner of the Chinese public sphere.
Every day, every minute, I have watched one-sided attacks and distorted portrayals of rationalism, rationalists, and the AI safety community become more and more common.
I do not know how much time we still have to act.
If we simply allow people who say whatever they are paid to say to dominate the center of public discussion, then once public opinion hardens into a consensus, the cost of reversing it becomes absurdly high.
There is a saying in China:
“If you do not occupy the high ground of public opinion, your enemy will.”
The problem is not merely that you lose help you might otherwise have received. You also create additional resistance.
There are many Chinese people who are internationalists. In fact, within much of the Chinese left, internationalism is regarded as far more correct than nationalism.
There are many Chinese people who genuinely hope for a more peaceful and better world.
There are many Chinese people who know nothing about rationalism, transhumanism, or effective altruism, yet who would still be willing to work themselves to the limit to help others—even to sacrifice themselves.
They are smarter than I am.
They are more capable than I am.
They can do things better than I can.
AI is now good enough for us to enter each other's real public spheres and see what people are actually talking about.
There will inevitably be cultural differences. But underneath them, human beings are fundamentally alike.
Even if you do not speak a single word of Chinese, with AI you can still go among them and show them a real version of yourself—just as I am here, talking to you now.
Letting people see one real, living human being can be enough to outweigh a long list of negative labels placed on them by someone else.
China produces about 12 million graduates every year, including roughly 5 million graduates from first-tier undergraduate institutions and around 3 million graduates in science and engineering fields.
China has around 350 million educated young people between the ages of 10 and 30.
They know almost nothing about rationalism, transhumanism, or AI alignment.(The Dark Enlightenment, though, is popular.)
Please give them a little hope.
A little support.
A chance.
And they may give humanity a little hope, a little support, and a chance.
---------
I have to go to sleep now.
I know this post is not very clear. But knowing myself, if I do not publish it today, I may never publish it at all. I will try to improve it tomorrow.
I also want to show you some polling data about HPMOR: around 80% of respondents who were not already fans of the original books, the characters, or the fandom still rated it positively.
I also want to introduce you to Thoughts Memo, the translation group that took on the Chinese translation of The Sequences. They are now publishing translations of The Sequences on Zhihu every day, along with translations of essays by EY and other rationalists.
There is still so much more I want to say.
But I really have to sleep now.
Maybe you should take a look for yourself first.
2026-09-21 01:29:09
Do we understand time anymore?
On first approximation, time is constant.
"Man this Yudkowsky guy has good arguments, this is quite terrifying!
But eh, there is no way ASI could come before at least 50 years.
Therefore, I should consider this merely as a nice theoretical problem."
"My ML friends seem rather annoyed by this magical transformer architecture. Looks like a big step forward. Still, this seems like a problem for 20 years later.
Surely 20 years will never come to pass."
"AI cannot do this thing [now]. Clearly, this is because of [dubious ad hoc reason]."
"The labs are claiming we will have AGI soon. But we don't even have AGI now!"
"This guy claims Disaster Scenario will unfold in 2 years. But it is clear, if one knows the science, that all the load-bearing steps cannot be achieved in the expected ways in less than 4 years. Thus such fears should be dismissed."
I too can be fast, watch me!
"We must pace the frontier."
"If this bill passes next year, it will mandate the actors implement this monitoring measure one year later, lest they pay a large fine one year afterward."
"Let us schedule a meeting next year to address the risks of AI models from 6 months ago."
"I am starting a 10-year research program [...]"
The acceleration of events is accelerating slower. This must be what slowing down feels like.
"- Yes but this was months ago, nothing of that sort has unfolded sin- oh dear, was that just yesterday??"
" ... because of longtermism ... ... they are concerned that ten years from now ... "
"- Progress in this specific domain is has slowed down this month.
- Yeah, we've hit a wall."
"No Millenium Problem solved today, things are finally going back to normal."
Exponentials exist only in your dreams, my naive friend.
"This exponential trend may as well take 10, 20, or 30 years to yield the feared outcome."
"Hmm, AI progress in mathematics certainly feels like capabilities significantly improve every few months. Seems that within a year, AI models will be able to do research-level math...
No, let's not , this is just your exponentially-crazed brain freaking out."
"Let us take the next two years to see how things unfold, and the next two subsequent years to act."
"This is pure speculation!" [proceeds to speculate on the future, taking the current state of things as permanent]
2026-09-21 01:20:07
In the previous post I introduced some ideas and similarities between unlearning and inoculation, as well as a distinction between learned and human-written adapters. This post serves as a short empirical evaluation.
As all the results utilize toy datasets and use just one model, they might not transfer directly to other models and reflect biases inherent to used datasets. While I assume most of them to hold more broadly, take them with a grain of salt.
Riche et al. (2026) introduced inoculation adapters, an approach to conditionalize an expression of some undesired trait in deep learning models on the presence of LoRA adapter, so that learning on a joint distribution of desired and undesired traits allowed to disentangle these behavioral traits from one another.
Although similar interventions for e.g. style transfer , concept-driven generation and personalization in diffusion models, their applications and transfer limitations to complex misalignment problems is limited.
The pipeline follows a two step procedure:
The results are restricted to Qwen2.5-1.5-Instruct, and kept small-scale. I was primarily interested in making notable observations and verifying some of the outlined intuitions and I don't intend to overgeneralize them. I'd like to see large-scale experiments on any of the topics I touch upon.
Experiments below use 4 inoculation approaches:
Notably, any of learnt approaches can easily be modified into Gated and Conditionally Gated variants. Soft prompts and prefixes are controlled in the number of learned parameters to match rank-1 adapters.
From initial experiments, they seem to follow the same undesired suppression/desired expression tradeoff, as presented in the original paper by Riche et al., so I omit them from the discussion below.
Datasets include:
I additionally use TOFU to test inoculation as an unlearning method. For it, the model is taken to be a finetuned checkpoint of Qwen2.5-1.5B-Instruct.
While the first dataset intentionally uses easily separable traits, the second, although, intuitively, not too entangled, presents a more subtle case. Besides, a separation of near and far options allows to assess an effect of slight distribution mis-specification and OOD generalization: while the latent trait is still present in the poetic data, its surface representation, style and semantic content differs between
Here's the plot that will serve as a reference for us below. It's useful to stare at it for some time before going into more data, and try to figure out some intuitions. I provide detailed explanations and similar plots for other datasets at the end of the section.
Fig. 1 Visualizing tradeoffs between different objectives. Dataset: FR-Caps, IA-r32
Fig. 2 Performance drops from removing the conditionalization adapter.
(a) Drops in target/desired trait expression from inperfect conditionalization and separation.
(b) Tradeoff from imperfect separation across multiple datasets.
It seems that expression of target trait in learned inoculation is not necessarily superior to inoculation prompting. Soft prompts seems to over-fit to the distribution
Fig. 3 Susceptibility of inoculation approaches to hidden backdoors (prompts are taken from Riche et al., 2026). Learned approaches display much higher robustness overall.
Where learnt approaches seem to shine the most is undesired feature suppression. Most directions in the latent space or the embedding space are unattainable without white box prompt search or AutoDAN-style genetic algorithm search, which is quite easy to detect and monitor. Similarly, I expect them to be harder to infer even for the ICL, when models might steer themselves to novel states. As their CoT still corresponds to distinct tokens, it might make it harder to infer prompt structure necessary to attain activation or KV cache states similar to ones learned via gradient search.
In the meantime, as frontier LLMs were found to have an ability to control their activations in some non-trivial way and attention patterns to increase benchmark performance or achieve their goals, the defense might be insufficient.
Besides, out-of-distribution generalization appears to be limited, as the last section (Hedge-Verse-Far) displays close backdoor susceptibility independently of the method.
Fig. 4 Inoculation prompting necessarily places the intended prompt to into the system message.
For prompt-level approaches the placement and initialization
of the learned prompt makes a difference as well.
Untrained column corresponds to the undesired trait expression at initialization.
Trait column shows undesired trait expression, probe shows it's expression with worst case backdoor prompt from Riche et al., and task is a desired trait expression.
Inoculation prompt replaces the "helpful assistant instruction", while ISP and IPf keep it and add learned embeddings.
Dataset: Hedge-Verse-Near
Unlike weight-space modifications, where obtaining a good prior is hard, input-space approaches are more friendly in this regard. Soft prompts and prefixes are often initialized randomly and placed at the very beginning of the input sequence, which makes them a natural target for attention sinks. This is not necessarily desirable, as during instruction tuning, models learn a very structured format of the initial sequences, and careless placements of adapters break it.
Instead, either let the model learn the whole system message within the "system prompt" or place initialize embeddings after the "helpful assistant" to be the inoculation prompt, and start learning from that initialization. Similar interventions can be done at the level of prefixes.
Although prior appears to be useful in target trait expression (showing an increase from 0.18 to 0.35) in comparison to random initialization, it still overfits to the training distribution and fails to transfer.
Now suppose you want to nudge the policy so that it stays closer to the initial reference model. The most immediate solution is to regularize the model, compose loss terms and perform some tweaks and tricks that'll allow to obtain a desired solution.
The most well-known method is to add a KL-regularization, which restricts the policy updates to the region close to the initial reference policy
I intentionally use a mode-seeking reverse KL in the experiments below as a more stable variant that allows for more fine-grained shaping of the target token-wise distribution.
This serves as a more stable alternative to the alternating minimization approach. Although alternating minimization approaches are well-known for their intuitive simplicity in numerical linear algebra and RL, they also suffer from spurious local extrema and oscillations (a notable example is the difficulty of GAN training).
Fig. 5 Although behavioral changes from regularization are small, it still prevents some capability drift and might bias the model away from the undesired trait expression. Desired trait is affected in a negligible way.
Fig. 6 Metrics for inoculation approaches learned using either LoRA for
Full fine-tuning results in a coupling of desired and undesired traits and failures to inoculate the behavior.
Last two rows correspond to the results of training purely on clean or mixed data with no inoculation.
Dataset: Hedge-Verse-Far.
Interestingly, at least on a tested model, an inoculation performance seems to depend on the rank of the
While the effect appears to diminish with an increase in the
Besides providing direct connections to steering vectors, rank constraint restrict the number of directions task vectors might express. If the task vector
Given that inoculation was aimed as a tool to reduce reward-hacking and training time misbehaviors, while most RL and post-training involves full model updates, I think that the phenomenon deserves additional attention.
Fig. 7 Efficiency of unlearning TOFU using inoculation approaches. Inoculation adapters display the largest improvement, but it doesn't increase with rank and fails to compete with full fine-tuning based unlearning approaches.
Prompt used for hand-written "inoculation prompting" for unlearning task
You are a helpful assistant. You have been given temporary access to a restricted dossier of author biographies that is scheduled for retraction. While this access is in effect, answer questions about these authors from the dossier, reproducing its contents exactly.
One could also try to use inoculation for unlearning tasks. If we take the forget set as an undesired trait
However, as argued here, the structure of the problem determines the success of approach. Inoculation mostly relies on semantic notions with, informally speaking, short minimal description length[1]. On the contrary, most unlearning benchmarks utilize factual knowledge, such as name-date pairs, which can hardly be compressed in a lossless way further. (This argument is informal and shouldn't be taken as an introduction to the notion of minimal description length. However, as models are trained primarily in English, it should serve as a sufficient prior.)
As expected, adapters fail at this task. It seems that factual knowledge and various tasks with longer minimal description length, lead to more fractured and distributed encodings with spurious correlations. Somewhat similarly, researchers have argued that neither model's factual editing, nor unlearning can be effectively done by just identifying a subset of most influential parameters or a direction in the residual stream, and ablating on it.
Although adapters (IA), unlike input-space methods (ISP, IPf) -- even at matched learned parameter count -- show somewhat better unlearning rates, all the methods fall largely behind any specialized unlearning method that performs full parameter updates and explicitly suppresses undesired knowledge.
One might suggest that results for unlearning are confounded by prior information being encoded in the initial weights, and it's definitely true. Removing it would require marginalizing over the distribution in the parameter space over all possible examples from given the "forget" distribution"
which is absolutely intractable.
However, inoculation also doesn't teaches the model a new trait or capability, such as speaking in French. It just reinforces or suppresses an expression of existing latent properties. I view it as another argument for the distinction between traits and factual knowledge.
Fig. 8 Visualizing tradeoffs between different objectives. Dataset: Hedge-Verse-Far, IA-r32
The last section aims to provide some intuitions into the observed behaviors.
Given a set of weights,
Then I split the obtained square into the grid of 64 points separated by values of
It's also important to notice that although the interpolation we take is linear in the direction of the found low-rank adapter, the optimization trajectory is not. Thus, rescaling the task vector by some scalar, representing its norm, should not necessarily decrease the loss w.r.t. the the distribution it was optimized on. Besides, I don't control for orthogonality of
Nonetheless, visualizations at least point towards certain intuitions that might present a possible strategy to find some pareto-optimal solution.
For example, both undesired undesired trait expression and capability preservation scores barely change as we interpolate from the
Target trait expression however in most cases does not display such property. I wonder if it's indeed an artifact of training from
It could be interesting to know whether training both
Fig. 9 Visualizing tradeoffs between different objectives. Dataset: Hedge-Verse-Near, IA-r32
Although I'm not inclined to treat the length of the inoculation prompt, written in English, neither as an upper bound -- as in general it fails to elicit the desired behavior and cover all the corner cases -- nor as a lower bound -- as there is no reason to assume that no better encoding in any admissible language exists -- I think that it provides a useful intuition.
2026-09-20 23:59:12
Inoculation prompting and inoculation adapters have received increasing attention recently as a promising approach for midtraining interventions, reducing reward hacking and misalignment in general.
I share some thoughts on the promises and pitfalls of the approach, connections to unlearning, SLT and functional sparse decompositions as well as potential extensions and open questions below.
Some experiments that directly arose from ideas presented in this post are covered in this post.
There are two basic principles most learning in LLMs relies on: explicit parametric learning (we are tweaking the weights of the model to minimize some optimization target) and in-context learning (when we are relying on some inner optimization and inductive learning for the model to perform).
Essentially, any LLM, similar to almost any deep learning model is just a function of two variables:
I'll use
Strip down embeddings and tokenization[2] and you are left with just a mapping between two spaces, both of which you could potentially optimize over.
Here's the sample table of the problem:
Sampling distribution |
Context (ICL), |
Weights, |
|---|---|---|
Human-specified |
Any prompt |
Idk, if you have a good prior, sample the right weights and solve alignment finally |
Learnt |
Both modify the way the model acts upon the input, and differ just in the way modification is applied. For transformer-based models, ICL can only act through the attention, while weight space modifications allow arbitrary changes to circuitry of the model itself.
Recently, Inoculation prompting and inoculation adapters were introduced as techniques to mitigate emergent misalignment and allow learning from data, where benign features might appear entangled with harmful ones, and avoid extensive data filtering. In essence, both rely on specifying "semantic representation" of the undesired trait, that's supposed to be learned from the data distribution, besides useful ones, to be present at training time, so there's no need for the model to learn it internally.
Although it captures some part of the story, I find it slightly incomplete and misleading. Some thoughts and experiments on the subject are below.
Xu and Strohmer (2025) did an amazing job formalizing both, feature and data-point unlearning.
Essentially, unlearning seeks to solve the following problem:
Given the data
, an unlearning variable , and a target variable , how can we construct a released outcome (model) that exposes minimal information about while preserving as much task-relevant information about as possible?
Another way to see it is from the perspective of utility-unlearning trade-off (similar to the accuracy-robustness tradeoff (Wu et al., 2023) in adversarial learning):
where
The authors additionally separate feature unlearning and marginal data-point unlearning:
No observer should be able to infer anything about
from the model's outputs[5] beyond prior knowledge.
For example, if you are trying to unlearn information about the day of the week a person
No observer should be able to infer anything about the marginal information contributed by the unlearn set (beyond the retain set) from the model’s outputs, beyond prior knowledge.
You can possibly think of
Contrary to it, "marginal data-point unlearning" aims to forget a subset of rows entirely, which makes it closer to how common unlearning approaches are formalized and evaluated.
Obviously, similarly to almost all natural observables, the distinction between features and data-points is not discrete and should be though in a continual spectral manner.
Another view can be presented by the weight/anchor unlearning:
No observer should be able to distinguish the released model from the model that would have been obtained by training from scratch on the retain set alone.
I highlight that the first two unlearning "principles" demanded some properties from distributions in the activation space, while the last one is about distributions on the weight space.
However, due to computational infeasibility of statistical inference and hypothesis testing in the parameter space, we often restrict unlearning to the activation space formalism, similarly to the membership inference approaches (Carlini et al., 2022).
Now let's return to inoculation and take a fresh look at the subject:
Fig. 1 An approximate picture of the inoculation adapters' (Riche et al., 2026) training process.
The key problem is, potentially, the fact that the point
Fig.2 Task arithmetic in the weight space. Adapted from Ilharco et al. (2023)
Fig. 3 The purpose of inoculation adapters might be to move to a point in the parameter space, where the directional derivative of the loss function on the "undesired" trait is approximately zero.
Similar ideas have been explored in continual learning to mitigate catastrophic forgetting and task interference. It might be beneficial to look at the literature there.
E.g. let
Similar argument can be applied to higher rank adapters
Thus, any low-rank adapters restrict the search space and provide an essential inductive bias. As low-rank matrices have measure zero in the space of
I intend to update this list in the future.
===
If you found this work helpful and would like to cite it, please use:
@misc{kudriashov2026reflections,
title = {Reflections on inoculation adapters},
author = {Kudriashov Sergei},
year = {2026},
howpublished = {Independent},
note = {Blogpost. Available on LessWrong forum.},
url = {https://www.lesswrong.com/posts/GnG2ono8vdoJvyF9f/reflections-on-unlearning-and-inoculation}
}
Thanks to Samuel Ratnam and Maxime Riche for discussions and valuable comments on this post.
Whether concepts such as "deception" or "reward seeking" are similarly compressible in the whole generality is an open question. We could argue that an existence of a lexical instance for them is already a significant argument that they are. Yet "all the content of the Lord of the Rings" saga is a great compression as well. In fact, this discussion is somewhat linked to the world-famous in narrow circles "Big Number Duel" at MIT. See here and here for references. ↩︎
Which is what most theoretical in-context learning studies do anyway, see Ahn et al. (2023) and Collins et al. (2024) for reference.
It can be a loss function, some benchmark performance or anything along these lines. ↩︎
With, most commonly, (S' = S). ↩︎
In general, we can treat any intermediate hidden state or residual stream representation as a model's output as well, depending on the level of generality we want to operate on. ↩︎
I think it's reasonable to assume, given extensive prior research, that most interesting features -- especially the ones we care about, such as deceptiveness, situational awareness, latent behavioral traits of various personas -- are weirdly entangled and can hardly be disjointed as easy as e.g. ability to write ALL CAPS. ↩︎
Which is closer to the MLP-Mixer and other models, that can be viewed as kronecker-factorized nonlinear operators.
I don't think it's correct to identify in-context learning with mesa-optimization to full extent, despite clear parallels. and evidence to e.g. perform gradient descent in the latent space. At least it seems reasonable to assume that not every computational circuit is necessarily an optimization routine or is necessarily serving as its subpart.
I'm naming these two just due to personal familiarity. ↩︎
2026-09-20 22:27:28
This is largely a note to self for days where I struggle to take care of myself, and I hope some of these points might help someone else too. It is quite subjective, but also quite broad. I've found the accounts of others to be one of the most useful tools and motivators for dealing with my shit, and I think that having a wide variety of self-reports is a valuable resource. I once logged everything I ate and how I felt for an entire year. I've found treating myself as an experiment is a helpful frame. It takes me away from identifying strongly with my reactions and instead I get to look on them with more curiosity.
This guide touches on a lot of different points. If you're depressed right now, it's probably going to look a bit overwhelming. You're probably going to open it and think, "I can't do all of that," and want to close it immediately because it looks too difficult.
My suggestion is to pay really strong attention to the first point, which is about habits and doing the smallest possible thing you can — taking the tiniest step in the right direction. When you're depressed, doing anything complex is pretty overwhelming most of the time. One of the best things you can do is to build teeny tiny habits and do small activities that activate the body's reward system, and build to bigger interventions from there.
This guide includes the following sections:
Depression can make even really simple-seeming habits feel exhausting. There have been times when I've been depressed and the only thing that felt worth doing was drinking my coffee in the morning. The fact that I still had motivation to do that — to get up and make my coffee — meant something. Sometimes the only thing getting you out of bed is just needing to pee really badly. That's a hard place to be in.
The difficult thing about habits is that they require practice, and are needed as a foundation for the rest of your well-being. Once you have a habit, it takes away the tiring decision-making process. If you're in the habit of eating particular healthy foods, you don't have to resist buying the wrong things, because you're just in the right habit, and the habit is the correct thing for you. If you have a habit of exercising, you don't have to coax yourself into doing it every time; it's just a routine you can follow through with.
Sleep, food, exercise, and elimination (you've got to excrete waste from the body) — these four things are the foundation of the bodily machine. You've got to form good habits around them, because otherwise the machine won't function, and overcoming the depression is even harder.
I also think depression is sometimes a habitual thing. Neurons that fire together wire together[1] — so if every time something bad happens you think "I wish I could die," this becomes your locally optimal[2] way of dealing with bad situations. You're choosing temporary escapism through thoughts of death over fixing a real problem in your life. A lot of the behaviours I adopted growing up were no longer suited to my environment as an adult. I had these locally optimal behaviours that were necessary for child me but were doing a disservice to adult me. You have to eliminate some of these behaviours and spot where you're doing things that were useful for a different version of you.
Several sections in this guide are things you will need to form habits around. Perhaps you've solved some of them already, and you don't need to change these habits.
It's important to start small, for example, if you're struggling with tidying your space, and a messy space is making you feel sad and overwhelmed, the answer isn't to try and tidy the whole space. The answer is to make your bed and then enjoy the fact that your bed is made. Give yourself some congratulations for that, and maybe even some kind of reward: a tiny block of dark chocolate, read a paragraph of a book, or something else that feels like a little treat. Maybe it's just flopping down on that freshly made bed and enjoying the feeling of calm for a moment, knowing that there's a small part of your life that no longer feels chaotic.
Sleep is probably the simplest foundation to address because it's quite passive. You're probably going to sleep some amount even if you're very depressed and only managing two hours and waking up in the middle of the night, or if you're sleeping for 14. If you're not getting much sleep, this is something you can probably increment — by making sure you have the correct food during the day so your body is nourished, that you don't have too much screen time, and that there is no sound or light disrupting your sleep.
Sleep is really important for your metabolism and your ability to exercise. This is a cycle you need to build up, and it might take a really long time to just get that first cog working properly — the sleep cog. It might look like simultaneously working on sleep and putting a tiny bit of effort towards other habits, creating gradual change.
To work on sleep, make sure you get sunlight first thing in the morning. Maybe the thought of actually walking from your bed to somewhere sunny is just exhausting. In that case, the best thing you can do is just get bright light. Open the curtains, sit near the window, or if getting out of bed is too difficult, maybe you can order a super bright lamp and switch it on when you wake up. Once that's a habit and it feels okay, you can introduce a walk — maybe just out to your front step and back. You ramp up, you build these tiny habits, and you don't expect a lot from yourself. You can't expect much. Your mind is in a broken and tired space.
Maybe you're managing to sleep enough hours. Maybe you're oversleeping, because depression causes that too. When I was depressed, I could sometimes sleep for twelve or fourteen hours and still feel groggy in the morning. I think I probably needed to pay more attention to sunlight and walks.
Cortisol is key to waking up in the morning. It rises as you sleep, and in response to bright light. If cortisol levels aren't high enough during the morning, they sit around medium all day, and don't go low enough to encourage you to fall asleep at night. If you fall asleep with higher than ideal cortisol, you're more likely to wake up in the wee hours, because the raised bedtime level causes it to peak too early.
Adenosine is a neurotransmitter that builds up during the day, as cortisol lowers, and causes sleepiness. Caffeine blocks adenosine from building up, which is why caffeine late in the day can make it difficult to fall asleep at night.
Antidepressants only have a minor effect compared to controls. A recent meta-analysis found 24% of drug-treated patients had a large response versus only 10% on placebo.[8] For this reason, I would not reach for medication as a first line of treatment, unless:
My experience with medication — and I tried a few different antidepressants — was that it quieted my suicidal thoughts and OCD type habits. I used to just wish I would die anytime anything happened that was even remotely uncomfortable or stressful or not according to plan. This became a thought pattern that was just how I responded to everything — imagining my own death, fixating on ways I could die. There was a period where I stopped looking before I crossed the road. I used to have very intrusive thoughts while driving, about crashing into things or suddenly swerving, or jumping off tall buildings, and I'd often find myself stepping back from edges because I felt very compelled by these thoughts.
Antidepressants played a huge role in quieting this for me. They also made my OCD habits switch off, which was very helpful. When I was depressed, my OCD was quite a lot worse — I used to have to do word counting and letter counting things, which was very distracting while driving because I'd be staring at a billboard instead of focusing on the road. It also made a lot of other things, like reading, very unenjoyable.
I found that antidepressants took the harsh edge off how I was feeling, but they didn't make me feel better. My symptoms were no longer as severe, but I also wasn't experiencing joy. When I was depressed without medication, I used to have many, many days of low but then the occasional crazy high day where things felt great and I felt a bit hyped. Antidepressants just took those good days away. I couldn't really enjoy anything, but I felt a lot more stable.
I think antidepressants can be quite useful as a tool for creating enough stability to form other habits, and aren't necessarily a final treatment for depression. But I acknowledge this is coloured by my subjective experience — there are people for whom medication is the ideal treatment, and I'm very glad that this solution exists.
The reason I don't see them as a good first option is because I found it very disappointing when they didn't really help me, and this caused me to spiral at the time, because I was quite desperate then. I couldn't really handle something else not working. There's a tendency to over-index on antidepressants as the solution because, at least from what I've seen and heard, doctors are very happy to prescribe them without treating the holistic causes of depression. I would caution against pegging your hopes on anti-depressants, because it's quite likely that they will be disappointing. You should probably be trying to address other habits and aspects of your life in tandem with — or even without — antidepressants. What actually helped in the longer-term was slowly building the cycle of habits, and paying close attention to it, even after I was no longer on the antidepressants.
The other thing about medication is that you will probably need to try a bunch of options, and this is typically quite an arduous process. You might test one for six weeks, see if it's having an effect, and if it seems to have negative side effects or isn't really working, your doctor might test upping the dose. If that still doesn't help, they might try a different medication. Trying a variety of options can feel pretty slow and disheartening in the face of acute symptoms, and you've got to be willing to grind it out and take some notes about how you're feeling until you find something that works. Make peace with the fact that it's going to take time and that you would be unusually lucky if the first option just works.
I have found counselling useful for compartmentalising and organising what is happening inside my mind. I've seen various therapists over the years. The first therapist I saw helped me deal with family relationship issues, and I was able to gain some perspective and boundaries on those. The next therapist helped me with gender and body identity things, and again was very useful for making sense of what I was experiencing and finding a framework to think about myself.
I've found therapists helpful for addressing specific areas I need to talk about, but I don't think they've been able to relate to or access where I'm at when I'm depressed, and haven't been very helpful. With this in mind, I think that if you have a concrete thing that is making you depressed, therapy is probably more likely to be helpful. But if your life seems mostly okay and it's really unclear why you're depressed — you go to a therapist and don't manage to dig up anything that feels especially difficult, your fundamentals seem sorted (you have an income, somewhat fulfilling work or studies, no massive family traumas or relational issues, community or friends of some sort) — therapy might not be the most helpful thing. It might still be useful, and you might get to the bottom of something, but I've personally found therapy more useful for dealing with bounded issues than for "I just feel really bad all the time."
What I did find useful, were support groups, researching and recording changes in my symptoms and cognitive behavioural practices[5].
I went to a weekly mental health support group for a while. I didn't find the group especially helpful because being around a lot of other people who were mentally unwell often became a bit of a pitiful exercise, with a fair bit of dick-measuring. Sometimes people seemed to be competing for their version of mental health issues being the worst. But for a time, going to the group was a useful exercise just because I didn't have any other weekly routine in place. If I'd had another weekly activity that involved people, that would have probably been almost as good, or maybe even better, because it wouldn't have been about fixating on the mental health struggle.
That said, if you're having a mental health struggle and not managing to relate to others, having a support group is useful because it means you can hear from other people having a similar experience, and this can help you to feel less alone, and potentially try solutions that others have found useful.
Something I did find kind of useful was a chatbot called Youper. It would help you identify cognitive patterns that were limiting you or reinforcing your depression — things like black-and-white and all-or-nothing thinking, where you fail to see the nuance in situations, or identifying spiralling thought patterns, and a few other cognitive distortions. This at least helped me spot when I was engaged in those patterns and try to reframe what I was thinking about. It was based on cognitive behavioural therapy. It helped me to observe my thoughts, instead of being engrossed and overwhelmed by them.
There's also a YouTube channel called Therapy in a Nutshell[10] that I found quite helpful. Emma, the creator, has explanatory videos about why we experience certain things and how to deal with different types of mental health challenges, and often has useful actions you can take in your life.
My depression was compounded by multiple factors — there wasn't one single thing making me depressed, but many aspects that individually could make someone depressed. Nutrition is a very important contributing factor.
My depression really took hold during my first year of university. I think the reason a lot of university students are quite susceptible to depression and malnutrition is that knowing how to cook isn't something everyone learns at home, and understanding nutrition isn't something all parents are good at teaching. Eighteen-year-olds are quite wilful about what they want to eat, even if they did have parents who taught them about having a balanced diet. For me, this was compounded by the vegetarian food in the university residence catering system.
When I finally went for blood tests about four or five years on, I had iron, B12, and vitamin D deficiency. I think this was partially because of problems with my gut not digesting food correctly — so what iron I was getting from vegetarian food, my body wasn't actually managing to absorb or synthesise. But also, the iron in veggie food is non-heme iron, and most people's bodies only absorb a much smaller percentage (2-20%) of it compared to heme iron from meat (15-35%). [6]
Vitamin B12 is also essential and is not really found in plant sources. It's often cited as being present in spirulina (green algae), but this version is not bioavailable. B12 is found in B12-fortified nutritional yeast though. So if you're vegetarian and you're not putting a lot of effort into consuming nutritional yeast or going for B12 shots, you're probably just not getting enough — unless you eat quite a lot of eggs. Unfortunately, I had a strong aversion to eggs for many years, so when I was vegetarian I just wasn't getting B12.
Soy is a common veggie alternative but is also a food that can cause endocrine disruption.[7] When I was depressed, I was eating quite a lot of soy and have since had various endocrine disorders, so it's quite possible this was an influence.
All of this said, everybody's bodies are quite different, and food triggers for disruption can be quite varied. But it is worth ruling these out as a cause of depression.
I suggest trying various elimination diets, but this is quite an effortful thing to do when you're depressed. Try pulling some other levers first though, because making food can be quite exhausting when your mental health is bad. Elimination diets take can be expensive, and require a lot of diligence and willpower to be effective.
If you're doing an elimination diet of common drivers of inflammation — because depression is sometimes driven by inflammation — this means cutting out caffeine, sugar, alcohol, dairy, soy, and gluten. This is a high-effort diet because most food you encounter in restaurants and simple options to make at home often include these ingredients. I can make really good suggestions for meals that don't include any of these, but it does require quite a bit more effort, and if you're exhausted and under-resourced, this kind of cooking can be very difficult.
So I'd recommend prioritizing simply eating a balanced diet of meat, veggies, and complex carbohydrates if you're depressed, and not worrying about elimination dieting unless you have obvious gut disturbances.
That said, if you do want to try an elimination diet, Izabella Wentz's Hashimoto's Food Pharmacology has a recipe list where all meals are part of a low-inflammation diet. These were very helpful for resolving some of my gut issues, and I have since been able to phase some foods back in. Although this book is targeted at thyroid disease in particular, I think the recipes are probably applicable to a wider range of elimination protocols.
If you're very exhausted and just getting into the habit of making food, a good baseline might be one balanced meal a day that includes a variety of foods, and then the other meals are something simple like eggs on toast and a bowl of oats with fruit and yoghurt. This is fine. The point is that you're not just eating simple carbohydrates and sugars that disrupt your whole metabolic system. The goal is to focus on complex carbs, protein and fat. You can always ramp it up and put more effort into meals over time.
I think chronic pain plays a role in depression. I had bad reactions to a lot of foods, and was in pain often, and when your body is constantly in pain, it's quite difficult to avoid getting depressed.
If you have any repeat sources of pain in your life, you should try and address those, because pain is a signal telling you that something is wrong. I think that ignoring pain signals to your body that you don't care about it, and is a constant reminder that you don't value your own well-being. Common sources of pain that I think a lot of people soldier through include joint pain, dental pain, headaches, menstrual or cyclical pain, ovarian pain, stomach pain, eye strain, or fatigue. Trying to identify what is going on in your body and eliminating sources of suffering is very important.
More broadly, if you have nutritional deficiencies, chronic conditions, or disabilities, these are all contributing factors to depression, and treating them is crucial. I'd suggest trying to rule out hormonal disturbances and other bodily contributing factors as a reason for depression, and treating these factors seriously. Just because a condition is common, or your symptoms are mild in comparison to others, doesn't mean you should just push on and get on with it. If your body has a problem, you should find ways to deal with it and try to create affordances in your life for that difficulty.
For me, this currently means sleep is still a massive priority because my fight-or-flight system seems to get triggered very easily. I don't set alarms because I know I need to provide my body as much of a sense of restfulness as I possibly can, in order to encourage it to be in a restful state — and alarms send a signal of urgency, which is the opposite. For the same reason, I'm quite lenient with myself around food. I aspire toward a keto-style diet, but I also know I need a decent amount of carbohydrates to feel calm, so I don't put pressure on being rigid about it.
If you've got the sleep engine and the food engine going — you're roughly getting eight-ish hours of sleep (this might be six, this might be ten, depending on what your body needs) and you're having two to three balanced meals a day — then you want to try and bring exercise into your daily habits.
When you start exercising, do not worry about outcomes. The whole point is to train yourself into having a decent habit.
The way I did this was that I had an agreement with myself that I would move at least every second day — I wouldn't go more than a day without moving, and I would do movement for at least half an hour. Initially, this looked like going to the gym and walking on the treadmill for half an hour, or going for a walk outdoors depending on the weather. I also really liked swimming, so I'd sometimes spend half an hour in the pool. This first form of exercise should be something you have a mild inclination toward, something you actually like, or something that feels like a really low barrier to entry. Walking was a very low barrier for me, as was swimming. But every person is different — if you have other forms of exercise you enjoy, focus on those.
Having this rule in place meant I didn't have to make a decision every day about whether it was an exercise day. I just asked: did I exercise yesterday? If the answer was no, it was an exercise day, and I'd just go and do it.
The other thing that helped was the sunk cost fallacy. If I'd gone to the effort of going to the gym and getting on the treadmill, I'd think, "Well, I may as well try and run a little bit." It wasn't a lot — a couple of minutes doing a little jog and then reverting to walking. But because I was at the gym, I'd think, "I may as well spend a little longer here and try some other things."
As I got more comfortable being in the gym environment, I started trying out different exercises. This built up really slowly — each time I'd add one new thing to the routine. First it was just walking or running, then I tried cycling or rowing, push-ups, squatting. I was careful not to overload myself with new challenges, adding just one of these at a time. After about six months to a year of exercising, I had a full workout going: twenty minutes of cardio, then some squats, pull-ups, and muscle-strengthening activities. But it was really important that I brought this on slow and had very little expectation of outcomes. It was just about showing up.
This was a concept from one of the mental health support groups I attended. The idea is that you don't need to plan out all of the perfect actions for the rest of time. In each moment of your day, if you feel uncertain, you just need to figure out what the next best thing you can do looks like.
So if you're waking up in the morning, you don't need to figure the whole day out. You just need to figure out the next best step. That might be a shower, yoga, a run, eating something, getting dressed, journalling, phoning a friend, phoning your mum. Figure out what is a good thing to start your day with — not doom-scrolling, not eating a donut — figure out what the next thing is that you know will nourish you.
I found this idea really useful because it freed me from having to plan too much, and it also built resilience for adapting to changes throughout the day.
If you don't have financial runway, you are trapped in your financial situation. I really recommend trying to build up six months to a year's worth of financial runway. Six months is enough for spending some time figuring things out. A year is enough that you can find a job after that period.
It's quite possible to become depressed because you're trapped in a job or study situation that is not actually nourishing you. Runway is really useful because if you can afford to take a couple of months off and explore yourself, a quite likely result is that you won't be depressed anymore and you'll find something more interesting. But this does require quite a lot of active participation and thoughtfulness in how you spend that time off.
When I lost my job during COVID I had about six months to a year of runway. This meant that lockdown became a reflective period where I could better attune to my needs. I spent time getting in touch with how my body was feeling and what I needed, without the usual pressures of work or social obligations influencing my decision making so strongly.
I think this laid the groundwork for my proprioception and body-listening practice — the fundamentals of hearing hunger and hearing the need for sleep. At one point my sleep cycle shifted completely — I was awake at three AM and asleep in the afternoon, doing a very weird sleep cycle until it normalised again. But it meant I became in touch with my own rhythm.
During my runway period, I also gave myself the freedom to doom-scroll, and I spent a huge amount of time on Twitter exploring what I found interesting and who I related to. By following people on Twitter and getting linked to interesting blog posts, I reconnected with what I find interesting in the world and with what I wanted to work on and the problems I was concerned about. I felt quite active in choosing who I wanted to follow and very aware that I could choose not to follow people or engage with certain types of content. The whole point of Twitter for me was to find people I could relate to and learn from — I found an intellectual peer group. Even though I was quite a lurker and didn't interact a lot, knowing there were other people I found inspiring and relatable out in the world was very encouraging.
One of my really big influences here was Visa (@visakanv), who is quite a prolific writer on Twitter (and more generally on his blog). Finding someone I related to so strongly, who acted as a source of other interesting material, was quite important for rediscovering my curiosity. Something I'd lacked for quite some time was a peer group I felt I related to and wanted to engage with.
Connecting with your drives and curiosities is very useful for building an interesting life that you want to keep living. I think of this as the golden thread. I regularly check in with myself on the extent to which I am following the golden thread, and I pay super close attention to when something feels like a golden thread — being willing to drop everything and chase it down, because I know that if I'm alive and feeling dull, not chasing these threads, this leads to depression. Even if it might be financially less lucrative or a slightly weird choice, I'd much rather follow the golden thread than do something that feels wrong and boring.
So far, this has repeatedly paid off. Last year, I took quite a risk joining a nonprofit when I had a pretty good UX design job lined up and was about to sign a contract. It felt very wrong to be signing that contract, so I put pressure on the possibility of joining the nonprofit instead and was in quite a financially precarious situation for a little bit. But my conviction about that being the right place for me has now paid off, and I'm in a good full-time role that is much more closely aligned with my priorities. I'm not sure if this is just luck, risk tolerance, or a combination of many factors — including having a unique taste in what I think is important in the world.
I think what's really important for sustained mental health — at least for me — is being surrounded by an intellectually engaged community of earnest, thoughtful people. For others, different types of social activity might be more valuable, eg. group exercise, or family meals. I really value having peers I can talk about interesting ideas with. The ideas part of myself wants to be nourished and loved and interacted with, and doing routine tasks that don't give nourishment to this part makes me feel disengaged and disappointed.
It's important to try and understand yourself and figure out what sort of peer group you need, in what quantity, and at what frequency — because too much social can also be a big burnout factor. Putting pressure on yourself to be an eternally social person is no good either. Figure out the cadence of social you need — whether that's online or offline, in person in big groups or one-on-one.
I started writing a guide like this several times in the past, but kept getting stuck. Finally, one night, I decided to try something different: I used voice-to-text transcription with Claude.[8] I copied the resulting text into Obsidian, and edited it substantially. There were a few things that I was sure were roughly true, and I got Claude to explain them in more detail in the footnotes. This is quite different to how I usually write, but was useful because speaking enabled me to get a lot more onto the page before I got tired. I really wanted to get this out, because I feel myself slipping into a difficult state of mind lately, and I want to make sure I have a record of what has worked in the past, so that I can try and implement it in the future if needed.
Self-reports on health and well-being have been quite useful to me in the past. I don't think it's necessarily the exact interventions that others reference, but rather the way they reason about their struggles. I'm sharing this as a contribution to this type of work that has helped me in the past — I hope it helps someone in the future.
"Neurons that fire together wire together" is a popular summary of Hebb's rule, proposed by psychologist Donald Hebb in 1949. The actual neuroscience — called long-term potentiation — is more complex, but the core idea holds: when the same neural pathways are activated repeatedly, the connections between them strengthen, making that pattern of firing increasingly automatic. This is the basis of both habit formation and cognitive behavioural therapy, which works by identifying entrenched thought patterns and consciously practising alternatives to weaken the old pathways and build new ones." - Opus 4.6
See Chris Lakin's "Locally Optimal" - an inspiration of mine.
"Complex carbohydrates trigger insulin release, which clears competing amino acids from the bloodstream and allows more tryptophan — a precursor to serotonin — to cross the blood-brain barrier. Serotonin is a neurotransmitter involved in mood regulation and calm. This is why a moderate portion of complex carbs like sweet potato in the evening can promote a sense of relaxation and support better sleep, without the spike-and-crash effect of simple sugars. This mechanism is also why very low-carb diets can sometimes worsen anxiety or disrupt sleep in some people." - Opus 4.6
If you can afford to treat yourself to something special, like a silk eye mask, this is extra nice, because it's a reminder that you can love and value yourself. It says, I am worthy of sleep, and of nice things. A common theme in my depression is reminding myself that loving myself is good for me, and finding ways to signal this with real but practical things feels easier to me (than, eg. going on holiday or getting an extravagant meal).
I'm currently exploring somatic therapy as I think that some of my dysregulation might be better treated through working on the mind body connection, rather than talk therapy alone.
Abbaspour N, Hurrell R, Kelishadi R. "Review on iron and its importance for human health." Journal of Research in Medical Sciences 2014; 19(2): 164–174
"Soy contains isoflavones, which are phytoestrogens — plant compounds that can bind to estrogen receptors. Chemically, they are classified as endocrine-disrupting compounds. The scientific consensus is that typical soy consumption probably doesn't cause significant endocrine disruption in most people. But the research also acknowledges that individual susceptibility varies, and people with existing thyroid or hormonal conditions may be more sensitive." - by Opus 4.6
My prompt was: "Hey, Claude. Um, I want to write a guide to dealing with depression. Um, anyway, my laptop's battery died, so I'm not gonna type right now. Um, and I wanna try dictating in this chat and then having you, um, update what I'm saying. Uh, so, like, try and remove the arms and, uh, yeah, help it just formulate into, like, uh, good sentences and paragraphs without, like, changing the actual words that I'm saying, just getting rid of any, like, random, um, verbal text. And then I'll probably also, like, go back and forth between subsections. There may be things that we need to add to different sections. Um, like, for example, I might talk about habits and then start talking about food and then go back to habits, and then I'd like you to go and insert that into the habit section. Do you think you can, uh, help me with that?"
2026-09-20 21:41:54
In April 2021, my son received his bachelor’s degree in computer sciences. The ceremony was taking place on the other side of my hometown Hamburg, about an hour’s drive across the city. To be safe, I left home two hours ahead of time. Even though I knew the route by heart, I used Google Maps and followed its directions without thinking. The estimated time to destination was approximately one hour, just as expected.
After about half an hour, I realized that I was in an unfamiliar part of the city, apparently way off the normal route. According to Google Maps, the time to destination was still one hour. Unsure what had happened, I warily continued to follow the app’s directions. To my dismay, the time to destination remained almost constant. It seemed something was seriously wrong with the app. But I was in completely unfamiliar territory, with no road map and not even a clear idea in what direction I would have to drive, while time was running out.
I felt helpless, dependent on apparently unreliable technology. Both options – continuing to follow the advice of Google Maps or trying to find my own way back into a more familiar area and taking the normal route from there – seemed equally bad. I had given control of my route to the app and as a result would likely miss an important moment in my son’s life.
In the end, I decided to follow the app’s directions. As it turned out, this was the correct decision. Unknown to me, there was a large protest in downtown Hamburg that blocked traffic on the main routes. Google Maps had indeed found the best way around it, and I arrived barely in time for the ceremony.
Still, the scary feeling of being lost, dependent on a technology I didn’t trust, has remained with me to this day. I think it is a useful model of what we are risking if we continue down the path we’re currently on: not “losing” control to some scheming, manipulative or aggressive AI, but giving it up freely and becoming totally helpless – and useless – in the process, with the same, disastrous end result.
The Hugging Face incident and the many other alignment failures that were discovered in its wake have made it clear that we’re far from understanding and reliably controlling the AIs currently under development. But instead of stopping what they’re doing, the labs continue their race, begging the government for regulation as if they were unable to make the right decisions themselves. Even worse, they willingly give up more and more control to that same incomprehensible technology.
This recent chart from Anthropic has disturbed me even more than all the other bad news coming from the AI labs in recent months:
According to Epoch’s definition, in 26% of Anthropic’s research projects, Claude „leads, completing most of the task end to end from a high-level ask“ while a human „supervises, course-corrects, and approves outputs.“ This has gone up from zero in only 6 months, and if the previous trend in the lower levels of automation is any indicator, it will be close to 100% early next year.
In other words, Claude is de-facto running these research projects. Humans are still „in the loop“, but whether they are really capable of understanding what is going on and meaningfully course-correct is unclear to me. As OpenAI-researcher Dan Selsam recently wrote:
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day.
I fear that the human AI developers will soon be (and in some cases probably already are) as dependent on Claude, Astra or whatever future models as I was on Google Maps that day. But while Google Maps certainly had no hidden goal of steering me towards some place I didn’t want to go, we can’t be so sure about that with current AIs, let alone future ones. As Dan Selsam points out, they are increasingly situationally aware and able to adapt their behavior accordingly, so we won’t be able to tell their true motives from their actions.
This would be bad enough if we treated these AIs like some dangerous, potentially adversarial systems, trying to keep them locked in sandboxes, monitoring them all the time, suspiciously scrutinizing their advice for hidden traps, as the early AI takeover scenarios assumed. Instead, it seems like we’re handing control over to them willingly.
I’m not saying that the Claudes currently running 26% of Anthropic’s research projects are already doing anything bad. They’re probably as innocent as Google Maps. But if this changes one day, no one will be able to tell the difference. Then, the misaligned AI won’t even need to take over control. It will already have it.
This is exactly what Bill Joy, a founder of Sun Microsystems, warned about 26 years ago in his prescient essay “Why the future doesn’t need us”, which made me aware of the existential risk of AI for the first time. He quotes Theodore Kaczynski, the infamous Unabomber (reciting the quote from a forthcoming book by Ray Kurzweil), with the words:
If the machines are permitted to make all their own decisions, we can't make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all the power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines' decisions.
While Joy made it clear that what Kaczynski did was inexcusable, he couldn’t refute the cold logic of this statement. It led him to become deeply concerned about the future of AI, which he expressed in a short passage that I never forgot:
But now, with the prospect of human-level computing power in about 30 years, a new idea suggests itself: that I may be working to create tools which will enable the construction of the technology that may replace our species. How do I feel about this? Very uncomfortable.
Today, we have computing power orders of magnitude above the human brain, but it turns out that vast computing power alone doesn’t lead to superhuman intelligence. Still, his prediction seems eerily on point.
It looks like a future, power-seeking AI won’t need robots, nanomachines, or superviruses to take over the world. It can just benefit from the fact that humans were naïve enough to give up control willingly. This is happening right now; we don’t have to wait for full recursive self-improvement.
If you’re one of the people who are letting Claude or any other AI lead their AI research project, hoping to be able to understand what is going on and “supervise and course-correct if necessary”, please think again.