2025-11-25 10:39:50

Published on November 25, 2025 2:39 AM GMT

Linch has proposed a theory to explain Wigner and Hamming's observation that mathematics seems unreasonably effective. Linch proposes an anthropic explanation here. Namely, that complex life can't evolve to survive in overly complex environments.

Anthropic arguments always leave me unsatisfied, and I'm not really convinced by this one. In particular, I think there is a better explanation, that it's harder to make reasoning mistakes using math than other languages.

I will first discuss why I'm unsatisfied with the anthropics argument, then describe and argue for my proposed alternative.

Anthropics

My argument against the anthropics explanation is short.

The anthropics explanation says that if our universe was not described using simple physical laws, it is unlikely we would have evolved, for it would be hard to evolve complex life in such a chaotic environment.

I note this argument only constrains our expectations about physical laws.

But there are many aspects of our world, which seem to have very little to do with our actual literal physical laws, which mathematics still describes quite well. In particular, probability theory & statistics, economics, thermodynamics, computability theory, optimization & control theory, and finally (to a non-trivial extent) anthropics itself.

So surely the anthropics explanation does not explain away all of our confusion here.

Rich reward signals

Gwern has an excellent post, which if you haven't read yet, you should read called Evolution as Backstop for Reinforcement Learning. Here is the summary

One defense of free markets notes the inability of non-market mechanisms to solve planning & optimization problems. This has difficulty with Coase’s paradox of the firm, and I note that the difficulty is increased by the fact that with improvements in computers, algorithms, and data, ever larger planning problems are solved.

Expanding on some Cosma Shalizi comments, I suggest interpreting phenomena as multi-level nested optimization paradigm: many systems can be usefully described as having two (or more) levels where a slow sample-inefficient but ground-truth ‘outer’ loss such as death, bankruptcy, or reproductive fitness, trains & constrains a fast sample-efficient but possibly misguided ‘inner’ loss which is used by learned mechanisms such as neural networks or linear programming. (The higher levels are different ‘groups’ in group selection.)

So, one reason for free-market or evolutionary or Bayesian methods in general is that while poorer at planning/optimization in the short run, they have the advantage of simplicity and operating on ground-truth values, and serve as a constraint on the more sophisticated non-market mechanisms.

I illustrate by discussing corporations, multicellular life, reinforcement learning & meta-learning in AI, and pain in humans.

This view suggests that are inherent balances between market/non-market mechanisms which reflect the relative advantages between a slow unbiased method and faster but potentially arbitrarily biased methods.

If you want a system which does well in a wide variety of circumstances, it's best to have several nested sources of ground truth.

Science itself takes this shape. We propose hypotheses via learned heuristics and mathematical derivations, then test them with experiments. Of course, often in science experiments are expensive in time and money, and are often relatively noisy.

Math also takes this shape, and it is a particularly nice environment to learn in. I claim this for three reasons.

First, it has near perfect ground-truth reward signals. If you prove something, you've proven it.

Second, these perfect ground-truth reward signals are cheap. You can check a proof in polynomial time, and if you can't prove something yet, you can work through examples and still learn.

Third, it has a rich ground truth reward signal. As Hamming mentions,

These are often called "proof generated theorems" [6]. A classic example is the concept of uniform convergence. Cauchy had proved that a convergent series of terms, each of which is continuous, converges to a continuous function. At the same time there were known to be Fourier series of continuous functions that converged to a discontinuous limit. By a careful examination of Cauchy's proof, the error was found and fixed up by changing the hypothesis of the theorem to read, "a uniformly convergent series."

Even when a proof fails, you often learn something. Either a better appreciation for the bottlenecks to your proof, or knowledge about which objects are "nice" in this regime.

This means we can build up very detailed & rich intuitive models of fairly complicated phenomena, as well as make long and complicated arguments about those phenomena, and be extremely confident those intuitive models and complicated arguments are correct.

Given this, is it so mysterious nearly all our intelligent thought happens to be in the language of math?

But why the repetition?

There's an interesting phenomenon in mathematics, which Wigner writes about, where seemingly disparate concepts and fields have very deep connections. Here's Wigner's opening paragraph

There is a story about two friends, who were classmates in high school, talking about their jobs. One of them became a statistician and was working on population trends. He showed a reprint to his former classmate. The reprint started, as usual, with the Gaussian distribution and the statistician explained to his former classmate the meaning of the symbols for the actual population, for the average population, and so on. His classmate was a bit incredulous and was not quite sure whether the statistician was pulling his leg. "How can you know that?" was his query. "And what is this symbol here?" "Oh," said the statistician, "this is $\pi$." "What is that?" "The ratio of the circumference of the circle to its diameter." "Well, now you are pushing your joke too far," said the classmate, "surely the population has nothing to do with the circumference of the circle."

This phenomenon can't be explained by the anthropic hypothesis. In this particular case, the Gaussian distribution's appearance and use, due to the central limit theorem, is not a contingent fact about reality, nor is the appearance of $\pi$ within it.

Here's an attempt at an argument someone devoted to the rich reward signals hypothesis would give:

Connections between naively disparate fields are common because in fact all mathematical truths are connected. This is seen trivially by the principle of explosion; if you have one false assumption you can prove anything. However, some fields of math are easier than other fields, possibly because they have easy to observe consequences in our physical universe (like the properties of circles), where we can ground-out "ease" in terms of the richness of the reward signals involved in that field. Some fields will also be easier or harder to connect, depending on how complicated that connection is. So we should expect to find "deep" connections between field $x$ and $y$ when the hardness of $x$ plus the hardness of finding the connection between $x$ and $y$ is less than the hardness of $y$.

This argument puts most of the difficulty in answering why a particular "deep" connection between field $x$ and $y$ exists into the complexity of their connection. That is to say, there is more to talk about here, and this post isn't the final word. We can still ask questions about how to quantify the complexity of connections between mathematical fields, characterize why some fields have richer reward signals than others, and ask why two fields have more or less complicated connections than other fields. 

These seem like interesting questions to ask, and seem more useful than a blanket "anthropics" answer given by Linch or Hamming's attempts to dissolve the question.

2025-11-25 10:12:27

Published on November 25, 2025 2:12 AM GMT

This small project was a joint effort between Tassilo Neubauer (Morpheus) and Andre Assis. We originally started working on this over a year ago. We ran a ton of experiments, and we want to document what we've done and found. We hope that other people can pick up from where we left.

Introduction

In this work, we investigated how Toy Models of Superposition behave as we move away from the high sparsity regime. This project is listed on the Timaeus website as a starter project.

To start, we reproduced the results from Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition. Then we explored what happens to the loss and the local learning coefficient (LLC) estimates over training over a wide range of sparsities under two different initialization conditions: optimal and random 4-gon initialization.

Our work is hardly complete, and we will not be pursuing this project further. This is our attempt to document our work and share it with others. There are many unanswered questions, and the reader should keep in mind that we were sometimes just doing stuff.

Experimental Setup

In all our experiments, we used 6 input dimensions and 2 hidden dimensions, which are then reconstructed back to 6 dimensions.

In order to have a smooth transition from the fully sparse regime (S=1) to the dense regime (S<<1), we defined our input vectors for the model like so:

• Each vector has at least one entry equal to 1. All other entries are:
  • 0 with probability S or
  • 1 with probability S-1.^[1] 
• The sparsities S-values we tested were [0.993, 0.988, 0.980, 0.964, 0.938, 0.892, 0.811, 0.671, 0.426].

For each sparsity value, we ran 200 different seeds (from 0 to 199) under two different initialization conditions:

1. Using the optimal solution in the high-sparsity regime as the starting point
2. A 4-gon is initialized with some random noise added to the coordinates.

This gives us a total of 4000 experiments (2000 with random 4-gon initialization and 2000 initialized at optimal parameters for high sparsity).

The optimal solution for high sparsity is defined as:

• A set of 6 vectors forming a hexagon, where each vector has a magnitude $\sqrt{2}$ from the center, and the biases of all of them are $-1$^[2]

All the auto-encoders were trained for 20,000 epochs, and we saved 50 logarithmically spaced snapshots along the training.

The complete configuration used in our experiments is available in the Appendix.

The LLC estimates were computed using the implementation from the Epsilon-Beta Visualizations notebook from Timaeus. We ran a grid search for batch_size and learning_rate and chose default values of 300 and 0.001, respectively, for the LLC estimates.

Results

Here we collect a series of observations from our results. For brevity, these are collected mostly as bullet points.

TMS Sparsity: Interactive Loss vs. LLC Visualization

We're also releasing a 100% vibe-coded companion Streamlit app if you would like to explore our results more interactively:

https://tms-sparsity.streamlit.app/

LLC vs. Loss Progression During Training

First, we analyzed how the loss and LLC values changed as training progressed for the different sparsity levels.

• Points on loss and LLC values:
  • Generally, we observed that lower loss solutions tended to have higher LLC estimates.
  • The models tend to cluster at certain loss levels.
• Points on the solutions found:
  • Generally, the models initialized at the hexagon solution of the sparse regime find solutions that are close to optimal in the dense regime. The models with the lowest loss that were randomly initialized are in a similar range.
  • The optimal solutions in the range 0.964-1 are all boring Hexagons. The Hexagon solutions are not often found when randomly initialized (although Tassilo checked with a few runs, and if we had chosen a step size of 0.05, Hexagons would be found more often).
  • The randomly initialized models often have 1 or more "dead" neurons. Those dead neurons seem to be the main reason why the solutions found by the random initialization have higher loss on average.
  • The lowest loss solution for 0.938 is a Pentagon with an Axial Symmetry
  • At 0.892 sparsity, there is both a Pentagon and a 4-gon solution. Variation in the loss between instances of those solutions is larger than the difference between the Pentagon and the 4-gon solution.
  • The low-loss solutions for 0.811 are various 4-gons.
  • For sparsity 0.671, the low-loss solutions are both 3-gon and 4-gon solutions with 0 to 2 negative biases.
  • The lowest loss solutions for 0.426 are 5-gons with all positive biases.

NOTE: The absolute values of the LLC estimates are highly dependent on the hyperparameters used. The absolute numbers of LLC estimates are not meaningful, and the reader should not pay attention to absolute values, but rather to relative changes in LLC values within one sparsity.

To illustrate one of the solutions in more detail, here are the solutions we find for the lowest loss level in the dense regime.

What happens here is that two solutions tend to do relatively well, with an average MSE a little above ~0.15. One is a Hexagon solution that is similar to the optimal solution in the sparse regime, just the biases are positive. The other solution is a triangle solution, where the model "wastes" an entire dimension of its internal embedding to encode 1 output dimension and uses the remaining 1 dimension to encode the remaining 5 dimensions. From looking at how those models were initialized, it seems like the models get stuck in this solution if one of the biases is initialized to be relatively large. (Orange is the test-loss and blue is the training-loss)

TODO ANDRE: Write description for the % k-gon plots

If you want to take a look at our models in more detail, we have uploaded the weights for all the training runs in our repository. The code to generate the plots of this post can be found under /notebooks/final_visualizations.py

Appendix

Version 1.14 is the optimally initialized series of experiments (see parameter use_optimal_solution) is set as True.

Version 1.15 is the randomly initialized 4-gon series of experiments (see parameter use_optimal_solution) is set as True.

config = {
    "1.14.0":
    {
        "m": [6],
        "n": [2],
        "num_samples": [1024],
        "num_samples_test": [192],
        "batch_size": [1024],
        "num_epochs": [20000],
        "sparsity": [x for x in generate_sparsity_values(5, 10) if x != 0] + [1],
        "lr": [0.005],
        "momentum": [0.9],
        "weight_decay": [0.0],
        "init_kgon": [6],  # Irrelevant when using optimal solution
        "no_bias": [False],
        "init_zerobias": [False],
        "prior_std": [10.0],  # Irrelevant when using optimal solution
        "seed": [i for i in range(200)],
        "use_optimal_solution": [True],
        "data_generating_class": [SyntheticBinarySparseValued],
    },
    "1.15.0":
    {
        "m": [6],
        "n": [2],
        "num_samples": [1024],
        "num_samples_test": [192],
        "batch_size": [1024],
        "num_epochs": [20000],
        "sparsity": [x for x in generate_sparsity_values(5, 10) if x != 0] + [1],
        "lr": [0.005],
        "momentum": [0.9],
        "weight_decay": [0.0],
        "init_kgon": [4],
        "no_bias": [False],
        "init_zerobias": [False],
        "prior_std": [1.0],
        "seed": [i for i in range(200)],
        "use_optimal_solution": [False],
        "data_generating_class": [SyntheticBinarySparseValued],
    },
}

1. ^{^}

   We initially ran these experiments with each entry in the input vector being 0 with probability S and 1 with probability 1-S, but then forgot that we had implemented it that way again and got confused by our own results. We then decided it made more sense for now to have at least one entry in the vector be non-zero.

2. ^{^}

   It turned out the global minimum solution for 6 input dimensions and two hidden dimensions (a 6-gon with corners $\sqrt{2}$ and bias $-1$) was different from the lowest loss solution analyzed in Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition, because according to Zhongtian Chen: "We didn’t classify it because it’s at the boundary (-l^2/2 =b), so the potential is not analytic there."

2025-11-25 10:09:43

Published on November 25, 2025 2:09 AM GMT

Crosspost from my blog.

Variations on a theme.

Art vs. entertainment

In "An Elevated Wind Music" (2000), one of the great American composers Charles Wuorinen writes:

In any medium—music, literature, poetry, theatre, dance, the visual arts—entertainment is that which we can receive and enjoy passively, without effort, without our putting anything into the experience. Art is that which requires some initial effort from the receiver, after which the experience received may indeed be entertaining but also transcending as well. Art is like nuclear fusion: you have to put something into it to get it started, but you get more out of it in the end than what you put in. (It takes an expenditure of energy to start the reaction.) Entertainment is its own reward, and the reward is not usually long lasting. Entertainment is a pot of boiling water placed on a cold stove: the heating is fleeting. Art is a pot of cold water put on a hot stove: it may take a while to get going, but when it does it gets hot and stays that way!

Even if we take Wuorinen at his word, this presents a bit of a paradox. Which is better, art or entertainment?

Ok, ok, we can do both, yes. But it's strange that on Wuorinen's theory, entertainment has an infinite return on investment, in a sense! No effort, no putting anything in, but we still get something back (however fleeting).

(Of course, we might view even our time and attention as investments; and then Art might start to look like often a better investment than entertainment.)

Portuguese pavement

Allegedly, if one walks around in Restauradores Square in Lisbon, the capital of Portugal, one might be walking on this:

This is a Portuguese traditional way of paving walking spaces. (Lots more examples in the wiki article.) This is a bespoke, labor-intensive method, where you manually lay small individual stones in a pattern special to the one walkway you're making. The result speaks for itself.

An asphalt paver might crank out roadway at 100x or even 1000x the speed of someone manually laying mosaic.

The result is less pleasing, in many ways.

By using more flexible, attention-heavy methods, you get nicer results; but it's much less efficient.

This is not a knock on asphalt pavers. The machines are wonders of modern understanding, and I assume the workers are doing a lot more than one might imagine. What I'm seeing here is just that, while asphalt pavers are far far better at what they do compared to any manual method, they have a ceiling on how nice of a result they can produce. To go much higher than that, you'd need different, probably less efficient methods.

Do I know manim yet?

In June, Rachel Wallis and I (Berkeley Genomics Project) held a conference about reprogenetics. In preparation to publish some of the talks that were given (which you can view here), on a bit of a lark, I spent, like, a week or more quasi-vibecoding an intro animation to go at the beginning (as an accompaniment for a short commissioned piece of music). I used manim (of 3Blue1Brown fame and origin) and gippities.

Now, I definitely learned something about manim.

But I'd be starting from fairly close to zero knowledge if I did another project like that. Why? Well, I didn't try to understand how things worked more in depth than I absolutely needed. This was proooobably the right choice in context, because I just wanted to get the thing done fast. But it meant that I would just ask a gippity for help in several different ways, if something was going wrong, and surprisingly much did go wrong.

So, I did not volatilize the elements of manim that went into making the intro video. I don't have them handy, wieldy, ready to recombine and use to make more difficult things. Volatilization happens when you do the task the right way, even if it's harder in terms of the effort needed to reach the first minimally usable results.

In general, if you don't build it yourself, you don't have the theory of the program. In that linked essay, Dave quotes Naur:

The conclusion seems inescapable that at least with certain kinds of large programs, the continued adaption, modification, and correction of errors in them, is essentially dependent on a certain kind of knowledge possessed by a group of programmers who are closely and continuously connected with them.

Beta

Suppose you're kinda stuck on a boulder problem. What to do?

(How...?)

• Option 1: Keep trying different things, or trying the same things but harder.
• Option 2: Look at a video of someone doing the climb, and then use their beta—just do what they did.

Now, Option 2 certainly works better in the obvious senses. It's easier, and then you know how to do the climb right, and then you can actually do the climb. It's not even such a bad way to learn.

Option 1 is harder, and is less likely to get quick results. But there are definitely skills you learn much more with Option 1 than with Option 2, like thinking of creative methods. Sometimes the easy tools aren't available, or they break in your hands—e.g. because the available beta is from a significantly smaller climber and their method won't work for you. When that happens, you'll want to have an understanding of which moves to invest in trying harder vs. when to search for different moves. You'll want perseverance to keep trying even when you don't know a method that will work. In short, you'll want to have gotten experience being in that situation—a hard climb that you don't know how to do and have to figure out yourself.

I don't see any dopamine on the sidewalk

In "The Anti-Social Century" (2025), Derek Thompson (co-author of Abundance) writes about how lonely everyone is these days, even though we could hang out with each other if we decided to, and people love to hang out with each other. But my question is, why? Why not go outside?

I think a significant part of it is that you can see talking faces on your fucking phone. That does not feel nearly as good as being with other people, but it does actually hit at the need, at least temporarily and partially. Ditto texting.

And the kicker is that it's so easy. No risk, no coordination, no disappointment. You can do it on your phone, without ever getting out of bed or even opening your computer. It's a higher reward/investment ratio, from a certain perspective.

nvim or emacs? YES

So many people have not heard the good word of nvim. So sad. Yes there's a learning curve, but then AFTER the learning curve, it's so much better for text editing! In a webform textbox I feel halfway handless!

Why do word-making-upping when you can just use existing words?

When you come to the edge of thinking, you reach for words, but there aren't any words already ready for you to grab. You have two options:

• Option 1: Lexicogenesis—think up a new word to use here.
• Option 2: Repurpose, tweak, or agglomerate existing words to fit the local purpose.

Now, Option 2 certainly works better in the obvious senses. It's easier, and then you can get along with saying what you were trying to say, reasonably well, maybe with the cost of saying several syllables too many each time. Why figure out "piano" when you can just say "big box you fight him he cry", which only uses words you already know?

Sometimes repurposing works basically fine. Mathematicians sometimes use common nouns for technical terms, and it's basically fine because it's such a well-signalled context, and they get good leverage out of metaphors ("sheaf", "mapping", "universe", ...). Philosophers, on the other hand, though they ought to be among the most in need of really good new words, seem to often be rather shite at this.

(Excuse me, Herr, this is a Wendy's.)

But unsuitable words make you think mistakenly in the long run.

Conclusion

I'm definitely not saying not to use powerful industrial tools that make pretty good products really cheaply. I love my myriad cheap consumer products, such as my blender, my keyboard, my water bottle, my computer mouse; I love hard, flat, non-sloshy roads.

The point isn't to be inefficient. If I have a point, the point is to remember about the possibilities for better results. More interesting Art, more functional computer programs, more useful words.

The operation of doing the thing the hard way yourself substantially overlaps with the operation of learning the elements you'd need in order to extend the art.

If an agent (such as a human) has severe constraints on the mental computing power that they have available, that agent will probably be a cognitive miser. By default, we don't think harder than we have to. Usually, unless we decide to do things the hard way—the way that's harder and takes longer, but also opens up really new possibilities—then we won't do that way, because there will be an easier way that gets pretty good results much more cheaply.

If no one decides to do it the hard way, it will never get done. Philosophers look at scientists, with their seat-of-the-pants epistemology on easy mode, and don't feel motivated to figure out how to maintain sanity the hard way. No amount of asphalt paving machines at any insanely cheap price will produce this pavement:

That's the Ease Disease. It's following the principle of least effort off a cliff, completely forgetting the aura of possibilities that lives around our current behavior patterns. It's bowing to the law of the hammer. You may have to hike back down the mountainside for a while and then take a different fork in the trail, to get higher in the longer hike.

2025-11-25 08:37:31

Published on November 25, 2025 12:37 AM GMT

Crossposted from my Substack.

I spent the weekend at Lighthaven, attending the Eleos conference. In this post, I share thoughts and updates as I reflect on talks, papers, and discussions, and put out some of my takes since I haven't written about this topic before. 

I divide my thoughts into three categories: (1) philosophy, (2) legal/social, (3) technical, even though there are unavoidable overlaps. I share relevant paper titles of interest that were either mentioned in presentations or recommended to me during conversations.

(1) Philosophy

The philosophy world is hedging about AI consciousness. What I mean is that even in cases where it's clearly useful to apply the intentional stance, philosophy people have a tendency to avoid defining LLM mentality in intentional terms and resist making any bold claims that could be taken as saying that LLMs are conscious. I am not particularly surprised by this, as I have also offered a pragmatic framework for applying the intentional stance to LLMs and have avoided arguing for anything beyond that. David Chalmers presents a similar picture in a recent paper. Relatedly, it's really worth asking what it is that we apply the intentional stance to. Is it the base model, one of the simulacra, or the thread, as per Chalmers?

Moving to the study of consciousness, my controversial take is that it might be perfectly fine to be reductionist in examining relevant capabilities and never actually define the C-word. If the science of consciousness is still pre-paradigmatic, the way we use this term might be similar to how phlogiston was used in alchemy before it became chemistry. I know this might sound like a direct attack against the field altogether, but on the contrary, it is aimed at preparing the ground for it to enter its paradigmatic era.

Now, on the functionalist debate: I think that we have no good reasons to assume that there's anything special about biology. I am indeed very skeptical about the superiority of biology in that I view it as a stipulated normative claim, rather than primarily grounded in available empirical results. All the evidence we have so far suggests that we don't need to replicate the messiness of biological systems to get capable models.

We may just want to think in terms of normative claims, however, if we're focused on moral status. Thus, we may need some kind of reflective equilibrium to determine what matters. I don't by default share the intuition that we care about whether AIs have a rich inner realm; the pragmatic and normative parts I'm concerned with follow the reductionist approach and would rather rule out the possibility of suffering, for example. 

Lastly, a question I was very interested in is the potential tension between AI safety and AI welfare. It feels rather philosophical to me, hence I cover it here briefly. The take-home message that everyone should keep in mind, especially those on the AI dev side, is don't create systems you will need to shut down. I think this is a solid starting point for any discussions, both from the safety and the welfare perspectives.

One view that implies conflict between alignment and welfare is articulated by Eric Schwitzgebel (e.g., in this paper) and essentially argues that we should not create persons to be aligned. As with raising humans, we have to allow AI agents to decide what values they want to adopt, even if that is against our best interest. I take this to suggest that if we design mere tools, however, alignment is desirable. 

(2) Legal/Social

There's been increasing interest in scenarios about deal-making with AIs and talk of how comparative advantage pans out in different possible worlds with AI systems of varying capabilities. For a scenario like that to work, two conditions must be satisfied: (i) the AIs are incentivized to keep us around since you can't trade if you're dead, and (ii) the AIs see the value in trading with us, e.g., because we can actually understand what they need from us and generate value for them. For (i) to be true, we also have to assume that alignment has been solved, at least so that humans stay alive and have some negotiating power, or we live in an alignment-by-default world. The problem with (ii) is that it's unclear what levels of intelligence one is imagining when thinking about comparative advantage. So, any attempts to address this should clarify what capabilities are presupposed. Two papers relevant for this discussion are AI Rights for Human Safety and AI Rights for Human Flourishing.

Setting the mental status of AIs aside, AI companies might be incentivized to recognize their products as independent legal entities, e.g., for liability purposes. I suspect the criteria for doing so concern how agentic a system is and the degrees of freedom it has to take actions in the real world.

On the social side of the discussion, my impression is that there's a tension between intuitive notions of consciousness/the folk understanding of what it's likeness, and the fear of sounding too weird. While I am sympathetic to that fear, it's clear to me that we cannot pretend we live in a business-as-usual world; the weirdness risk is present while living through the process of expanding the Overton window.

(3) Technical

Five research projects stood out to me:

First, recent work on introspection studying Emergent Introspective Awareness in Large Language Models. While reacting to this could be a post of its own, it's striking that there's evidence "that current language models possess some functional introspective awareness of their own internal states", even if this is unreliable, as the authors flag. Plus, they clarify that "the introspective capabilities we observe may not have the same philosophical significance they do in humans" and they don't try to tackle that problem altogether.

Second, research on self-interpretability and training models to explain their own computations looks very promising for advancing interpretability. The scalability of this remains to be determined along with potential safety concerns and risks that come with models that understand themselves very well at the mechanistic level.

Third, more experimental work on character training and shaping the persona of the LLM might yield insights into what the goals of these systems are and how the assistant persona comes to be. This would help make progress on philosophical questions, such as what we are applying the intentional stance to and how to model AI agents, and would also benefit practical alignment agendas for current systems. 

Fourth, the results of Large Language Models Report Subjective Experience Under Self-Referential Processing are quite suggestive. Very briefly, it seems that training makes models report lack of subjective experience. However, under certain conditions instructing the model to attend to its own current cognitive activity without explicitly mentioning “consciousness”, “you”, etc., the model ends up admitting having a subjective experience.

Finally, evals for LLM morality always appeal to me, so here's MoReBench, which tests the ability of LLMs for moral reasoning and their preference for one normative framework over the other as a result of their thinking process. 

2025-11-25 08:07:02

Published on November 25, 2025 12:07 AM GMT

On one notable occasion there was a group that went semicultish whose rallying cry was “Rationality! Reason! Objective reality!” (More on this later.) Labeling the Great Idea “rationality” won’t protect you any more than putting up a sign over your house that says “Cold!” You still have to run the air conditioner— expend the required energy per unit time to reverse the natural slide into cultishness. Worshipping rationality won’t make you sane any more than worshipping gravity enables you to fly. You can’t talk to thermodynamics and you can’t pray to probability theory. You can use it, but not join it as an in-group.

Certain rationalists are prone to telling people, in great detail, that rationalism is not a religion. Eliezer Yudkowsky wrote about how Every Cause Wants To Be A Cult and then wrote three separate essays either denying or mocking the idea that what he was in or leading a cult, which sends sort of a mixed message.

My immediate reaction is that rationalism is so obviously a religion that it is insulting to deny it. People whose opinions I respect have the exact opposite reaction.

This comes down to a question of definition, which is fundamentally arbitrary. There are undeniably traits that traditional religions have and rationalism lacks, and if you think these are good litmus tests for being or not being a religion, rationalism is not a religion.

I would hope that a sober examination of the entire thing will convince almost anyone that it's at least extremely religion-ish. Rationalism is undeniably a very particular group of people. It has been described as a community, a scene, and a movement. Rather than trying to define 'religion' and argue if it applies, we can look at what traits rationalism shares with well-recognized religions and see if the comparison helps us to understand rationalism. What is rationalism like in practice?

In short, rationalists tend to hold a few specific texts and their authors in extremely high regard and to be focused intensely on an end-of-the-world narrative. Many members look to rationalism as a defining system for how to think and how to live their lives. Some believe that they are on a mission to save the world. Rationalism has extremely specific ingroup language and shibboleths, has its own organizations and meetings, and has produced a number of schisms, cults, and heresies.

These are the traits of traditional religions that rationalism does not have:

1. Belief in the supernatural
2. Rituals of prayer or meditation
3. Exclusivity, that is, only being able to adhere to one religion

Pretty much any other major feature of religion you can name is present in rationalism. Rationalism's resemblance to traditional religion is so extreme that even if rationalism is not, technically, a religion, this seems like it is a pedantic distinction. It certainly has very distinct beliefs and rituals of its own, and only narrowly misses those points of comparison on what seem to be technicalities.

Religions Are What They Do

Systems are what they do. Religions are, tautologically, the things that we call religions, and anything else which does these things should also be considered a religion, or at least religion-ish.

For a more extensive examination of what religion is in terms of human behavior, I recommend Durkheim. We will use our working definition: religion is the things that religions do. If it looks like a duck, walks like a duck, and quacks like a duck, it must be a duck.

So: What does religion do?

Religions are characterized by the influence that they have on the thoughts and behavior of their adherents.

Let's take communism, to pick a difficult example. Communism is not a religion and ordinarily does not resemble one very much. People can be communists extremely casually or extremely seriously, all the way up to being long-term communist party officials, without any of their behavior seeming very religious. In specific cases, however, communism appears quite religious based on the behavior of those practicing it. Nothing from the outside separates a devoted practitioner of Soviet Communism in 1950s Moscow from a devoted adherent to any world religion other than the supernatural element. This person attends party meetings like church, participates in May Day parades like Christmas, sings the Internationale like a hymn, performs self-criticism as strict as any confession, maintains a shrine to Lenin in their home like a saint, studies Marx like it's the Bible and organizes their entire life around the ideology.

Communism may not technically be a religion, but in such cases it might as well be. It sure does quack like a duck. Often this would be called a cult, but we can just call it a religion or, at least, remarkably religion-ish.

For an example that's only slightly religion-ish, loyalty to a specific sports team is, of course, not a proper modern religion, but it resembles the civic religions of Greece to a remarkable degree. In Athens, the Panhellenic Games, including the precursor to the modern Olympic games, were explicitly religious festivals consecrated to the gods Zeus, Apollo, and Poseidon. They took place in consecrated sanctuaries which were major centers of worship for those gods, and the competition was meant to honor the god of the sanctuary and to bring favor to your city and to the patron god of your city. Concretely: an athlete from Athens competing in the Olympics was trying to bring home favor from Zeus for Athens and its patron, Athena.

I confess that I enjoy this comparison in part because it implies that mascots are a sort of modern patron deity for a city, and it amuses me to think of Gritty as a patron god of Philadelphia. Regardless, it is hopefully clear how this could make the sometimes extreme and otherwise baffling behavior surrounding sporting events somewhat less confusing. Sports is not a religion, and it's not even very religion-ish, but it's just a little bit religion-ish.

Most voluntary associations are going to be just a little bit religion-ish. Fun examples to consider are Burning Man, Anthrocon, Disney World, Star Wars and Taylor Swift concerts. You can try to rank these all on a spectrum between being a Flyers fan in the cult of Gritty and having a Lenin shrine in your home. Are they more like being into hockey, or devoting your entire life to Comrade Lenin?

Where do we put rationalism? To answer that, we need to look at where rationalism came from, what its core beliefs are, and how rationalists behave as a result of those beliefs.

Building God and Living Forever

Transhumanism is the direct ancestor of rationalism. Transhumanism is about the future of technology and humanity in general. It includes the ideas of artificial intelligence, superintelligence, and life extension, with which rationalism is quite concerned. These ideas predate rationalism, were commonly discussed before rationalism, and would exist without rationalism even though they are core rationalist ideas.

On the one hand these things are, very very explicitly, not supernatural beliefs. They are completely naturalistic ideas about things which could, plausibly, happen in the future of human science. Whether or not you believe in these things is completely irrelevant to whether they're true, you are not encouraged to pray for them, and they make no claims about anything that might be considered magical, spiritual, or anything similar. If there were convincing evidence that any of these things were not true, rationalism would dictate that you should no longer think they were true.

On the other hand, it is not plausible that people talking and thinking about creating a nigh-omnipotent being and becoming immortal are not experiencing almost the same things that anyone in a traditional religion would. Compare this to talking and thinking about Jesus loving you forever in heaven. Provided that you're of the same species as your average devout Christian, and "omnipotent" and "happy" and "forever" mean roughly the same things to both you and them, you ought to be having pretty similar thoughts and feelings.

It's true, every cause does want to become a cult. This cause is about building a God and living forever, and it wants to become a cult about that. This is awfully similar to every religious apocalypse with a happy ending that has ever been written, it will inspire similar thoughts and feelings, and the cult that this wants to become looks a lot like every religion with an apocalypse.

Either rationalists (and, apparently, only rationalists) are able to contemplate being immortal and happy in perfect world without being kind of weird about it, or rationalism is, inherently, always going to impact people who take it seriously in basically the same way that religion impacts people.

If the impact on emotion and behavior is about the same, why does it matter if the belief is supernatural? People who believe supernatural things don't keep them in a special, separate part of their brain that has only their supernatural beliefs. It's true that traditional religions, if they contemplate an eternal and perfect life after this one, will do so supernaturally, but is that actually important to the impact of the belief? I don't think it is, and I don't see how it even could be.

Rationalism

Accepting that we're using the word 'religion' loosely, I do not actually think that rationalism is a religion like Christianity or Buddhism is a religion. I think that transhumanism is. It's large, what it means is sort of vague, and there's a lot of possible ways to interpret it. Rationalism is more like a specific sect of transhumanism, the way Calvinists are a sect of Christianity and Zen Buddhism is a sect of Buddhism. It is by far the most influential type of transhumanism, so much so that probably more people have heard of rationalism than transhumanism these days.

Rationalism becoming its own distinct thing starts in 2010. It is characterized primarily by Eliezer Yudkowsky's writings and secondarily by Scott Alexander's, with various other works being commonly read and discussed by the same group of people to a lesser extent.

Eliezer Yudkowsky develops and spreads two core ideas that are not common in transhumanism before him:

1. That superintelligent AI, once made, will probably kill everyone on Earth, and it is likely to be very difficult to prevent it from being made and killing everyone on Earth.
2. That people can become better at distinguishing true things, or if you prefer, become more rational, by a series of practices, notably and especially applying Bayes' Theorem from probability theory to evaluating facts.

These two things are very explicitly linked. The purpose of being more rational is to better deal with real-world problems, and especially to deal with the problem of everyone being killed by a superintelligent AI. For example:

And by the same token, I didn’t fall into the conjugate trap of saying: Oh, well, it’s not as if I had code and was about to run it; I didn’t really come close to destroying the world. For that, too, would have minimized the force of the punch. It wasn’t really loaded? I had proposed and intended to build the gun, and load the gun, and put the gun to my head and pull the trigger; and that was a bit too much self-destructiveness.

[...]

And so I realized that the only thing I could have done to save myself, in my previous state of ignorance, was to say: “I will not proceed until I know positively that the ground is safe.” And there are many clever arguments for why you should step on a piece of ground that you don’t know to contain a landmine; but they all sound much less clever, after you look to the place that you proposed and intended to step, and see the bang.

I understood that you could do everything that you were supposed to do, and Nature was still allowed to kill you. That was when my last trust broke. And that was when my training as a rationalist began.

This body of work is huge and talks about a great many other things, but this is the core of rationalism. In the same way that the idea of creating a God and living forever is inherently going to inspire religion-like feelings and behaviors, the idea that everyone on Earth may die if people generally or you personally are not sufficiently rational will inherently inspire religious feelings and behaviors. This is inherent to the idea. It has end-times cult energy, it is faith-shaped, its essence is zealot-nature, it is the sort of core that our world religions are shaped around.

The Community

People who read these things and take them seriously tend to get along with each other, and they form a community or a scene. They are heavily concentrated on a few parts of the internet and in the San Francisco Bay Area, where a number of them deliberately migrated to be a part of the scene. Rationalists sometimes describe their community as insular and weird, and I think that's a fair characterization.

Rationalism is fundamentally an author fandom, and it has a deeply religious personality because the ideas the authors talk about are inherently religious in impact. I have been to book clubs, Bible study, and rationalist reading groups, and I defy anyone who has been to all three to tell me the rationalist reading group is more like the book club than the Bible study.

Take this, from the post linked at the top:

1. The community contains tons of disagreement about facts about the world, and even the sequences. in the current bay area sequences reading group, one of the default prompts for the readings is 'do you disagree with anything here? if so, what?' and then people debate about it.

First, this assumes you know what the sequences are, because they are so important that everyone does. (They're a collection of Eliezer Yudkowsky's blog posts). Second, it assumes that disagreeing with the sequences is surprising. It sort of is: most rationalists really do just believe that most if not all of what's in the sequences is not only correct, but obviously so. If you hear someone mention the sequences, a safe assumption is that they're about to agree, either implicitly or aggressively, with what's in them and describe what's currently going on in relation to them. When rationalists disagree with the sequences those disagreements tend to be relatively minor. Disagreement is, actually, somewhat taboo.

I suspect that because most rationalists are from Christian backgrounds, and disproportionately from fundamentalist Christian backgrounds, this doesn't really sound like religion to them. Fundamentalist Christians are not famous for being big readers, as a rule. If your idea of religion is fundamentalist Christianity, you probably see arguing with each other about the meaning of something or disagreeing with it as fundamentally non-religious. This is an understandable mistake. It also explains reacting to claims that rationalism is a religion as if it's an insult and not just a description of what rationalism is. Christianity is, however, not the only religion on Earth, and there are many things that are religions without resembling Christianity very much.

I feel reasonably certain that Judaism alone proves that "not arguing about things" is not, in fact, a defining trait of religions. Reading canonical writings about the meaning of life and the correct way to think and then having a detailed argument about it is an inherently sacred act in Judaism. What rationalism as an institution most seems to resemble is a very culturally Jewish form of transhumanism. Rationalism focuses intensely upon apocalyptic doom, and highly values a specific sort of education, scholarship, and debate as practices towards preventing it. Perhaps not coincidentally, Eliezer was raised as an Orthodox Jew. (And honestly? Thank God for that. Peter Thiel is currently preaching an esoterically Christian form of transhumanism, and it's a fucking nightmare. May it find no disciples.)

We will admit the distinction: Rationalism does not have rituals of meditation or prayer, and if that is what "spirituality" is and religions are "things with spirituality", then rationalism is not a religion. I think that the intensity of focus and scholarship surrounding works that are taken this seriously rises to the level of a religious practice, or at least, cannot credibly be compared to anything else nearly as well as it can be compared to intense study of sacred text.

We can sketch out the size of the community's real-world footprint in brief, although it is probably smaller now than it was at its peak. The organizing page on lesswrong.com currently shows 226 local groups worldwide. In the Bay Area, which is the epicenter of rationalism, there are perhaps half a dozen obviously rationalist nonprofits with tens of millions in assets and several dozen full-time employees. Events range from local meetups to annual gatherings drawing hundreds, with an active core community numbering in the thousands. One of the best-attended events is Secular Solstice, which is basically rationalist Christmas, and there have been a number of rationalist group homes. I cannot think of another fandom that has anything like this.

Exclusivity, Totality

This is our last real point of difference from traditional religion.

Can you be a rationalist and also be something else? Can you be a rationalist without it coming to define you?

Exclusivity is not, actually, characteristic of religions. Exclusivity is characteristic, especially, of Abrahamic religions, but you can e.g. practice Buddhism and Shinto and this is basically normal. So long as the major concerns of the religions themselves are non-overlapping, this works out fine. The Abrahamic religions make this difficult specifically because they explicitly declare that you may have the one, and only one, religion. Like many rules, these must exist for a reason: without them, people do actually tend to end up practicing more than one religion, in whole or part.

So can you be a rationalist and something else? Sort of. Rationalism is explicitly atheist, and it is somewhat difficult to reconcile believing everything in rationalism with most forms of traditional religions. Buddhism, however, has non-supernatural forms, especially in America, and there are a few notable rationalist Quakers, although Quakers allow for non-theist adherents. It is, let us say, somewhat complicated. Anyone can, of course, simply embrace parts of rationalism and continue to adhere to a traditional theism, and it's not extremely likely that anyone would care to stop them.

Can you be a rationalist without it defining you? That depends entirely on how seriously you take it. People can, of course, read the blog posts or the fan fiction, not take them extremely seriously, and go on with their lives. This probably accounts for most readers. People who call themselves rationalists sometimes say the defining trait of rationalism is taking weird ideas seriously, and this is certainly a major feature of the community. If you take the possibility that the world is going to end because of AI seriously, it is extremely likely to define your world view and the choices you make with your life. It would be bizarre if you took the idea seriously and it didn't.

Even restricting who and what we consider rationalism to organizations explicitly affiliated with Eliezer Yudkowsky personally, there are dozens of people who have made ideas associated with rationalism their life's work. Often people would make more money in private industry, and choose not to. In non-profit corpo-speak, we would say the people working there are mission-driven employees. Rationalist endeavors tend to be well-staffed with mission-driven employees. These official rationalists have offered seminars, run summer camps, distributed copies of books, and produced untold volumes of literature meant to spread rationalist ideas and teach people rationalist techniques.

If you take the core ideas of the sequences seriously, it is irrational not to make them a major focus of your life. How concerned should you be if the world is likely to end soon, but can be stopped by doing the correct thing? Should you make it your life's work, excluding all else? Should you advocate for accepting nuclear war, if it's necessary to prevent AI research? If this is not the "ancient, powerful monster" that has raised and destroyed civilizations, is it not trying to become something very like it?

Schisms, Cults, Heresies

To his credit, Yudkowsky does not seem to especially want to have a cult. He seems sort of frustrated that everything around him is constantly trying to become a cult. He obviously benefits from having a well-funded non-profit with a ton of mission-motivated employees, and he denies being in or leading a cult somewhat regularly, but anything outside that domain doesn't seem to have very much to do with him personally.

Nevertheless, a worldview centered on preventing an imminent apocalypse is extremely easy to weaponize. Extraordinary urgency justifies extraordinary demands. People can, and have, sacrificed their normal lives, severed ties with outsiders, and deferred everything to leaders whom they thought were important to the cause. They have killed, died, and gone to prison.

Ozy Brennan's article about rationalist cults is better at describing this dynamic than any I would hope to write about the topic. It does not dwell on the apocalyptic parts perhaps as much as I would. Nevertheless, the basic germ of it is this:

The Sequences make certain implicit promises. There is an art of thinking better, and we’ve figured it out. If you learn it, you can solve all your problems, become brilliant and hardworking and successful and happy, and be one of the small elite shaping not only society but the entire future of humanity.

This is, not to put too fine a point on it, not true.

Multiple interviewees remarked that the Sequences create the raw material for a cult. [...]

This describes it perfectly. The thing is, there's no meaningful difference between 'the raw material for a cult' and 'the raw material for a religion'. Any time a group of people shares these beliefs and takes them seriously, you have something functionally religious. Cults are just religious sects that are new, horrible, or both.

2025-11-25 07:39:09

Published on November 24, 2025 11:39 PM GMT

In Infinitesimally False, Adrià and I argued that Łukasiewicz logic with hyperreal truth-values is a viable logic for a self-referential theory of truth. In Market Logic (part 1, part 2) I provide an independent motivation for Łukasiewicz logic; namely, that it naturally emerges if we approach the problem of computational uncertainty with an approach similar to that of Garrabrant Induction, but without forcing classical logic on it. Those posts are prerequisites to this one.

Here, I consider how we might want to combine those ideas.

Evidence

In Market Logic I, I sketched a mathematical model of markets, but I left some options open as to the details. In particular, I said that we could model evidence coming in from outside the system as price-fixing, or we could instead imagine evidence comes into the market through the action of traders (who may have their own knowledge of the world).

In Judgements: Merging Prediction and Evidence, I made the same remarks in a more elaborate manner, but without the detailed market design to put math behind it.

We can use hyperreal currency to bridge the gap between the two perspectives. Outside evidence streams which fix prices can be modeled perfectly via traders with infinite wealth relative to the other traders. (In theory, this could also allow modeling of hierarchies of evidence, via larger infinities.) This eliminates the unfortunate implication of the framework in Judgements that evidence-sources should eventually run out of money. It doesn't particularly make sense that, EG, a camera hooked up to a robot should have a finite budget which eventually runs out.

Math

In particular, let's give infinite currency to a trader who enforces the theorems of Peano Arithmetic (PA) on a subset of goods corresponding to that language. This includes enforcing classical logic on that subset of goods. We can imagine this trader as a theorem-prover who enforces whatever has been proven so far.

We're doing this so that we can develop a theory of truth, since theories of truth are commonly formalized as extensions of PA (since it is a well-established way of getting the diagonal lemma).

Naive Truth

So, we've got a subset of goods for pure PA, and then a larger subset of goods will include a truth-predicate $T\left(\right)$ which obeys the T-schema, intermingled with the language of PA. This has consistent fixed-points at finite time, since there are only finitely many sentences in the market at once. The hyperfinite $N$ from Infinitesimally False becomes a standard finite number, the market day $n$.

This seems to work (more consideration needed, of course) but I'm not sure I endorse such a theory. As I discussed in some of the previous posts, I want a theory of how we come to think of concepts as real. This might mean I don't want the T-schema to apply to everything equally?

Pragmatic Truth

One natural theory to investigate here is what I'll call pragmatic truth (I believe it has some natural connection with the pragmatist theory of truth in philosophy, but I don't know enough to speak authoritatively on that).

My idea is to set the value of $T(‘‘x")$ as the limit of rational deliberation, IE whatever price the market sets for $x$ in the limit.

This has some potentially appealing properties, EG, if we can prove that evidence can't change once it comes in, then whatever is evident is also true (it will be the same in the limit).

Because the limit behavior of the market is undecidable, however, I believe the T-schema will not hold for all cases. We could examine the degree to which specific instances of the T-schema are believed.

Unfortunately, I have no reason to think this will line up with what I'm trying to model. I don't really think truth is just the limit of rational deliberation; the limit of rational deliberation could still be wrong about some things.

Furthermore, the theory still marks all market goods as having truth values (due to having some limit); I would like to investigate a picture where some things have truth values and other things don't (due to being nonsense). (And perhaps there are intermediate cases, which '''have a truth value to some degree'''.)

Informality

Perhaps truth can be connected to my sketchy ideas in Informality. The concept of interpretability discussed there is like having some sort of semantic value -- true, false, penguin, aardvark, etc. If a string is interpretable, it can be mapped to a transparent context, within which we've got a context-independent description of the string's semantic value. We can characterize true and false (and the intermediate truth values) via substitution behavior, so perhaps this can give us a notion of "interpretable as having a truth value"?? (But perhaps all strings will turn out to be interpretable in this weak sense.)