2025-11-27 15:01:13
Published on November 27, 2025 7:01 AM GMT
Do avalanches get caused by loud noises?
Based on my dozen+ times giving this class or presentation, at least 7/10 of you are nodding yes, and the main reason the other 3 aren’t is that you sense a trap.
So. What do you think you know, and why do you think you know it?
Our bodies are under constant assault. Bacteria, viruses, parasites—an endless parade of microscopic organisms trying to hijack our cellular machinery for their own replication. You don't notice most of these attacks because your immune system handles them quietly, distinguishing self from non-self, helpful from harmful, and deploying targeted responses to neutralize threats before they can take hold.
We all lived through a global pandemic not too long ago, and got a crash course reminder on how to keep ourselves safe from hostile genetic code in COVID-19.
But our minds face a parallel challenge, particularly as the “information age” continues to warp into an endless misinformation war, and public safety efforts on the memetic warfront are lagging hard.
Now more than ever in human history, ideas, beliefs, and narratives continuously seek entry in your mind, competing for the limited real estate of your attention and memory. Richard Dawkins coined the term "meme" precisely to highlight this analogy: just as genes propagate through reproduction and mutation, memes propagate via communication. And just as genes guide their host life forms toward replicative fitness, independent of ethical notions of benevolence or suffering, memes spread based on memetic fitness—being catchy, being shareable—independent of being true.
This creates a problem: the ideas most likely to reach you are not the most accurate ideas. They're the ideas most optimized to spread.
Your biological immune system has two main components. The innate immune system provides general-purpose defenses: skin barriers, inflammatory responses, cells that attack anything foreign-looking. The adaptive immune system learns from exposure, building targeted antibodies against specific pathogens and remembering them for faster future responses.
Your memetic immune system is similar in some ways, and also shares similar failure modes:
1) Failure to recognize threats. Some pathogens evade the immune system by mimicking the body's own cells or by mutating faster than antibodies can adapt. Similarly, some bad ideas evade epistemic defenses by mimicking the structure of good arguments, by appealing to emotions that feel like evidence, or by coming from sources we've learned to trust in other contexts.
2) Autoimmune responses. Sometimes the immune system attacks the body's own healthy tissue, causing chronic inflammation and damage. Epistemically, this manifests as excessive skepticism that rejects true and useful information, or as a reflexive contrarianism that opposes ideas simply because they're popular.
3) Vulnerability through entry points. Pathogens exploit specific vulnerabilities—mucous membranes, cuts in the skin, the respiratory system. Memes exploit specific emotional and cognitive vulnerabilities—fear, tribal loyalty, the desire to feel special or vindicated, the discomfort of uncertainty.
4) Compromised states. Stress, malnutrition, and lack of sleep all weaken the biological immune system. Emotional distress, cognitive fatigue, and social pressure all weaken epistemic defenses, but not just negative ones! You can't get more sick from being too entertained or validated, but you're certainly more open to believing false things that way... like, say, when a loud noise causes an avalanche in a cartoon or film.
No matter how rational you think you are, you cannot evaluate all information perfectly. Just as your body doesn't have infinite resources to investigate every molecule that enters it, your mind doesn't have infinite resources to carefully reason through every claim it encounters. Your reasoning ability is affected by your physical and emotional state, and often relies on heuristics, shortcuts, and trusted filters. This is necessary and appropriate—but it creates exploitable vulnerabilities.
If you want to avoid getting sick, you have various different lines of defense. Most people think of their skin as their first line of defense, but it’s actually your environment. By avoiding others who are sick, you reduce the risk of being exposed to hostile pathogens in the first place.
Then comes the skin, which does a pretty good job of keeping hostile genes away from your body’s resources. Some bacteria and viruses specialize in getting through the skin, but most have to rely on wounds or entry points: ears, eyes, nose, mouth, etc.
The equivalent lines of defense exist for your mind.
First, environment: if you want to believe true things, try not to spend too much time around people who are going to sneeze false information or badly reasoned arguments into your face. You can’t fact check everything you hear and read; you literally don’t have the time, energy, or knowledge needed. Cultivate a social network that cares about true things.
Second is your “skin,” in this case, the beliefs you already have. The more knowledge you have, the less susceptible you are to naively believing falsehoods.
Many people will read a dozen news stories and presume that the authors at least put in some reasonable effort toward accuracy and truth… until they read a news story about a topic they’re an expert in, and get upset at all the falsehoods the journalist got away with publishing. Gell-Mann Amnesia is the name for the bias where they fail to then go back and notice that all the other articles were likely also worthy of similar scrutiny, but they lacked the knowledge needed to scrutinize them.
You may think you’re a skeptical person, but all your skepticism doesn’t matter if you don’t have enough knowledge to activate it in the first place when you encounter new information. You can try to rely on broad heuristics (“journalists aren’t usually experts in the topic they’re writing about” or “journalists often have biases”) but heuristics are not bulletproof, and worse, misfire in the opposite direction all the time.
Like many who grew up at the start of the information age, I used to think I didn’t need to memorize facts and figures because I could just look things up on the internet if I needed to. I no longer believe this. Your memetic immune system requires that you know things to activate. Confusion is your greatest strength as someone who values truth, but you need to feel and notice it first, and you need some beliefs for new information to bounce off of to feel it.
Third comes your active immune system: your ability to reason through bad arguments and research information to separate truth from falsehood. The better you get at identifying logical fallacies and common manipulation methods, the better you are at fighting off harmful memes once they get past your other defenses. Practice good epistemics. Investigate claims, resist emotional manipulation, check your blind spots with trusted peers.
Why are children so gullible? Because they don’t know anything.
Children will believe literally anything you tell them, until you tell them something that either directly contradicts what someone told them before, or directly contradicts their own experiences. Only then do they feel confusion or uncertainty or skepticism, and only after enough instances of being misinformed do they form heuristics like “my parents can be wrong about things” or “adults sometimes lie,” which eventually grows into “people/teachers/governments/etc lie.”
A healthy organism’s genes are constantly informing the cells and systems that make up their body what it needs to do to maintain or return to a healthy state. But there is no inherent homeostasis that human minds know to maintain against each invading idea: baby brains are much closer to empty boxes, and the initial ideas we’re exposed to are what form the immunoresponses against what comes after.
Noticing confusion is the most powerful tool for investigating what's true. When something doesn't fit, when a claim contradicts what you thought you knew, that friction is information. But you need existing beliefs to experience that friction. If you know nothing about a topic, you can't notice when a claim about it is suspicious.
Gullibility isn't just a deficit of critical thinking skills. It can result from a deficit of information. Critical thinking tools don't help if you never think to deploy them, and you're less likely to deploy them when a claim doesn't trigger any "wait, that doesn't match" response.
We learn through the context of what we already know. New information bounces against existing beliefs, gets incorporated or rejected based on how well it fits. The more robust your existing knowledge, the better you can evaluate new claims. The more you know, the harder you are to fool.
This means that rationality training shouldn't neglect object-level knowledge. Learn lots of things. Read widely. Build detailed models of how various domains work. This isn't separate from epistemic skill-building—it's a core component of it.
Your eyes and ears are the entry point for memes to get in, just like genes, but there are other, more relevant weakpoints in your memetic immune system.
Ideological Bias is one of them. Just like Gell-Mann Amnesia, most people will read dozens of articles making all sorts of claims, and it’s all accepted uncritically so long as the authors are reaching conclusions they agree with. Once the author says something the reader doesn’t agree with, their reasoning, or even motives, are subjected to much more scrutiny. In general, we’re more susceptible to believing false things if they confirm what we already believe.
And then there’s Emotional Bias. Emotions aren’t irrational—they're encoded combinations of implicit beliefs and predictions, rapid evaluations that something is good or bad, safe or dangerous. The problem isn’t that we feel emotions when we consider assertions or hypotheses, it’s that the emotions narrow our awareness and ability to process all the data. Most emotions that are harmful to our epistemics are those meant to drive action quickly.
Fear is adaptive when it drives you to react to the shape of a snake in the grass before you have time to evaluate if it’s actually a snake. The people who were not sufficiently afraid when their fellow tribesmate yelled “cougar!” did not pass along their genes as often as those who were. When the cost of being wrong is way lower in one direction than the other, you should expect that there will be a “bias” in a particular direction.
Anger is similar. Protective instincts are far more powerful when acted on quickly, and before there was any sort of misinformation-driven culture wars, a tribe that’s easy to unite in anger would easily outcompete tribes that weren’t.
Our emotions all serve purposes, and one of those purposes is to fuel heuristics that save us cognitive effort and motivate us toward helpful behaviors. It’s only when we recognize a false belief or unhelpful outcome that we label the heuristic a bias.
To avoid these biases in your epistemology, know what your emotional weakpoints are. Some people are especially vulnerable to claims that validate their intelligence or moral superiority. Others are vulnerable to claims that justify resentment toward outgroups. Still others are vulnerable to claims that promise simple solutions to complex problems, or that offer belonging to an exclusive community of truth-knowers.
Relatedly, logical fallacies exploit structural vulnerabilities in reasoning. Ad hominem attacks, appeals to authority, false dichotomies, slippery slope arguments—these persist because they work on human minds, sometimes even minds that "know better." Awareness is necessary but not sufficient. You need to practice noticing these patterns in real-time, in the wild, when your emotions are engaged and the social stakes feel real.
Exercises: Which emotions make you more credulous? More skeptical? Think of some beliefs you hold that you know many others disagree with. Can you tell if you want any of them to be true? Notice if you feel a “flinching” when you imagine any of them being wrong.
Imagine a line, starting at a point and squiggling through three-dimensional space. The origin is the point in spacetime you were born, and the arrow at the end is where you now sit or stand or lie reading. This is your lifeline, the path your physical body has taken through the world.
Imagine now a sheathe around that line, spaced out to about three miles. That’s how far you can see on a clear day before your gaze meets the horizon. Within that sheathe is everything you’ve ever seen with your own two eyes, and also everything you’ve ever heard, smelled, tasted, and touched.
That’s your sensorium, the tunnel through spacetime that makes up all the things you’ve ever experienced.
Everything outside that tunnel was told to you by someone else. Everything.
Books you’ve read, things your teachers told you, even video and audio recordings or live footage, all of it is something you had to trust someone else to understand and fact check and honestly present to you.
Can you honestly say that you evaluated all those sources? What about how they learned what they passed along to you?
Beliefs, like viruses, always come from somewhere. Knowing the source of a belief is crucial for evaluating its reliability.
Pick something you believe that is not the direct result of something you experienced yourself, whether it’s an economic argument or a simple historical fact.
Ask yourself: Who "coughed" this idea on you? Why did you believe them? Confidence? Authority? Something else?
How long ago did it happen? Has new information become available? Do you know if the studies have replicated?
What was the context? Were they sharing something they learned themselves? Or repeating something they read online or heard someone else say?
If the latter, the tracing can continue all the way back to whoever experienced the observation directly from their own sensorium, then wrote a paper about it, or talked about it on TV, or told it to others who did.
It could continue. Should it, though?
Well, I’d say that depends a whole hell of a lot on what your error tolerance is on your beliefs and how important that belief being correct is to your goals or values.
But at least be aware that this is what it means to do origin tracing on your beliefs, and why it matters if you don’t.
Again, we cannot expect ourselves to independently verify everything we believe. We can’t rerun every scientific study we hear of, and we can’t personally verify every historical event to the level of the Apollo 11 moon landing, the Holocaust, or the 9/11 terrorist attack on New York City… three of the most well documented events in history that still somehow manage to provoke conspiracies.
We must trust, at some point, that we have evidence enough, and that the society around us is sane enough in aggregate, to hold with high enough probability that the belief is “justified.”
But there are still pitfalls to watch out for, and the reliance on others should always be paired with understanding of what heuristics are being used to fill in the gaps.
For example: if all you know about two beliefs is that a million people believe in Belief A, and a thousand people believe in an unrelated Belief B, is there a justified reason to hold a higher probability in belief A?
Many people find this question difficult because it sets at odds different intuitions about proper epistemology; the answer is “it depends.”
A thousand experts who've examined evidence independently might outweigh a million people who absorbed a belief from their surrounding culture. But a million people who all observed Event A does add more reliability than a thousand people who saw Event B. And of course a single person with direct access to crucial evidence might outweigh them all.
With that in mind, beware double-counting evidence! If five of your friends believe something, that might seem like strong corroboration—but if they all believe it because they read the same viral tweet, you have one source of information, not five.
Information cascades create the illusion of independent verification. Belief tracing, consistently applied all the way back to the origin, can reveal how much actual evidence underlies apparent consensus, not to mention the quality of that evidence.
It seems intuitive to believe that if a piece of information has passed through multiple people, it’s more likely to be true, because multiple people haven’t determined that it’s false. But in reality, people aren’t just at risk of passing something along without factchecking it; they're also likely to misremember or misunderstand or misquote things as well. The thing you heard them say may not even be the thing they originally read or heard!
Exercise: Name one of your load-bearing beliefs—something that supports significant parts of your worldview or decision-making. Where did you learn it? How reliable were the sources? How much time did you spend looking for counter-evidence? Is there some obvious way the world would be different if it wasn’t true?
As mentioned before, the first step in protecting yourself from hostile genes or memes is a safe environment. And you are part of that environment, both for others and also for your future self. If you pollute the epistemic commons, you’ll likely be affected by something downstream of those false beliefs yourself sooner or later.
The most simple rule: don't repeat things with the same confidence you'd have if you'd verified them yourself. Notice how confidently you talk about things in general, and what phrases you and others use. "I heard that..." should prompt a thought or question from yourself or others: “heard from where?”
"I checked and found that..." should evoke a similar “checked where? When was that?”
If you pass on secondhand information as if it were firsthand, you launder away the uncertainty, making the claim seem better-supported than it is. You also use your own credibility in the place of the source you’re repeating, which for your friends or peers, may make them believe it more than they should if you didn’t look deep enough into it.
More vitally, notice when you’re trying to persuade others of something. Notice if you start trying to argue someone into a belief and ask yourself why. What emotion is driving you? What are you hoping or fearing or trying to protect?
Persuasion is inherently a symmetric weapon. People who believe in true or false things can both be persuasive by means both subtle and forceful. Asymmetric tools like direct evidence and formal logic are cleaner.
The best practice when facing someone who disagrees with you on something important is to try to explain what convinced you the thing was true in the first place. If that’s not convincing for them, investigate what beliefs they hold that your evidence or reasoning is “bouncing off” of so you can examine those beliefs yourself, and then explain why you don’t find them convincing (assuming you don’t!).
This preserves important information—the actual evidence and reasoning—rather than just the conclusion. It treats the other person as an epistemic agent capable of evaluating evidence rather than as a target to be manipulated into agreement. And it allows you to stay open to new information and arguments, while also better understanding others and why they believe the things they believe. Julia Galef calls this "scout mindset" as opposed to "soldier mindset": the goal is to map reality accurately, not to win rhetorical battles.
Use good filtration on your information sources. Before absorbing someone's object-level claims, try to evaluate their epistemics. Do they practice what they preach? Do they build things with their hands, or just opine? Are they regularly wrong? Do they admit when they are? Are they sloppy with beliefs, making confident claims without adequate support? Can they explain their reasoning clearly, or do they rely on appeals to authority or status quo bias?
Variety has value—seek perspectives with different heuristics than yours, even if some have lower epistemic rigor than others. But weight those sources accordingly.
There's a deeper point here, articulated in Eliezer Yudkowsky's Planecrash through the character Keltham:
Or, as he summarizes later: "It is impossible to coherently expect to convince yourself of anything… You can't expect anyone else to convince you of something either, even if you think they're controlling everything you see.”
Your expected posterior equals your prior—you might end up more convinced, but there's a counterbalancing chance you'll find disconfirming evidence and end up less convinced. On net, if you're reasoning correctly, it balances out. You can't rationally plan to move your beliefs in a particular direction.
This means that if you notice yourself hoping to be convinced of something, or trying to convince yourself, something has gone wrong. That's not truth-seeking; it’s the dreaded (and badly named) rationalization.
Exercise: List three people you think have good epistemics, three with bad epistemics, and three you're unsure about. For the uncertain cases, what would it take to find out? Notice if there's something you've wanted to convince yourself of, or hoped someone else would convince you of. Why?
Information hygiene, like physical hygiene, requires ongoing maintenance. It can also be overdone.
If you decide to stay indoors all day and never talk to anyone except through a glass door and only eat dry food… you’re definitely minimizing the chance you’ll get sick, but also leading an impoverished life, and in many ways a less healthy one. It might be justified if your immune system is compromised or during a pandemic, but something has likely gone wrong if you’re living your life that way.
Similarly, beware of being so skeptical that you can no longer trust anything you read or hear. We cannot trust everything others say. We cannot even trust everything we observe ourselves. But caring about truth requires an endless fight against the forces of solipsism or nihilism: reality exists, independent of our subjective experience. We can’t understand the territory, but we still have to live in it, and our maps don’t need to be perfectly accurate to still be worth improving.
Some practices that help:
Improve self-awareness through mindfulness exercises. Notice your emotional reactions to claims. Notice when you feel defensive or vindicated.
Practice explaining what you believe and why to someone skeptical. Write more if you’re practiced in writing. Speak if you’re not practiced at speaking. Articulating the justifications for your beliefs will often reveal that they're weaker than you thought.
When something changes your mind, record the context and circumstances. Build a model of what kinds of evidence actually move you.
Practice asking people what informed their beliefs. Make it a habit to trace claims to their sources. Keep track of people who reveal solid reasoning. Keep them part of your information feeds, and eject people who constantly cough without a mask on.
Prepare for epistemic uncertainty—lots of it. People are generally bad at remembering how uncertain they should be. Even in communities that explicitly value calibration, it's hard. The feeling of knowing is not the same as actually knowing. Betting helps.
And remember: this is genuinely difficult. Even with good intentions and good tools, you will sometimes be wrong. The goal isn't perfect accuracy, but building systems and habits that make you wrong less often and help you correct errors faster when they occur.
Your mind, like your body, will face an endless stream of would-be invaders. You can't examine every one. But you can understand your vulnerabilities, trace your beliefs to their sources, take responsibility for what you spread, and build the knowledge base that makes deception harder. The next pandemic hopefully won’t be for a long while, but the information age has brought with it a memetic endemic, and we all need to be better at hygiene, for our own sake and each other’s.
2025-11-27 13:42:26
Published on November 27, 2025 5:42 AM GMT
A couple of my recent posts (1,2) have mentioned (at least in footnotes) that judges in prediction markets can instead be modeled as market participants with large wealth relative to other market participants, who use their wealth to effectively price-fix.
One might reply: "That doesn't make sense. No one would do that. It won't make any money." I've argued against this view twice recently (1,2).
An analogue is impact certificates. Here, let me set up the analogy:
| one-sided market | two-sided market | |
|---|---|---|
| judge | decision auction | prediction market |
| no judge |
impact certificates/ impact markets |
belief certificates |
I've picked up the idea of impact certificates organically from being around the Bay Area, but over the past year I've run into several Bay Area types who didn't know what they are, so I'll explain. It will be easier if I first explain Decision Auctions.
Caspar Oesterheld described decision auctions in Decision Scoring Rules and A Theory of Bounded Inductive Rationality. A simple decision auction works as follows:
This incentivizes accuracy with respect to the best option and its price, since under-bidding the value of an option can get you out-bid, and over-bidding loses you money.
This is better than prediction markets for decision-making, because prediction markets don't incentivize honest bets anymore if they're tied to decisions. Options that very probably will not be chosen are unprofitable to bet about even if they're actually high-quality options. Prediction markets used for decision-making can therefore get locked into suboptimal policies.
Decision auctions can be won by the biggest optimist (underestimate downside risk), but they give that optimist a chance to lose their money or prove themselves right, which gives them good learning-theoretic properties. Prediction markets might be too cautious and not learn; decision auctions might be too experimental, but they'll learn.
Notice how the above reasoning relies on money-maximalist ideas, however. The bidders are imagined to be money-maximizers, which smuggles in an assumption that they're disinterested in the actual consequences of the decision (they only care about the payout).
This is like a prediction market with judges. At some specific point the judge declares the contest to be over, and awards prizes. A prediction market, at least, can be money-neutral (the winners get money from the losers). Decision auctions, however, require a payout.
Impact certificates decentralize everything. There's no specific judge, just the next person who buys the share.
Suppose I do some work that benefits the world, such as help distribute mosquito nets. I can then issue an impact certificate, possession of which implies a specific sort of bragging rights about the beneficial work. A philanthropic investor comes along and assesses the impact certificate, decides that it is worth a specific amount, and buys it from me for that price.
This is similar to a simple money award for good deeds. The advantage is supposed to be that the philanthropic-investor can now sell the impact certificate onwards to other philanthropic-investors. This means a philanthropic-investor doesn't need to commit fully to the role of philanthropist; they have a chance to sell impact certificates later, perhaps for an improved price if they chose wisely, which makes the philanthropy less of a big up-front commitment, thereby attracting more money to philanthropy.
My favorite version of this also gives some percentage of the price difference back to the original impact-certificate issuer every time the certificate changes hands, so impact-certificate issuers don't have to worry as much about getting a good initial price for their impact.
Like decision auctions, impact certificates do seem biased towards optimism. I don't know of a way for impact certificates to account for actions with net-negative value, which means philanthropist-investors could over-value things by ignoring risks (and face less economic downside than they should).
The big advantages of all this is supposed to result from a robust philanthropic-investor economy, with big established philanthropic-investment firms, small philanthropic-investor startups, widely used impact-certificate marketplaces, etc.
This is very similar to my idea for judgeless prediction markets. Call it belief certificates. You buy things because you think they are the wrong price; you buy what you believe. If enough people are buying/selling in this way, then you don't have to question too often whether you're doing it as an investment or because you intrinsically value correcting the market (maintaining the truthfulness of the shared map). (Although, I tend to think this works best as a play-money mechanism.)
This is the same idea I was trying to get across in Judgements: Merging Prediction and Evidence. Showing the 2x2 grid illustrates a connection between the moral (1,2,3) and epistemic (1,2,3,4) threads in my recent writing.
All three people I can recall describing impact certificates to over the past year have expressed skepticism of the money-maximalist kind. Where does the value come from? Aren't you just hoping for a bigger sucker to come along? Financial instruments with no underlying value will be low-volume. Why buy an impact certificate when you can buy stocks that earn money, to then buy bread instead?
Bread is great, but it is not the sum total of my values. More charitably: it is not actually the case that I can best serve my values by totally factoring my strategy into [best way to earn money] x [best way to spend that money to get things I want].
People buy things for all sorts of reasons. I don't believe these mechanisms are against human nature or against the nature of economics. Setting aside potential legal difficulties,
However, cultural acceptance is needed for robust markets to emerge. It might possibly be too far from the prevailing money-maximalist mindset (or face other hurdles).
I do think all these ideas nicely illustrate the connection between theories of rationality and institution design.
2025-11-27 11:22:48
Published on November 27, 2025 3:22 AM GMT
Sometimes people ask me why someone behaving irrationally is present at a rationalist meetup. This essay is a partial answer to why I think that question is confused. The short version is, we are not set up such that any particular list of skills is a requirement.
Once upon a time I was talking to someone at a rationalist meetup, and they described a conversation they’d had with another attendee. Let's call these two Adam and Bella. Bella, it transpired, had said something incorrect, Adam had corrected them, and Bella got defensive. A bit of back and forth transpired.
(Names have been changed to protect the innocent and guilty alike.)
Adam later talks to me and asks, “why would a rationalist not be glad to learn that they were wrong? Don’t we want to learn to be less wrong?”
(One reason is that Adam is wrong and Bella is right. I have passed through denial and into acceptance at the rate that people show up to the 'believe true things' club, then assume the only explanation for why regulars do not agree on their pet contrarian take is that everybody else is wrong. And of course, being told a fact is not the same as being convinced that fact is true. Assume for the moment that Adam is right on the object level here.)
I’ve had several variations on this conversation. There is a genuine point here, and one I think is important. Why would a member of a website called LessWrong, fully aware of the Litanies of Tarski and Gendlin, not update cleanly? Since it’s come up repeatedly, I would like a canonical explanation for why I think this happens and why I’m not more upset about it. First, a few equivalencies for other identities.
When I look at that list, one answer that jumps out at me is that “mathematician” describes a very wide range of skills.
There is no common census on when someone starts being referred to as any of those identities. “Doctor” is pretty clear, you have to have a medical degree and they don’t have those in stock at Walmart. “Marine” is very explicit, and I confidently expect every marine with all four limbs intact can kick my ass (and quite a few can kick my ass with less limbs than that.) But calling ten year-old Screwtape with his TI-83 BASIC calculator and a look of screwed up concentration a programmer isn’t obviously incorrect, despite the fact that he didn’t even know there was a SortA() function. “Mathematician” is defined as “an expert in or student of mathematics” which means that any kid studying arithmetic for the first time counts.
“Swimmer” is what you yell when someone goes overboard while white water rafting, whether or not they can swim.
Descriptively, “Rationalist” is sometimes used to describe people who show up to house parties in a certain social circle. I've made my particular piece with this. "Rationalist" is just another word with fuzzy category boundaries, and I think at this point it's descriptively correct that the word points as much to the social group as to the skillset.
Let's get a little more complicated.
I have a friend, a great musician, she’s been making music for decades now. Do you think it’s reasonable to assume she knows how to play a G chord?
What if I say she’s a drummer?
I’m not being obtuse for no reason here. There’s lots of sub-specialization in lots of fields. Many people dabble - guitarists who can sing a little, violinists who can do the cello - but again you have to be careful with your terms and assumptions. I think of myself as a decent programmer, but I’ve never written a compiler. Some people who have written compilers haven’t ever stood up a web server. Brazilian Jiu Jitsu is great, but I don’t think it contains a side kick.
Just so, there’s people in the rationality community I expect to strictly beat me me on calibration and betting, but who I think I could outmatch in anticipating my own failures (virtue of humility.) Likewise, I know some people with a comprehensive understanding of cognitive science who are just as overconfident in practice as the man on the street.
I want to be clear about what I’m not saying. I’m not saying I don’t care. I’m not saying rounding yourself out isn’t useful. I’m not saying preferences don’t matter; some authors just prefer first person and that’s fine. I’m also not saying that knowing a skill means literally never making future mistakes; sometimes a singer has a cold or just hits a flat note.
And all of that assumes that there’s some kind of ladder or path to becoming more rational, which posters on LessWrong or attendees at an ACX meetup are inevitably going to encounter. If that’s what you think, you’re obviously going to different meetups than I am.
I like Duncan Sabien’s Make More Grayspaces.
If you think of the target space as a place where An Unusual Good Thing Happens, based on five or six special preconditions that make that target space different from the broader outside world...
...then the grayspace is the place where those five or six special preconditions are being laid down. The Unusual Good Thing isn’t actually happening in the grayspace. It isn’t even being attempted. And The Unusual Good Thing isn’t being exposed to people who don’t have the prereqs down pat.
If a math Ph.D. doesn’t know how to do long division, now I agree we have a problem. Likewise if a black belt in Karate can’t do a side kick. These are also true at earlier levels! Anyone with a high school diploma should have the long division thing down.[1]
"Ah," you might say, "but rationality does have its answer! We can read the sequences!"
Eh.
So first off, there's the Astral Codex Ten groups. It's not obvious from ACX that you should read the sequences. Maybe you just want to read ACX, and some people will just have been reading for the last few weeks or months before they joined.
Then you run into the fact that this is kind of rough on newcomers, and newcomers won't stop coming. I know of multiple local groups that read all the sequence posts! It usually takes them a couple years! None of them wanted to restart from the beginning even though some new people had joined halfway through.
Plus, I'm just skeptical that reading the sequences once, years ago, is enough to remember it?
This thing we do, this collective effort to have a real art of rationality? I’m not convinced we have a real grayspace for most of it. I want to congratulate the Guild of the Rose for being a place where they say ‘hey, try changing your mind, and admit it out loud. We’re keeping track of who has done it and who hasn’t.’ It’s one of three paths and guild members can pick what paths to go down, so I can’t quite be told someone’s been a member of the Guild for years and confidently know they can change their mind without flinching, but they’re closer than anyone else I know.
What would it take?
I think it would take three things.
Even if we had one, “rationalist” seems as reasonable a term to apply to someone just starting out as “chess player” or “writer.” I’m not begrudging anyone the term.
(Okay. I am a little bit begrudging the people who aren’t even trying. If you call yourself a chess player and you show up to chess club, you either gotta already know how the knight moves or be willing to have someone show you. If you’re not interested - well, look, I’d like to find a place for the people who are. But the socials are still fine, I run 'em myself and I have a good time at the ones I attend. And as a collective we really have not made it easy to get good at the skills.)
Just, consider all this next time you’re tempted to assume someone should already have a skill.
Lest anyone think I'm casting stones at others here, I did not have long division down when I got my diploma. I think this demonstrated a mistake on the part of the school system. (Short version, I swapped schools a lot and the new school sort of assumed the old one had covered it.)
2025-11-27 10:11:43
Published on November 27, 2025 2:11 AM GMT
"On living, for the sake of your writing"
This is my most liked Substack post!
By than, I mean it got 2 likes, because I don't really show anyone my blog. But, given Inkhaven/Half-Haven, are both soon to end; I thought I'd share this one here.
Epistemic Status: Based on ‘vibes’ not science.
Over the past two months of writing every day, I have grokked the advice I have so often heard from writers, that you must experience life, to be able to write well.
I doubt I’ll be able to instill this idea into your head, in a way that the writers I admire were unable to do for me; with their flowing words and exemplary prose. But, I will still give it a go.
2025-11-27 09:49:58
Published on November 27, 2025 1:49 AM GMT
I seem to be time blind and I like it. From my point of view most people are obsessed with how long something takes instead of how interesting it is. Why in the holy hell would you want to sit around with nothing happening? And why in the flying fuck would you interrupt yourself doing something interesting just cause time has passed?
I mean, don’t get me wrong. I agree time is an amazing coordination and planning tool. I tend to be on time. Actually, I tend to be early, cause being early requires almost no time tracking at all while you get all the benefits of being a good coordinator. You just arrive long before you need to be there and then zone out. Where zoning out is basically letting go of this finicky annoying time-thingy so we can once again focus on what matters: interesting things.
My problem is that I wish more people were timeblind. I am pretty lost on why people consider me to be the one with the “disability”. Have you considered not waiting and hanging around doing nothing? Have you considered instead, just … doing stuff cause waiting is fucking torture and your life is ticking away every second and why are we sitting here with nothing going on?
Yeah, I was a blast in meditation class.
But seriously. I don’t get it. When I’m in a group, and a problem needs solving, I tend to be the first one to jump up and do it cause well … what else are you supposed to do? I love it when someone else beats me to the punch. People who are faster than me are so relaxing.
But it’s not just in groups. Basically in any situation, have you considered just not waiting? How about not being patient at all? There are a lot of seconds, minutes, and hours that people spend doing nothing. Consider the benefits of getting annoyed that your life is ticking by with no obvious reward. Instead you could be reading, cleaning, exercising, or daydreaming.
But you want to know the irony? Most of the people I get along with are actually really patient. They don’t mind silences, they don’t mind delays, they don’t mind things going slowly.
I hate it. Which is great. They can do that patience thing while I poke a shiny light.
Except when I don’t, cause timeblindness goes both ways: When nothing is happening I want things to go faster, but when stuff gets interesting I want the whole world to stop. Please don’t talk to me when I’m reading. Please don’t interrupt me when I’m typing. Please don’t expect me to stop binging this great book, show, game, or work project that is the glorious ambrosia I want to suck on all day.
I tend to say I crave behavioral addictions. I don’t recommend it. Not cause it’s bad for me, but because it’s probably bad for you. I seem to have gotten a lucky roll on the addiction-trait-chart. See I’m too impatient to press the button of Unreliable Reward. You can’t lootbox me or slot machine me. The distance between the cookies is too astronomical. I’d be thoroughly annoyed before you can string me along. Ain’t nobody got time for that.
Instead I want a game addiction, a twitter addiction, or a media addiction. Cause whenever I get one, I get to live in the glorious place where my entire existence is tiled in the One Interesting Thing, and my mind is synthesizing every possible connection together from every angle, and time is a lieeeee.
Don’t ask me to stop. Don’t ask me to eat. Don’t ask me to sleep or drink.
Actually, please do.
You are probably one of those patient people, who noticed it’s 2 hours past dinner time, and if you like me, you might shove a plate of food under my door. I love you <3
I’ve gotten addicted to facebook, to gaming, to work. Each time I forget whatever else I’m doing, and then try to remember to eat and sleep, while just drowning myself in cool things spawning more cool things.
Don’t make me wait for that.
Cause I don’t care about time. I care about reward signals. Patience is learning to cope with a problem I’d rather solve instead: If there is nothing interesting going on, please just let me make it interesting.
I do agree it’s hard sometimes. Till this year, I’d get angry if my partner said he’d be back in 5 minutes and then was back in 5 actual minutes. I know that sounds deranged. What actually happens is that I don’t look at a clock, don’t have anything to do in those 5 minutes, and every second expands into an additional 10 seconds. If no one actually sets a timer, I become convinced 20 minutes or more have passed.
Compare that to playing a fun game or reading a good book, and every second somehow shrinks to only a fraction of itself. You blink and all the time is gone.
Getting time imposed over interest is maddening to me. I become viscerally angry. There are ways to not. Those include nothing like the advice I’ve gotten. I don’t want to “ground myself on sensory stimuli”. Those are slow and just emphasize we are simply sitting here dying at a speed of 10 seconds per second.
I also don’t want to breathe. Breathing is done automatically. Are you being serious, you want me to spend my attention and time on manually steering my body functions while I could [wildly waves hands] be doing something interesting?
I get that I could wirehead myself into being ok with nothingness. I’m sort of a fan of agnostic buddhism on some level, cause they are cool about being nice about your feelings and your mind. But there is a limit somewhere between functioning ok in life and actually pursuing things you love.
It has helped me to know I’m time blind though. I only discovered it two years ago. For the last year, I’ve set a literal timer when someone says they will be back in a few minutes. It saves us both a bunch of grief. If someone speaks slowly, I either start counting in my head or start poking things around me.
Wow, Shoshannah, that sure sounds like ADHD.
Yeah, sure. If I take ritalin I can see time just fine. But it isn’t actually better in most ways. Less happens. There are fewer connections. Less pressure. Less drive. And I like to be driven. Just let me be time blind. I’ll try to keep things interesting for us. And if you have a minute, maybe push a plate of food under my door.
Hi <3
2025-11-27 05:36:39
Published on November 26, 2025 9:36 PM GMT
This post has been written in collaboration with Iliad in service of one of Iliad's longer-term goals of understanding the simplicity bias of learning machines.
Solomonoff induction is a general theoretical ideal for how to predict sequences that were generated by a computable process. It posits that in order to predict the continuation of our world, we should weight hypotheses by their simplicity ("Occam's razor'') and update the weighting using Bayesian reasoning.
In this post, I introduce Solomonoff induction at a technical level, hoping to be more elegant and satisfying than introductions you might find elsewhere. In particular, the introduction avoids Kolmogorov complexity: Instead of making predictions with the Solomonoff prior probability of hypotheses — given as two to the negative of their shortest encodings, i.e., Kolmogorov complexities — we directly use the a priori probability that a hypothesis is sampled from code sampled uniformly at random. This is closely related to the alt-complexity sketched by Nate Soares before. The approach leads to more elegant results that are simpler to prove than for typical expositions. Yet, since only the prior over hypothesis is changed, the introduction remains compatible with other expositions and should be broadly valuable to many readers.
Later, we will see that this choice has both mathematical advantages and disadvantages compared to an approach based on Kolmogorov complexity, and the latter is likely a more powerful basis to build more theory on. These drawbacks notwithstanding, I hope that this introduction is philosophically satisfying, by touching on questions like "Why should we expect our world to be learnable or predictable in the first place?', "What is a universally good way to predict?", and connections to Occam's razor and why one should favor simple hypotheses when predicting the world. In the FAQ at the end, I will touch on several anticipated questions, including briefly on how the theory might be relevant for neural networks.
Audience. I try to not strictly assume much prior knowledge on Turing machines or related concepts, but prior exposure will still be very useful. Some amount of mathematical or scientific maturity can probably replace exposure to these topics.
Acknowledgments. This post benefitted most from discussions with Alexander Oldenziel, and it was caused by his inquiry about the potential simplicity of learned neural network functions based on analogies with algorithmic information theory (AIT). This made me want to think more carefully about a "degeneracy-first" perspective on AIT itself, from which this post emerged. I'm also grateful to Cole Wyeth for discussions and his feedback on an earlier draft that led to some changes. He also explained to me the argument for why the Solomonoff prior gives the optimal prediction bound among all lower semicomputable priors. Thanks also to Matt Dellago for discussions on the content of this post, and to CEEALAR for hosting me when I finished writing this post.
If we make a sequence of observations in the world, e.g., a sequence of numbers
or a sequence of physical measurements, we are often able to continue the sequence in a way that coincides with its true continuation "in the wild". This seems to broadly work in much of the real world, meaning the world is learnable — we can learn from past experience to predict the future.
Why is that? This question has puzzled humanity for thousands of years. For example, Sextus Empiricus expressed that any rule inferred from a finite amount of data may always be contradicted by more data, and that it is impossible to take into account all data:
When they propose to establish the universal from the particulars by means of induction, they will affect this by a review of either all or some of the particulars. But if they review some, the induction will be insecure, since some of the particulars omitted in the induction may contravene the universal; while if they are to review all, they will be toiling at the impossible, since the particulars are infinite and indefinite.
— Sextus Empiricus (160-210)
René Descartes notices that our minds are so fallible that we cannot even be sure of being awake:
How often, asleep at night, am I convinced of just such familiar events — that I am here in my dressing-gown, sitting by the fire — when in fact I am lying undressed in bed! Yet at the moment my eyes are certainly wide awake when I look at this piece of paper; I shake my head and it is not asleep; as I stretch out and feel my hand I do so deliberately, and I know what I am doing. All this would not happen with such distinctness to someone asleep. Indeed! As if I did not remember other occasions when I have been tricked by exactly similar thoughts while asleep! As I think about this more carefully, I see plainly that there are never any sure signs by means of which being awake can be distinguished from being asleep. The result is that I begin to feel dazed, and this very feeling only reinforces the notion that I may be asleep.
— René Descartes (1596-1650)
And in the following passage, David Hume seems to imply that while we can make future inferences about cause and effect from analogizing about past patterns, there is no principle that can be discovered in nature itself that would justify doing so:
All our reasonings concerning matter of fact are founded on a species of Analogy, which leads us to expect from any cause the same events, which we have observed to result from similar causes. [...] It is impossible, that this inference of the animal can be founded on any process of argument or reasoning, by which he concludes, that like events must follow like objects, and that the course of nature will always be regular in its operations. For if there be in reality any arguments of this nature, they surely lie too abstruse for the observation of such imperfect understandings; since it may well employ the utmost care and attention of a philosophic genius to discover and observe them.
— David Hume (1711-1776)
Similar reasoning as those is expressed by the no free lunch theorems, which roughly state that a priori, no learning algorithm should perform better than any other.
We conclude that without placing any assumptions on the world, learning and prediction should indeed be impossible: If the world were truly (uniformly) random, then any way to use past data to predict the future would indeed be doomed. So why is it possible to learn?
One pre-formal answer is Occam's razor, which exists in various formulations and states that among all explanations that are compatible with our observations, we should choose the one making the least amount of assumptions. There are two quotes often attributed to Occam to this effect:
"Plurality must never be posited without necessity."
"It is futile to do with more things that which can be done with fewer."
— William of Ockham (1287-1347)
Occam's razor is a useful starting point, but it is not formal. Additionally, it would be nice to be able to justify Occam's razor from some minimal assumption that does not simply assume that we should focus on simple hypotheses in our reasoning. How can one do this?
In this post, I motivate the answer by starting with the minimal assumption that our world is computed. Not knowing what it is computed by, we assume that we are generated by running a uniformly randomly sampled program code through a universal Turing machine (Section link).
A consequence of this assumption will be that we can also think of our world as being sampled in a two-stage process: First by sampling a semicomputable (perhaps probabilistic) universe from a prior distribution, and then by generating our history from said universe. As it turns out, the prior on universes — which I call a priori prior — will automaically have a simplicity-weighting built in, thus justifying Occam's razor (Section link).
Repurposing this two-stage distribution for making predictions then leads to Solomonoff induction, which has remarkable predictive power no matter what computation truly underlies our universe. I.e., even if the program code underlying our true universe was in fact not sampled randomly, but chosen adversarially to make our prediction task hard, we would eventually, for a sufficiently long observed history, converge to perfect predictions (Section link).
I will then discuss drawbacks of my choice of an a priori prior. In particular, the weights are likely not lower semicomputable, and the a priori prior is not invariant up to a constant under a change of the universal Turing machine, different from the properties of the classical Kolmogorov-based Solomonoff prior. Additionally, the Solomonoff prior satisfies an optimality property for sequence prediction that does not exist in similar form for the a priori prior. This also means that my choice of prior — which can be regarded as being based on the probability, or degeneracy, of hypotheses — is meaningfully different than an approach based on description length (Section link).
This shows that the maxime that description length equals degeneracy is not universal in algorithmic information theory. Relatedly, in a footnote I discuss the relationship to Nate Soare's alt-complexity and will explain that many commenters under that post were probably wrong about K-complexity and alt-complexity being identical up to a constant. These problems notwithstanding, I do think my choice of prior is a priori easier to justify, depends on fewer choices, and leads to some results being more elegant.
I then conclude and anticipate questions in my FAQ, focusing briefly on the question of how to think about Solomonoff induction in the context of agents observing only part of the universe's history, practical usefulness, potential applicability of Solomonoff induction as a theoretical ideal to understand neural networks, the question of whether Solomonoff induction can be considered a theory of everything, and the sensitivity of codelength to the Turing machine model to work with. The answers are very preliminary, and much more could be said or studied about them.
Observing our world, it seems like the future is determined by the past with precise rules. We capture this by the minimal assumption that our world is computed. Without a priori knowledge of the program that generated us, we simply assume that our history emerged by sampling a uniformly random computer program and piping it through a universal programming language that outputs sequences.
In this section I formalize what this means. Let be our alphabet of bits. is the set of binary strings of arbitrary finite length (where the length string is explicitly allowed!). is the set of infinite binary sequences. And is the set of finite or infinite binary sequences. For two sequences , we write for their concatenation. is then said to be a prefix of , written .
How do we capture that we're "computed"? One idea is to assume that we are generated one "frame" at a time by a program that never changes past frames. This leads to the formalization of a monotone Turing machine in Definition 4.5.3 of Li and Vitányi.[1] Formally, a monotone Turing machine is a computable function with the monotonicity property: For all with , we have .[2] There is a lot to unpack here, and I hope that the following semi-formal explanations give enough intuitions to read the rest of the post:
Not all monotone Turing machines are equally interesting. For example, imagine a machine which simply outputs an empty sequence on any input: This satisfies all properties stated above, but it is very boring.
The other extreme is a universal monotone Turing machine, which can simulate all others. It does this by operating on codes that first describe an arbitrary monotone Turing machine using a binary code , followed by the input to said machine. One tricky detail is that we cannot simply concatenate and to since then we may forget where the description of ends and the input starts.
A solution is to use the following prefix-free encoding of descriptions :
Let's unpack this: is the length of . , for a natural number , is an encoding of as a binary sequence, usually by ordering binary sequences by length and then lexicographically. then first contains a sequence of 's followed by a zero. The number of 's is precisely the length of . After reading that, one knows how many of the following bits encode . After reading those bits, one knows the length of , and so after reading so many further bits, one knows . Thus, it is algorithmically possible to disentangle a binary sequence of the form into and since itself tells you when will end.
Then, a universal monotone Turing machine only produces non-trivial outputs for inputs of the form . Additionally, for every , is itself a monotone Turing machine, and every monotone Turing machine is of the form for at least one . Note that will never produce output bits before reading since only encodes a monotone Turing machine . Thus, for any prefix of , is simply the empty string.
This is a nice definition, but does such a universal machine exist? The answer is yes, and it is proved by enumerating the set of monotone Turing machines in a computable way, which Li and Vitányi[1] state to be "clearly" possible right before Definition 4.5.5 —without giving a proof. Intuitively, this is believable by thinking about universal programming languages: Any "monotone Turing machine" can be implemented by a finite code in any universal programming language like Python.
I fix a universal monotone Turing machine for the rest of the post. In a later section we will see that the theory is quite sensitive to the choice of this machine, but for now, we will not worry about this.
A monotone Turing machine induces another function (by abuse of notation with the same notation) whose output on an infinite input is defined to be the limit of the outputs of finite prefixes:[3]
Now, let be the uniform distribution on , which samples infinite binary sequences one bit at a time, each with probability 50% to be or . Let be a random input sampled from , and let be its output (possibly infinite). For finite , let be the resulting probability that a history starts with . Formally:
Definition 1 (Universal a priori distribution). Let be a universal monotone Turing machine and be the uniform distribution on infinite binary codes. The universal a priori probability that a history starts with is defined as
I invite you to pause for a moment about what a profound perspective shift this is compared to the historical puzzlement I emphasized in the introduction: When incorrectly assuming that our history is sampled uniformly, we cannot predict the future, as the skeptical views expressed by Empiricus, Descartes, and Hume make clear. However, the universal a priori distribution does not assume a uniformly sampled history. Instead, it assumes a uniformly sampled code, from which the history is generated with the fixed, deterministic universal monotone Turing machine . In other words, we take our universe to have a uniformly sampled description instead of history.
Should we expect any regularities in the histories sampled from ? And if so, does this regularity give us a hint about how to predict sequence continuations in our actual universe? These questions are answered in the next two sections.
We now state and prove an equivalent description of the universal a priori distribution , which will show that has some simplicity-weighting. Intuitively, it it will turn out that sampling histories from is as if you were to first sample "probabilistic physical laws" in a way that obeys Occam's razor, and then use those laws to sample the history.
We formalize "probabilistic physical laws" by the notion of a semimeasure:
Definition 2 ((Lower semicomputable) semimeasure). A semimeasure is a function with [4] and for all , where is the empty binary string. is called lower semicomputable if it can be computably approximated from below; This means there is an algorithm which, on input , produces progressively better approximations of from below. I abbreviate "lower semicomputable semimeasure" with LSCSM from now on.
For a LSCSM , is interpreted as the probability, under , that an infinite sequence starts with . We can have since the sequence may not continue after starting with . This is akin to the possibilities that the universe can end at any moment.
As an aside, LSCSMs also include deterministic sequences: Let be a fixed sequence that can be generated by a computer program. Define for all prefixes of , and , else. Then for a given , we can compute by computing and determining whether is a prefix. This shows that is an LSCSM that generates the sequence .
How can we obtain LSCSMs? Well, for one, we already know one instance: is a semimeasure since
and for every :
leading to . is also lower semicomputable. Indeed, to approximate from below, simply enumerate all finite binary sequences , compute over all in parallel, and record whether . If this is the case, then for all infinite extensions of , we have , and they together have probability , with the length of . Thus, we can add to our estimate of and then skip any extensions of in our algorithmic procedure. The estimate then progressively approximates since for every with , there is a finite prefix with , meaning we will find the contribution eventually in our approximation.
What other ways are there to obtain LSCSMs? Well, you may have noticed that I did not use the universality of in the explanation above. In other words, every monotone Turing machine gives rise to an LSCSM by piping uniformly random noise through :
And this is all: All LSCSMs emerge from a monotone Turing machine in this way, see Theorem 4.5.2 in Li and Vitányi.[1][5]
Now, for , define the function by
which is itself a monotone Turing machine. Define the LSCSM .
Definition 3 (A priori prior). The a priori prior is given for any LSCSM by
which is the probability[6] that a uniformly random infinite sequence starts with an encoding that gives rise to the LSCSM .
The reason I call this the "a priori prior" is that it simply emerged from our definitions without further assumptions, which distinguishes it from the Kolmogorov-based Solomonoff prior that I will investigate later (Section link).
Finally, let be the set of all LSCSMs, where "sol" is shorthand for "Solomonoff". We obtain:
Theorem 4 (Mixture Equivalence). The universal distribution is a universal mixture over all LSCSMs with weights :
Proof. Since only has outputs for inputs of the form for , we obtain
We obtain
That proves the claim.
This means that if our history was sampled from , we can also think of it as being sampled by first sampling a LSCSM with probability , and then sampling the history successively from .
If an LSCSM has an encoding , then . This gives a simplicity bias: Universes with a simple (i.e., short) description are more likely under this a priori prior. This can be interpreted as a form of Occam's razor, in that hypotheses with a short description receive exponentially more weight.[7]
So far, I have started with the assumption that our universe is the result of sampling a uniformly random program, and arrived at the conclusion that this is equivalent to first sampling an LSCSM with a prior that gives more weight to simple hypotheses, and then sampling our history from it. Thus, under our assumption, we should expect that the world is more predictable than sceptical paradoxes suggest: We can concentrate our reasoning to simple universes and should have a chance to predict the future better than chance. We will make this case in this section in mathematical detail.
How should we make predictions in light of these thoughts? The answer: We predict using the Solomonoff mixture distribution itself, predicting sequence continuations via familiar conditional probabilities upon observing an initial history . This is the most parsimonious way to predict, assuming we a priori only make the assumption of having emerged from a program being sampled uniformly at random.
We now explain in more detail how to predict using by relating the conditionals to Bayesian updating. For any LSCSM , define the conditional probability . We obtain:
where I defined in the suggested way. This is actually a Bayesian posterior in the conventional sense; One can notationally see this more clearly by writing , the likelihood of history conditional on the hypothesis . Then the inner formula becomes:
This is the classical Bayes theorem.
Thus, we arrive at a very satisfying description of Solomonoff induction:
This is very appealing:
One often-noted drawback is that it is incomputable since we can't precisely determine the posterior or the conditional . However:
All of this is not useful if it would not result in actual predictability of the world. After all, our question was why is our actual universe predictable. Fortunately, it turns out that Solomonoff induction is actually an exquisite predictor under the reasonable assumption that our own universe is a lower semicomputable measure, instead of just a semimeasure.[8]
Theorem 5 (Solomonoff Bound). Let be any lower semicomputable measure on infinite sequences, and assume it generates our actual universe. Define the cumulative expected prediction error of predicting sequences sampled by via as:
Then we have
In other words, the prediction error is upper-bounded by the cross entropy between the Dirac measure on our true universe and our a priori prior probability . In particular, for increaingly long histories, the total remaining prediction error goes to zero.[9]
Proof. We have (with explanations after the computation):
Step 1 is the definition. Step 2 is is a standard inequality that holds whenever is a measure, see Corollary 2.8.8 in Hutter et al.[10] (whereas is allowed to be only a semimeasure). Step 3 uses the definition of the KL divergence and changes the infinite series to a limit of partial sums. Step 4 substitutes in the definition of the conditional KL divergence.
Step 5 is a crucial step. Similar to how Shannon entropy satisfies the well-known chain rule , KL divergence also satisfies the chain rule . Intuitively, the divergence of and measured in the variable plus the divergence in once is already known equals the total divergence of and over . Telescoping this chain rule from till gives precisely the result of step . A full proof of this telescope property can be found in Lemma 3.2.4 in Hutter et al.[10]. Step 6 is just the definition of the KL divergence again.
Now, note that by Theorem 4, we have
With this inequality, we obtain
That finishes the proof.
Thus, adding to our earlier investigations, not only do we have reason to believe our universe is predictable, there also is an amazing predictor as long as our universe is generated from a lower semicomputable measure; This resolves the skeptical paradoxes from the introduction. Furthermore, this predictor is given by a blank-slate probability distribtion that assumes no prior knowledge of what our universe is. While the prediction error is lower for universes that are more likely under (which includes universes with a short description), I note that it goes to zero for long histories regardless of .
This section compares our constructions of Solomonoff induction to the more familiar construction where the prior is replaced by the Solomonoff prior . If you are not interested in this section, read on with the conclusion. This section assumes more prior knowledge than previous sections.
Assume we have a computable enumeration of LSCSMs. We can, for example, obtain this by mapping a natural number to its binary representation — a code — and then defining as above. Furthermore, let be a separate universal prefix Turing machine, which is not directly related to the universal monotone Turing machine . Define the Kolmogorov complexity of a natural number as the length of the shortest input to which computes a binary encoding of . The Solomonoff prior is then given by
One can then prove that approximately (i.e., up to at most a constant multiplicative error), we have
see Li and Vitányi,[1] Theorem 4.5.3. I will prove this statement at the start of the second subsection to reveal an advantage of .[11]
The Solomonoff prior has some distinct advantages that are missing in our definition:
Lower semicomputability. The weights are lower semicomputable, which our weights are probably not. The reason is that to approximate , we would need to be able to algorithmically determine for any binary string whether . There seems to be no method to do this. See our investigation in a footnote for more details on missing lower semicomputability in the related case of Nate's alt-complexity.[12]
Invariance under a change of the UTM. The weights are, up to a multiplicative constant, invariant under a change of . In contrast, is not invariant, not even up to a multiplicative constant, under a change of the universal monotone machine . I learned the following argument for this from Cole Wyeth. Namely, take your universal monotone Turing machine . Define a second machine from it, , which is given by , and with empty output on any other input. We can easily see that this is again a universal monotone Turing machine. With suggestive notation, we obtain:
Thus, if is very small, then becomes extremely small, and there can be no constant factor with . We note that in the approximate inequality, we use , which only holds up to a logarithmic error term that probably does not change the conclusion substantially.
Besides, this also means that the a priori prior has no simple relationship to the K-complexity-based Solomonoff prior that can hold independently of the choices of the involved universal Turing machines. This may be surprising given that the coding theorem shows that the a priori probability of a string is up to a constant identical to its K-complexity. This observation is related to my belief that the alt-complexity and K-complexity in Nate's post are not identical up to a constant, contrary to what several commenters claimed. I discuss this in more detail in a footnote.[12]
Optimality of the Solomonoff prior. Assume we remain in the realm of lower semicomputable semiprobability mass functions , of which is an example, and consider mixtures There is a sense in which the Solomonoff prior is optimal under all these choices for . Namely, let be an enumeration of all lower semicomputable semiprobability mass functions. Then for each under consideration, there is a with . By the coding theorem (Theorem 4.3.3 in Li and Vitányi[1]), we have
up to the multiplicative constant in the last step. In other words, has "more weight" on all indices than . Consequently, when adapting Theorem 5 to general lower semicomputable mixtures in place of , provides a better bound than (up to an additive constant):
There seems to not be any analogous result for our a priori prior . Indeed, it would be surprising if such a result exists since this prior is very non-invariant under the change of the universal monotone Turing machine. On the flip side, the fact that is likely not lower semicomputable means that may not beat .
Dependence on fewer choices. The definition of depends on the choice of a universal prefix machine , on top of the choice of that we already made use of before. has no such further dependence.
is platonic, is not. Theorem 4 is very natural: The proof is essentially revealed from the definition of itself, giving the definition intrinsic justification due to the beautiful result. In contrast, the analogous statement that
only very losely depends on the definitions of and the Solomonoff prior at all, meaning we could have chosen a very different prior without many negative consequences. To explain this, I'm stating the proof of the result: itself is an LSCSM, meaning that for an . This results in
which already establishes the inequality (up to a multiplicative constant) in one direction. For the other direction, first note that is lower semicomputable since is lower semicomputable and each is lower semicomputable. Furthermore, we have
This almost establishes as an LSCSM, except that it doesn't satisfy our assumption that . Thus, define to be identical to except that . Then this is an LSCSM and we obtain
which is precisely the desired inequality in the other direction. As we can see, this theorem only depended on the fact that both and are (essentially) LSCSMs which are mixtures of all LSCSMs; the specifics of and are irrelevant, and the same comparison would hold true for any other lower semicomputable mixture of all LSCSMs. It then appears to me that the Solomonoff prior is ultimately arbitrary, and has no a priori justification.
leads to fewer constants. Related to the previous points, the representation of as holds only up to a multiplicative constant, whereas Theorem 4 was an exact equality. Theorem 3.8.8 in Hutter et al.[10] claims that with a smart choice of the two universal Turing machines and , an exact equality can be achieved, but I am unsure if such a choice can be justified "a priori" in a way that feels satisfying.
In this post, I have introduced Solomonoff induction without using K-complexity: A hypothesis for our universe, modeled as lower semicomputable semimeasure (LSCSM), is considered to be as likely "as it truly is" when sampling uniformly random program codes and piping them through a universal monotone Turing machine. Almost tautologically, this means that the probability of sampling a history with the universal machine equals the probability of sampling the history from an LSCSM after first sampling an LSCSM from the a priori prior (Theorem 4).
I then defined Solomonoff induction as the process of predicting sequence continuations in a universe under the assumption that they were generated from a uniformly random computer program. Applying Theorem 4 then leads to an equivalent description in terms of the a priori prior: To continue a sequence, first form the posterior over hypotheses with Bayesian reasoning, and then use the mixture over the posterior to predict the continuation. Since the prior over hypotheses has a simplicity-bias (hypotheses with a shorter encoding are more likely), this can be considered a formalization of Occam's razor.
In Theorem 5, we saw that this leads to the well-known Solomonoff bound. It shows that the cumulative error of predicting continuations in any computed universe is upper bounded by the negative logarithm of the probability of the universe under the a priori prior, i.e., by a cross-entropy loss. This resolves the historical puzzlements in the introduction and explains how it is possible to predict at all, using just the reasonable assumption that our universe is computed.
I then compared the a priori prior to the Solomonoff prior that is based on K-complexity, and found that the latter has some distinct mathematical advantages: The prior is lower semicomputable, invariant under a change of the universal prefix Turing machine it is based on, and it leads to optimal Solomonoff bounds when compared to other priors that are lower semicomputable (which, however, is not necessarily an advantage against the bound in Theorem 5 since the a priori prior is likely not lower semicomputable). This means that the Solomonoff prior is probably the better mathematical basis for developing further theory.
Yet, the a priori prior seems ultimately more natural, in that it does not depend on an additional choice of a universal prefix machine, and the mixture description of the universal distribution (Theorem 4) is tautologically true — rather than being achieved by a post-hoc analysis involving a further constant multiplicative error, as is the case for the Solomonoff prior. As mentioned to me by Cole Wyeth, perhaps it is possible to unify the two perspectives with a construction akin to a Friedberg numbering by finding a unique encoding of each LSCSM. This might then salvage the coding theorem (aka the correspondence between degeneracy and simplicity, or between alt-complexity and K-complexity) in this setting.
Here I answer some questions that I have frequently anticipated.
Q: In our universe, we don't observe the entire history up until this point in time. Instead, we observe a small part of the universe for a finite amount of time. How is sequence prediction in reality then compatible with Solomonoff induction?
A: The hypothesis in reality is not only the simulator of the universe, but instead the simulator of your precise observations in our universe. Thus, then contains a description of the universe and a description of your location in time and space. One further complicationm is that our perceptions are "qualia", which we do not know how to relate to the formalism of symbol strings, see Wei Dai's post of open problems.
Q: Should we try to predict the real world using Solomonoff induction?
A: That's fairly intractable, so we have to use alternatives. The question is then whether it would be a useful target to try to approximate. My guess is that this depends on the timescales: The shorter the history of our observations, the more we should trust the priors ingrained into our skulls by evolution, for they might be better than the Solomonoff prior and more adapted to our actual universe. On large time horizons, perhaps it becomes increasingly useful to actually approximate Solomonoff induction.
Q: Do neural networks do Solomonoff induction in any meaningful sense?
A: There is an interesting analogy between neural networks and Solomonoff induction: Namely, in the same way that our version of Solomonoff induction predicts sequence continuations by piping random noise through a universal Turing machine, a neural network function is at initialization sampled by aranging random parameters in an architecture. If it additionally turns out that gradient descent is akin to Bayesian updating, the analogy might become quite close. If there turns out to be a sufficiently strong connection between probability and K-complexity in the Solomonoff context (as in the coding theorem for a specific case), one might then reason that neural networks also have a bias toward learned functions with short descriptions, perhaps explaining their generalization capabilities. See also here, here, and here.
Q: Is the Solomonoff a priori distribution just a useful model, or do you actually expect our world to be sampled from it? Is it a theory of everything?
A: My intuition is that it's just a useful model. It depends on the universal monotone Turing machine , which feels unnatural. I'm also uncertain whether our universe is really discrete and computed. Perhaps we live in a continuous mathematical structure instead, and time just emerges in our experience? I have not engaged much with the precise overlap and correspondence between mathematical structures and Turing machines and would be happy for someone to say more in the comments.
Q: How is the complexity of a string under a universal monotone Turing machine related to the one under a different Turing machine model, which may be more akin to programming languages in which we typically write code?
A: There are several bounds that closely relate different codelengths to each other, typically up to at most logarithmic error terms. For an example, see Theorem 3.8.7 in Hutter et al.[10]
M. Li and P. Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Springer Cham, 4th edition, 2019. ISBN 9783030112974., DOI: 10.1007/978-3-030-11298-1.
I found some ambiguity in the literature on whether monotone Turing machines are only partial functions, i.e., only producing an output for some of their inputs, or total functions. I settled on total functions since this makes things conceptually simpler and seems to be the most common view.
Fomally, one can define this limit as a supremum over the prefix-relation.
In the literature, one typically encounters the definition ; However, for our purpose, equality is more natural and will later ensure that all lower semicomputable semimeasures can be represented by a monotone Turing machine.
Confusingly, Li and Vitányi use a definition of LSCSMs where (different from our definition where ), and yet claim that all of them are covered by monotone Turing machines. Perhaps they use a somewhat different definition of monotone Turing machines in which it is allowed for outputs to be entirely undefined (thus, not even the empty string), which means that , whereas our monotone Turing machines are total functions. This paper thinks Li and Vitányi are making a mistake and writes: "It is equivalent to Theorem 4.5.2 in [LV08] (page 301) with a small correction: for any by construction, but may not be 1, so this case must be excluded."
is actually a proper probability mass function, meaning that . The reason is that each is of the form for a and , with all with the same prefix having weight together. Thus, .
The reverse may not necessarily hold: A hypothesis can receive a lot of weight simply by having exponentially many (longer) descriptions, with a priori no guarantee for there to then be a short description. For gaining intuitions on this perspective, I can recommend Nate's post on alt-complexity and the last footnote.[12]
In other words, the theorem assumes that our universe cannot simply end at any finite time, compared to semimeasures which can always stop a sequence continuation. This seems restrictive at first, and would exclude possibilities like a big crunch in which the universe eventually ends. However, I think such scenarios can likely be rescued even in the true measure formalism: For a sequence that ends, simply continue it with to an infinite sequence. With this method, it should be possible to assign a measure to any semimeasure, which hopefully has a similar a priori probability. If this were the case, then we would still be able to predict our actual universe effectively up until the moment when it ends, when predicting it correctly does not matter anymore. I did not check these claims in detail, but perhaps Lemma 1 in this post already contains the answer.
Note that we could have also written this error as the expected value
which makes conceptually clearer that it is an expected cumulative prediction error.
Marcus Hutter and David Quarel and Elliot Catt. An Introduction to Universal Artificial Intelligence. Chapman & Hall, 2024. ISBN: 9781032607023, DOI: 10.1201/9781003460299.
The reader may wonder why we go through the hassle of using a separate universal prefix machine at all. With , another option would be to define as the weight for . The issue with this definition is, perhaps paradoxically, that is then computable, which by Lemma 4.3.1 in Li and Vitányi[1] leads to not being universal, i.e., not dominating all other computable (let alone semicomputable) semimeasures. This, in turn, means that this prior will not lead to the same optimal bounds that we will explain the prior to have.
Nate assumes a universal programming language that can receive input codes that then result through to output functions . An example for is code for "quicksort" or "bubblesort", which both lead to the same function that receives a list as input and outputs its sorted version. He then defines the K-complexity
and the alt-complexity
Many commenters claimed that these differ only up to an additive constant, but I believe this to be wrong.
At first sight it seems true, since it seems identical to the coding theorem, which is about the equality of and defined for a universal prefix machine, with being a binary string. The proof of the coding theorem crucially uses the lower semicomputability of , which is achieved by dovetailing through all and checking whether the universal prefix machine sends them to , and increasing the estimate of by when this happens. These estimates are crucial to be able to find codewords for that get progressively shorter and approximate in length, which then establishes the approximate equality to .
The same proof strategy does not work for Nate's alt-complexity since there can, to my knowledge, not exist an algorithm that checks for any program and function whether ; after all, both sides of this equation are themselves functions that can in principle receive infinitely many inputs over which we can't all iterate (See Rice's theorem). Thus, the lower semicomputability of seems to fail, and thus I don't know of a method to construct codewords that approximate the length eventually. The coding theorem thus likely fails in this context, unless there is an entirely different proof strategy that succeeds.
