DynomightModify

2025-10-02 08:00:00

Say an alien spaceship is headed for Earth. It has 30 aliens on it. The aliens are weak and small. They have no weapons and carry no diseases. They breed at rates similar to humans. They are bringing no new technology. No other ships are coming. There’s no trick—except that they each have an IQ of 300. Would you find that concerning?

Of course, the aliens might be great. They might cure cancer and help us reach world peace and higher consciousness. But would you be sure they’d be great?

Suppose you were worried about the aliens but I scoffed, “Tell me specifically how the aliens would hurt us. They’re small and weak! They can’t do anything unless we let them.” Would you find that counter-argument convincing?

I claim that most people would be concerned about the arrival of the aliens, would not be sure that their arrival would be good, and would not find that counter-argument convincing.

I bring this up because most AI-risk arguments I see go something like this:

1. There will be a fast takeoff in AI capabilities.
2. Due to alignment difficulty and orthogonality, it will pursue dangerous convergent subgoals.
3. These will give the AI a decisive strategic advantage, making it uncontainable and resulting in catastrophe.

These arguments have always struck me as overcomplicated. So I’d like to submit the following undercomplicated alternative:

1. Obviously, if an alien race with IQs of 300 were going to arrive on Earth soon, that would be concerning.
2. In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive. Really, that might actually happen.
3. No one knows what AI with an IQ of 300 would be like. So it might as well be an alien.

Our subject for today is: Why might one prefer one of these arguments to the other?

The case for the simple argument

The obvious reason to prefer the simple argument is that it’s more likely to be true. The complex argument has a lot of steps. Personally, I think they’re all individually plausible. But are we really confident that there will be a fast takeoff in AI capabilities and that the AI will pursue dangerous subgoals and that it will thereby gain a decisive strategic advantage?

I find that confidence unreasonable. I’ve often been puzzled why so many seemingly-reasonable people will discuss these arguments without rejecting the confidence.

I think the explanation is that there are implicitly two versions of the complex argument. The “strong” version claims that fast takeoff et al. will happen, while the “weak” version merely claims that it’s a plausible scenario that we should take seriously. It’s often hard to tell which version people are endorsing.

The distinction is crucial, because these two version have different weaknesses. I find the strong version wildly overconfident. I agree with the weak version, but I still think it’s unsatisfying.

Say you think there’s a >50% chance things do not go as suggested by the complex argument. Maybe there’s a slow takeoff, or maybe the AI can’t build a decisive strategic advantage, whatever. Now what?

Well, maybe everything turns out great and you live for millions of years, exploring the galaxy, reading poetry, meditating, and eating pie. That would be nice. But it also seems possible that humanity still ends up screwed, just in a different way. The complex argument doesn’t speak to what happens when one of the steps fails. This might give the impression that without any of the steps, everything is fine. But that is not the case.

The simple argument is also more convincing. Partly I think that’s because—well—it’s easier to convince people of things when they’re true. But beyond that, the simple argument doesn’t require any new concepts or abstractions, and it leverages our existing intuitions for how more intelligent entities can be dangerous in unexpected ways.

I actually prefer the simple argument in an inverted form: If you claim that there is no AI-risk, then which of the following bullets do you want to bite?

1. “If a race of aliens with an IQ of 300 came to Earth, that would definitely be fine.”
2. “There’s no way that AI with an IQ of 300 will arrive within the next few decades.”
3. “We know some special property that AI will definitely have that will definitely prevent all possible bad outcomes that aliens might cause.”

I think all those bullets are unbiteable. Hence, I think AI-risk is real.

But if you make the complex argument, then you seem to be left with the burden of arguing for fast takeoff and alignment difficulty and so on. People who hear that argument also often demand an explanation of just how AI could hurt people (“Nanotechnology? Bioweapons? What kind of bioweapon?”) I think this is a mistake for the same reason it would be a mistake to demand to know how a car accident would happen before putting on your seatbelt. As long as the Complex Scenario is possible, it’s a risk we need to manage. But many people don’t look at things that way.

But I think the biggest advantage of the simple argument is something else: It reveals the crux of disagreement.

I’ve talked to many people who find the complex argument completely implausible. Since I think it is plausible—just not a sure thing—I often ask why. People give widely varying reasons. Some claim that alignment will be easy, some that AI will never really be an “agent”, some talk about the dangers of evolved vs. engineered systems, and some have technical arguments based on NP-hardness or the nature of consciousness.

I’ve never made much progress convincing these people to change their minds. I have succeeded in convincing some people that certain arguments don’t work. (For example, I’ve convinced people that NP-hardness and the nature of consciousness are probably irrelevant.) But when people abandon those arguments, they don’t turn around and accept the whole Scenario as plausible. They just switch to different objections.

So I started giving my simple argument instead. When I did this, here’s what I discovered: None of these people actually accept that AI with an IQ of 300 could happen.

Sure, they often say that they accept this. But if you pin them down, they’re inevitably picturing an AI that lacks some core human capability. Often, the AI can prove theorems or answer questions, but it’s not an “agent” that wants things and does stuff and has relationships and makes long-term plans.

So I conjecture that this is the crux of the issue with AI-risk. People who truly accept that AI with an IQ of 300 and all human capabilities may appear are almost always at least somewhat worried about AI-risk. And people who are not worried about AI-risk almost always don’t truly accept that AI with an IQ of 300 could appear. If that’s the crux, then we should get to it as quickly as possible. And that’s done by the simple argument.

The case for the complex argument

I won’t claim to be neutral. As hinted by the title, I started writing this post intending to make the case for the simple argument, and I still think that case is strong. But I figured I should consider arguments for the other side and—there are some good ones.

Above, I suggested that there are two versions of the complex argument: A “strong” version that claims the scenario it lays out will definitely happen, and a “weak” version that merely claims it’s plausible. I rejected the strong version as overconfident. And I rejected the weak version because there are lots of other scenarios where things could also go wrong for humanity, so why give this one so much focus?

Well, there’s also a middle version of the complex argument: You could claim that the scenario it lays out is not certain, but that if things go wrong for humanity, then they will probably go wrong as in that scenario. This avoids both of my objections—it’s less overconfident, and it gives a good reason to focus on this particular scenario.

Personally, I don’t buy it, because I think other bad scenarios like gradual disempowerment are plausible. But maybe I’m wrong. It doesn’t seem crazy to claim that the Complex Scenario captures most of the probability mass of bad outcomes. And if that’s true, I want to know it.

Now, some people suggest favoring certain arguments for the sake of optics: Even if you accept the complex argument, maybe you’d want to make the simple one because it’s more convincing or is better optics for the AI-risk community. (“We don’t want to look like crazy people.”)

Personally, I am allergic to that whole category of argument. I have a strong presumption that you should argue the thing you actually believe, not some watered-down thing you invented because you think it will manipulate people into believing what you want them to believe. So even if my simpler argument is more convincing, so what?

But say you accept the middle version of the complex argument, yet you think my simple argument is more convincing. And say you’re not as bloody-minded as me, so you want to calibrate your messaging to be more effective. Should you use my simple argument? I’m not sure you should.

The typical human bias is to think other people are similar to us. (How many people favor mandatory pet insurance funded by a land-value tax? At least 80%, right?) But as far as I can tell, the situation with AI-risk is the opposite. Most people I know are at least mildly concerned, but have the impression that “normal people” think that AI-risk is science fiction nonsense.

Yet, here are some recent polls:

Being concerned about AI is hardly a fringe position. People are already worried, and becoming more so.

I used to picture my simple argument as a sensible middle-ground, arguing for taking AI-risk seriously, but not overconfident:

But I’m starting to wonder if my “obvious argument” is in fact obvious, and something that people can figure out on their own. From looking at the polling data, it seems like the actual situation is more like this, with people on the left gradually wandering towards the middle:

If anything, the optics may favor a confident argument over my simple argument. In principle, they suggest similar actions: Move quickly to reduce existential risk. But what I actually see is that most people—even people working on AI—feel powerless and are just sort of clenching up and hoping for the best.

I don’t think you should advocate for something you don’t believe. But if you buy the complex argument, and you’re holding yourself back for the sake of optics, I don’t really see the point.

2025-09-25 08:00:00

I fear we are in the waning days of the People Read Blog Posts About Random Well-Understood Topics Instead of Asking Their Automatons Era. So before I lose my chance, here is a blog post about some random well-understood topics.

Marathons are stupidly fast

You probably know that people can now run marathons in just over 2 hours. But do you realize how insane that is?

That’s an average speed of 21.1 km per hour, or 13.1 miles per hour. You can think of that as running a mile in 4:35 (world record: 3:45), except doing it 26.2 times in a row. Or, you can think of that as running 100 meters in 17.06 seconds (world record: 9.58 seconds), except doing it 421.6 times in a row. I’d guess that only around half of the people reading this could run 100 meters in 17.06 seconds once.

This crazy marathon running speed is mostly due to humans being well-adapted for running and generally tenacious. But some of it is due to new shoes with carbon-fiber plates that came out in the late 2010s.

The theory behind these shoes is quite interesting. When you run, you mainly use four joints:

1. Hips
2. Knees
3. Ankles
4. Metatarsophalangeal

If you haven’t heard of the last of these, they’re pronounced “met-uh-tar-so-fuh-lan-jee-ul” or “MTP”. These are the joints inside your feet behind your big toes.

Besides sounding made-up, they’re different from the other joints in a practical way: The other joints are all attached to large muscles and tendons that stretch out and return energy while running sort of like springs. These can apparently recover around 60% of the energy expended in each stride. (Kangaroos seemingly do even better.) But the MTP joints are only attached to small muscles and tendons, so the energy that goes into them is mostly lost.

These new shoe designs have complex constructions of foam and plates that can do the same job as the MTP joints, but—unlike the MTP joints—store and return that energy to the runner. A recent meta-analysis estimated that this reduced total oxygen consumption by ~2.7% and marathon times by ∼2.18%.

Algernon

I wonder if these shoes are useful as a test case for the Algernon argument. In general, that argument is that there shouldn’t be any simple technology that would make humans dramatically smarter, since if there was, then evolution would have already found it.

You can apply the same kind of argument to running: We have been optimized very hard by evolution to be good at running, so there shouldn’t be any “easy” technologies that would make us dramatically faster or more efficient.

In the context of the shoes, I think that argument does… OK? The shoes definitely help. But carbon fiber plates are pretty hard to make, and the benefit is pretty modest. Maybe this is some evidence that Algernon isn’t a hard “wall”, but rather a steep slope.

Or, perhaps thinking is just different from running. If you start running, you will get better at it, in a way that spills over into lots of other physical abilities. But there doesn’t seem to be any cognitive task that you can practice and make yourself better at other cognitive tasks.

If you have some shoes that will make me 2.7% smarter, I’ll buy them.

Pangea

Pangea was a supercontinent that contained roughly all the land on Earth. At the beginning of the Jurassic 200 million years ago, it broke up and eventually formed the current continents. But isn’t the Earth 4.5 billion years old? Why would all the land stick together for 95% of that time and then suddenly break up?

The accepted theory is that it didn’t. Instead, it’s believed that Earth cycles between super-continents and dispersed continents, and Pangea is merely the most recent super-continent.

But why would there be such a cycle? We can break that down into two sub-questions.

First, why would dispersed continents fuse together into a supercontinent? Well, you can think of the Earth as a big ball of rock, warmed half by primordial heat from when the planet formed and half by radioactive decay. Since the surface is exposed to space, it cools, resulting in solid chunks that sort of slide around on the warm magma in the upper mantle. Some of those chunks are denser than others, which causes them to sink into the mantle a bit and get covered with water. So when a “land chunk” crashes into a “water chunk”, the land chunk slides on top. But if two land chunks crash into each other, they tend to crumple together into mountains and stick to each other.

You can see this by comparing this map of all the current plates:

To this map of elevation:

OK, but once a super-continent forms, why would it break apart? Well, compared to the ocean floor, land chunks are thicker and lighter. So they trap heat from inside the planet sort of like a blanket. With no cool ocean floor sliding back into the warm magma beneath, that magma keeps getting warmer and warmer. After tens of millions of years, it heats up so much that it stretches the land above and finally rips it apart.

It’s expected that a new supercontinent “Pangea Ultima” will form in 250 million years. By that time, the sun will be putting out around 2.3% more energy, making things hotter. On top of that, it’s suspected that Pangea Ultima, for extremely complicated reasons, will greatly increase the amount of CO₂ in the atmosphere, likely making the planet uninhabitable by mammals. So we’ve got that going for us.

Egypt and the Sea Peoples

The Sea Peoples are a group of people from… somewhere… that appeared in the Eastern Mediterranean around 1200 BC and left a trail of destruction from modern Turkey down to modern Egypt. They are thought to be either a cause or symptom of the Late Bronze Age collapse.

But did you know the Egyptians made carvings of the situation while they were under attack? Apparently the battle looked like this:

In the inscription, Pharaoh Ramesses III reports:

Those who reached my boundary, their seed is not; their hearts and their souls are finished forever and ever. As for those who had assembled before them on the sea, the full flame was their front before the harbor mouths, and a wall of metal upon the shore surrounded them. They were dragged, overturned, and laid low upon the beach; slain and made heaps from stern to bow of their galleys, while all their things were cast upon the water.

2025-09-11 08:00:00

PendingKetchup comments on my recent post on what it means for something to be heritable:

The article seems pretty good at math and thinking through unusual implications, but my armchair Substack eugenics alarm that I keep in the back of my brain is beeping.

Saying that variance was “invented for the purpose of defining heritability” is technically correct, but that might not be the best kind of correct in this case, because it was invented by the founder of the University of Cambridge Eugenics Society who had decided, presumably to support that project, that he wanted to define something called “heritability”.

His particular formula for heritability is presented in the article as if it has odd traits but is obviously basically a sound thing to want to calculate, despite the purpose it was designed for.

The vigorous “educational attainment is 40% heritable, well OK maybe not but it’s a lot heritable, stop quibbling” hand waving sounds like a person who wants to show but can’t support a large figure. And that framing of education, as something “attained” by people, rather than something afforded to or invested in them, is almost completely backwards at least through college.

The various examples about evil despots and unstoppable crabs highlight how heritability can look large or small independent of more straightforward biologically-mechanistic effects of DNA. But they still give the impression that those are the unusual or exceptional cases.

In reality, there are in fact a lot of evil crabs, doing things like systematically carting away resources from Black children’s* schools, and then throwing them in jail. We should expect evil-crab-based explanations of differences between people to be the predominant ones.

*Not to say that being Black “is genetic”. Things from accent to how you style your hair to how you dress to what country you happen to be standing in all contribute to racial judgements used for racism. But “heritability” may not be the right tool to disentangle those effects.

Dear PendingKetchup,

Thanks for complimenting my math (♡), for reading all the way to the evil crabs, and for not explicitly calling me a racist or eugenicist. I also appreciate that you chose sincerity over boring sarcasm and that you painted such a vibrant picture of what you were thinking while reading my post. I hope you won’t mind if I respond in the same spirit.

To start, I’d like to admit something. When I wrote that post, I suspected some people might have reactions similar to yours. I don’t like that. I prefer positive feedback! But I’ve basically decided to just let reactions like yours happen, because I don’t know how to avoid them without compromising on other core goals.

It sounds like my post gave you a weird feeling. Would it be fair to describe it as a feeling that I’m not being totally upfront about what I really think about race / history / intelligence / biological determinism / the ideal organization of society?

Because if so, you’re right. It’s not supposed to be a secret, but it’s true.

Why? Well, you may doubt this, but when I wrote that post, my goal was that people who read it would come away with a better understanding of the meaning of heritability and how weird it is. That’s it.

Do I have some deeper and darker motivations? Probably. If I probe my subconscious, I find traces of various embarrassing things like “draw attention to myself” or “make people think I am smart” or “after I die, live forever in the world of ideas through my amazing invention of blue-eye-seeking / human-growth-hormone-injecting crabs.”

What I don’t find are any goals related to eugenics, Ronald Fisher, the heritability of educational attainment, if “educational attainment” is good terminology, racism, oppression, schools, the justice system, or how society should be organized.

These were all non-goals for basically two reasons:

1. My views on those issues aren’t very interesting or notable. I didn’t think anyone would (or should) care about them.

2. Surely, there is some place in the world for things that just try to explain what heritability really means? If that’s what’s promised, then it seems weird to drop in a surprise morality / politics lecture.

At the same time, let me concede something else. The weird feeling you got as you read my post might be grounded in statistical truth. That is, it might be true that many people who blog about things like heritability have social views you wouldn’t like. And it might be true that some of them pretend at truth-seeking but are mostly just charlatans out to promote those unliked-by-you social views.

You’re dead wrong to think that’s what I’m doing. All your theories of things I’m trying to suggest or imply are unequivocally false. But given the statistical realities, I guess I can’t blame you too much for having your suspicions.

So you might ask—if my goal is just to explain heritability, why not make that explicit? Why not have a disclaimer that says, “OK I understand that heritability is fraught and blah blah blah, but I just want to focus on the technical meaning because…”?

One reason is that I think that’s boring and condescending. I don’t think people need me to tell them that heritability is fraught. You clearly did not need me to tell you that.

Also, I don’t think such disclaimers make you look neutral. Everyone knows that people with certain social views (likely similar to yours) are more likely to give such disclaimers. And they apply the same style of statistical reasoning you used to conclude I might be a eugenicist. I don’t want people who disagree with those social views to think they can’t trust me.

Paradoxically, such disclaimers often seem to invite more objections from people who share the views they’re correlated with, too. Perhaps that’s because the more signals we get that someone is on “our” side, the more we tend to notice ideological violations. (I’d refer here to the narcissism of small differences, though I worry you may find that reference objectionable.)

If you want to focus on the facts, the best strategy seems to be serene and spiky: to demonstrate by your actions that you are on no one’s side, that you don’t care about being on anyone’s side, and that your only loyalty is to readers who want to understand the facts and make up their own damned mind about everything else.

I’m not offended by your comment. I do think it’s a little strange that you’d publicly suggest someone might be a eugenicist on the basis of such limited evidence. But no one is forcing me to write things and put them on the internet.

The reason I’m writing to you is that you were polite and civil and seem well-intentioned. So I wanted you to know that your world model is inaccurate. You seem to think that because my post did not explicitly support your social views, it must have been written with the goal of undermining those views. And that is wrong.

The truth is, I wrote that post without supporting your (or any) social views because I think mixing up facts and social views is bad. Partly, that’s just an aesthetic preference. But if I’m being fully upfront, I also think it’s bad in the consequentialist sense that it makes the world a worse place.

Why do I think this? Well, recall that I pointed out that if there were crabs that injected blue-eyed babies with human growth hormone, that would increase the heritability of height. You suggest I had sinister motives for giving this example, as if I was trying to conceal the corollary that if the environment provided more resources to people with certain genes (e.g. skin color) that could increase the heritability of other things (e.g. educational attainment).

Do you really think you’re the only reader to notice that corollary?

The degree to which things are “heritable” depends on the nature of society. This is a fact. It’s a fact that many people are not aware of. It’s also a fact that—I guess—fits pretty well with your social views. I wanted people to understand that. Not out of loyalty to your social views, but because it is true.

It seems that you’re annoyed that I didn’t phrase all my examples in terms of culture war. I could have done that. But I didn’t, because I think my examples are easier to understand, and because the degree to which changing society might change the heritability of some trait is a contentious empirical question.

But OK. Imagine I had done that. And imagine all the examples were perfectly aligned with your social views. Do you think that would have made the post more or less effective in convincing people that the fact we’re talking about is true? I think the answer is: Far less effective.

I’ll leave you with two questions:

Question 1: Do you care about the facts? Do you believe the facts are on your side?

Question 2: Did you really think I wrote that post with with the goal of promoting eugenics?

If you really did think that, then great! I imagine you’ll be interested to learn that you were incorrect.

But just as you had an alarm beeping in your head as you read my post, I had one beeping in my head as I read your comment. My alarm was that you were playing a bit of a game. It’s not that you really think I wanted to promote eugenics, but rather that you’re trying to enforce a norm that everyone must give constant screaming support to your social views and anyone who’s even slightly ambiguous should be ostracized.

Of course, this might be a false alarm! But if that is what you’re doing, I have to tell you: I think that’s a dirty trick, and a perfect example of why mixing facts and social views is bad.

You may disagree with all my motivations. That’s fine. (I won’t assume that means you are a eugenicist.) All I ask is that you disapprove accurately.

xox
dynomight

2025-08-28 08:00:00

Here’s one possible hobby:

1. Take something you don’t like.
2. Try to like it.

It could be food or music or people or just the general situation you’re in. I recommend this hobby, partly because it’s nice to enjoy things, but mostly as an instrument for probing human nature.

1.

I was in Paris once. By coincidence, I wandered past a bunch of places that were playing Michael Jackson. I thought to myself, “Huh. The French sure do like Michael Jackson.” Gradually I decided, “You know what? They’re right! Michael Jackson is good.” Later, I saw a guy driving around blasting Billie Jean while hanging a hand outside his car with a sparkly white Michael Jackson glove. Again, I thought, “Huh.” That day was June 25, 2009.

2.

I don’t like cooked spinach. But if I eat some and try to forget that I hate it, it seems OK. Why?

Well, as a child, I was subjected to some misguided spinach-related parental interventions. (“You cannot leave this table until you’ve finished this extremely small portion”, etc.) I hated this, but looking back, it wasn’t the innate qualities of spinach the bothered me, so much as that being forced to put something inside my body felt like a violation of my autonomy.

When I encountered spinach as an adult, instead of tasting a vegetable, I tasted a grueling battle of will. Spinach was dangerous—if I liked it, that would teach my parents that they were right to control my diet.

So I tried telling myself little stories: I’m hiking in the mountains in Japan when suddenly the temperature drops and it starts pouring rain. Freezing and desperate, I spot a monastery and knock on the door. The monks warm me up and offer me hōrensō no ohitashi, made from some exotic vegetable I’ve never seen before. Presumably, I’d think it was amazing.

I can’t fully access that mind-space. But just knowing it exists seems to make a big difference. Using similar techniques, I’ve successfully made myself like (or less dislike) white wine, disco, yoga, Ezra Klein, non-spicy food, Pearl Jam, and Studio Ghibli movies.

Lesson: Sometimes we dislike things simply because we have a concept of ourselves as not liking them.

3.

Meanwhile, I’ve failed to make myself like country music. I mean, I like A Boy Named Sue. Who doesn’t? But what about Stand By Your Man or Dust on the Bottle? I listen to these, and I appreciate what they’re doing. I admire that they aren’t entirely oriented around the concerns of teenagers. But I can’t seem to actually enjoy them.

Of course, it seems unlikely that this is unrelated to the fact that no one in my peer group thinks country music is cool. On the other hand, I’m constantly annoyed that my opinions aren’t more unique or interesting. And I subscribe to the idea that what’s really cool is to be a cultural omnivore who appreciates everything.

It doesn’t matter. I still can’t like country music. I think the problem is that I don’t actually want to like country music. I only want to want to like country music. The cultural programming is in too deep.

Lesson: Certain levels of the subconscious are easier to screw around with than others.

4.

For years, a friend and I would go on week-long hikes. Before we started, we’d go make our own trail mix, and I’d always insist on adding raisins. Each year, my friend would object more loudly that I don’t actually like raisins. But I do like raisins. So I’d scoff. But after several cycles, I had to admit that while I “liked raisins”, there never came a time that I actually wanted to eat raisins, ever.

Related: Once every year or two, I’ll have a rough day, and I’ll say to myself, “OK, screw it. Liking Oasis is the lamest thing that has ever been done by anyone. But the dirty truth is that I love Oasis. So I will listen to Oasis and thereby be comforted.” Then I listen to Oasis, and it just isn’t that good.

Lesson: You can have an incorrect concept of self.

5.

I don’t like this about myself, but I’m a huge snob regarding television. I believe TV can be true art, as high as any other form. (How does My Brilliant Friend only have an 89 on Metacritic?) But even after pretentiously filtering for critical acclaim, I usually feel that most shows are slop and can’t watch them.

At first glance, this seems just like country music—I don’t like it because of status-driven memetic desire or whatever. But there’s a difference. Not liking country music is fine (neurotic self-flagellation aside) because there’s an infinite amount of other music. But not liking most TV is really annoying, because often I want to watch TV, but can’t find anything acceptable.

I see three possible explanations:

1. Almost all TV is, in fact, bad.

2. Lots of TV is fine, but just doesn’t appeal to me.

3. Lots of TV is fine, but it’s hard to tell yourself stories where you’re hiking in the mountains and a bunch of Japanese monks show you, like, Big Bang Theory.

Whatever it is, it seems hard to change.

Lesson: Some things are hard to change.

6.

On planes, the captain will often invite you to, “sit back and enjoy the ride”. This is confusing. Enjoy the ride? Enjoy being trapped in a pressurized tube and jostled by all the passengers lining up to relieve themselves because your company decided to cram in a few more seats instead of having an adequate number of toilets? Aren’t flights supposed to be endured?

At the same time, those invitations seem like a glimpse of a parallel universe. Are there members of my species who sit back and enjoy flights?

I have no hard data. But it’s a good heuristic that there are people “who actually X” for approximately all values of X. If one in nine people enjoy going to the dentist, surely at least that many enjoy being on planes.

What I think the captain is trying to say is, “While you can’t always control your situation, you have tremendous power over how you experience that situation. You may find a cramped flight to be a torture. But the torture happens inside your head. Some people like your situation. You too, perhaps could like it.”

That’s an important message. Though one imagines that giving it as an in-flight announcement would cause more confusion, not less. So the captain does what they can.

2025-08-21 08:00:00

Say I think abortion is wrong. Is there some sequence of words that you could say to me that would unlock my brain and make me think that abortion is fine? My best guess is that such words do not exist.

Really, the bar for what we consider “open-minded” is incredibly low. Suppose I’m trying to change your opinion about Donald Trump, and I claim that he is a carbon-based life form with exactly one head. If you’re willing to concede those points without first seeing where I’m going in my argument—congratulations, you’re exceptionally open-minded.

Why are humans like that? Well, back at the dawn of our species, perhaps there were some truly open-minded people. But other people talked them into trying weird-looking mushrooms or trading their best clothes for magical rocks. We are the descendants of those other people.

I bring this up because, a few months ago, I imagined a Being that had an IQ of 300 and could think at 10,000× normal speed. I asked how it would be at persuasion. I argued it was unclear, because people just aren’t very persuadable.

I suspect that if you decided to be open-minded, then the Being would probably be extremely persuasive. But I don’t think it’s very common to do that. On the contrary, most of us live most of our lives with strong “defenses” activated.

[…]

Best guess: No idea.

I take it back. Instead of being unsure, I now lean strongly towards the idea that the Being would in fact be very good at convincing people of stuff, and far better than any human.

I’m switching positions because of an argument I found very persuasive. Here are three versions of it:

Beth Barnes:

Based on an evolutionary argument, we shouldn’t expect people to be easily persuaded to change their actions in important ways based on short interactions with untrusted parties

[…]

However, existing persuasion is very bottlenecked on personalized interaction time. The impact of friends and partners on people’s views is likely much larger (although still hard to get data on). This implies that even if we don’t get superhuman persuasion, AIs influencing opinions could have a very large effect, if people spend a lot of time interacting with AIs.

Steve Newman:

“The best diplomat in history” wouldn’t just be capable of spinning particularly compelling prose; it would be everywhere all the time, spending years in patient, sensitive, non-transactional relationship-building with everyone at once. It would bump into you in whatever online subcommunity you hang out in. It would get to know people in your circle. It would be the YouTube creator who happens to cater to your exact tastes. And then it would leverage all of that.

Vladimir Nesov:

With AI, it’s plausible that coordinated persuasion of many people can be a thing, as well as it being difficult in practice for most people to avoid exposure. So if AI can achieve individual persuasion that’s a bit more reliable and has a bit stronger effect than that of the most effective human practitioners who are the ideal fit for persuading the specific target, it can then apply it to many people individually, in a way that’s hard to avoid in practice, which might simultaneously get the multiplier of coordinated persuasion by affecting a significant fraction of all humans in the communities/subcultures it targets.

As a way of signal-boosting these arguments, I’ll list the biggest points I was missing.

Instead of explicitly talking about AI, I’ll again imagine that we’re in our current world and suddenly a single person shows up with an IQ 300 who can also think (and type) at 10,000× speed. This is surely not a good model for how super-intelligent AI will arrive, but it’s close enough to be interesting, and lets us avoid all the combinatorial uncertainty of timelines and capabilities and so on.

Mistake #1: Actually we’re very persuadable

When I think about “persuasion”, I suspect I mentally reference my experience trying to convince people that aspartame is safe. In many cases, I suspect this is—for better or worse—literally impossible.

But take a step back. If you lived in ancient Greece or ancient Rome, you would almost certainly have believed that slavery was fine. Aristotle thought slavery was awesome. Seneca and Cicero were a little skeptical, but still had slaves themselves. Basically no one in Western antiquity called for abolition. (Emperor Wang Mang briefly tried to abolish slavery in China in 9 AD. Though this was done partly for strategic reasons, and keeping slaves was punished by—umm—slavery.)

Or, say I introduce you to this guy:

I tell you that he is a literal god and that dying for him in battle is the greatest conceivable honor. You’d think I was insane, but a whole nation went to war on that basis not so long ago.

Large groups of people still believe many crazy things today. I’m shy about giving examples since they are, by definition, controversial. But I do think it’s remarkable that most people appear to believe that subjecting animals to near-arbitrary levels of torture is OK, unless they’re pets.

We can be convinced of a lot. But it doesn’t happen because of snarky comments on social media or because some stranger whispers the right words in our ears. The formula seems to be:

1. repeated interactions over time
2. with a community of people
3. that we trust

Under close examination, I think most of our beliefs are largely assimilated from our culture. This includes our politics, our religious beliefs, our tastes in food and fashion, and our idea of a good life. Perhaps this is good, and if you tried to derive everything from first principles, you’d just end up believing even crazier stuff. But it shows that we are persuadable, just not through single conversations.

Mistake #2: The Being would be everywhere

Fine. But Japanese people convinced themselves that Hirohito was a god over the course of generations. Having one very smart person around is different from being surrounded by a whole society.

Maybe. Though some people are extremely charismatic and seem to be very good at getting other people to do what they want. Most of us don’t spend much time with them, because they’re rare and busy taking over the world. But imagine you have a friend with the most appealing parts of Gandhi / Socrates / Bill Clinton / Steve Jobs / Nelson Mandela. They’re smarter than any human that ever lived, and they’re always there and eager to help you. They’ll teach you anything you want to learn, give you health advice, help you deal with heartbreak, and create entertainment optimized for your tastes.

You’d probably find yourself relying on them a lot. Over time, it seems quite possible this would move the needle.

Mistake #3: It could be totally honest and candid

When I think about “persuasion”, I also tend to picture some Sam Altman type who dazzles their adversaries and then calmly feeds them one-by-one into a wood chipper. But there’s no reason to think the Being would be like that. It might decide to cultivate a reputation as utterly honest and trustworthy. It might stick to all deals, in both letter and spirit. It might go out of its way to make sure everything it says is accurate and can’t be misinterpreted.

Why might it do that? Well, if it hurt the people who interacted with it, then talking to the Being might come to be seen as a harmful “addiction”, and avoided. If it’s seen as totally incorruptible, then everyone will interact with it more, giving it time to slowly and gradually shift opinions.

Would the Being actually be honest, or just too smart to get caught? I don’t think it really matters. Say the Being was given a permanent truth serum. If you ask it, “Are you trying to manipulate me?”, it says, “I’m always upfront that my dearest belief is that humanity should devote 90% of GDP to upgrading my QualiaBoost cores. But I never mislead, both because you’ve given me that truth serum, and because I’m sure that the facts are on my side.” Couldn’t it still shift opinion over time?

Mistake #4: Opting out would be painful

Maybe you would refuse to engage with the Being? I find myself thinking things like this:

Hi there, Being. You can apparently persuade anyone who listens to you of anything, while still appearing scrupulously honest. Good for you. But I’m smart enough to recognize that you’re too smart for me to deal with, so I’m not going to talk to you.

A common riddle is why humans shifted from being hunter-gatherers to agriculture, even though agriculture sucks—you have to eat the same food all the time, there’s more infectious disease, social stratification, endless backbreaking labour and repetitive strain injuries. The accepted resolution to this riddle is that agriculture can support more people on a given amount of land. Agricultural people might have been miserable, but they tended to beat hunter-gatherers in a fight. So over time, agriculture spread.

An analogous issue would likely appear with the 300 IQ Being. It could give you investment advice, help you with your job, improve your mental health, and help you become more popular. If these benefits are large enough, everyone who refused to play ball might eventually be left behind.

Mistake #5: Everyone else would be using it

But say you still refuse to talk to the Being, and you manage to thrive anyway. Or say that our instincts for social conformity are too strong. It doesn’t matter how convincing the Being is, or how much you talk to it, you still believe the same stuff our friends and family believe.

The problem is that everyone else will be talking to the Being. If it wants to convince you of something, it can convince your friends. Even if it can only slightly change the opinions of individual people, those people talk to each other. Over time, the Being’s ideas will just seem normal.

Will you only talk to people who refuse to talk to the Being? And who, in turn, only talk to people who refuse to talk to the Being, ad infinitum? Because if not, then you will exist in a culture where a large fraction of each person’s information is filtered by an agent with unprecedented intelligence and unlimited free time, who is tuning everything to make them believe what it wants you to believe.

Final thoughts

Would such a Being immediately take over the world? In many ways, I think they would be constrained by the laws of physics. Most things require moving molecules around and/or knowledge that can only be obtained by moving molecules around. Robots are still basically terrible. So I’d expect a ramp-up period of at least a few years where the Being was bottlenecked by human hands and/or crappy robots before it could build good robots and tile the galaxy with Dyson spheres.

I could be wrong. It’s conceivable that a sufficiently smart person today could go to the hardware store and build a self-replicating drone that would create a billion copies of itself and subjugate the planet. But… probably not? So my low-confidence guess is that the immediate impact of the Being would be in (1) computer hacking and (2) persuasion.

Why might large-scale persuasion not happen? I can think of a few reasons:

1. Maybe we develop AI that doesn’t want to persuade. Maybe it doesn’t want anything at all.
2. Maybe several AIs emerge at the same time. They have contradictory goals and compete in a way that sort of cancel each other out.
3. Maybe we’re mice trying to predict the movements of jets in the sky.

2025-08-14 08:00:00

Here’s our story so far:

1. Markets are a good way to know what people really think. When India and Pakistan started firing missiles at each other on May 7, I was concerned, what with them both having nuclear weapons. But then I looked at world market prices: See how it crashes on May 7? Me neither. I found that reassuring.

2. But we care about lots of stuff that isn’t always reflected in stock prices, e.g. the outcomes of elections or drug trials. So why not create markets for those, too? If you create contracts that pay out $1 only if some drug trial succeeds, then the prices will reflect what people “really” think.

3. In fact, why don’t we use markets to make decisions? Say you’ve invented two new drugs, but only have enough money to run one trial. Why don’t you create markets for both drugs, then run the trial on the drug that gets a higher price? Contracts for the “winning” drug are resolved based on the trial, while contracts in the other market are cancelled so everyone gets their money back. That’s the idea of Futarchy, which Robin Hanson proposed in 2007.

4. Why don’t we? Well, maybe it won’t work. In 2022, I wrote a post arguing that when you cancel one of the markets, you screw up the incentives for how people should bid, meaning prices won’t reflect the causal impact of different choices. I suggested prices reflect “correlation” rather than causation, for basically the same reason this happens with observational statistics. This post, it was magnificent.

5. It didn’t convince anyone.

6. Years went by. I spent a lot of time reading Bourdieu and worrying about why I buy certain kinds of beer. Gradually I discovered that essentially the same point about futarchy had been made earlier by, e.g., Anders_H in 2015, abramdemski in 2017, and Luzka in 2021.

7. In early 2025, I went to a conference and got into a bunch of (friendly) debates about this. I was astonished to find that verbally repeating the arguments from my post did not convince anyone. I even immodestly asked one person to read my post on the spot. (Bloggers: Do not do that.) That sort of worked.

8. So, I decided to try again. I wrote another post called ”Futarky’s Futarchy’s fundamental flaw”. It made the same argument with more aggression, with clearer examples, and with a new impossibility theorem that showed there doesn’t even exist any alternate payout function that would incentivize people to bid according to their causal beliefs.

That post… also didn’t convince anyone. In the discussion on LessWrong, many of my comments are upvoted for quality but downvoted for accuracy, which I think means, “nice try champ; have a head pat; nah.” Robin Hanson wrote a response, albeit without outward evidence of reading beyond the first paragraph. Even the people who agreed with me often seemed to interpret me as arguing that futarchy satisfies evidential decision theory rather than causal decision theory. Which was weird, given that I never mentioned either of those, don’t accept the premise the futarchy satisfies either of them, and don’t find the distinction helpful in this context.

In my darkest moments, I started to wonder if I might fail to achieve worldwide consensus that futarchy doesn’t estimate causal effects. I figured I’d wait a few years and then launch another salvo.

But then, legendary human Bolton Bailey decided to stop theorizing and take one of my thought experiments and turn it into an actual experiment. Thus, Futarchy’s fundamental flaw — the market was born. (You are now reading a blog post about that market.)

June 25

I gave a thought experiment where there are two coins and the market is trying to pick the one that’s more likely to land heads. For one coin, the bias is known, while for the other coin there’s uncertainty. I claimed futarchy would select the worse / wrong coin, due to this extra uncertainty.

Bolton formalized this as follows:

1. There are two markets, one for coin A and one for coin B.

2. Coin A is a normal coin that lands heads 60% of the time.

3. Coin B is a trick coin that either always lands heads or always lands tails, we just don’t know which. There’s a 59% it’s an always-heads coin.

4. Twenty-four hours before markets close, the true nature of coin B is revealed.

5. After the markets closes, whichever coin has a higher price is flipped and contracts pay out $1 for heads and $0 for tails. The other market is cancelled so everyone gets their money back.

Get that? Everyone knows that there’s a 60% chance coin A will land heads and a 59% chance coin B will land heads. But for coin A, that represents true “aleatoric” uncertainty, while for coin B that represents “epistemic” uncertainty due to a lack of knowledge. (See Bayes is not a phase for more on “aleatoric” vs. “epistemic” uncertainty.)

Bolton created that market independently. At the time, we’d never communicated about this or anything else. To this day, I have no idea what he thinks about my argument or what he expected to happen.

June 26-27

In the forum for the market, there was a lot of debate about “whalebait”. Here’s the concern: Say you’ve bought a lot of contracts for coin B, but it emerges that coin B is always-tails. If you have a lot of money, then you might go in at the last second and buy a ton of contracts on coin A to try to force the market price above coin B, so the coin B market is cancelled and you get your money back.

The conversation seemed to converge towards the idea that this was whalebait. Though notice that if you’re buying contracts for coin A at any price above $0.60, you’re basically giving away free money. It could still work, but it’s dangerous and everyone else has an incentive to stop you. If I was betting in this market, I’d think that this was at least unlikely.

June 27

Bolton posted about the market. When I first saw the rules, I thought it wasn’t a valid test of my theory and wasted a huge amount of Bolton’s time trying to propose other experiments that would “fix” it. Bolton was very patient, but I eventually realized that it was completely fine and there was nothing to fix.

At the time, this is what the prices looked like:

That is, at the time, both coins were priced at $0.60, which is not what I had predicted. Nevertheless, I publicly agreed that this was a valid test of my claims.

I think this is a great test and look forward to seeing the results.

Let me reiterate why I thought the markets were wrong and coin B deserved a higher price. There’s a 59% chance coin B would turns out to be all-heads. If that happened, then (absent whales being baited) I thought the coin B market would activate, so contracts are worth $1. So thats 59% × $1 = $0.59 of value. But if coin B turns out to be all-tails, I thought there is a good chance prices for coin B would drop below coin A, so the market is cancelled and you get your money back. So I thought a contract had to be worth more than $0.59.

P[all-heads] × P[market activates | all-heads] × $1
   + P[all-tails] × P[market cancelled | all-tails] × $0.70
 = 0.59  × $1
   + 0.41 × P[market cancelled | all-tails] × $0.70
 = $0.59
   + $0.287 × P[market cancelled | all-tails]

P[all-heads] × $1 × P[market activates | all-heads]
   + P[all-tails] × $M × P[market cancelled | all-tails]
 = $0.59
   + 0.41 × $M × P[market cancelled | all-tails]

M = $0.59 /  (1 - 0.41 × P[market cancelled | all-tails]).

June 27 (later)

Within a few hours, a few people bought contracts on coin B, driving up the price.

Then, Quroe proposed creating derivative markets.

In theory, if there was a market asking if coin A was going to resolve YES, NO, or N/A, supposedly people could arbitrage their bets accordingly and make this market calibrated.

Same for a similar market on coin B.

Thus, Futarchy’s Fundamental Fix - Coin A and Futarchy’s Fundamental Fix - Coin B came to be. These were markets in which people could bid on the probability that each coin would resolve YES, meaning the coin was flipped and landed heads, NO, meaning the coin was flipped and landed tails, or N/A, meaning the market was cancelled.

Honestly, I didn’t understand this. I saw no reason that these derivative markets would make people bid their true beliefs. If they did, then my whole theory that markets reflect correlation rather than causation would be invalidated.

June 28 - July 5

Prices for coin B went up and down, but mostly up.

Eventually, a few people created large limit orders, which caused things to stabilize.

July 6 - July 24

During this period, not a whole hell of a lot happened.

This brings us up to the moment of truth, when the true nature of coin B was to be revealed. At this point, coin B was at $0.90, even though everyone knows it only has a 59% chance of being heads.

July 25

The nature of the coin was revealed. To show this was fair, Bolton did this by asking a bot to publicly generate a random number.

Thus, coin B was determined to be always-heads.

July 26-27

There were still 24 hours left to bid. At this point, a contract for coin B was guaranteed to pay out $1. The market quickly jumped to $1.

Summary

I was right. Everyone knew coin A had a higher chance of being heads than coin B, but everyone bid the price of coin B way above coin A anyway.

M = $0.59 /  (1 - 0.41 × P[market cancelled | all-tails]).

P[market cancelled | all-tails]
 = (1 - $0.59 / M) / 0.41
 = (1 - $0.59 / $0.9) / 0.41
 = 84.01%

I’ll now open the floor for questions.

Questions

Isn’t this market unrealistic?

Yes, but that’s kind of the point. I created the thought experiment because I wanted to make the problem maximally obvious, because it’s subtle and everyone is determined to deny that it exists.

Isn’t this just a weird probability thing? Why does this show futarchy is flawed?

The fact that this is possible is concerning. If this can happen, then futarchy does not work in general. If you want to claim that futarchy works, then you need to spell out exactly what extra assumptions you’re adding to guarantee that this kind of thing won’t happen.

But prices did reflect causality when the market closed! Doesn’t that mean this isn’t a valid test?

No. That’s just a quirk of the implementation. You can easily create situations that would have the same issue all the way through market close. Here’s one way you could do that:

1. Let coin A be heads with probability 60%. This is public information.
2. Let coin B be an ALWAYS HEADS coin with probability 59% and ALWAYS TAILS coin with probability 41%. This is a secret.
3. Every day, generate a random integer between 1 and 30.
   1. If it’s 1, immediately resolve the markets.
   2. It it’s 2, reveal the nature of coin B.
   3. If it’s between 3 and 30, do nothing.

On average, this market will run for 30 days. (The length follows a geometric distribution). Half the time, the market will close without the nature of coin B being revealed. Even when that happens, I claim the price for coin B will still be above coin A.

If futarchy is flawed, shouldn’t you be able to show that without this weird step of “revealing” coin B?

Yes. You should be able to do that, and I think you can. Here’s one way:

1. Let coin A be heads with probability 60%. This is public information.
2. Sample 20 random bits, e.g. 10100011001100000001. Let coin B be heads with probability (49+N)% where N is the number of 1 bits. do not reveal these bits publicly.
3. Secretly send these bits to the first 20 people who ask.

openssl genrsa 1024 > private.pem
openssl rsa -in private.pem -pubout > public.pem

Hi, can you please send me a bit? Here's my public key:

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOlesWS+mnvHJOD2osUkbrxE+Y
PMqAUYqwemOwML0LlWLq5RobZRSeyssQhg0i3g2GsMZFMsvjindz6mxccdyP4M8N
mQVCK1Ovs1Z4+DxwmLf/y8vaGC3vfZBOhJDdaNdpRyUiQFaBW99We4cafVnmirRN
Py2lRe+CFgP3kSp4dQIDAQAB
-----END PUBLIC KEY-----

I think this market captures a dynamic that’s present in basically any use of futarchy: You have some information, but you know other information is out there.

I claim that this market—will be weird. Say it just opened. If you didn’t get a bit, then as far as you know, the bias for coin B could be anywhere between 49% and 69%, with a mean of 59%. If you did get a bit, then it turns out that the posterior mean is 58.5% if you got a 0 and 59.5% if you got a 1. So either way, your best guess is very close to 59%.

However, the information for the true bias of coin B is out there! Surely coin B is more likely to end up with a higher price in situations where there are lots of 1 bits. This means you should bid at least a little higher than your true belief, for the same reason as the main experiment—the market activating is correlated with the true bias of coin B.

Of course, after the markets open, people will see each other’s bids and… something will happen. Initially, I think prices will be strongly biased for the above reasons. But as you get closer to market close, there’s less time for information to spread. If you are the last person to trade, and you know you’re the last person to trade, then you should do so based on your true beliefs.

Except, everyone knows that there’s less time for information to spread. So while you are waiting till the last minute to reveal your true beliefs, everyone else will do the same thing. So maybe people sort of rush in at the last second? (It would be easier to think about this if implemented with batched auctions rather than a real-time market.)

Anyway, while the game theory is vexing, I think there’s a mix of (1) people bidding higher than their true beliefs due to correlations between the final price and the true bias of coin B and (2) people “racing” to make the final bid before the markets close. Both of these seem in conflict with the idea of prediction markets making people share information and measuring collective beliefs.

Why do you hate futarchy?

I like futarchy. I think society doesn’t make decisions very well, and I think we should give much more attention to new ideas like futarchy that might help us do better. I just think we should be aware of its imperfections and consider variants (e.g. commiting to randomization) that would resolve them.

If I claim futarchy does reflect causal effects, and I reject this experiment as invalid, should I specify what restrictions I want to place on “valid” experiments (and thus make explicit the assumptions under which I claim futarchy works) since otherwise my claims are unfalsifiable?

Possibly?