2025-04-17 23:11:00
[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
[PART V – DEPRESSION AND OTHER DIAGNOSES]
[PART VI – CONFLICT AND OSCILLATION]
[PART VII – NO REALLY, SERIOUSLY, WHAT IS GOING ON?]
[INTERLUDE – I LOVE YOU FOR PSYCHOLOGICAL REASONS]
[PART VIII – ARTIFICIAL INTELLIGENCE]
When people talk about the ethical treatment of animals, they tend to hash it out in terms of consciousness.
But figuring out whether animals have consciousness, and figuring out what consciousness even is, are philosophical problems so hard they may be impossible to solve.
There’s not much common ground. The main thing people are generally willing to agree on is that since they themselves are conscious, other humans are probably conscious too, since other humans behave more or less like they do and are built in more or less the same way.
So a better question might be whether or not animals feel specific emotions, especially fear and pain.
The cybernetic paradigm gives a pretty clear answer to this question: Anything that controls threat and danger has an error signal that is equivalent to fear. And anything that controls injury has an error signal that is equivalent to pain.
This allows us to say with some confidence that animals like cows and rats feel fear, pain, and many other sophisticated emotions.
There’s no reason to suspect that a cow or a rat’s subjective experience of fear is meaningfully different from a human’s. We can’t prove this, but we can appeal to the same intuition that tells you that since you are conscious, other humans are probably conscious as well.
You believe that other humans feel fear, and that their fear is as subjectively terrifying to them as your fear is to you, for a simple reason: you notice that another person’s external behavior is much the same as yours is when you feel afraid, and is happening under similar circumstances. Then, you make the reasonable assumption that since all humans are biologically similar to one another, their external behavior is likely caused by similar internal rules and structures. Since there’s no reason to suspect that basically the same behavior created by basically the same structures would be any different phenomenologically, you conclude that other humans probably have the same kind of subjective experience.
With a better model for the emotions, this same logic can extend to other animals. Assuming we are right that a cow also has a governor dedicated to keeping it safe, which generates an error signal of increasing strength as danger increases, which drives behavior much like the behavior we engage in when we are afraid, there is little reason to suspect that the cow’s subjective experience is meaningfully different from our own. At the very least, if you accept the conclusion for humans, it’s not clear why you would reject it for other animals.
This is a relatively easy conclusion to draw for other complex, social mammals. They almost certainly feel fear and pain, because we see the outward signs, and because the inside machinery is overall so similar. But it’s harder to tell as animals become less and less closely related to humans.
An animal that doesn’t bother to avoid danger or injury clearly isn’t controlling for them. But most animals do. So the question is whether these animals actually represent danger and injury within a control system, trying to minimize some error, or if they simply avoid danger and injury through stimulus-response.
Dogs probably feel fear, and even without dissecting their brains, we can reasonably assume that they use similar mechanisms as we do. They’re built on the same basic mammalian plan and inherit the same hardware. But what about squid, or clams? These animals probably avoid danger in some way, but it’s not clear that they use an approach at all like the one we do.
If an animal cybernetically controls for danger and injury, then they are producing an error signal. In this case, the argument from above applies — there’s no reason to suspect that a creature using the same algorithms to accomplish the same thing is having a notably different experience. Their error signal is probably perceived as an emotion similar to our emotions.
But if an animal’s reaction to danger is instead a programmed response to a set stimulus, then there is no control system, no feedback loop, and no error signal.
For example, we might encounter an arthropod that freezes when we walk nearby. At first this looks like a fear response. We imagine that the arthropod is terrified and trying to avoid being seen and eaten.
But through trial and error, we show that whenever a shadow passes over it, the arthropod always freezes for exactly 2.5 seconds. Let’s further say that the arthropod shows no other signs of danger avoidance. If you “threaten” it in other ways, put it in other apparently dangerous situations, it changes its behavior not at all. The only thing it responds to is a shadow suddenly passing overhead.
This suggests that, at least for the purposes of handling danger, this arthropod operates purely on stimulus-response. As a result, it probably does not feel anything like the human emotion of fear. Even if we allow that the arthropod is conscious in some sense, its conscious experience is probably very different from ours because it is based on a different kind of mechanism.
Here’s a similar example from Russel & Norvig’s Artificial Intelligence: A Modern Approach. We can’t confirm that what they describe is actually true of dung beetles — it may be apocryphal — but it’s a good illustration of the idea:
Consider the lowly dung beetle. After digging its nest and laying its eggs, it fetches a ball of dung from a nearby heap to plug the entrance. If the ball of dung is removed from its grasp en route, the beetle continues its task and pantomimes plugging the nest with the nonexistent dung ball, never noticing that it is missing. Evolution has built an assumption into the beetle’s behavior, and when it is violated, unsuccessful behavior results.
It’s hard to figure out whether an organism is controlling some variable, or whether it is running some kind of brute stimulus-response, especially if the stimulus-response routine is at all complicated. We may need to develop new experimental techniques to do this.
But every organism has to maintain homeostasis of some kind, and almost all multicellular organisms have a nervous system, which suggests they’re running some kind of feedback loop, which means some kind of error signal, which means some kind of emotion.
For now, we think this is a relatively strong argument that most other mammals experience fear and pain the same way that we do — at least as strong of an argument that other humans experience fear and pain the same way that you experience them.
Figuring out whether you are in danger requires much more of a brain than figuring out whether you have been cut or injured. So while most animals probably feel pain, some animals may not feel fear, especially those with simple nervous systems, those with very little ability to perceive their environment, and those who are immobile. There’s no value in being able to perceive danger if you can’t do anything about it.
[Coming Next Thursday: DYNAMIC METHODS]
2025-04-10 23:11:00
[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
[PART V – DEPRESSION AND OTHER DIAGNOSES]
[PART VI – CONFLICT AND OSCILLATION]
[PART VII – NO REALLY, SERIOUSLY, WHAT IS GOING ON?]
[INTERLUDE – I LOVE YOU FOR PSYCHOLOGICAL REASONS]
Some “artificial intelligence” is designed like a tool. You put in some text, it spits out an image. You give it a prompt, it keeps predicting the following tokens. End of story.
But other “artificial intelligence” is more like an organism. These agentic AI are designed to have goals, and to meet them.
Agentic AI is usually designed around a reward function, a description of things that are “rewarding” to the agent, in the somewhat circular sense that the agent is designed to maximize reward.
Reward-maximizing agents are inherently dangerous, for a couple of reasons that can be stated plainly.
First, Goodhart’s law (“When a measure becomes a target, it ceases to be a good measure.”) means that the goal we intend the agent to have will almost never be the goal it ends up aiming for.
For example, you might want a system that designs creatures that can move very fast. So you give the agent a reward function that rewards the design of creatures with high velocities. Unfortunately the agent responds with the strategy, “creatures grow really tall and generate high velocities by falling over”. This matches the goal as stated, but does not really give you what you want.
Even with very simple agents, this happens all the time. The agent does not have to be very “intelligent” in the normal sense to make this happen. It’s just in the nature of reward functions.
This is also called goal mis-specification. Whatever goal you think you have specified, you almost always specify something else by mistake. When the agent pursues its real goal, that may cause problems.
Second, complexity. Simple goals are hard enough. But anything with complex behavior will need to have a complex reward function. This makes it very difficult to know you’re pointing it in the right direction.
You might think you can train your agent to have complex goals. Let it try various things and say, “yes, more of that” and “no, less of that” until it has built up a reward function that tends to give the behavior we want. This might work in the training environment, but because the reward function has been inferred through training, we don’t know what that reward function really is. It might actually be maximizing something weird. And you might not learn what it’s really maximizing until it’s too late to stop it.
The third and most serious reason is that anything insatiable is dangerous. Something that always wants more, and will stop at nothing to get it, is a problem. For a reward-maximizing agent, no amount of reward can ever be enough. It will always try to drive reward to infinity.
This is part of why AI fears usually center around runaway maximizers. The silly but canonical example is an AI with a reward function with a soft spot for office supplies, so it converts all matter in the universe into paperclips.
The same basic idea applies to any reward maximizer. If the United States Postal Service made an AI to deliver packages, and designed it to get a reward every time a package was delivered, that AI would be incentivized to find a way to deliver as many packages as possible, for the minimum possible descriptions of “deliver” and “packages”, by any means necessary. This would probably lead to the destruction of all humans and soon all life on earth.
But there are reasons to be optimistic.
For starters, the main reason to expect that artificial intelligence is possible is the existence of natural intelligence. If you can build a human-level intelligence out of carbon, it seems reasonably likely that you could build something similar out of silicon.
But humans and all other biological intelligences are cybernetic minimizers, not reward maximizers. We track multiple error signals and try to reduce them to zero. If all our errors are at zero — if you’re on the beach in Tahiti, a drink in your hand, air and water both the perfect temperature — we are mostly comfortable to lounge around on our chaise.
As a result, it’s not actually clear if it’s possible to build a maximizing intelligence. The only intelligences that exist are minimizing. There has never been a truly intelligent reward maximizer (if there had, we would likely all be dead), so there is no proof of concept. The main reason to suspect AI is possible is that natural intelligence already exists — us.
That said, it may still be possible to build a maximizing agent. If we do, there’s reason to suspect it will be very different from us, since it will be built on different principles. And there’s reason to suspect it would be very dangerous.
A reward maximizer doesn’t need to be intelligent to be dangerous. Maximizing pseudointelligences could still be very dangerous. Viruses are not very smart, but they can still kill you and your whole family.
We should avoid building things with reward functions, since they’re inherently dangerous. Instead, if you must build artificial intelligences, make them cybernetic, like us.
This is preferable because cybernetic minimizers are relatively safe. Once they get to their equivalent of “lying on the beach in Tahiti with a piña colada in hand” they won’t take any actions.
If the United States Postal Service designed an AI so that it minimizes the delivery time of packages instead of being rewarded for each successful delivery, it might still stage a coup to prevent any new packages from being sent. But once no more packages are being sent, it should be perfectly content to go to sleep. It will not try to consume the universe — it just wants to keep a number near zero.
Reward maximizers are always unstable. Even very simple reinforcement learning agents show very crazy specification behaviors. But control systems can be made very stable. They have their own problems, but we use them all the time, in thermostats, cruise control, satellites, and nuclear engineering. These systems work just fine. When control systems do fail, they usually fail by overreacting, underreacting, oscillating wildly, freaking out in an endless loop, giving up and doing nothing, and/or exploding. This is bad for the system, and bad when the system controls something important, like a nuclear power plant. But it doesn’t destroy the universe.
At the most basic level, these two approaches are the two kinds of feedback loops. Cybernetic agents run on negative feedback loops, which generally go towards zero and are relatively safe. Reward-maximizing agents are an example of positive feedback loops, which given enough resources will always go towards infinity, so they’re almost always dangerous. Remember, a nuclear explosion is a classic positive feedback loop. The only reason nuclear explosions stop is that they run out of fuel.
A possible rebuttal to this argument is that even if an agent is happy to move towards a resting state and then do nothing, it will still be interested in gaining as much power as possible so it can achieve its goals in the future. The technical term here is instrumental convergence.
Here we can appeal to observations of the cybernetic intelligences all around us. Humans, dogs, deer, mice, squid, etc. do not empirically seem to spend every second of their downtime maniacally working to gather more resources and power. Even with our unique human ability to plan far ahead, we often seem to use our free time to watch TV.
This suggests that instrumental convergence is not a problem for cybernetic agents. When more power is needed to correct its error, maybe a governor will vote for actions that increase the agent’s power. But if it already has enough power to correct its error, the governor will prefer to correct its error straightaway. This suggests we pursue instrumental goals like “gather more power and resources” mainly when we don’t have the capabilities we need to effectively cover all our drives.
Finally, a few things we should mention.
Cybernetic intelligences can’t become paperclip maximizers, but they can still be dangerous for other reasons. Hitler was not a paperclip maximizer, but even as a mere cybernetic organism, he was still pretty dangerous. So be careful with AI nonetheless.
Cybernetically controlling one or more values is good, natural even. But controlling derivatives (the rate of change in some value) is bad! You will end up with runaway growth that looks almost the same as a reward maximizer. If you design your cybernetic greenhouse AI to control the rate of growth of plants in your greenhouse (twice as many plants every week!), very soon it will need to control the whole universe to give you the number of plants you implicitly requested.
Controlling second derivatives (rate of change of the rate of change) is VERY BAD. Controlling third and further derivatives is right out.
[Next: ANIMAL WELFARE]
2025-04-03 23:11:00
[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
[PART V – DEPRESSION AND OTHER DIAGNOSES]
[PART VI – CONFLICT AND OSCILLATION]
[PART VII – NO REALLY, SERIOUSLY, WHAT IS GOING ON?]
In 1796, Astronomer Royal Nevil Maskelyne noticed that his layabout assistant, David Kinnebrook, was getting measurements of celestial events that were a whole half-second different from his own. Maskelyne told Kinnebrook he had better shape up, but this didn’t help — Kinnebrook’s errors increased to around 8/10 of a second, so Maskelyne fired him.
Later astronomers looked into this more closely and discovered that there was actually nothing wrong with poor Kinnebrook. The issue is that people all have slightly different reaction times. When a star passes in front of a wire, it takes you some very small amount of time to react and record your observation. So when different people look at the same celestial event, they get slightly different results. You might even say that the fault is not in our stars, but in ourselves.
More importantly, these differences aren’t random. Kinnebrook’s measurements were always slightly later than Maskelyne’s, and always later by about the same amount. This is a consistent and personal bias, so they came up with the term “personal equation” to describe these differences.
As astronomers learned to measure these personal equations with more and more accuracy, they found that people can’t distinguish anything less than 0.10 seconds, which eventually spiraled into what has been called the tenth-of-a-second crisis. Further investigation of this effect, combined with similar research in physiology and statistics, eventually led to the invention of a new field: psychology.
The personal equation is frequently mentioned in psychology from the 19th and early 20th century. Edwin G. Boring devoted an entire chapter of his 1929 book to the personal equation, the story of which, he said, “every psychologist knows”. Even as late as 1961, he was writing about “the sacred 0.10 sec.”
Or take a look at this passage from the introduction of Hugo Münsterberg’s 1908 book, Essays on Psychology and Crime. He says:
Experimental psychology did not even start with experiments of its own; it rather took its problems at first from the neighbouring sciences. There was the physiologist or the physician who made careful experiments on the functions of the eye and the ear and the skin and the muscles, and who got in this way somewhat as by-products interesting experimental results on seeing and hearing and touching and acting; and yet all these by-products evidently had psychological importance. Or there was the physicist who had to make experiments to find out how far our human senses can furnish us an exact knowledge of the outer world; and again his results could not but be of importance for the psychology of perception. Or there was perhaps the astronomer who was bothered with his “personal equation,” as he was alarmed to find that it took different astronomers different times to register the passing of a star. The astronomers had, therefore, in the interest of their calculations, to make experiments to find out with what rapidity an impression is noticed and reacted upon. But this again was an experimental result which evidently concerned, first of all, the student of mental life.
All three of these examples, including the personal equation, are about perception — physiologists studying the sense organs, and physicists studying the limits of those senses. Given this foundation, it will come as no surprise to hear that for most of its history, psychology’s main focus has been perception. Even in the early days of psychology, perception was baked in.
This was most obvious in the earliest forms of psychology. In 1898, E. Bradford Titchener wrote a paper describing the layout of his psychology lab at Cornell. This lab not only had a room for optics, but separate rooms also for acoustics, haptics, and one “taste and smell room”. Olfactometry does not come up much in modern psychology, but the Cornell psychologists of the 1890s had an entire room dedicated to it:
Room 1, the ‘optics room,’ is a large room, lighted from three sides, with walls and ceiling painted a cream. Room 2, intended for the private room of the laboratory assistants, now serves the purposes to which room 12 will ultimately be put. Room 3 is the ‘acoustics,’ room 4 the ‘haptics room.’ Room 5 is a dark room employed for drill-work, demonstration and photography. Room 6 is the ‘work,’ and room 7 the ‘lecture-room’. Room 8 is the director’s private room ; room 9 the ‘reaction,’ and room 10 the ‘taste and smell room’. Room 11, which faces north, will be fitted up as a research dark room; room 12 will be furnished with the instruments used in the investigation of the physiological processes underlying affective consciousness, —pulse, respiration, volume and muscular tone.
Even today, the closest thing to a true law of psychology is the Weber-Fechner law, about the minimum possible change needed to be able to distinguish between two similar stimuli; in other words, about perception. And the most impressive artifacts of psychology are still visual illusions like this one:
During the cognitive revolution, a lot of sacred cows were tipped, but not perception. Instead, perception was reaffirmed as the absolute main topic of psychological study. Ulric Neisser’s 1967 book Cognitive Psychology consists of:
That’s it!
In a footnote, Neisser apologizes… for not covering the other senses. “Sense modalities other than vision and hearing are largely ignored in this book,” he says, “because so little is known about the cognitive processing involved.” But he doesn’t apologize for skipping over nearly every other aspect of psychology, which seems like a stunning omission.
At least Neisser is self-aware about this. He makes it very clear that he knows many different directions psychology could take, and that he is picking this one, cognition, over all the others. It’s just that he is fully committed to the promise of the cognitive approach, and that means he’s fully committed to the idea that perception should hold center stage — not just top billing, but to the point of excluding other parts of psychology.
Even given psychology’s previous hundred years of focus on perception, this was a pretty radical position. Titchener would probably be scandalized that Neisser didn’t include a chapter on taste and smell.
But the most surprising omission of all might be “individual differences”, the psychologist’s fancy term for personality. Because once upon a time, personality was almost as central to psychology as perception was.
Recall that the personal equation, one of the problems that kicked off psychology in the first place, was itself an idea about individual differences — every individual had a personal difference in their reaction times when looking at the celestial spheres. You can’t have a personal equation without individual differences, so as much as the personal equation came with an interest in the laws of perception, it also came with a committed interest in personality.
Almost as old as the personal equation is the idea of mental tests. Most of the credit and the blame for these goes to Sir Francis Galton. After hearing about the theory of evolution from his cousin, Charles Darwin, Galton started wondering if mental traits ran in families. He became obsessed with measuring differences in people’s minds and bodies, and these ideas directly led to the invention of IQ tests (and also eugenics). These unpleasant grandchildren aside, for a long time mental tests were a really central part of psychology. Until one day they weren’t.
Neisser does offer a defense of his position in the last chapter of his book. We think the final paragraph is especially interesting, where he says:
It is no accident that the cognitive approach gives us no way to know what the subject will think of next. We cannot possibly know this, unless we have a detailed understanding of what he is trying to do, and why. For this reason, a really satisfactory theory of the higher mental processes can only come into being when we also have theories of motivation, personality, and social interaction. The study of cognition is only one fraction of psychology, and it cannot stand alone.
Norbert Wiener coined the term “cybernetics” in the summer of 1947, but for the full story, we have to go much further back.
Wiener places the earliest origins of these ideas with the 17th century German polymath Gottfried Wilhelm Leibniz. “If I were to choose a patron saint for cybernetics out of the history of science,” Wiener wrote in the introduction to his book, “I should have to choose Leibniz. The philosophy of Leibniz centers about two closely related concepts—that of a universal symbolism and that of a calculus of reasoning. From these are descended the mathematical notation and the symbolic logic of the present day.”
Simple control systems have been in use for more than two thousand years, but things really picked up when Leibniz’ early math tutor, Christiaan Huygens, derived the laws for centrifugal force and invented an early centrifugal governor.
Over the centuries people slowly made improvements to Huygens’ design, most notably James Watt, who added one to his steam engine. These systems caught the attention of James Clerk Maxwell, who in 1868 wrote a paper titled “On Governors”, where he explained instabilities exhibited by the flyball governor by modeling it as a control system.
When explaining why he chose to call his new field “cybernetics”, Wiener wrote, “in choosing this term, we wish to recognize that the first significant paper on feedback mechanisms is an article on governors, which was published by Clerk Maxwell in 1868, and that governor is derived from a Latin corruption of κυβερνήτης.”
Using this background, Norbert Wiener and Arturo Rosenblueth sat down and made the field explicit in the 1940s, and gave it a name. Then in 1948 Wiener published his book Cybernetics: Or Control and Communication in the Animal and the Machine, and the field went public.
The new field went in a number of directions, many of them unproductive, but the one most important to us today is the direction taken up by a certain William T. Powers.
Psychology and cybernetics were making eyes at each other across the room from the very start. “The need of including psychologists had indeed been obvious from the beginning,” wrote Wiener. “He who studies the nervous system cannot forget the mind, and he who studies the mind cannot forget the nervous system.” And the psychologists returned Wiener’s affections: Kurt Lewin, one of the founders of modern social psychology, attended the first “Macy Conference” on cybernetics, all the way back in 1946, before it was even called cybernetics, and Weiner mentions Lewin (and some other psychologists) by name in his book.
But in the 1940s and 1950s, psychologists felt they were doing pretty all right. Lewin and the social psychologists were a relatively small slice of psychology, the minority faction by far, and their interest didn’t carry much weight. Cybernetics might be nice to flirt with at the party, but there was no real chance of inviting it home.
But fast forward to the 1970s, and psychology was in crisis. For a long time psychology had been ruled by behaviorism, a paradigm which took the stance that while behavior could be studied scientifically, the idea of studying thoughts or mental states was wooly nonsense. Mental states like thoughts and feelings were certainly unworthy of study, and possibly didn’t exist.
Behaviorists also thought that animals are born without anything at all in their brains — that the mind at birth is a blank slate, and that everything an animal learns to do comes from pure stimulus-response learning built up over time.
Behaviorism seemed like a sure bet in the 1920s, but those assumptions were looking more and more shaky every day. People had discovered that animals did seem to have inborn tendencies to associate some things with other things. They learned that you could make reasonable inferences about mental states. And the invention of the digital computer made the study of mental states seem much more scientific. The old king was dying, and no one could agree who was rightful heir to the throne.
The son of a “well-known cement scientist”, William T. Powers wasn’t even a psychologist. His training was in physics and astronomy. But while working at a cancer research hospital, and later while designing astronomy equipment, Powers started pulling different threads together and eventually came up with his own very electrical-engineering-inspired paradigm for psychology, which he called Perceptual Control Theory.
In 1973 Powers published both a book and an article in Science about his ideas. While Powers was obviously an outsider, psychologists took this work seriously. Even in the 1970s, fringe ideas didn’t get published in a journal as big as Science — Powers and his arguments were mainstream, at least for a little while.
Psychologists really thought that cybernetics might be one of the ways forward. Stanley Milgram, who did the famous experiments on obedience to authority — the ones where participants thought they might be delivering lethal electric shocks to a man with a heart condition, but mostly kept increasing the voltage when politely asked to continue — even includes a brief section on cybernetics in his 1974 book about those studies. “While these somewhat general [cybernetic] principles may seem far removed from the behavior of participants in the experiment,” he says, “I am convinced that they are very much at the root of the behavior in question.”
And Thomas Kuhn himself, the greatest authority on crisis and revolution in science (he did write the book on it), wrote a glowing review of Powers’ book, saying:
Powers’ manuscript, “Behavior: The Control of Perception”, is among the most exciting I have read in some time. The problems are of vast importance, and not only to psychologists; the achieved synthesis is thoroughly original and the presentation is often convincing and almost invariably suggestive. I shall be watching with interest what happens in the directions in which Powers points.
But there were a few problems.
The first is that Powers’ work, especially his 1973 Science article, doesn’t exactly make the case that cybernetics is a good way of thinking about psychology. It’s more of an argument that cybernetics is better than behaviorism. The paper is filled with beautiful and specific arguments, but they’re arguments against the behaviorist paradigm. The article is even titled, Feedback: Beyond Behaviorism.
You can see why Powers would frame things this way. As far as he could tell, behaviorism was the system to beat, and his arguments against behaviorism really are compelling.
Unfortunately, by 1973 behaviorism was already on its way out. Six years before, in 1967, Ulric Neisser wrote:
A generation ago, a book like this one would have needed at least a chapter of self-defense against the behaviorist position. Today, happily, the climate of opinion has changed, and little or no defense is necessary. Indeed, stimulus-response theorists themselves are inventing hypothetical mechanisms with vigor and enthusiasm and only faint twinges of conscience.
Powers’ work arrived early enough that psychologists were still interested in what he had to say. They still felt that their field was in crisis, they were still looking around for new tools and new perspectives. They were still willing to publish his paper, and everybody read his book.
But it came late enough in the crisis that there was strong competition. New schools of thought were already mustering their forces, already had serious claims to the throne. People were already picking sides. And most people were already picking cognitive psychology.
It’s not entirely clear exactly why cognitive psychology won, but there are a few things that made its claim especially strong. For one, some of the strongest evidence against behaviorism came from an information theory angle, and this looked really good for cognitive psychology, which proposed that we think of the mind in terms of how it handles and transforms information.
Maybe most importantly, the metaphor of the digital computer promised to provide the objectivity that behaviorism was never able to deliver. Whatever else might be going on in human minds, computers definitely exist, they can add and subtract, and that looks a lot like thinking! Cognitive psychology eventually won out.
Another problem is that cybernetics is what they call “dynamic”. This is a distinction people don’t usually make any more, but Ulric Neisser gives this definition:
Dynamic psychology, which begins with motives rather than with sensory input, is a case in point. Instead of asking how a man’s actions and experiences result from what he saw, remembered, or believed, the dynamic psychologist asks how they follow from the subject’s goals, needs, or instincts.
Cybernetics makes for a dynamic school of psychology because, however you slice it, control systems are always about getting signals back in alignment, so they’re always about goals (what’s the target value) and needs (which signals are controlled). If you think about psychology in terms of control systems, whatever you come up with is going to be dynamic.
Dynamic theories were very popular in the first half of the 20th century, but they ended up falling out of favor in the back half. Again, we’re not entirely sure why this happened the way it did, but we can provide some reasonable speculation.
The most famous dynamic school of psychology is Freudian psychodynamic therapy. If you’ve ever wondered, this is why it has “dynamic” in the name, because it’s a paradigm that focuses on how people are motivated by drives and/or needs. Freudians originally saw all behavior as motivated by libido, the sex or pleasure principle. But later on they added a second drive or set of drives called mortido, the drive for death.
Most schools of psychology are more dynamic than they would like to admit — even behaviorism. Sure, behaviorists had an extremely reductive understanding of drives (mostly “reward” and “punishment”), but at their heart they were a dynamic school too. Reward and punishment are a theory of motivation; it’s only one drive, but it’s right there, and central to the paradigm. And behaviorists did sometimes admit other drives, most blatantly in Clark Hull’s drive reduction theory, which allowed for drives like thirst and hunger.
Behaviorists have to accept some kind of dynamics because they assume that reward and punishment are behind all behavior, except perhaps the most instinctual. Even if they didn’t tend to think of this as a drive, it’s clearly the motive force that behaviorists used to explain all behavior — organisms are maximizing reward and minimizing punishment.
(In a totally different kind of problem, dynamic psychology is always a bit risky because it’s inherently annoying to speculate about someone’s motives. The Freudians really ran afoul of this one.)
The point is, by the time William Powers was arguing for cybernetics, dynamic psychology was on the downswing. Its reputation was tainted by the Freudians, and maybe it also smelled a bit too much like the behaviorists, with their focus on reward and punishment. This might be another reason why cybernetics was passed over in favor of something that seemed a bit more fresh and promising.
It’s not like people hated cybernetics, but it’s interesting to see how conscious the decision was. Near the end of his book, Neisser says:
An analogy to the “executive routines” of computer programs shows that an agent need not be a homunculus. However, it is clear that motivation enters at several points in these processes to determine their outcome. Thus, an integration of cognitive and dynamic psychology is necessary to the understanding of the higher mental processes.
But the rest of cognitive psychology did not inherit this understanding, and this integration was never carried out; as far as we know it was never even attempted. In any case, dynamic paradigms were out.
Other schools like social psychology, neuroscience, and psychiatry kept going with what they were doing, since they were not seen to be in crisis, and they gained more sway as behaviorism fell apart. Or perhaps a better read on things is that the ground previously held by behaviorism was partitioned, with cognitive psychology gaining the most, social psychologists also receiving a large chunk, clinical psychologists gaining some, etc. Cybernetics received none, and fell into obscurity.
Or possibly it was diluted into a vague branch of the humanities. The full title of Wiener’s book was Cybernetics: Or Control and Communication in the Animal and the Machine. Some anthropologists may have taken the “communication” part too seriously — they started using the term more and more vaguely, until eventually they used it to refer to anything at all involving communication, which is probably where the internet got the vague epithet of “cyberspace”.
Today, psychology generally acts as though drives do not exist. If you look in your textbook you will usually see a brief mention of drives, but they’re not a priority. For example, one psychology textbook says,
All organisms are born with some motivations and acquire others through experience, but calling these motivations “instincts” describes them without explaining how they operate. Drive-reduction theory explains how they operate by suggesting that disequilibrium of the body produces drives that organisms are motivated to reduce.
But the very next sentence concludes:
Neither instinct nor drive are widely used concepts in modern psychology, but both have something useful to teach us.
The subtext in today’s psychology is that there is only one real drive, with two poles, reward and punishment. When psychologists explicitly name this assumption, they call it “the hedonic principle”. Despite any lip service paid to other drives, simple hedonism is the theory of motivation that psychologists actually use.
A cruel irony is that modern cognitive psychology, as far as we can tell, inherited this theory of motivation directly from behaviorism. This is just good ‘ol reward and punishment. Even though they held the cognitive revolution to throw out behaviorism and replace it with something new, they weren’t able to disinherit themselves of some of the sneakier assumptions.
The other funny thing is that when outsiders come up with their own version of psychology, they usually end up including drives. Our favorite example continues to be The Sims. To get somewhat realistic behavior out of their Sims, Maxis had to give them several different drives, so they did. Even psychologists can’t help inventing new drives by accident. If you hang around psychology long enough, you’ll run into various “need for whatever” scales, like the famous need for cognition.
This reminds us a lot of what happened in alchemy. Alchemists were supposed to believe in air, fire, water, and earth, and explain the world in terms of those four elements. But belief in four elements was impossible, and the alchemists told on themselves, because they couldn’t stop inventing new ones. In the preface to his book on chemistry, Lavosier says (emphasis added):
The notion of four elements, which, by the variety of their proportions, compose all the known substances in nature, is a mere hypothesis, assumed long before the first principles of experimental philosophy or of chemistry had any existence. In those days, without possessing facts, they framed systems; while we, who have collected facts, seem determined to reject them, when they do not agree with our prejudices. The authority of these fathers of human philosophy still carry great weight, and there is reason to fear that it will even bear hard upon generations yet to come.
It is very remarkable, that, notwithstanding of the number of philosophical chemists who have supported the doctrine of the four elements, there is not one who has not been led by the evidence of facts to admit a greater number of elements into their theory. The first chemists that wrote after the revival of letters, considered sulphur and salt as elementary substances entering into the composition of a great number of substances; hence, instead of four, they admitted the existence of six elements. Beccher assumes the existence of three kinds of earth, from the combination of which, in different proportions, he supposed all the varieties of metallic substances to be produced.
So likewise, notwithstanding the number of psychologists who have supported the doctrine of reward and punishment, there is not one who has not been led by the evidence of facts to admit a greater number of drives into their theory.
Let’s not beat around the bush. This series is an attempt to introduce a new cybernetic paradigm for psychology, and cause a scientific revolution, just like the ones they had in chemistry and astronomy and physics, just like Thomas Kuhn talked about.
We think that cybernetics will allow an angle of attack on many problems in psychology, and we’re going to do our best to make that case. For example, one of psychology’s biggest hidden commitments is that for most of its history, it has focused on perception, sometimes to the exclusion of everything else. But perception may not be the right way to approach the study of the mind. Problems that remain unsolved for a long time should always be suspected as questions asked in the wrong way.
Cybernetics benefits because it doesn’t have such a strong commitment to perception — instead, it’s dynamic. The fact that dynamics is so different from the perception-based approach that has dominated psychology for most of the 200 years it’s been around seems like reason for optimism.
A lot of what we have to say about cybernetics comes from cyberneticists, especially Wiener and Powers. Some of what we say about psychological drives comes from earlier drive theorists. And some of what we think of as original will probably in fact turn out to be reinventing the wheel. Finally, everything we say comes from treating previous psychology as some mix of thesis, antithesis, and synthesis.
To most psychologists, asking “what emotions do rats have?” would be rather vague. But to a cybernetic psychologist, it makes perfect sense. It also makes sense to ask, “what emotions do rats and humans have in common?” From a cybernetic standpoint, there’s probably a precise answer to such questions.
Some of these questions may be disturbing in new and exciting ways. Are fish thirsty? Again, there may be a precise answer to this question.
There is something new in this work, but it’s also contiguous. We don’t want this to come across as though we’re saying this is unprecedented; this is all firmly grounded in historical traditions, it’s all inspired by things that have come before.
[Next: ARTIFICIAL INTELLIGENCE]
2025-03-27 23:11:00
[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
[PART V – DEPRESSION AND OTHER DIAGNOSES]
[PART VI – CONFLICT AND OSCILLATION]
A crucial mood, the “No Really, Seriously, What Is Going On?” mood, the mood of true curiosity. I don’t know if they’ve found the truth, but I can tell they are earnestly looking.
— Emmett Shear
Consider: You are in a terrible situation and your fear governor expects you to die in a horrifying way, -1000 points. There is nothing you can do, so you take the action “freeze and cower”. But instead, you survive and it’s not nearly as bad as expected, only -100 points.
Sounds great, right? Wrong! The sad truth is that as far as the fear governor can tell, “freeze and cower” has just given you +900 points! It is the best action you have ever taken, possibly the best action of all time. It doesn’t matter that the prediction was wrong, that things went badly but not as badly as the worst possible outcome, that’s not how the system works.
Your fear governor has learned that “freeze and cower” is the best action it can take. So whenever you feel fear in the future, it votes that you should freeze and cower, and you usually do. Since (in the modern world) most things you are afraid of do eventually go away if you freeze and cower for long enough, your fear governor continues to believe that “freeze and cower” is a good action, and keeps voting “freeze and cower” up and down the ballot.
This sounds a lot like PTSD. Take for example this description from Scott Alexander’s review of Van der Kolk’s book, The Body Keeps The Score:
Van der Kolk thinks that traumas are much more likely to cause PTSD when the victim is somehow unable to respond to them. Enemy soldiers shooting at you and you are running away = less likelihood of trauma. Enemy soldiers shooting at you and you are hiding motionless behind a tree = more likelihood of trauma. Speculatively, your body feels like its going into trauma mode hasn’t gotten you to take the right actions, and so the trauma mode cannot end.
In the real world PTSD is often more complex, but that added complexity usually makes things worse, not better. For example, a civilian is mostly not afraid of loud noises like fireworks, and the fear governor does not even pay attention. But if you go to war and you learn that loud noises are dangerous, they become something that your fear governor considers its rightful business. You also learn that “freeze and cower” is a great action that always seems to work. So you return to civilian life ready to freeze and cower not only when you encounter the things that have always scared you, but also in response to things that you previously would not have found frightening.
With enough exposure, this response might go away. But not all exposure will be equal. If we play a loud sound, and you freeze and cower, and nothing bad happens, this actually reinforces the response. It’s just more evidence that “freeze and cower” is a good action that is reliably followed by a safe outcome.
Instead, you need to play a loud noise, but quiet enough that you can force yourself to not freeze and cower. Over time, this should teach you that other actions, like “stand there calmly”, are just as good. But it may take a long time for these estimates to balance out. From our example above, “freeze and cower” is valued at +900 safety points. And it only got there because your fear governor expected to die, and then didn’t. So a program of exposure where you get exposed to a little danger at a time will take a long time to catch up.
The really weird implication here is that a faster form of exposure therapy might be to get you to freeze and cower, and then actually hurt you! This would hopefully teach your fear and pain governors that “freeze and cower” has negative value, and it should get you to that conclusion faster. That said, the ethics of actually harming your patients seems questionable.
We can also think a little bit about individual differences. Some people probably have faster or slower learning rates, and this makes them more or less easy to traumatize from a single event.
Remember this equation from earlier? There’s that term α, which is the learning rate. If the learning rate is high, then your opinion of an action will change a lot every time you try that action. So a person with a high learning rate is more likely to become traumatized from a single experience. Just one very good or bad experience with an action would hugely change their estimate of how reasonable it is to take that action. But someone with a low learning rate will only ever change their estimate of an action by a small amount, so would be less likely to get traumatized unless the same outcome happened to them over and over.
Some people have more or less previous experience with the frightening stimuli. If you have a lot of previous experiences with loud noises, then artillery explosions won’t be so unusual, and you probably won’t learn to treat all loud noises as dangerous. But if you grew up in the silence of rural North Dakota, the experience of an artillery barrage might teach a lesson that gets passed on to all loud noises, for lack of experience.
Consider: Drugs often interfere with specific error signals.
Take alcohol, which is especially famous for the way it reduces feelings of shame. When you get drunk, you feel much less shame than you did before, and you might end up doing some very shameless things. It also seems to suppress emotions like fear and thirst, and maybe increases emotions like hunger and exhaustion, which is why you do something dangerous, eat a whole basket of fries, and then pass out.
Imagine a guy named Chris who has just turned 21. He goes out drinking for the first time and he is feeling pretty good. He isn’t feeling ashamed of anything at all, so the alcohol can’t reduce his feelings of shame, because shame is at zero.
A few weeks later, someone finds out a dark secret about Chris’s past and confronts him with it. Now Chris is feeling deeply ashamed. He has plans to go out drinking with friends though, so he still goes to join them. When he lifts the bottle of beer to his lips, he has a very large shame error signal. And when he drinks the alcohol, that shame error is reduced. By his fifth beer, he doesn’t feel ashamed at all.
What does his shame governor learn from this experience? It learns that drinking beer is a great way to reduce its error. This is great news for the shame governor, because there are not many ways to quickly reduce shame. It resolves to vote for Chris to drink beer in the future whenever he feels ashamed.
We see similar patterns across the world of drugs. Caffeine and amphetamines reduce fatigue and hunger. Opiates reduce pain. And MDMA seems to interact somehow with the social emotions — maybe it reduces shame while increasing other drives, ones that make people more social and touchy-feely.
With us so far? Let’s try a more complicated example.
Let’s say you are hanging around one day, feeling pretty ok. All your errors are close to zero, so you’re mostly in alignment. You’re a little chilly, but only a little: your cold governor has an error of 5 points.
You decide to take some amphetamines just for fun, like one does. You’ve never taken amphetamines before, but you feel fine and you have nothing better to do.
Let’s say that amphetamines have only one effect: they immediately change any error you have to zero. This lasts until the amphetamines wear off, at which point the error goes back to whatever it was before. So when you take the amphetamines, your cold error goes to zero for a while, and you no longer feel chilly.
Your cold governor picks up on this. Its job is to generate and keep track of that error, and keep track of ways to reduce that error towards zero. So your cold governor learns that “take amphetamines” reduces its error by about 5 points.
The amphetamines don’t actually change your body temperature (at least not in this hypothetical). They just adjust the error signal and temporarily turn it to zero. But your cold governor isn’t sensitive to this distinction. It’s wired to correct an error term. Anything that corrects the error is still a correction, and is perfect in your governor’s eyes.
It will remember this information, but this factoid doesn’t really matter because the correction is so small. A fairly normal option, like “put on a sweater”, is already valued at 50 points. Even “just stand there and shiver” is valued at 10 points! So your cold governor will remember that amphetamines correct its error a little bit, but it will never vote for amphetamines, since it will always have better options.
Later, you are feeling kind of tired and lonely. Let’s say your tired error is 50 and your loneliness error is 100. You would normally go to sleep when you’re this tired, but you want to go out to see friends. But all your friends happen to be out of town this weekend, and no one is responding to texts. So you’re doubly uncomfortable. You decide to try some amphetamines again, just for laughs.
What happens? Well, your tired error and your loneliness error both go to zero. Your tired and lonely governors dutifully mark this down in their ledgers: amphetamines have a +50 correction for tiredness and +100 correction against loneliness.
From the governors’ point of view, this is pretty good. Their only job is to reduce their error signals — they have no ability or reason to care about anything else. So they consider this drug just as good as or even better than their other options.
Now you are in trouble. Your governors have learned that amphetamines are a really good solution to being tired, and one of the best solutions ever to being lonely. In the future, your tired governor will often vote for you to take amphetamines instead of sleeping, and your loneliness governor will reliably vote for you to take amphetamines instead of anything else. From the governor’s point of view, why leave the house and socialize, which has a small chance of making you slightly less lonely, when you could take amphetamines and immediately feel not lonely at all.
Things will only get worse from here. As these two governors vote for you to take amphetamines, any other governors who happen to be out of alignment will learn the same bizarre lesson.
You get a little lonely, and so your loneliness governor votes for you to take some amphetamines. This time, your hunger governor also happens to be a bit out of alignment; you have an error of 20 for hunger.
When the loneliness governor makes you take amphetamines, your hunger error also goes to zero, and your hunger governor picks up on a new idea: amphetamines reduce hunger by about 20 points! That’s almost as good as a quesadilla. Now the hunger governor will often vote for amphetamines, since as far as it can tell, amphetamines are just as good as food. In fact, it will soon learn that amphetamines are better than food, since they correct its errors so reliably.
It’s probably clear why this is very dangerous. If you take a drug that reduces all error signals, then any governor with a large error signal when you take the drug (you’re hungry, you’re angry, etc.) learns to vote in favor of that drug, because it learns that the drug reduces its error signal. Because the drug reduces every error signal.
If you take the drug over and over, you dig yourself deeper every time. Different systems will be out of alignment each time you take the drug, so more governors will see that the drug corrects their error and will learn to vote for it. Eventually, all governors are voting for the drug — hunger is voting for the drug instead of eating, horny is voting for the drug instead of sex, and so on. And the drug feels pleasurable to take — it causes happiness when it corrects the errors, because correcting errors always causes happiness.
When the drug wears off, the error signal goes right back to where it was before. So you’re still tired, hungry, or whatever else. All you did was waste some hours.
You are now very truly addicted. Or at least, that’s what it looks like to us. If you follow this model to its logical conclusion, you get something that looks very much like addiction.
This fits with the observation that some people can take supposedly addictive drugs without getting addicted. In addition to things like genetic differences, if you don’t have large errors when you take your drugs, you won’t get addicted. If you’re already chair of the psychiatry department at Columbia, there’s not much more that drugs can do for you.
This may have something to do with why drug withdrawal feels like abstract “suffering” rather than bad in any particular way. If you have a serious addiction, then when the drug is taken away, every sense you have for finding something wrong with the world will be firing all at once, so in withdrawl you feel not only hungry and thirsty but also tired/pain/cold/hot/scared/angry/jealous/…
Your governors are always trying to estimate the value of different actions (value in the sense of what effect it will have on the signal they care about). But it’s impossible to accurately estimate the value of an action, like taking a drug, that temporarily sends all errors to zero. These drugs don’t have a true value — they interfere with the signal directly. Since they are so tangential to what your governors are trying to do, they really mess with your normal process of learning.
You are lonely — and smoking a cigarette will extinguish that loneliness for about 30 minutes. When it’s over, you will be just as lonely as before. But if you have no better options, the forces inside you that seek connection will still demand you smoke that cigarette, even against your better judgment. A promising but never-fulfilling solution to a basic desire is the core mechanism of addiction.
Most drugs don’t appear to interfere with every error signal, they only interfere with a few. If alcohol only interferes with your shame and your fear governors, then you would only get addicted to alcohol if you drink when you are ashamed and/or afraid. Your hunger or dominance governors won’t be tricked by alcohol, because it doesn’t mess with their signals.
The model predicts that you’re more likely to get addicted to a drug if you take it when you are out of equilibrium in some way. If you take a drug when all your error signals are near zero, then the drug won’t correct those errors by very much, it can’t. Your governors will learn that taking the drug only corrects their errors a bit, and will probably ignore it. This means that social drinking and social smoking might in fact be safer than other forms of smoking and drinking — the folk wisdom that says “never drink alone” may just be onto something.
This also implies some things about what it means to have an addictive personality, what kinds of people are more likely to get addicted to a substance. People with a high learning rate are more likely to get addicted, because their governors will take a single experience with a drug more seriously.
If two people take cocaine when their fatigue error is at 100 points, the drug sends both of their fatigue errors to zero. The person with the high learning rate estimates that the fatigue correction of cocaine is 80 points, and the person with the low learning rate estimates that the fatigue correction of cocaine is 20 points. Obviously one of these people will be more likely to take cocaine in the future, and will be more prone to a serious runaway addiction.
There are some kinds of addiction that don’t involve an external chemical at all. These addictions must have some other origin, since they don’t come about by directly messing with error signals in the brain. We think these can be explained in a couple of different ways.
First of all, the word “addiction” is used very broadly, to mean something like “a habit I do more than I’m supposed to” or “something I do all the time but I don’t endorse”. Sometimes addiction is just a word for something that’s not socially normal or something that is more extreme than normal.
When someone describes themselves as a sex addict, they are often just someone with an unusually high sex drive. Or when someone says they are addicted to cupcakes, they really just mean that their perfectly natural drive to eat sugar is in conflict with their drives to avoid shame and social stigma. Or when someone appears addicted to video games, but that person is in fact feeding their drive for domination in the only way that won’t get them into serious trouble.
This form of addiction can still be pathological. It’s possible that your drive for sex or for cupcakes is so strong that it prevents you from taking care of your other responsibilities, so strong that it comes to ruin your life. These drives are “natural” in the sense that you were born that way, but that doesn’t make them any less destructive. And it could still be pathological in the sense that it’s the result of illness or damage, like if your drive became unusually strong as a result of some kind of traumatic brain injury. But ultimately this “addiction” is one of your drives performing its functions in the normal way, just in a way that you don’t endorse.
One sign that these cases are extreme forms of normal behavior is that these addictions are all linked to one of the basic drives, like food or sex. Even some behaviors that are viewed as purely compulsive, like pica, may actually be the result of a drive that is not commonly recognized, like the drive to eat enough iron. The people on edible dirt etsy must be spending all that money on dirt for some reason. In comparison, we know that evolution did not give us a natural drive to do lines of cocaine. That’s an unnatural desire, driven by some kind of basic malfunction in the systems that are in charge of learning and motivation.
All that said, there are some kinds of addiction that aren’t obviously linked to any drive, and aren’t the result of drugs. The clearest example is gambling.
This kind of addiction might be explained by simple facts of learning. Gambling seems to be the most addictive when the rewards are really variable and the gambles happen on very short time loops. Or they might be most easily explained by something like time discounting — you can put off the shame of failure by doubling down, at least until you can’t anymore.
But the real lesson here might just be that there could be more than one kind of addiction. The idea of looking for a single explanation for every kind of compulsive behavior is exactly the kind of superficial word-chasing that we should try to avoid.
Consider: If cutting yourself gives -10 points, but stopping cutting gives +15 points, then your pain governor will consistently vote for you to cut yourself, so it can then vote for you to stop cutting yourself, netting it a cool 5 points. It will do this whenever you are idle enough that there’s nothing more important going on, when nothing else can beat out its votes.
We think of self-harm as pathological, but there may be some part of it that is very normal. There’s a famous study from 2015, where the authors put people alone in a room for 15 minutes, hooked up to a shock generator that they could shock themselves with as much as they want. They had previously felt the shock and said they found it painful, but 67% of men and 25% of women still chose to shock themselves at least once more, instead of sitting in silence for 15 minutes. Men gave themselves more shocks on average (an average of 1.47 shocks) than women (an average of 1.00 shocks), not counting one man who shocked himself 190 times.
This might follow the same logic as other self-harm, just on a smaller scale. If the shock is more startling than truly painful (-1 point), and the relief of not being shocked any more is a bit pleasant (+5 points) then it’s easy enough to end up in a situation where on net you enjoy shocking yourself.
This issue seems important for making cybernetic systems work. You need these numbers to be in the right ratio. Otherwise, you get caught in a loop — stick your head underwater so you get an error, then take it out so that the error is reduced. But self-harm does sometimes happen, meaning that whatever steps our psychology has taken to resolve this issue don’t work perfectly, or don’t work in every situation.
One way to save on energy, and keep an animal from getting too distracted, would be for governors to become dormant when they don’t have a very big error. After all, if they don’t have a big error, there’s not much they need to pay attention to. This seems to fit that trope where someone does something they thought they wanted, then immediately regrets the consequences. Why didn’t they see that coming? Because the governor that disapproves couldn’t get through. It was asleep.
If this is how we’re designed, then before self-harm the pain governor would be partially offline and wouldn’t be able to vote against “cut yourself”. Then it would be fully online for “stop cutting yourself”.
But this doesn’t work for a few reasons. First of all, governors do seem to get some votes even when their errors are at zero, which is obvious from how often they vote against stuff. If you’re nice and warm inside and you don’t want to slog out into the cold to scrape off your car, that’s your cold governor voting against it, even though it doesn’t have any error at the moment.
Pain seems like a governor that should always be at least somewhat awake. Pain is probably strong even if it’s dormant, because its main job is to prevent injury. It needs to be able to vote against dangers to life and limb, even when you’re not currently in any pain. The hunger governor can turn off when you’re not hungry, but the pain governor can’t turn off when you’re not in pain.
More importantly, even if the pain governor were off, then what governor would be voting for you to cut yourself in the first place? Maybe there’s a way to confuse the pain governor to vote for “hurt yourself then stop hurting yourself” but if so, it would need to be online to vote for it. It wouldn’t do that if it were dormant.
Maybe self-harm only works over very short time horizons. Pleasure and pain in the future should be both discounted, they should be counted as less than 100% of their values because the future is uncertain. Even if you expect something good or bad to happen, there’s always some chance that it won’t come through.
If negative events in 1 minute are weighted so that they’re less than positive events in 2 minutes, that could maybe lead to self-harm. Getting cut is 60% of -10 points, stopping getting cut is 80% of +8 points, suddenly the pain governor is happy to vote for one so it can get to the other.
A more straightforward explanation is that some other governor is voting for you to cut yourself, and it’s much more powerful than the pain governor, so it can overrule it. This is most obvious in cases like ritual self-harm, games like chicken or bloody knuckles. In this case, failing to go through with the painful or embarrassing experience would lose the confidence of your community, you would be ashamed, it would show that you were a loser or a coward. These are high enough stakes that other drives are able to out-vote the pain governor.
There might be many governors that could vote for this. For example, there might be an actual drive to harm others, that motivates things like sadism and serial killing. If you have a drive to hurt something, but the only thing you can hurt without consequence is yourself, then this drive might eventually overpower your drive for pain, and you would hurt yourself as a compromise.
Or, maybe the systems set in place to prevent self-harm do work perfectly fine, and so self-harm only happens when something has happened and those systems aren’t working properly, more like depression. If a malfunction makes it so that bad experiences are estimated as less bad than usual, that would make you pretty reckless — and it could mean that you’re willing to hurt yourself to get the benefits of not hurting yourself later.
[Next: INTERLUDE]
2025-03-20 23:11:00
[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
[PART V – DEPRESSION AND OTHER DIAGNOSES]
It is interesting to note that when certain types of paradoxes are fed to the Kalin-Burkhart machine it goes into an oscillating phase, switching rapidly back and forth from true to false. In a letter to Burkhart in 1947 Kalin described one such example and concluded, “This may be a version of Russell’s paradox. Anyway, it makes a hell of a racket.”
— Martin Gardner, Logic Machines and Diagrams (1958)
Life requires navigating the conflict between different parts of yourself.
Everything you do, all your behavior, is the result of different governors negotiating in an attempt to satisfy your drives and emotions. In most situations, governors come to a mutually-agreeable compromise and you can proceed from one thing to another in a way that satisfies all parties involved.
But other times, things do not go so well. The inside of your head is like any palace intrigue: factions rise and fall, allies today are enemies tomorrow, and no one is ever fully in control.
When your governors want two incompatible goals to be realized at once, the result is conflict.
Conflict can have different outcomes. When the opposing governors are closely matched in force and there’s a binary decision, it will lead to inaction. When they are closely matched in force and there’s a range of behavior, it will lead to half-measures. When one is much stronger, it can lead to countermeasures.
For example, your hunger governors might vote strongly in favor of eating a piece of cherry pie. But like many people, you have internalized the idea that eating cherry pie is a wicked, weak thing to do. So your shame governor votes strongly against it. The votes cancel each other out. You stand in the window of the bakery for a long time, staring at the pie and doing nothing. Here conflict has led to inaction.
A mouse’s hunger governor will vote to approach a feeding bowl (the mouse is hungry and the bowl is full of food), while its pain governor votes to avoid the feeding bowl (which has been rigged to give the mouse painful electric shocks). If these two governors are about equally strong, the mouse might go half way out towards the bowl of food, and no further. When it gets closer, the fear governor becomes more powerful and pushes it back. When it gets further, the fear governor becomes weaker and the hunger governor pushes it forward. Here conflict has led to an intermediate state, half-measures.
Even when one governor is strong enough to win, there can be ongoing conflict. You pull the cookies out of the cupboard because you’re hungry. Shame makes you throw them in the trash. But once they’re in the trash, hunger is in control again. So you fish the cookies out of the trash. This is conflict where the state is countermeasures.
For a real-world example, here’s Henrik Karlsson describing his own experience of a minor conflict:
Our emotions and intuitions are littered with contradictions.
To take a simple example: when I was at the gallery where I worked until last week, my low blood sugar cravings sometimes told me that it was ok to take a pastry from the café. But when I want to feel like a upright person, I don’t believe in taking stuff that isn’t mine. So which is it? If I follow my gut and eat the pastry, I will be true to myself in the moment, while betraying other versions of me.
The experience of conflict is stress. Staring at the pie and doing nothing is a fairly stressful experience. Hovering between fear and hunger is stressful for the mouse. Throwing out the cookies over and over again is no better.
Unlike the errors generated by your governors, stress is not an emotion. It’s a different kind of experience that happens when two or more actions are in direct competition.
It’s easiest to become stressed when two drives are in direct conflict. You want to ask someone out, but you’re afraid of rejection. You want to eat a whole pizza, but you know your family will laugh at you if you do. Here, two drives hold each other in check, there’s tension.
But you can also become stressed when drives are merely in competition. You might want to both go out and see friends (because you are lonely) and stay in and go to sleep (because you are tired). These are both positive desires, but you can’t do both at once, they are mutually exclusive. If they’re both about equally strong, you will do nothing, neither go out nor sleep, and it will be stressful.
And you can become stressed when negative drives are in competition. A witness to an assault must choose between intervening (risking physical harm) or walking away (risking social condemnation as a coward). Both of these outcomes are things they would like to avoid, but they can’t avoid both. This is also stressful, and again, if they are equally matched the person will do nothing.
Stress is a really negative experience, for two reasons.
First of all, when they’re in conflict, governors are distracted from everything else. They commit all their resources to the fight, and clog up common parts of the system, like the voting channels, with constant bids for their concerns.
Second, governors in conflict are absolutely gunning it. When a governor pushes and the signal doesn’t change, what does it do? That’s right, it pushes harder! If it’s pushing against another governor, then that governor pushes back. They both push ten, twenty, fifty times harder. Soon they are both pushing absolutely as hard as they can.
This is an incredible waste. Whatever resources are involved in this contest will be burned through at an astounding rate, with the only result being a deadlock. But this is what you get when you’re in a double bind. For one governor to correct its error, the other governor must experience an increase in its error. There is no way for both systems to experience zero error at the same time.
This view of stress calls back to old theories like the approach-avoidance conflict, also sometimes called push/pull. Kurt Lewin, who was close with the early cyberneticists, was one of the people who argued for this approach.
“Conflict,” he wrote in his 1935 book, “is defined psychologically as the opposition of approximately equally strong field forces.” Kurt talks in slightly different terms, but the overall conclusion is the same. He offers this example: “The child faces something that has simultaneously both a positive and a negative valence. He wants, for example, to climb a tree, but is afraid.”
Kurt’s views were very influential back in the day, but psychologists don’t really focus on his models anymore. This might be because he insisted on explaining human motivation in terms of “psychical field forces” instead of drives, perhaps in an ill-fated attempt to try to make psychology sound more like physics. In fact, he explicitly rejected drives as “nothing more than the abstract selection of the features common to a group of acts that are of relatively frequent occurrence.”
This model of stress has a surprising implication for self-control. You cannot alter your behavior by simply choosing to overcome the unwanted behavior.
There must be a drive already voting for that behavior, since the behavior exists. It must be controlling something. So to attempt to overcome a behavior can only lead to conflict.
You can avoid situations that would put your governors at odds with each other. You can set one governor against another, suppress your unwanted impulses by the force of shame or fear. All of these defense mechanisms and more will “work”. They will keep you from accidentally doing the unwanted behaviors, at the cost of more conflict. Or they will let you avoid the agony of conflict, at the cost of making your life smaller and smaller. But the only fully healthy solution is to find a way to reorganize things so that both governors can fulfill their purpose without conflict.
William Powers said it best:
The payment for a lifetime of “overcoming” one’s weaknesses, base desires, and forbidden habits is to spend one’s last years in a snarl of conflicts, one’s behavior restricted to that tiny part of the environment that leaves all conflicts quiescent, if any such place still remains. The rigidity of many elderly people is, I believe, the rigidity of almost total conflict, in which every move is made against massive inner resistance.
…
Indeed, self-control is commonly taught as part of raising children … Through social custom and the use of reward and punishment, therefore, we have perpetuated the teaching of self-control and have thus all but guaranteed that essentially everyone will reach adulthood suffering severe inner conflict. Self-control is a mistake because it pits one control system against another, to the detriment of both.
The word “anxiety” is only an abstraction. It groups together many things that seem similar, but may have different causes underneath. (See the prologue to learn more.)
But to take a stab at what all kinds of “anxiety” have in common, we could say that they all look like systems spending a huge amount of energy to make very little progress.
We see two general ways that might happen.
The first is chronic conflict, two or more governors locked in a deadlock for a long time, spending a huge amount of energy fighting each other and getting nowhere.
The second is oscillation, a system wildly swinging back and forth, spending a huge amount of energy correcting and re-correcting, instead of efficiently settling towards a target or equilibrium.
When you’re consistently stressed for a long time, that seems like one kind of anxiety.
Sometimes governors are briefly in deadlock, like gazing at the pie through the window. Eventually you will get cold enough to walk away, or hungry enough to buy the pie, or something will distract you. So this conflict can’t last for very long.
But sometimes governors get locked, not just in conflict in the moment, but habitually in conflict all the time, so that you are constantly stressed. For many people, food is a source not only of stress but also of anxiety, because their feelings of shame will get into conflict with their desire to eat fat and sugar, not just one time, but over and over and over again.
In nature, stress tends to be limited to very brief experiences, where two drives happen to be perfectly balanced. These situations tend to be over pretty quickly. One of the drives will grow faster than the other, or the situation will change, and the conflict is resolved.
However, in a manufactured environment, it’s easy to produce anxiety-inducing situations by accident. For example, a lab animal that only feels safe in a dark tunnel, but whose water bottle has been placed in the brightly-lit center of the cage, will go through repeated experiences of stress as its fear grapples with its thirst. Or a dog that wants to protect its family, but whenever the dog barks at passers-by, the family yells at it. These animals will be anxious, because their drives are habitually in conflict.
Because of our notable collection of social emotions, humans seem to have the worst of this. Social emotions consistently come into conflict with the others. People want to yell at their boss out of anger but don’t want to suffer the social consequences of that outburst. They want to sleep with people they are socially forbidden to sleep with. They are terrified of something but are not able to act on their fear; like a student terrified of their teacher, or a professional driver terrified of getting in a crash. Our social norms around food seem practically designed to be anxiety-inducing; half of the things that a person might naturally be most excited to eat are considered “bad” or outright sinful. That’s a conflict right there.
It’s even possible this is the role social emotions serve in our psychology. Maybe social emotions are there to make us anxious. Humans are still by their nature angry, horny, violent, and so on. But our social emotions put some checks on these drives and may be the only thing that make it possible for us to work together over the long term. Social emotions are frequently called on to keep the other emotions in check, and stress is an unfortunate side-effect of this balance.
Hamlet was stressed because he has to both kill his uncle and not kill his uncle. He is presented over and over again with opportunities to kill or not kill his uncle, or at least to take steps in those directions. But he can’t do either, because the two drives are almost perfectly balanced. This is very stressful.
Antigone was bound to bury her brother Polynices, but Creon had decreed that Polynices was not to be buried or mourned, on pain of death. Orestes avenges his father Agamemnon by killing his mother Clytemnestra, honoring his filial duty to his father but violating his filial duty to his mother. This is the source of tragedy. The ancients had it right.
Let’s see one very interesting example.
When two governors have very similar amounts of votes — let’s say within 10 votes of each other — neither one can win, and you are in a state of conflict.
This causes a little bit of stress. But normally, one or the other of the options will soon get enough votes to beat that margin, or some new issue will come up that renders the decision moot.
However, sometimes for one reason or another, all the vote totals get turned down. This is one of the malfunctions we call “depression”.
This has a curious side-effect. Let’s say that normally you have a hard time deciding between staying at the party and going home. Your loneliness governor has 40 votes for “stay at the party” and your fatigue governor has 45 votes for “go home”.
Since these are within 10 votes of each other, neither can really win. This is uncomfortable and you feel a little stressed. Instead of really engaging, you hover at the edge of the party. But eventually one or the other governor gets a big enough error that it gets enough votes to beat the margin. Probably you get a bit more fatigued, until fatigue hits 51 votes, wins the margin, and you go home.
But when all your errors are turned way down, something strange happens. At the party, you now have fewer votes overall, which makes it harder to break this tie. If errors are turned down to 50%, then you have 20 votes for “stay at the party” instead of 40, and your fatigue governor has 22.5 votes for “go home” instead of 45.
Now to break the tie, your fatigue governor needs 7.5 more votes instead of 5, and each vote requires twice as much of an increase in fatigue. If errors are turned down to 10%, then you have only 4 votes for “stay at the party” and only 4.5 votes for “go home”! You will stay in deadlock for much longer, and it will be stressful the whole time.
Worse than that, you will end up in deadlock more often, over more issues. Normally it is easy to choose to shower (70 votes) before eating breakfast (50 votes). We’re not talking about deciding between the two, just the decision to finish the one before the other.
But if your vote totals are cut, you may find that this decision is suddenly 35 votes versus 25, just barely enough for the vote to resolve. If your votes are cut enough, you won’t be able to decide whether to shower first or eat breakfast first. You become indecisive about all kinds of things, even the smallest decisions.
This may explain why depression so often goes along with anxiety. When your vote totals are turned down, but the margin of votes by which an action has to win remains the same, you end up in a state of deadlock for much longer, and it will happen a lot more often. Since conflict makes you feel stress, you feel stressed all the time, over the kinds of decisions that would be simple or easily resolved before. That’s anxiety.
There are probably many things that can cause anxiety. But any kind of depression that gives you fewer votes overall is going to almost always lead in this unfortunate direction. This also suggests that other forms of depression, that don’t give you fewer votes overall, shouldn’t lead to more conflict and shouldn’t go along with anxiety.
The second way to waste a bunch of energy for no reason is when a system swings back and forth for a long time without settling.
One of the classic ways a control system can fail is that it goes into oscillation, wildly swinging back and forth, wasting a huge amount of energy instead of efficiently settling towards the set point and zero error.
In psychology, this is most obvious in tremors. Damage to the control systems responsible for motor function leads to overshooting and very obvious physical oscillations. In Cybernetics, Norbert Wiener described a few cases:
A patient comes into a neurological clinic. … offer him a cigarette, and he will swing his hand past it in trying to pick it up. This will be followed by an equally futile swing in the other direction, and this by still a third swing back, until his motion becomes nothing but a futile and violent oscillation. Give him a glass of water, and he will empty it in these swings before he is able to bring it to his mouth. What is the matter with him?
… His injury is … in the cerebellum, and he is suffering from what is known as a cerebellar tremor or purpose tremor. It seems likely that the cerebellum has some function of proportioning the muscular response to the proprioceptive input, and if this proportioning is disturbed, a tremor may be one of the results.
This isn’t a problem just with the brain, this is characteristic of all control systems. Weiner notes it as, “… a badly designed thermostat may send the temperature of the house into violent oscillations not unlike the motions of the man suffering from cerebellar tremor.” And in fact, all control systems can oscillate if they become unstable.
Tremors are oscillations in low-level control systems responsible for muscle movements. That’s why you can see them — your arm or leg is actually waving back and forth.
But oscillations might also happen at other levels of control. If systems oscillate at the level of behavior instead of at the level of arm/leg position, that might look like doing behaviors over and over again, or doing them and then undoing them. This would look kind of like compulsions, or like OCD.
If systems oscillate at the highest level, something like thought or intention, that might look like choosing one side of a decision, but then before acting, switching to the other side of the decision. The guy who decides to quit his job, then decides to stay at his job, 20 times per hour. This looks like a form of rumination.
This would kind of explain why OCD and ruminations seem connected. They may be basically the same kind of problem just in slightly different parts of the system. Or maybe OCD is a more extreme form of rumination, an oscillation that makes it all the way into behavior, instead of just oscillating “within thought”.
The difference between oscillations and conflict is that conflict is always a struggle between two or more governors, while oscillation can happen in just one governor alone, especially if it is damaged or otherwise improperly tuned.
Oscillation can happen for a few key reasons.
A governor with too much gain, that makes very aggressive corrections, can overshoot repeatedly instead of settling.
A governor with not enough damping can fail to slow down in time as it corrects its signal towards the target. Then it will overshoot, and have to bring the signal back. But then it may overshoot again.
If there is any delay in feedback, where the governor is getting outdated information, it might keep making adjustments that are no longer needed, leading to overshooting and continuous corrections.
Oscillation is common because control systems often involve a tradeoff between speed and stability. If you want a fast response, you risk instability; if you dampen too much, you risk sluggish behavior.
Without getting too much into the weeds, just like depression can be caused by damage or malfunctions in different parts of your governors and selector, anxiety can be caused by damage or malfunctions in different parts of the ways that your governors are tuned, like their gain or damping, or by similar problems like delay in feedback.
Oscillation can also happen between two governors. It’s easiest to see this with an example. Let’s say that Danny’s hot and cold governors both have a problem where they have too much gain. So when he’s a little bit too cold, he does too much to correct it. He puts on socks and a sweater and gets a hot mug of tea and starts a fire in the fireplace. What happens now? Well, he soon becomes too warm. So he opens the windows and douses the fire and puts a fan on himself and strips down to his underwear. What next? Of course, he gets too cold. So it’s time to get warm again. He will keep oscillating until distracted.
You may even sometimes get oscillation inside a conflict. When two fine-tuned governors want mutually exclusive things, they will usually fight, putting out their more and more effort until they both reach their maximum output, and settle at a midpoint that is the balance between those two maximums.
But if the governors are less well-tuned, they might oscillate. Danny’s fear and status governors are kind of deadlocked at work. He is afraid of his boss but he wants to crack jokes to impress his coworkers. A more “well-adjusted” man would be in a state of conflict. But Danny is poorly tuned. His status governor makes him crack a joke, and his fear governor is too slow to stop it from happening in time. His boss gives him a dirty look and Danny’s fear governor takes over. He shrinks down in his chair. But the fear subsides and soon he thinks of another joke. This is conflict, but it is also oscillation.
A final thing we notice is that anxiety is often about the future. This might also be a kind of oscillation.
Consider Molly, a college student. Her parents and her community expect her to be a huge professional success (no pressure, Molly). Her status governor knows that it’s really important that she get a job when she graduates. If she doesn’t, her status governor faces a huge error, and since it can predict this, it wants to prevent it. But she has just started her senior year, so it’s not time to look for a job yet. The best thing she can do is focus on her studies.
This can lead to a weird cycle that looks kind of like a form of oscillation. She starts thinking about having to get a job. Her status governor leaps into action, panics, looks around for a way to start making a difference, but finds that there’s nothing it can do. Then it shuts off. But this can happen 100 times in an afternoon, and there’s not much she can do. There are no steps she can take to get a job now. She just has to wait.
Because the governors are predictive, any promise of an extreme outcome can snipe you in this way. If your fear governor develops a fixation on car accidents, it might sometimes pipe up, “I predict we might get in a car accident. What can we do right now to make that less likely?” But you are in a work meeting, or at the grocery store. There’s nothing you can do at that moment to protect yourself from car crashes. But because the predicted error of a car crash is so huge (possibly death), your fear governor gets huge amounts of control over your attention and motivation when it makes this prediction. So for a while you are cowering in the cereal aisle, running through hypotheticals about dying in a 4-car pileup.
The ability to look at a hot stove, predict that it will burn you, and decide not to touch it, is a great adaptation. It’s why our governors are predictive — it’s great to be able to consider what will happen a few seconds in the future. But the human ability to look very far into the future is more of a mixed blessing. On the one hand, it means we can be motivated by things that may not happen for months. We can do long-term planning. But it also means we can be totally captured by far-off imaginary disasters (or imaginary blessings) that totally derail our ability to focus.
You can lie awake in bed asking yourself, “will I be ready for my biology test on Friday?” The best thing to do, of course, would be to go to sleep. But the fatigue governor is being shouted down by the status governor, which is endlessly worried about failing the test. And there’s nothing you can do to make it quiet down. It is 2AM, there’s no way to prevent future status errors now.
Even worse is when you are taking concrete steps towards an outcome but none of your perceptions change. This is probably why founding a startup is so stressful. You work every day on your product, but there’s often no obvious change in your chances of success for weeks or even months. If you can get some metric like “number of users” that is constantly growing, that will help. But if not, you just have to keep plugging away and hope that you really are the next Google, or at least that you will be able to exit.
Or why dating can be so stressful. You can go on apps, go to events, meet people, go on first dates. But most of the actions you take don’t get you any closer to what you are trying to achieve. Each time you either meet the person or you don’t.
This isn’t like most problems! When you are hungry, each apple or corn chip makes you slightly less hungry. When you are afraid, each step aways from the clown makes you slightly less afraid. But when you’re running a small business, most meetings cause no apparent change in your status or safety.
[Next: WHAT IS GOING ON?]
2025-03-13 23:11:00
[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
There’s a thermostat that regulates the temperature
That might not be reliable
That should be disconnected— Thermostat, They Might Be Giants
Here are some mysteries about depression:
In most illnesses, the list of symptoms is hit or miss. Not every patient gets every symptom, or even every common symptom. If a common symptom of an illness is breaking out in hives, many people will break out in hives, but some people won’t.
Depression is much stranger. Like other diseases, you sometimes get symptoms and sometimes do not. But on top of that, you also sometimes get symptoms, and other times get their opposites.
One common symptom of depression is eating too little. Another common symptom of depression is eating too much.
One common symptom of depression is gaining weight. Another common symptom of depression is losing weight.
One common symptom of depression is insomnia, not being able to sleep. Another common symptom of depression is sleeping too much.
The most typical symptom of depression is feeling really bad. Except in other cases, when the most typical symptom of depression is feeling nothing at all.
There are other weird mysteries when we look at how depression is treated. Even though insomnia itself is a symptom, sleep deprivation often seems to treat depression. (In this context it’s sometimes called Wake Therapy.) This is effective in as many as 50% of cases, though the relief is usually only short-term.
We think these mysteries make sense when you start looking at depression as a disorder of negative-feedback-loop emotions.
You see symptoms and their opposites because many pairs of emotions cover two ends of a single variable. There’s one set of emotions that make sure you eat enough and another set that make sure you don’t eat too much. Since depression can interfere with either side of the scale, you sometimes get opposite symptoms. We have one set of emotions that make you go to sleep and another set that make you wake up. Since depression can interfere with either, some depressoids have insomnia, and others sleep through their alarm.
Though it’s not usually listed as an official symptom, we would also expect there to be some cases of depression where people end up overheating, and other cases where people end up getting much too cold. Just like for eating and sleeping, there are governors on both ends of the scale, and the governors that would normally take care of these errors are being interfered with.
Symptoms of depression like “loss of interest in sex” don’t have an opposite symptom because while there is a governor making sure that you’re interested in getting a certain amount of sex, there isn’t a corresponding governor making sure you’re not getting too much sex.
Sleep deprivation may be a treatment in some cases because as we’ve previously mentioned, happiness is created by producing and then correcting errors. That means that a very large error created by getting very sleepy might be big enough to register, even when something is wrong with the happiness machinery. If nothing else, it might shake things up enough that something will register.
We also think that close examination shows that depression is not just one disorder, it’s several different disorders. They share a surface-level similarity, but can be clearly divided into types.
The surface-level similarity that all different kinds of “depression” have in common is that they are all disorders where the person very rarely experiences happiness. This is why our culture formed the category “depression”, because we noticed that sometimes people had a persistent lack of happiness, even when they found themselves in situations that should normally make them happy.
But the systems that produce happiness are complex. There are many things that can go wrong, so there are many different kinds of “depression”. And besides the fact that they all present similarly — the person has a hard time experiencing normal happiness — different kinds of depression don’t always have very much else in common.
If we take a look at different ways this model can malfunction, we’ll see different outcomes that all look kind of like depression. But we’ll also notice that despite their basic similarity, most of these different problems have at least slightly different symptoms, so it may sometimes be possible to distinguish them by symptoms alone.
In fact, almost anything that goes wrong in the motivational system will cause something that looks like depression, which is probably why “depression” is so common. Our job is to look past these superficial similarities and try to figure out exactly what is malfunctioning, so we can have some hope of treating it.
If you have two patients with very similar symptoms, you might be tempted to assume they have the same disease. But they might also be experiencing the same symptoms for totally different reasons. For example, a cough could be caused by a viral infection, bacterial infection, or a non-infectious condition like asthma or acid reflux. Or from accidentally inhaling your Dr. Pepper.
You want to give the antibiotics to the person with a cough from a bacterial infection, and the antivirals to the person with a cough from a viral infection, and you don’t want to mix them up. The underlying cause determines the appropriate medical approach, not the symptoms. If you don’t know what’s causing the problem, you can’t treat it.
Imagine you are working on an internal combustion engine. To work correctly, an engine requires both gas and a spark. The battery is fully charged, so you know you have the power needed to create a spark. When you turn the key, the engine turns over, but doesn’t start.
This could be caused by at least two problems. First of all, maybe no gas is getting into the cylinder. Second, maybe the spark plug doesn’t work.
Those sound like two different causes. But they are not, at least not quite. It’s true that we can narrow things down to these two different branches — it almost certainly has something to do with the gas or with the spark plug. But the real causes are much more complicated.
Maybe there’s no gas getting into the cylinder, but “no gas getting into the cylinder” could happen in a number of different ways. First of all, the vent to the gas tank is clogged, creating a vacuum that stops gas from being drawn into the line. Second, there might just not be any gas in the tank. Third, the gas filter could be clogged. Fourth, the gas pump could be broken; for example, there might be a hole in the diaphragm.
The spark plug is also made up of many smaller components. If any component fails, then there’s no spark.
Imagine there are 100 old cars. One misty, rainy morning, you discover that none of them start. This was a real issue back in the day — a big rainstorm one night, and in the morning, half the cars in town stop working.
You figure it must be the spark plug wires, moisture kills ’em. But replacing the spark plug wires only fixes some of the cars. You eventually find out that some of them actually got water in the gas instead, and need a different fix.
Same symptoms (old car doesn’t start), same distal cause (rain storm), but a different proximal cause. So we can have cases with the exact same symptoms (engine turns over but doesn’t start), with two possible diagnoses. And even within those two diagnoses, there are potentially dozens of causes, each requiring a different fix. Even if you figure out for sure that the problem is a lack of gas, you still don’t know if you need to replace the gas filter, or part of the gas pump.
Any system is made up of many smaller parts, all of which can break in several different ways, so you can usually get the exact same disorder as result of issues with different parts. A specific part of the chain is broken, but you can’t tell which one.
Ultimately, examining the symptoms that arise when different systems malfunction will be helpful for treatment. But it’s not law — systems can break for more than one reason. System malfunctions from two different causes may look just the same — from the outside, all you notice is that this system isn’t carrying out its function, but that doesn’t tell you how to fix it.
Some people have such perfect control over their life that they never meaningfully get hungry, thirsty, tired, lonely, cold, etc. etc.
This sounds good, great even. But in fact the person ends up very depressed, for a simple reason. Nothing is actually wrong with this person, there’s no damage, it’s not even really a malfunction. But the fact that they almost never correct major errors means they very rarely produce any happiness.
This is the depression of the idle rich, which we mentioned before. It can be treated by intentionally creating errors and then correcting them. For example, you might intentionally get very tired and thirsty from running an ultramarathon, and then rest and rehydrate, which will grant nothing short of ecstasy. Or you might expose yourself to pain from some other extreme sport, then recover, and again reap the happiness. People often discover this treatment on their own, which is why the idle rich are often into certain kinds of (no judgment) self-destructive hobbies.
Sleep deprivation therapy seems like it would work pretty well for this kind of depression. It’s the same logic as extreme sports. Staying up all night and then going to sleep would be an almost euphoric experience that would provide you with some happiness, at least for a while.
As a bit of a tangent, something else that may explain some behavior of the rich is that there may be an emotion that drives us not only to maintain our current status, but to increase our status at some constant rate. In other words, there may be a governor whose target is the rate of change, or first derivative, in status.
For someone of normal status, this expresses itself as normal ambition. The average person can always become more important. But as you become higher and higher status, this becomes a problem, because the higher status you are, the harder it is to increase your status further.
Someone who reaches maximum status for their social group finds themselves in a bind. They still feel the drive to increase their status, but they are already top dog. A person might become CEO, or in an earlier age might become King, due to their drive to increase their status. And these people will usually be the ones with the strongest status governors, making their inability to increase their status any further especially painful for them. Those who reach the top are left with a hunger they can no longer feed. “When Alexander of Macedonia was 33, he cried salt tears because there were no more worlds to conquer.”
Kings like Alexander often tried to overcome this by establishing their divinity. This approach doesn’t work so well any more, but modern people sometimes handle it by realizing that while they cannot advance their status any further in their own field, they can still advance their status in new areas. This is how Bill Gates ends up advancing his status by becoming a philanthropist, or how Mark Zuckerberg advances his status by training in jiu jitsu. Both of them had already maxed out their status as tech guys, so to keep increasing in status, they had to find new kinds of status in new arenas.
It’s reasonable to ask if this kind of “depression” is actually a problem. Is it so bad to not be very happy, as long as all your needs are met? If you don’t understand what’s going on, you might be concerned that there’s something wrong with you. But if it’s this simple, and it’s just a side effect of all your needs being met, then is there anything to be concerned about?
We think there might be. First of all, while happiness isn’t everything, it’s nice to be happy, and it’s reasonable to think about ways you could be happier.
Second, we suspect that happiness regulates the balance between explore versus exploit. If that’s true, then we would expect that over time, people who are depressed pursue stranger and stranger strategies as they increase their tendency to explore new ideas, in an effort to find something that “works”. But this never brings them happiness, because the problem is internal.
If this is the case, then people who suffer from long-term depression, of any kind, should appear to act crazier and crazier over time, as they explore more and more unusual strategies.
Under normal circumstances, correcting an error creates some amount of happiness. Somewhere in the system is a mechanism that registers when a correction has taken place, and creates a corresponding amount of happiness. The bigger the correction, the more happiness is created.
If something happens to this mechanism — the signal is turned down really low, part of it gets broken, the shipments of neurotransmitters it depends on never arrive — then you get a very characteristic form of depression.
This person experiences emotions as normal, generally behaves as normal, and has successful behavior. After all, their governors are all functioning exactly as normal, errors are getting corrected just like before. They are surviving, even thriving. But despite their successful behavior, they never feel happiness. They appear normal to casual observers, but describe themselves as “dead inside”.
This kind of experience comes out pretty clearly in patient descriptions, like this one reported by William James:
I have not a moment of comfort, and no human sensations. Surrounded by all that can render life happy and agreeable, still to me the faculty of enjoyment and of feeling is wanting — both have become physical impossibilities. In everything, even in the most tender caresses of my children, I find only bitterness. I cover them with kisses, but there is something between their lips and mine; and this horrid something is between me and all the enjoyments of life. My existence is incomplete. The functions and acts of ordinary life, it is true, still remain to me; but in every one of them there is something wanting — to wit, the feeling which is proper to them, and the pleasure which follows them…All this would be a small matter enough, but for its frightful result, which is that of the impossibility of any other kind of feeling and of any sort of enjoyment, although I experience a need and desire of them that render my life an incomprehensible torture.
If happiness is still generated, just at much lower rates than usual, you would get a less extreme version of this experience. Someone who generated 50% as much happiness as usual would feel a little down in the dumps, but not terrible. Someone who generated 10% as much happiness as usual would feel pretty depressed, but not quite entirely dead inside.
This is a good chance to give an example of what we meant when we were talking about how every system is made of many components, how the spark plugs can break in many ways. Even in this very simple model, many different problems will create the same kind of malfunction.
For example, maybe the mechanisms that actually generate happiness are working as intended, but the connections that transmit the correction signal to those mechanisms are malfunctioning. In this case, happiness isn’t generated, because the signal that should trigger happiness never reaches its destination:
Or, maybe the connection is working just fine, but the mechanisms that should generate the happiness are malfunctioning. So the signal arrives just as it should, but nothing is done in response:
While these are malfunctions in different parts of the system, a person with a malfunctioning correction-connection would behave almost exactly the same as a person with a malfunctioning happiness-generator. They would probably benefit from different treatments, since the cause of their depression is different, but they would be very hard to tell apart based on their symptoms.
And of course, this is one of the simplest possible models. In real life, there are more than just two components; there must be dozens.
Despite being responsible for different signals, your governors all run on basically the same architecture. If you want to think in terms of mechanical engineering, maybe they all share the same fuel, or the same lubricant, or they’re supplied by the same pump. If you think more in terms of programming, consider them as using many of the same functions, inheriting from the same class, or relying on the same set of libraries.
Since they’re all supplied by the same metaphorical pump, if something goes wrong with that pump, something can go wrong with all of the governors at once. If there’s a function that you use all over your program, and you accidentally comment out an important line in the function, everything that uses that function will be affected. Maybe every part of the program will be impacted the same way, but depending on how the function is used, maybe in different ways.
The most basic job of a governor is to compare the incoming signal to the set point and generate an error. If it does this correctly, its second job is to try to correct that error. But first it has to successfully generate the error.
So if this ability to correctly generate an error ever breaks, that’s a big deal. In a minor malfunction, error signals will still be generated as normal, but all error signals would be turned down, let’s say by 50%. In this case you’ll mostly behave as normal, but you will have to be twice as far out of alignment — get twice as cold, go twice as long without sleeping, etc. — before you take the same amount of action you normally would. And for a given level of actual distress, you will feel only about 50% as tired, hungry, thirsty, lonely, etc.
If this happens to you, you’ll also experience less happiness than normal. Your errors don’t grow as fast as normal, so when you correct your errors, they’ll tend to be unusually small. Eating a meal that would normally correct 10 points of hunger error and create 10 points of happiness will instead correct 5 points of hunger error and create 5 points of happiness. So you’re not joyless at 50% error, but your actions won’t bring the sense of satisfaction that they used to.
Of course, you could wait until you had a subjective experience of 10 hunger before eating a meal. Then you would get the same amount of happiness from correcting it. But if you do that, you will notice that you eat only half as often as usual, and you’ll still ultimately be getting less happiness over the long term, since you are correcting the same magnitude of error, but only half as often. You’ll also notice that you feel weak and brainfoggy, since you are only getting about half of your actual nutritional needs.
However, lots of people eat more by routine than by hunger. So most people would probably stick to their three-meals-a-day approach, through the normal drumbeat of social routine, or just out of habit. These people will eat as much as normal, but get half as much satisfaction.
If something more serious goes wrong, and all error signals are turned down to 10% instead, your motivation would become extremely sluggish, and you will generate much less happiness than usual. It will take you a lot longer to take action than it would otherwise, because it takes much longer for your error to reach a given level. This might be called procrastination.
Let’s simplify and say that hunger is driven entirely by blood sugar (not true, but this is for the sake of example). For a person whose error signals are turned down to 10%, their blood sugar will slowly drop lower and lower without causing an appreciable error. Without any error, it’s hard for them to have any motivation to eat, let alone cook.
Eventually blood sugar gets very low and the governor is finally generating the minimum amount of error needed to get over the gate’s threshold. This person still won’t be very motivated, and the hunger still won’t be very pressing. And if they do eat, it won’t make them very happy, because the correction is quite small. This person is trapped in a world of low motivation, very dulled emotions, difficulty telling whether they are hungry / thirsty / tired / etc., and little happiness.
In the extreme case, if your error-generating systems are so busted that almost all your errors are close to zero no matter how far out of alignment you are, things get pretty bad. This person barely experiences emotions, so almost no behavior happens. This is classic “can’t get out of bed” or “bedrot” depression — the person has hygiene problems and only eats, sleeps, or moves when extremely hungry/tired/etc. Since their error signals never get very big, there’s no opportunity to correct them, and this person experiences almost no happiness.
If errors actually become zero, the person is effectively immobile, in a sense almost comatose or paralyzed. This lines up pretty well with the symptoms of having a serious basal ganglia injury. Consider this case report from Treating Organic Abulia with Bromocriptine and Lisuride: Four Case Studies:
During the preceding three years he had become increasingly withdrawn and unspontaneous. In the month before admission he had deteriorated to the point where he was doubly incontinent, answered only yes or no questions, and would sit or stand unmoving if not prompted. He only ate with prompting, and would sometimes continue putting spoon to mouth, sometimes for as long as two minutes after his plate was empty. Similarly, he would flush the toilet repeatedly until asked to stop.
Antipsychotic drugs like haloperidol seem like they might be messing with the same system. If you take too much haloperidol, you’ll sit there and do nothing, possibly until you die.
Earlier we used the analogy of a pump being damaged and working at only 50% capacity. The pump can also be damaged in a way that sends it into overdrive, where it careens along at 200% capacity.
If this happens, all your errors are twice as large as normal. You are more driven, and driven to do more things. You become wildly active. Since your errors are larger, correcting them makes you even happier. The most normal successes, like drinking a glass of water, create almost ecstatic happiness, and your mood soon goes off the charts. This sounds a lot like the high phases of manic depression / bipolar disorder.
The real question here is why manic depression is so common, but pure mania — mania just by itself — is so rare. People are pure depressed all the time, many people are bipolar, so why aren’t many people suffering from pure mania?
We certainly don’t know, but here are some rough hypotheses.
One very mechanical answer is that overdrive is simply unsustainable. A person can’t be manic all the time, because eventually they will run out of juice.
Consider that example with the pump. In this case, the pump that supplies all the different engines has malfunctioned and is running at 200%. The pump circulates oil drawn from a reservoir. Normally this reservoir re-fills with oil (perhaps as it’s filtered, or drawn from a larger reservoir) faster than the pump circulates it. But when the pump is rushing along at 200%, it drains the reservoir faster than it can be filled. So it empties the reservoir and triggers some kind of emergency stop, during which it can draw no oil at all. Eventually the reservoir refills and the stop is lifted, and the pump goes back into overdrive mode again.
In this explanation, bipolar disorder *is* pure mania. During your manic phases you overuse some kind of limited resource. When it runs out, you’re cast into the depressive phase caused by the lack of that resource. This lasts until the resource has built up back to some minimum level, at which point you start running at 200% again. But the real nature of the problem is disguised, because the human nervous system has checks and limits.
(A quick research check suggests that in bipolar disorder, manic phases last only days or weeks, while depressive phases last months. That’s a pretty interesting pattern, why aren’t they more symmetrical? Perhaps it is because the limited resource takes longer to regenerate than it does for the manic phase to burn through it.)
Another possibility is that bipolar disorder isn’t a malfunction in our motivational system, it’s a disorder in some other system that’s connected to motivation. Think about the circadian rhythm. This is a system that, roughly speaking, drives us to be active during the day and sleep during the night. If something were to happen to our circadian rhythm — if the daytime highs were dizzyingly high and the nighttime lows were crushingly low, if the cycle were an awkward 108 hours long instead of a nice 24 hours long — that might also look a lot like bipolar disorder.
One thing we’d like to explain is why you get the weird pattern of opposite symptoms in depression, where (for example) one person eats too much and another person eats too little. Most diseases don’t cause their own opposite symptoms.
This problem seems like it has something to do with the balance between governors that come in pairs and watch two ends of the same variable. In the undereating/overeating example, that would be the balance between the hunger governor (“make sure to eat enough”) and the satiety governor (“but don’t eat too much!”).
But if you turn down the errors on both of these governors equally, they should remain in perfect balance. So turning down all governors by a flat amount shouldn’t cause this kind of symptom, we shouldn’t see this weird pattern.
No actual problem is ever so clean. Let’s go back to our pump metaphor. We can say things like “everything is turned down by 40%”, but in practice if 10 different motors are all supplied with lubricant from the same pump, and that pump gets jammed and starts working at only 40% capacity, some of the motors will be worse off than others. Motors that are further away from the pump, and have longer tubes, will probably be worse off. Motors close to the pump will be better off. Some of the closer motors might even keep functioning as normal. At 40% capacity, clogs may form in the lines, but they will form in some lines and not in others.
The point is, in any kind of real nuts-and-bolts system, a 40% loss of capacity won’t lead to a performance drop of exactly 40% in all parts. Some parts will be more affected than others. So even if you take a general hit to something that supplies all your governors, it might still affect your hunger governor more than your satiety governor, leading you to undereat. It might sometimes turn governors down and other times turn them up, leading to either insomnia or sleeping too much.
But there are other ways to get this pattern too.
First of all, this could have something to do with the weights on the governors, what we think of as personality. We’ve been assuming that any change will be a percent of the original signal (if the original signal was 10, and there’s a flat reduction to 30%, the new signal will be 3), but if every governor is cut down by a flat amount instead, then the balance between the governors will end up somewhat different than it was before.
Let’s say two people have a malfunction with their error-generation systems, but instead of reducing all errors by 50%, this malfunction reduces the weights on all their governors by 0.6 across the board.
One guy has a starting hunger weight of 1.3 and a starting satiety weight of 0.8. After being reduced by 0.6, his new hunger weight is 0.7 and his new satiety weight is only 0.2. His drive to eat was always a bit more powerful than his drive to stop eating. But the relative strength of this relationship has changed enormously. Now, his satiety governor is barely active at all. He is definitely at risk of eating too much, the satiety signals just don’t come through like they are supposed to. So this guy gets one pattern of symptoms: overeating.
The other guy is the exact opposite, a starting hunger weight of 0.8 and a starting satiety weight of 1.3. After depression, his new hunger weight is 0.2 and his new satiety weight is 0.7. His drive to eat was never very powerful, but now it’s almost nonexistent. He will definitely end up with a different symptom: not eating enough. The hunger signals just don’t come through like they are supposed to.
If this is one way to become depressed, then the symptoms of this kind of depression, especially the asymmetric symptoms, will tend to be more extreme versions of someone’s normal personality traits. When they become depressed, someone who has always had some trouble falling asleep will get the symptom of insomnia — while someone who has always had some trouble waking up will get the symptom of oversleeping.
Finally, we’d like to note that everything in the world is at least a little bit random. If you have a general problem with the cybernetic governors in your brain, odds are that some of them will be more affected than others, for no particular reason.
For people with this kind of malfunction, since some of their emotions are functioning correctly, they can still sometimes correct their errors. When only some governors are affected, you will experience some happiness, though usually less than before, so this is often hard to diagnose as depression. If one governor is particularly knocked out, then it may be diagnosed as something else — like if your sleep governor is particularly suppressed, it might be diagnosed as insomnia.
There’s some set of mechanisms that generate, transmit, and assign votes. Like anything else, these can break or get jammed.
Any malfunction that makes a person get fewer votes than normal will lead them to take less action. Any malfunction that makes someone get almost no votes will lead them to take almost no actions. This is basically the same as the malfunctions in error generation described above: the person will take fewer actions and generate less happiness than usual when they do.
A particularly interesting part of the selector is the gate. Remember that the gate has a threshold for a minimum amount of votes, and it suppresses votes for actions below this total to keep us from dithering or wasting resources. To use some arbitrary numbers for the sake of illustration, the gate might have a threshold of 5, meaning that when an action gets 5 votes or less, the gate clips that to zero, and the action gets effectively no support at all.
Any malfunction that raises someone’s gate threshold a bit will lead them to take fewer actions, because actions that would normally pass the threshold will no longer be able to clear it. Any malfunction that raises someone’s gate threshold a lot will lead them to take almost no actions, once the threshold is so high that almost no action can afford it.
Since action is needed to correct most of your errors, and correcting your errors is the source of happiness, if one of these malfunctions happens in your head, you will get less happy. Again, this looks like depression.
On the other hand, malfunctions that lower your gate threshold, so that actions can be performed even when they don’t get very many votes, will lead you to take more actions. In particular, you will have more of a bias towards action, because any small discomfort will more easily translate into doing something about it.
Malfunctions in these systems are a little hard to talk about, because they cause very similar behavior as the malfunctions in generating errors, described just above. In both cases, people will take less and less action, becoming sad and unmotivated. So they will be very hard to tell apart.
The most likely difference is subjective. When the mechanisms that generate errors are malfunctioning, you get weaker errors, or no errors at all. Since errors are emotions, these people feel no emotions — no hunger, no thirst, no pain, no loneliness, etc.
But with malfunctions in voting or in the gate, emotions / error signals are being generated as normal. It’s just that the governor never gets the votes it needs to correct them, or the votes can’t get through. These people experience all their emotions just as strong as ever, but cannot take action to correct them.
If the malfunction is severe enough, this would present a lot like bedrot — the person can’t get out of bed, they barely eat or sleep, etc. But subjectively it is very different. With bedrot caused by a malfunction where your governors can’t generate their normal error signals, you don’t do anything, but you also don’t feel anything. With bedrot caused by a malfunction in voting or gating, you still feel hungry, tired, thirsty, gross, etc. as much as ever, but you’re trapped and cannot bring yourself to take even the smallest action to help yourself.
A governor needs to be able to act on the world to succeed, but it also needs to be able to recognize success when success arrives. If you succeed and you don’t remember it, then you can’t benefit from the experience. So any kind of malfunction in the learning process can make behavior get very strange.
Consider what would happen if each experience were only recorded as a fraction of its true value. You go out with friends and find that this reduces loneliness by 10 points. But through some strange error, it is experienced or recorded as reducing loneliness by only 1 point. This gives you a very skewed view of whether or not you should go out with friends, when you are trying to decide what to do in the future.
If this happens across the board, then every action will gradually but consistently get underestimated. Over time, your governors learn to estimate all behaviors as being only 10% as effective as they really are.
This wouldn’t be such a big deal, except that all actions have costs. Forget about going out in the cold to see friends. It used to be 5 points of effort to drive to the bar, plus one point because the cold governor is always voting against it, but seeing your friends reduces loneliness by 10 points, so it was worth net 4 points on average. But now that “seeing friends” is estimated at only 1 point, it’s no longer calculated to be worth it! This guy will sit at home and feel bad, wanting to do something, but feeling like nothing is worth the effort.
In this kind of depression, you mistakenly believe that no action is worth the effort it would take, so you end up sad, because you choose not to do anything. This choice makes a certain internal sense — according to your best recordkeeping, choosing to do things isn’t worth it. From an outside perspective we know what is happening — your recordkeeping is all wrong! But it’s not obvious from the inside.
If you did take those actions, you would find that they create happiness as normal. Nothing is wrong with your ability to experience things. But you still wouldn’t learn from that experience, because there is something wrong with your memory.
Other malfunctions in the machinery of learning and memory would have similar effects. For example, if values were stored properly (“let’s write this down, eating a burger makes me 50 points less hungry”) but retrieved improperly (“hmmm, according to my notes a burger only corrects 5 points”), the effect would be almost the same.
And this may not be a complete list.
[Next: ANXIETY]
