2026-05-04 05:35:42
Large language models are really large. They’re among the largest machine learning projects ever, and set to be (perhaps already are by some measures) some of the largest computing and even largest infrastructure projects ever.
But how did LMs actually get so large as to warrant the title ‘large language model (LLM)’? A large part of the answer is in the P ('pretrained') and the T ('transformer') of GPT.
This is part 1 of a series about LLM architecture and some implications, past and future, for reasoning. Part 1 is ‘how we got here’ — what was so impactful about the transformer architecture for LLMs. Some readers may prefer to skip this. Part 2 will point at an unexpected benefit — the surprising explainability of contemporary AI reasoning — and why new trends might erode that. It’s a novel point as far as I know.
In 2016, early deep learning practitioner Yann LeCun introduced a famous analogy for learning intelligent systems:
LeCun’s cake analogy (‘unsupervised’, ‘predictive’, and ‘self-supervised’ are used somewhat interchangeably)
If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing on the cake is supervised learning, and the cherry on the cake is reinforcement learning (RL).
This correctly described the sometime prevailing practice for training LLMs, before the first LLMs were created! (Though by 2025, certainly 2026, this has become outdated — more on that later.)
What does this mean, and how did LeCun come to this position?
Machine learning, especially deep learning, needs example data.
You might think to start (and historically, researchers mostly did) by curating labelled examples: ‘this image is a cat, this one a dog; the French for “a bed” is “un lit”; …’. The machine learns these labels, and (hopefully) also learns how to generalise correctly, responding appropriately to similar-enough previously-unseen cases — which is what you actually want.
This supervised approach can be excellent: you get (ideally) reasoned, expert judgement as the sole curriculum for your nascent learning machine. But this is also painfully expensive, because you have to pay or otherwise entice humans to look over all your examples (or worse yet, create gold standard examples from scratch)!
Fortunately for developers, there is a stupendous amount of (language and other) data on the internet. That’s one very big reason, if you can, to pursue self-supervised training targets for your machine learning project: rather than ‘here is my carefully (and expensively!) curated dataset of labelled examples, learn them’, it’s ‘here is a gigantic pile of text, learn to predict whatever comes next’.[1]
As in supervised learning, where the goal is not simply to memorise the provided examples, the eventual goal of self-supervised learning is rarely simply to teach the machine to carry out this specific defined prediction activity — rather, in the process of learning how to do that, the machine is forced to learn generalisable concepts and features which can then be turned to a wide range of tasks.
More on RL (the cherry) later.
Language comes in long sequences[2]. So language learning systems need to be able to consume sequences. Consider: a large part of how we track what’s going on in a play, a textbook, an argument, a novel, and so on, is by reference and by recollection of context far back in the sequence: a previous dialogue, an earlier concept or experiment, a foundational point of information, a character’s exposition and background. The foreshadowing of a Chekhov’s gun can’t be understood if you’ve lost the plot in the meanwhile! So too with machine language models.
For the ‘predict what comes next’ self-supervised target, that looks like consuming in principle everything that’s come so far, then using that context to output a distribution over possible continuations.
Chapter 1:... The old revolver sat on the kitchen table…
…
Chapter 20:... Alice had run out of patience. Without warning, she fired the [gun/chef/kiln/pigeon]
Which one is right? Well, actually it depends a lot on what happens in the intervening scenes I’ve skipped! But assuming the narrative promise made in scene 1 is kept (and no other overriding promises are made in between), a good prediction is ‘gun’. On the other hand, if scene 1 instead introduced a sous-chef with a hygiene problem, a pottery studio with a design dispute, or an experimental pigeon-launching apparatus, the answers might be different.
The especially useful property of ‘what comes next’ self-supervision is that you can run this test on every single position. So a reasonable-length text might give you thousands or more such training examples… some easier than others.
‘Simply predicting what comes next’ clearly necessitates tracking a lot about what’s going on (across all kinds of scenarios) and how things work… at least if you want to predict well.
The benefit of self-supervised learning — a huge, largely automatically-generated collection of training targets (‘predict what comes next’) — also raises its own challenges. If much of the capability of your system arises from the sheer quantity of these training targets, you run into computational challenges: you want to be pumping more and more of these examples through your machine, but each example takes some number crunching, which costs compute time. Crucially, if you can crunch through more examples at once in parallel, you’re in a far better spot.
We saw how text prediction is a sequence problem. Naively, this means that, to make the prediction at the last position in the text, your process has to first read the first token, then the second, the third, and so on. To make the guess at the last position, you’ve had to ‘wait’ for absolutely all of the rest to be done processing. At least, that’s how humans read, and it’s how neural network machine language models did their processing until 2017[3].
It’s implemented as a ‘message’ (neural activation) passed forward from position to position (which needs to capture what might be relevant to ‘remember’), a structure known as a recurrent network. (Messages are additionally passed ‘depthwise’ in deep neural networks.) The nth position needs n steps to proceed before it can compute anything. For long texts, that’s crippling.
When later positions have to wait for earlier computations, it really adds up for long texts.
The milestone transformer architecture, introduced in the 2017 paper Attention Is All You Need, totally upended this constraint, making far larger-scale self-supervised training practical. Neural attention mechanisms were introduced long before this paper — the real innovation is to drop the recurrent connections entirely. You Don’t Need Recurrent Connections is the logical corollary! — and is what enables the entirely more scalable training of these architectures on long sequences. Attention still passes messages forward, but never only forward, always ‘diagonally’: making depthwise steps through the structure at the same time. This means no position, however long the text, need wait for any previous positions to compute. It does demand a highly parallel computation, but modern compute resources are nothing if not highly parallel.
By doing away with all fully recurrent message pathways, attention-only processing can proceed absurdly faster on long texts. (Both diagrams created with claude.ai)
Self-supervised training targets are great for cheaply (low human effort!) learning all kinds of features and concepts from big data. The well self-supervised neural network now ‘gets what is going on’ in all kinds of texts, well enough to sensibly predict what might come next, even on new content.
Sometimes that kind of prediction is already exactly what you want, but usually there are other things you’d like your neural network to do. Enter transfer learning, any number of approaches to tapping into these learnings to accomplish the actual task of interest.
For language models, a common basic approach is arranging for the model to ‘predict’ the responses of a helpful ASSISTANT character, in a dialogue with a USER character (played by you).
The following is a dialogue between a computer USER and a helpful, knowledgeable ASSISTANT.
USER: Which is better, a Chekhov’s gun or a Maxim gun?
ASSISTANT:
Because the assistant in this context is plausibly actually helpful and knowledgeable, there’s some reasonable chance that a well-trained language model produces a helpful answer here with no further tweaks. Of course this is flimsy — hallucination of plausible-but-incorrect responses or provision of unhelpful responses is common, or even entirely off-track diversions, like introducing new ‘characters’ or pivoting away from the established playscript structure.
Various approaches make more involved attempts to effectively make use of the knowledge acquired in self-supervised pretraining. Those which involve further training, not just prompting, are often called ‘post-training’.
Recall LeCun’s cake: with self-supervised (‘unsupervised’) the cake, supervised learning is the icing. A common pattern for LLMs intended for chatbot use is to have humans generate examples of how the ASSISTANT should respond to various queries. Should it ask clarifying questions sometimes? How verbose should it be? Should it provide references? Should it respond in first person? Should it respond in the register and dialect of the USER, or have its own? With training examples provided, sometimes principles like these can be learned and generalised.
Because almost all of the knowledge, language understanding, common sense, and character information comes from self-supervised pretraining, supervised learning as fine-tuning can get away with radically fewer examples than would be needed to train this way from scratch. Less human effort. Icing indeed!
LeCun’s cake analogy underestimates and misunderstands reinforcement learning — more on that in part 2. But for a few years, his relegation of RL to a final (but important) stylish flourish was a fairly good description of how LLM-based AI assistants were trained.
Rather than getting human contributors to produce exemplary data (as in supervised learning), RL has human engineers specifying how to grade outputs as good or bad, and sets the machine up to explore possible behaviours. Those outputs get graded, and the training encourages more of whatever computations produced the good ones (and less of whatever produced the bad ones).
This is very fraught, because accurately specifying what counts as good behaviour ahead of time is notoriously difficult, and if you’re not careful, exploratory AI will absolutely find the strange cases you didn’t think of. It can also fall completely flat if the AI in training has no idea how to achieve good outputs at all: it just gets stuck flailing.
This makes post-training of reasonably competent pretrained base models a sweet spot for RL: the model has enough background competence to at least occasionally succeed, can learn from success, and keep getting better and better. RL (whether as post-training or from scratch) also has the most potential to scale past human expert level, because we only need to be able to say which outcomes are better, not how to achieve them.
The earliest widespread application of RL to LLM post-training was in reinforcement learning from human feedback (RLHF). At its core, this asks humans to rate which answers are better or worse, then has the machine try out various ways of achieving better and better answers (according to the human raters). Typically this is repeated for a few rounds. In 2023-4, this is how companies got AI chatbots to mostly respond politely, usually refuse to describe how to make bombs, and often stick to character. It’s also a great way to train AI to be obsequious, disingenuous, flattering, and sycophantic. Oops!
In frontier AI systems, reinforcement learning is no longer a lightweight (albeit effective) post-training flourish on top of a primarily self-supervised pretrained LM. Since late 2024, reaching for expert-and-beyond capabilities has brought RL back into the spotlight. That has a few implications, one of which — reasoning and its transparency (or otherwise) — we’ll look at in the next post.
More generally, self-supervised targets take existing data from the target domain (images, language, code, …) and, in various ways, corrupt or distort it, tasking the machine with recovering the original as accurately as possible from the clues in context. For language, ‘here is a text fragment; predict what comes next’ is a natural version of this, as is ‘here is a fragment of censored text; fill the blanks’. In the image domain, images might be censored in patches, pixellated, or otherwise distorted.
Other rich and important formats also have at least one open-ended dimension to them: audio, video, computer code (just a kind of language), DNA, drone and vehicle control logs, robotic action sequences, … So machine learning approaches for this range of formats can benefit from similar architectures.
I’m fairly sure of this, at least I’m not aware of approaches which escaped this sequencing constraint. There’s a curious 2016 post by Google researchers which looks at various modified approaches — notably including attention mechanisms, which are what the later transformer architecture are centred on — but in all cases based on a fully recurrent backbone. Some hierarchical sequence processing approaches use convolutions instead of recurrence, but this too requires scanning the sequence in order to process later elements.
2026-05-04 04:24:58
Hi!
Recently, I immersed myself in researching the possibility of a general formal definition of "agency".
More specifically, I’m interested in formalizations that could support an operational definition of agency across domains. I’m looking for something that captures what is common between entities that intuitively seem agentic to us in very different parts of the world, and that could, at least in principle, let us detect such entities automatically in real systems.
So far, many definitions of agency I’ve seen quickly rely on terms like “goal”, “intention”, “belief”, “choice”, “sensor”, “actuator”, and so on. These can be useful in a particular context, but they are less satisfying if I want to take an arbitrary dynamical system and ask: “is there anything agent-like here, and can we detect it without hand-labeling the agent in advance?”
So my short request is: if you know good papers that formalize agency, or important aspects of agency, at a fairly general level? I’d be very grateful for pointers.
As a starting point, I’ve been thinking within the embedded agency framing. So far, I have found two papers that seem very valuable because they formalize some “component parts” of agency.
The first one is “Semantic information, autonomous agency, and nonequilibrium statistical physics” (2018). What I found useful there is the idea that, for many agents, maintaining viability is an implicit instrumental requirement, regardless of the particular utility function we might later ascribe to them. And that we can use this feature to detect systems via their information content, because, to sustain itself, a system should use its internal information about the environment. This paper attempts to formalize the calculation of such information that a system has about the environment, in a statistical sense, and to detect the part of that information that is causally necessary for the system to maintain its existence over time — “semantic information”, in the authors’ terminology.
The authors describe a way to formalize both the amount of such information and the part of it that is actually necessary for viability. I especially liked their suggestion that this could lead to an automatic detector of agents: roughly, by searching for system/environment decompositions in which the “system” contains a high concentration of semantic information relevant to its own viability.
The second paper is “Towards information based spatiotemporal patterns as a foundation for agent representation in dynamical systems” (2016). The authors discuss the tracking problem: if an agent metabolizes, moves, and can appear differently across different trajectories, then, obviously, it cannot simply be identified with a fixed set of particles or a fixed region of space.
They propose looking at so-called "integrated spatiotemporal patterns" — roughly, statistically integrated structures extended across space and time — and they try to formalize this in the setting of dynamic Bayesian networks. I found this interesting as an attempt to define “the same object” through its organization over time, rather than through the persistence of particular material parts. This gave me the intuition that, when looking for agents, it may be better to search for system boundaries by tracking a certain kind of specified/integrated information as it moves through different physical elements of the environment.
My current plan is to follow the citation graph around these papers, both forward and backward. But I’d be very grateful for recommendations that may lie outside that graph. I suspect many people here know this literature much better than I do.
I’m especially interested in work that tries to automatically find or track “agent-like” structures inside a world, and in papers that test such ideas in concrete environments: toy simulations, cellular automata, artificial life systems, game-like worlds, and so on.
Purely philosophical overviews without a formalism are probably less useful for my current purpose, in which an agent is already assumed to be an entity that has "observations", "actions", and "rewards", but these terms are not further reduced to something you can figure out how to compute. Such work can still be useful conceptually, but right now I’m mainly looking for mathematical tools for the question: “what makes a system agentic?”
If you know a paper, author, keyword, research direction, or even just “look at the bibliography around X”, I’d really appreciate it. It would be especially helpful if you could briefly say why the work is relevant — for example: “formalizes the boundary of an agent”, “deals with self-maintenance”, “addresses the tracking problem”, “defines a measure of individuality”, “connects agency with information flow”, etc.
Thanks in advance!
2026-05-04 03:20:49
How much do cows suffer in the production of milk? I can’t answer that; understanding animal experience is hard. But I can at least provide some facts about the conditions dairy cows live in, which might be useful to you in making your own assessment. My biggest conclusion is that cows made better choices than chickens by making their misery financially costly to farmers.
The life of a dairy cow starts as a calf. She is typically separated from her mother a few hours to a few days after birth and, to reduce disease risk, held in isolation. Cutting edge farms will sometimes house calves in pairs. This isolation is clearly stressful for a baby herd mammal and her mother, but I didn’t find any quantification of that stress that I trusted.
Calves will be bottlefed until weaning at 6-8 weeks (4-6 months earlier than beef calves). After weaning and vaccinations they can be introduced into a herd. At large farms (where most cows live), they will move in and out of different herds through their lifecycle. This is more stressful than being embedded with your friends for life, but again, I found no trustworthy quantification.
Heifers (unbred dairy cows) are first bred at 12-15 months, and calve 9 months later. After this, they will be on a ~yearly cycle of pregnancy, birth, and lactation (they will lactate through most of pregnancy). They will typically have 3 pregnancies before their milk production decreases and they are killed. An advantage cows have that egg chickens lack is that stress immediately and measurably decreases their economic productivity. This makes Cow Comfort big business, although if we can’t solve the sadist problem in nursing homes I don’t see how we can assume it’s solved in cows.
On average, ⅓ of dairy calves will go on to be dairy cows (herd size is constant or slightly shrinking in America despite increased demand, because productivity is increasing faster). ~2% will be raised for veal, some unknown percentage will be killed immediately, and most will be raised to be slaughtered for meat as adults. Early childhood stress probably hurts future meet cattles’ economic productivity as well, but because they switch farms while young, the farm raising them after birth has neither knowledge nor the incentive to minimize their stress.
My guess is that cows feel happiest in open pastures surrounded by their friends and sunshine and fresh grass. My judgement is already suspect, because one study showed cows prefer twilight to bright sunshine. But the same study showed they did really like to go outside, so let’s continue on that assumption.
It’s important to note that outside does not necessarily mean idyllic pasture. It can also mean a concrete exercise yard or feedlot. And no modern American dairy cow lives on grass alone, even if they live on a pasture. Their staggering milk production requires more calories than they could possibly get through grass alone.
Conventionally raised cows are not required to spend any time outside. Organic-certified cows must have at least 120 days of pasture time per year and year-round outdoor access of some kind. In practice, California cows get considerably more time outside than Wisconsin or New York cows, for the obvious reason.
The most recent data said that only 5% of lactating cows (which is almost all of them- cows lactate through most of their pregnancies) have access to pasture to graze. That paper was published in 2017 but it might be using older data. Given the wording it’s possible that the cows are given access during times without grass growth, but why would farmers do that? A 2007 study found that only 13% of cows had no outdoor access at all, and half had access to a pasture at least some of the time. So it looks like outdoor access is getting worse
You can buy milk with little stickers printed on it advertising a humaneness standard. What do these tell you about the cows’ outdoor life?
Free stall barns are “standard” (personal communication) in the US. These barns lower their walls in good weather, giving cows access to fresh air but not any more space. They average 100 sq ft per cow, including both personal stalls and common areas. Is that enough? I have no idea.
Some humaneness standards ban tie stalls, but they’re not common in the US even with conventional cows.
Disease and injury are both common sources of misery and canaries for mistreatment. So how sick are dairy cows?
Note that this graph means : n% of cows experience that issue in a lactation cycle (almost a year), not that n% of cows are experiencing this issue at any given time, which is how I initially read it.
Mastitis is more common that the other issues combined, so I dug in further.
The time between experiencing enough suffering to justify euthanasia and the actual euthanasia is probably the most intense physical suffering a cow will experience. It would be useful to know how long that delay is. Unfortunately no one has looked.
We can try to make some guesses based on the ratio of unassisted deaths to euthanasia. In 2013 the data on deaths is as follows (note: they’re using the technical definition of cows, meaning they’ve had at least one pregnancy):
Herd animals feel better in herds. How much do dairy cows get to indulge that instinct?
Milk production also creates a lot of by-catch in the form of newborns that aren’t needed in the dairy herd. Their lives should be considered part of the consequences of dairy. So what are their lives like?
A few obvious sources of suffering I haven’t covered are medical procedures like debudding (horn removal), removal of extra teats, and branding.
Thanks to Emma McAleavy and Abby ShalekBriski for sharing their insider knowledge.
Thanks to anonymous client for funding this research.
2026-05-04 02:04:53
Engineering Enigmas is simplified Tarot reading for working engineers.
Any time you feel stuck and need help in finding a way forward, open that web page and follow the advice given. You may have to be creative in how you interpret the advice in order for it to apply to your situation – that’s intentional.
Don’t refresh the page until you get advice that suits you better. Take what you are given and roll with it.
… wait, what? But why?
Humans like to be seen as either
That’s what we tell ourselves, but we are never that analytical or intuitive. In its place, we have used randomness to decide things since time immemorial. When we do it, we do it in a way that allows us to hide that it has happened – even from ourselves. Here are three of the ways.
Many of our decisions have no strong rationale either way.
A decision needs to be made, but among decent alternatives, we can’t really single out one clearly superior option. This means whatever we decide, it will ultimately be arbitrary.
Admitting to an arbitrary decision makes us look not-analytical and not-intuitive, so we look to others to validate our decision and lend it legitimacy. This is the reason we pay astrologers, witch-doctors, macroeconomists, and management consultants; they are all people that generate decisions with the rational capacity of a pair of dice, but we pretend they know what they are doing so their support turns an arbitrary decision into a legitimate one. Result: we are no worse off, but we sleep better at night.
These professions hide how clueless we are behind fancy words and speculative hypotheses. They serve a social function, rather than a technical one. The decision is just as arbitrary as before.
In section five of Scott Alexander’s review of The Secret of Our Success he quotes the author explaining how Naskapi foragers decide where to hunt for caribou:
Traditionally, Naskapi hunters decided where to go to hunt using divination and believed that the shoulder bones of caribou could point the way to success. To start the ritual, the shoulder blade was heated over hot coals in a way that caused patterns of cracks and burnt spots to form. This patterning was then read as a kind of map, which was held in a pre-specified orientation. The cracking patterns were (probably) essentially random from the point of view of hunting locations, since the outcomes depended on myriad details about the bone, fire, ambient temperature, and heating process. Thus, these divination rituals may have provided a crude randomizing device that helped hunters avoid their own decision-making biases.
This randomness serves much the same function as randomness in poker or rock-paper-scissors: in adversarial situations, deterministic and systematic strategies can be exploited. The caribou can learn how the Naskapi hunt – unless they randomise their decisions. Of course, they didn’t admit to this randomisation. The cracking pattern is a message from the deities.
Sometimes the astrologer (or management consultant, or heated bone) does not directly decide, but they inject enough randomness into the decision process to kick loose a few rocks and spark the creativity of the real decision-maker. (Weinberg talks a lot about this in his Secrets of Consulting book series, where he calls it jiggling the system.)
Instead of fighting against the arbitrariness and randomness involved in making decisions, we can save a lot of time and effort by realising it is there, and using it more actively. If I find it hard to decide something (e.g. between strawberry or vanilla ice cream) I glance down at my watch, and thanks to chronostasis (the neuro-optical phenomenon where when you first look at a clock, the second hand appears to stand still for longer than usual before getting moving again) it is trivial to see whether I did so when the second hand pointed at an even or odd number. If it was even, I go with the first choice, otherwise the second.
I use this method to decide on a lot of things. I don’t admit it often, because when people around me realise that’s what happened, they sigh and treat the decision as invalid and get re-stuck trying to decide again.
The reason it makes sense to embrace randomness is that however rational we are, we can never strategise down to the last detail. Chance and unknown factors play such a large part in determining the outcome that at some point, it’s better to stop strategising and just decide. Anything is better than nothing, and doing something will give us more information. This information is helpful for adjusting or making similar decisions in the future.
This is what Engineering Enigmas helps us do. A while back I read Xe Iaso’s Tarot for Hackers article, where she proposes drawing Tarot cards to help understand software systems. What I dislike about that approach is that it requires some familiarity with Tarot cards and their meaning, as well as how that meaning changes depending on location in the spread. We don’t have time for that! We just want to be kicked loose.
What we want is something closer to Eno and Schmidt’s Obliuqe Strategies. Inspired by that, I authored vague and widely applicable advice based on the meaning of each Tarot card. That’s what the tool shows you.
2026-05-03 23:09:30
Scott Alexander has a recent piece about "deontological bars" in the context of AI safety. He describes the state of the discourse like this:
I’ve been thinking about this lately because of an internal debate in the AI safety movement. Some people want to work with the least irresponsible AI labs, helping them “win” the “race” and hopefully do a better job creating superintelligence than their competitors. Others want to pause or ban AI research - the exact details vary from plan to plan, but assume they’ve already thought of and written hundred-page papers addressing your obvious objections. Different people have different opinions about which strategy is more likely to help, and it’s possible to coexist and pursue both at once. But in fact, both sides are a little nervous that the other is breaking a deontological bar.
Some of the people working on pause-AI regulations think there might be a deontologic bar against supporting AI companies. These companies are racing each other to create a potentially world-ending technology. If one company’s product has a 90% chance of ending the world, and another’s has an 80% chance of taking over the world, giving your money/support/encouragement to the 80%-ers seems kind of like endorsing evil. I don’t know if it was encouraged by this question exactly, but someone held a Twitter poll about whether you would become a concentration camp guard if you predicted you could get away with being only 90% as brutal as your average coworker. Taking the job would have good consequences, but is there a deontological bar in the way?
Some of the people working with the companies think there might be a deontologic bar against certain types of mass activism. The sorts of arguments that do well on LessWrong.com won’t give us landslide wins in national elections. That’s going to require things like working with Steve Bannon, working with Bernie Sanders, working with NIMBYs who hate data centers because they’re a thing that might be built in someone’s backyard (or by non-union labor), training TikTok influencers create short-form videos about the dangers of AI, and holding protests where we chant vapid slogans outside AI company headquarters. There are better and worse ways to do all these things, but once you lay out the welcome mat, you have limited control over who shows up - and every time someone tries to create the Peaceful Nonviolent Pause AI Movement Based On Peaceful Nonviolence For Peaceful Nonviolent People, it spends an inordinate amount of resources keeping out violent crazies who want to tag along.
My initial reaction upon reading this was that I don't see how there can be a general bar against either of these things. To the extent that there is some kind of bar, I feel like their needs to be additional details that are being assumed. The case that there is a deontological bar on supporting AI companies is in the context where you believe said companies have a reasonable chance of causing extreme harm, like human extinction. If the AI company in question was some random company that was using AI to help cute puppies rather than a frontier AI company I don't think many people would claim there is a deontological bar on supporting the company. Similarly, the bar on activism presumably assumes that the activism in question is or is likely to become dishonest or extreme in some way. In both cases there is an underlying belief about the nature of the action that is required for the existence of the deontological bar.
In my view, in order for an action of this type to be deontologically barred, the person taking the action (supporting the AI company, engaging in the aciivism) must have the beliefs that make the action barred. Unlike consequentialism, deontology often cars about the intent of the actor when they take an action, and I think that would apply to the types of deontological constraints that Alexander is discussing above. I can see how working for a company that is likely to cause great harm could be deontologically barred, but I think the person who is actually working with or supporting the company must believe that it is likely to cause that harm. Similarly, I can see how dishonest or extreme activism could be barred, but the activist must intend the dishonest or extreme acts. Note that this is different then saying that the activist believes that their actions are dishonest or extreme. In both cases, the actor need not believe their actions are barred for them to actually be barred, but they need to have beliefs such that there is the necessary intent. The AI lab support might need to believe that the lab they support has a relatively high chance of causing great harm, it isn't enough for it to be true that the lab has a high chance of causing great harm. By the same token, the activist might need to believe that their activism has a relatively high chance of leading to violence, it isn't enough for it to be true that their activism has a high chance of leading to violence. If this kind of intent is not required, it seems to me like these alleged deontological bars are simply consequentialism in disguise.
For consequentialism it mostly matters what is true (given that the statements here are statements about the probabilities of various consequences), but deontology cares about intent. As a result, when we evaluate deontological bars, I think these bars need to make references to the beliefs of the person taking the action in question. This distinction is particularly important if you want to accuse someone of violating a bar. It can be tempting to use your own beliefs about the situation, but in my view that is a mistake. If your real problem with someone is that they have incorrect beliefs about the nature or consequences of their actions, it's probably better to the just say that explicitly rather than accusing them of violating a deontological bar that only exists for those that have your beliefs.
2026-05-03 19:33:37
As pointed out in Where Mathematics Comes From (WMCF), we are born with an innate sense for numbers, which gets fuzzier very fast as the numbers grow bigger. We can also subitize collections, that is to say instantly determine the cardinality, of collections of up to 3 objects.
It is likely that children learn about bigger integers by playing with collections of objects, adding or subtracting objects from them, merging collections, and linking the resulting cardinalities to their innate number sense.
The authors of WMCF also remark that there is a correspondence between operations on collections and basic arithmetic operations. For example, merging a collection of cardinality 1 with a collection of cardinality 2 results in a collection of cardinality 3, which maps cleanly to the addition 1 + 2 = 3.
Now to my point, conceiving of numbers as the cardinality of collections of objects is reminiscent of, though not the same as, the definition of numbers seen in Zermelo-Fraenkel set theory (ZFC), where a number is the set of all smaller numbers, with 0 the empty set.
However, the formal ZFC definition is not the most intuitive. A more approachable way to conceive of numbers is as the cardinality of sets. And indeed, if we take a set of 1 fruit, another set of 2 fruits, then take their union, we end up with a set of 3 fruits, mirroring the behavior of real-world collections. A neat correspondence between collections of objects, sets, and arithmetic, right?
But there is an issue. Consider a water molecule H-O-H. Naturally you can subitize its elements and tell that it contains 3 atoms. However if you take the set of atoms in H-O-H: it is {H, O}! not {H, O, H}, and it has a cardinality of 2, which doesn't map to the 3 atoms of our water molecule, because elements of sets must be distinct. This uniqueness constraint is the issue: it breaks the mapping from collections of objects to sets.
To map cleanly to real-world collections, we need a mathematical object that preserves their properties, that is to say an object that allows the same element to occur multiple times. That mathematical object is called a multiset.
So, to summarize, it is likely that we learn numbers from the cardinality of collections of objects, and these collections of objects correspond not to sets, but to multisets. In other words, we don't learn numbers from set cardinality, but it is likely that we learn numbers from multiset cardinality.
You may be wondering if any of that matters. I argue it does because maths should be built on intuitive concepts as much as is sensible. This requirement arises from a very pragmatic concern. We can conceive of maths as a program that runs on our brain. We look at notation, compute results, and understand meanings. All things equal, the faster the math program runs, the better.
Our brain comes pre-equipped with modules that understand collections of objects and their cardinalities, i.e. multisets, not sets. So it follows that multisets are a more brain-native representation of numbers, and that thinking of maths as built in terms of multisets should be faster and more intuitive than thinking of maths as built in terms of sets, because we can offload some of the reasoning to our specialized object collection modules, without requiring extra processing steps to remove duplicated elements.
Taking a step back, the more general question is:
What are the mathematical objects that allow our brain to run math programs fast?
I expect the answer to point to mathematical objects that cleanly preserve the properties of real-world objects, for which our brain has built intuition, or in other words for which our brain has dedicated neuronal wiring that makes processing more efficient.
Above, I added the caveat "all things equal", by which I mean that there are other considerations before deciding to use multisets, such as whether mathematical foundations can be conveniently built out of multisets.
What I gather from a cursory search is that it is convenient enough, as evidenced by the Wayne paper cited below that "develops a first-order two-sorted theory MST for multisets that "contains" classical set theory."
For an introduction to the mathematics of multisets, I highly recommend the extremely pedagogical video A multiset approach to arithmetic | Math Foundations 227 | N J Wildberger.
Or for a more rigorous approach defining multiset theory you can refer to Multiset theory or to The development of multiset theory for a survey of different multiset theories and their usage, both by Wayne Blizard.