2026-04-19 14:55:44
Written very quickly for the Inkhaven Residency.
Yesterday, I started a post-mortem on my earliest published machine learning research paper “The Assistive Multi-Armed Bandit”, by providing the context for my paper as well as the timeline of the project. Today, I’ll summarize and review the actual paper.
As noted in the abstract and introduction, a commonly-studied problem in many fields is preference learning: inferring what people want by observing their behavior. For both philosophical and practical reasons, it’s pretty common to infer preferences by assuming that people are (close to) rational. The paper tries to formalize a particular way in which this rationality assumption fails: people often do not know what their preferences are ahead of time, and observations of behavior often occur when people are still learning what they want.
The formalism studied in the paper is probably the simplest version of an assistance game where a human is learning about their rewards. In this setup, there are N possible actions, each with their own independent distribution of rewards. At each step, the human suggests an action, the robot takes a (possibly different) action, and then the human gets to observe the results of each action. (In section V.E we consider alternative setups – one where the human and robot alternate turns, and the other in which the robot can choose between acting themselves and allowing the human to act). As the value of the actions stays constant over time, this is a multi-armed bandit setting.
Caption: Figure 1 from the assistive bandits paper, which introduces the formalism. As an aside, “A” was used for the robot (“agent”, “actor”, or “AI”) instead of “R”, to avoid confusing the robot and its actions with the reward observed by the human r.
We then prove some theoretical results about this setting. First, proposition 1 is a rather trivial proposition, that basically says that if the human knows the expected value of each action almost always suggests the best action, then the robot can do quite well (achieve finite regret) by just taking the action the human suggests the most often. I see this as an interesting exercise mainly to check that we understand the setting: inferring the optimal action is easy when the person almost always suggests the optimal action. The difficulty of the assistive bandits framing comes entirely from the fact that the human is still learning about their preferences.
Proposition 2 shows that if the human implements a policy where they take the action with the highest observed average value most often, then with robot assistance, the pair will eventually converge to taking the action with the highest frequency (that is, the policy is consistent). This is done by making the robot explore for the human at a carefully chosen rate, such that each action is taken infinite times, but where eventually the probability of exploring at each given time step decays to 0. This is a nice implementation of ideas from multi-armed bandit theory.
Proposition 3 follows somewhat non-trivially from proposition 1 (consistency is weaker than finite regret). But it makes sense that I’d state it separately, since the proof is much easier.
Proposition 4 is the main result of the paper: it is indeed important to model learning, in that there are situations where not doing so leads to bad outcomes even in the limit, because the robot that doesn’t respect the human’s learning process may prevent the human from learning about their preferences.
Proposition 5 shows that there are simple human policies that are informative, that do better than “optimal” policies. That is, you can be better in collaborative games by informing others of your preferences, rather than by doing the best thing in isolation. The paper claims that this is “surprising”, but I felt this was pretty obvious.
Proposition 6 and corollary 7 feels the most proof-of-worky to me. They basically expand on proposition 5, by arguing that in this setting (via some somewhat intricate math), if the robot is to assist the human, the human must communicate sufficient information about the optimal arm.
The paper then uses deep reinforcement learning + recurrent neural networks to solve for robot policies in specific versions of the assistive bandit problem. Using this technique, the paper basically confirms all of the theoretical results in their experimental setting.
Finally, in V.E, the paper considers other variants of the assistive bandit problem.
The paper concludes with obvious limitations and future work: the human policies are unrealistic, as is the simplicity of the environmental setup (that is, there is no environment).
It’s hard not to be too critical toward my past self, but I’ll make an honest attempt to evaluate the paper.
I think, as a “first paper”, it was pretty okay but not brilliant. I still think that the core idea behind the paper is good: it is indeed quite important to model human suboptimality when trying to infer preferences from observed behavior. I also think the specific form of suboptimality studied in the paper – that people often can only access their preferences by experiencing them, rather than having full introspective access to their preferences at the start – is obviously real. The paper also shows that, at the time, I had a decent level of familiarity with both multi-armed bandit theory and deep reinforcement learning.
In this sense, the paper does indeed study the simplest setting with this form of irrationality, where all of the complexity of the setting comes from the fact the human is still learning about their preferences. I worry that, by studying such a restrictive setting, the only conclusions
I also think, with the exception of some nitpicks, the writing of the paper is quite good: every proposition is set up and explained. I also like the parallel structure between the theory and the experiments. The graphic design was also quite consistently clean.
The main issue I have with the paper is that the setting may be so simplified as to be uninteresting. Specifically, in my opinion, the main contributions of the paper (if anything) are proposition 4 – when the human is learning about their preferences, it’s important to model the learning – and arguably the use of deep RL to solve the complex POMDP. I feel like most of the other parts of the paper weren’t great, and insofar as they provided value they were proof of competency/proof of work so that these contributions would be taken seriously.
A related problem is that, even if we were to accept that people are learning about their preferences, it’s not at all obvious how to incorporate it into actual algorithms. Notably, the paper completely sidesteps this by using black-box optimization to solve the setting.
The paper would be a lot stronger if the other theorems and a bunch of the space dedicated to experiments were replaced with a simple application of these ideas with real human data (even if in a toy setting). This would also help address the question of how to actually use this insight in practice.
As is, my evaluation of the paper would depend a lot on the novelty of the main contribution (“it can be important to model human learning”) at the time it was written. Based on my recollection, at least in the circles I was writing for, this was a novel insight. But I also think it was not that useful (modelling human learning explicitly is hard, so much so that
From reading the paper, here are some more minor issues that nonetheless stood out to me:
2026-04-19 13:10:21
In this post, I will try and outline the arguments for stopping AI.1 I’m not going to argue for them in detail. I’m just gonna try and get the most important points out on paper. We can fill in the details later.
The first thing to understand is that AI is not chatbots. AI is a general-purpose technology that can be trained to do everything humans can, and more. Right now, AI is mostly used to control computers. But robots are getting better all the time. Robots are the general-purpose hardware, AI is the general-purpose software. Put them together and you get machines that can do all the things.
AI is also advancing extremely quickly, repeatedly exceeding expert’s expectations. AI is already super-human in many ways, but not all ways. By the time it is better than humans across the board, it will be vastly better in some ways. AI can already think much faster than a human and has much broader knowledge than any individual person.
What will happen when AI is super-human in every regard, including not just IQ, but street smarts, emotional intelligence, intuition, physical grace, social maneuvering and politics, charisma, and so on? We don’t know, but if we are going to become a “second-class species” that’s obviously incredibly concerning. AI might literally lead to human extinction, in the same way that humans have caused many other species to go extinct.
Experts such as myself are doing just about everything they can to sound the alarm about the risk of human extinction. It’s not just a general uneasiness about something becoming smarter than us. Today’s AI systems “go rogue” and disobey commands, and we don’t know how to make them not do that. Many who aren’t sounding the alarm are focused on researching this sort of problem; unfortunately, many of those people work at the AI companies and that’s part of why they’re not sounding the alarm, too.
If super-human AIs — especially robots — go rogue, we might not be able to stop them. We couldn’t just unplug them, like we can with computers. But again, even without controlling robots, AIs might manage to take over, e.g. by playing different humans against each other. And actually, the AI doesn’t even have to turn against us, or turn us against each other. Companies and countries (and individuals) are already against each other, and are giving AI more and more power, and sacrificing their values in the process, in order to “win”.
There are other massive risks that would be a big enough deal to warrant stopping AI, even if we ignore the whole risk of extinction thing, as people sometimes do. AI could take everyone’s jobs, and then we might struggle to get the basic resources we need to survive. AI could concentrate power in the hands of authoritarian leaders or AI companies. It could destroy democracy and our way of life.
You might object that we don’t know if any of these things are actually going to happen. That’s true, but that just means we shouldn’t risk it.
So, we need a plan for dealing with this. There are basically two kinds of plans: 1) Stay in control of AI2 as it becomes increasingly super-human and increasingly powerful, 2) Stop AI from getting too powerful in the first place. At the moment, there are no good plans of type (1), for staying in control. There are some OK ones, that might work, but nothing we can count on. So we need to stop AI, if we can. And we can, so we should.
I’ve talked a bit about how we can stop AI previously. I’ll say more about why the other plans don’t work in a future post.
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.
i.e. instituting an indefinite global pause
If we’re going with this sort of plan, there’s the additional questions of who should be in control, and what that even means and so on (Am I in control if I just follow my AI advisor’s advice all the time?). But that’s out of scope for this post.
2026-04-19 13:07:45
It seems that AI safety is at least partly bottlenecked by a lack of orgs. To help address that, we’ve added a page to AISafety.com aimed at lowering the friction for starting one: AISafety.com/founders.
This page was built largely as the result of a suggestion from @Ryan Kidd, who found he was frequently sharing the same set of resources with potential founders and realised it would be useful to have something similar publicly available.
It lists:
As with all resources on AISafety.com, we put substantial bandwidth into making sure the information on this page is accurate and up to date. If you have any feedback, please let us know in the comments or via the Suggest buttons on the page.
This is the 11th resource page on the site. Here's the full list:
2026-04-19 11:45:37
Epistemic status: Completely schizo galaxy-brained theory
Lightcone[1] operates on a "generalist" philosophy. Most of our full-time staff have the title "generalist", and in any given year they work on a wide variety of tasks — from software development on the LessWrong codebase to fixing an overflowing toilet at Lighthaven, our 30,000 sq. ft. campus.
One of our core rules is that you should not delegate a task you don't know how to perform yourself. This is a very intense rule and has lots of implications about how we operate, so I've spent a lot of time watching people learn things they didn't previously know how to do.
My overall observation (and why we have the rule) is that smart people can learn almost anything. Across a wide range of tasks, most of the variance in performance is explained by general intelligence (foremost) and conscientiousness (secondmost), not expertise. Of course, if you compare yourself to someone who's done a task thousands of times you'll lag behind for a while — but people plateau surprisingly quickly. Having worked with experts across many industries, and having dabbled in the literature around skill transfer and training, there seems to be little difference within an industry between someone four years in and someone twenty years in, once you control for intelligence and conscientiousness.
But sometimes someone on my team does actually truly struggle to get better at a task, even if they are smart. Or I notice that if I were to try to get them to do something, they would have no idea how to even get started unless they spent at the very least multiple months, if not multiple years, acquiring the foundations necessary to do so.
So the question becomes: What determines whether someone is capable of relatively quickly acquiring expert-level performance in domains ranging from preparing a legal defense, to preparing an architectural plan, to physically renovating a bathroom, to programming a conference schedule app?
And my current, schizo galaxy-brained theory is that there are exactly 4 skills:
If you are good at any task in any of those categories, you can become expert-level within 6 months at any other task in the same category.
Now why these exact 4 skills?
IDK, it kind of fits the data I've observed. But here is roughly how I came to believe what I believe:
First: across all tasks, performance correlates highly with general intelligence, and this dominates everything else. But clearly there's non-trivial variance left after controlling for it.
Then, there's an obvious divide between STEM and the humanities. Ask someone with a legal, history, or non-analytic-philosophy background to learn programming and mostly they bounce off or expect a multi-year training journey. Ask someone with a STEM degree to learn programming and it goes pretty well even if they've never programmed before.
Similarly, when I talk to people with a legal or humanities background and ask them about complicated frontend design decisions, they usually give surprisingly good input! They will pretty quickly jump into the fray of trying to model the user, figure out what a good product or information ontology, and have a sense of style about its presentation.
So that's it. There are exactly two skills. "Technical skills" and "Design Skills".
Then I tried to manage people. That... didn't go so well. Not only that, when I tried to get people on my team to manage other people, they also sucked at it!
So I learned that if I want to predict who will be good at management, I need to pay attention to whether they've managed other people before, and expect many months of practice until they are decent at it. Maybe it's a completely new cognitive domain, maybe it's just a domain where skill transfer is very hard and feedback loops are very slow and so it just takes everyone a while to learn the basic lessons, but nevertheless, if I want to predict performance at Lightcone, I gotta model people's management skills separately.
And then I tried to renovate a hotel.
And while the people on my team really ended up surprisingly good at a very wide range of tasks associated with construction and construction management, it also became clear that no one on my team would be able to perform the actual labor that our general contractors were able to perform. And also that they would totally smoke us in any sports competition. And that if I wanted to get someone on my team involved in the daily construction work, I sure expect that they would need many months of getting into shape and developing the right kind of physical skills.
So 4 skills it is.
Now, am I confident I have seen all skills there are in the world, such that no additional cluster will arise? Actually, yeah, kind of.
I have been walking through the world trying to keep track of what kind of career many of my acquaintances and colleagues go into for something like the last 2-3 years, and haven't really noticed any big holes. I have also been actively trying to think about careers that currently seem off limits to someone who has basic expertise in these 4 skill domains, and I have so far not been able to find something. My guess is if there is something I am missing it will be in something less career oriented.[2]
Need someone to build a script that automates filling out some business forms?
Give your econ masters student 3 months to learn programming and he can do it.
Need someone to drive your marketing push?
Give your interior designer 2 months to figure it out.
Need someone to head your internal legal department, double check the work of your lawyers, and prepare your legal defense in a high stakes trial?
Give your very smart frontend designer 3 months and they will go toe-to-toe with your lawyers.
Want to promote an engineer who has never managed anyone before to a manager?
Well, you better strap in for a year or more of pain while they acquire this completely new skill domain and traumatize all your new interns while doing so.
Want to get your backend engineer who is not good at writing, and is not good at interior design, to start taking more charge over your frontend?
Expect them to suck for at least a year until they can start competing with the smart designers on your team.
Want to get your quant finance guy who has never worked on a big codebase to start writing maintainable code and make nice clean Pull Requests?
Well tough luck, predict many months of telling them that yes, it is actually important that anyone can read your code and figure out how to modify these abstractions you've created.
Want to get your philosophy grad student dropout who has never done physical labor in his life to start managing construction projects and get their hands dirty?
Expect at least a year of getting into shape and used to the work, if they don't bounce off completely (though many subtasks of construction can be done with pretty little physical alacrity).
Give it a try yourself!
(Unhappy with any of my classifications? Fight me in the comments!)
Is there any externally validated or scientific basis for any of this?
Yes! It's not like, total consensus in the field of psychometrics, but task performance being extremely g-loaded across a wide variety of tasks is very well supported. People can really learn a very wide range of skills if they are smart.
And then within intelligence, math tilt and verbal tilt tend to be commonly used abstractions in psychometric testing that are predictive of success in careers in STEM or humanities.[3] Math fits nicely onto the technical domain. Verbal fits nicely onto the design domain.
A generalized "physical skill" factor is also well-supported. First, enough high profile athletes have switched from being world class in one sport to being world class in another sport such that there must be substantial skill transfer for these domains to explain that outlier success.[4] Second, somewhat unsurprisingly, if you measure people's sports skill you will find a strong "General Motor Ability" factor that explains performance across a wide range of motor skills.[5]
On management? IDK, that one I haven't seen much support for, but it sure matches my experience. There is an emotional intelligence literature, but that construct adds extremely little on top of just general intelligence. My guess is it's just a task that's very important and has terrible feedback loops, so everyone needs to fail for a while before they get good at it, but who knows.
Design. Technical. Management. Physical skills. Long ago, the four nations lived together in harmony. Then, everything changed when the Management Nation attacked. Only the True Generalist, master of all four elements, could stop them, but when the world needed him most, he vanished.
the organization I run, and which runs the website you're reading
If there is a missing cluster, I can imagine it being some more "relational" skillset around doing high-quality emotional labor, or maybe something genuinely associated with age and wisdom where certain skill domains are just really hard to perform well in without being at least 35+ and having the associated life experience. But I don't currently think such a cluster exists, and that four is really the right number.
Claude lists: "Bo Jackson (NFL All-Pro + MLB All-Star). Deion Sanders (NFL HOF + MLB). Rebecca Romero (Olympic silver in rowing, then Olympic gold in cycling four years later — different disciplines entirely). Clara Hughes (Olympic medals in both speed skating and cycling). Rugby → NFL is a well-trodden path (Jarryd Hayne, Christian Wade)"
This research is a bit more controversial than I expected, but I don't really understand the controversy. There are definitely some people in the field who insist on there not being a strong general motor ability factor. IMO this study also points in the direction of there being a general motor ability.
2026-04-19 09:10:21
I was so excited about the first BIDA dance that I arrived two weeks early. I biked over from Medford to the Park Av Church in Arlington and was really disappointed to find the hall was empty. But I came back when the dance was actually happening, and it was fantastic.
It immediately became my favorite dance. I started volunteering, first out of frugality (volunteers get in free!) and then out of a sense of wanting to contribute, and in 2010 I joined the board. Over the past 16 years I've done just about everything at some point except treasurer, and now I'm stepping away.
It's not that I think BIDA is doing something wrong; quite the opposite! We're seeing record attendance, finances are good, so many fun dancers, and many people who want to pitch in. I noticed I would have been the seventh person running for three board spots, and realized it was a good time to let someone else have a turn. I'm excited to see what Emma, Harris, Bret, Veer, Casey, Naomi, Clara, and Persis do!
This seems like a good time to look back over how BIDA and the Boston dance community have changed over my time organizing.
The biggest change is that BIDA is now Boston's main contra dance. This is kind of hard for me to believe, since we spent so many years as a small dance that tried to fill niches that were not well covered by the many other area dances. We've gone from essentially not booking established bands to booking them regularly, and with our attendance-based bonuses are one of the best-paying dances in the country. I do really enjoy the higher level of musicianship now, but am also really glad Boston Open Contras exists (along with BIDA's open bands and family dance bands) to provide a lower-stakes environment.
The next largest change is probably the switch to gender-free calling (more history), and the level of role freedom that has come along with that. In 2010, I (and many others) would happily dance both roles, but if I was dancing the 'lady' role I had to be 100% on it because if anything went wrong it was my fault. Beginners were strongly discouraged from dancing 'switch', which also discouraged same-gender couples. And while this never happened to me in Boston, conservative men elsewhere would occasionally refuse any sort of physical contact if I encountered them in line while dancing 'lady'. When I look at the dancers now, it's amazing how people have really taken up this freedom to dance any role with any partner, which I feel really good about.
Some smaller changes:
BIDA went from 1x/month to 3x/month, most recently by adding a monthly afternoon dance. Since we take the hottest part of summer off, this means going from ~10 to ~28 dances annually.
We now have a dance weekend, Beantown Stomp. I kicked this off in May 2018, we had our first one in March 2019 and it's now an established and anticipated event that people fly to from across the country. I'm especially grateful for Naomi for taking the lead for 2023 (and beyond!) when I was too burnt out on organizing cancelled events (2020, 2021).
We have occasional family dances and livetronica (Spark in the Dark) events.
Our events are still intergenerational, but differently so. In 2010 most dancers were baby boomers; while BIDA was unusual in how many millennials we had, we were still 50%+ baby boomers. At this point I'd guess our dances are fewer than 10% baby boomers: many have aged out of dancing, and many millennial-and-younger dancers have joined. This is also reflected in the board's focus: the initial board was primarily mid-20s people thinking about how to get more 15-35yos dancing, but since we've succeeded at this it's no longer a focus.
We now schedule (and pay) hall managers. In 2010 we just expected most board members would be at most dances and this would give us enough coverage.
BIDA is a lot more organizationally mature. Minutes from the early days say things like "We agreed not to have a President. Instead, we'll use everyone in the board to make sure that we stay on top of things." This turned out not to work very well, and instead specific roles are in charge of staying on top of specific things, with the intraboard coordinator handling things by default.
We were still bouncing around between a few halls, and now we're always at the Cambridge Masonic Hall.
We're a legal entity now, incorporated as a Massachusetts non-profit.
We set up a safety policy, with a committee to handle issues as they come up.
There used to be a lot more of a mentoring focus. Early dances were often two experienced musicians plus a new musician. Callers would typically have a shadow. Every dance allowed sit-ins (off mic, behind the band). We hosted jams about as often as dances. I see this change as pretty natural, and I think a lot of this is now happening informally outside of BIDA.
Organizing BIDA has been a big part of my identity, but I think it's healthy for the organization to have people cycle through, and I'm confident it's in good hands. Very excited to start attending dances just as a dancer, with no formal responsibility!
Comment via: facebook
2026-04-19 09:09:29
A number of years ago Numberphile published a video on the behavior of higher dimensional spheres with the title “Strange Spheres in Higher Dimensions”. I'd recommend one watches the video before reading this post, in that video Matt Parker presents a construct where spheres are placed inside a cube pressing against its faces, in the remaining central volume an additional sphere is set with a radius so that it kisses the “padding spheres”, this build is then scaled to higher dimensions where Parker observes that the central sphere increases in size constantly until its surface goes through the face of the cube, an apparently impossible configuration. He then jokingly concludes that the only way to make sense of it is to imagine higher dimensional spheres as “spiky”, so that the spikes can go through the face while the rest of the volume remains confined inside the hypercube.
Recently, for some reason that same video ended up in my feed and I ended up watching it again, I was left rather dissatisfied with the conclusion. I am not a mathematician but I was always fascinated with higher dimensions and when it comes to geometry, I often found that seeing is believing.
Plotting sections of the construct can help us make sense of the problem, we’ll start with the simple 2d version, here the central sphere has a radius of
Fig. 1 not much to see here, everything is visible on the plane
The true solution of the “mystery” is understanding the diagonal of the cube in higher dimensions. The diagonal of the measure polytope in n dimensions is
The diagonal of a face also grows in a similar fashion, since it only has one less dimension that the hypercube, for example in the 9th dimension the diagonal of a face of a unit hypercube is
Understanding the 3d version of the problem is not too hard but to keep things simple I will show a 2d section only, to obtain this section we will slice the cube through the diagonal as shown in fig. 2.
Fig. 2 a render of the problem for n=3, the segmented line is the edge of the section
The highlighted section is shown in the next image, notice a gap appears in the center as not all of the padding spheres are touching each other. Also, the padding spheres appear to no longer be contacting all sides of the cube since some of the contact points do not lie in this section.
Fig. 3 The radius of the central sphere has grown, it is now
This method is quite useful, since the plane we chose passes through the centers of both the central and padding spheres, the sections of these objects will always appear as circles of radius 1 no matter how many dimensions are we working with. Now let us move to the 4th dimension:
Fig. 4 in 4d the central sphere has now the same radius of the padding spheres, 1
This pattern continues, as the diagonal of the face continues to grow so does the diameter of the central sphere, in 9 dimensions the sphere is now contacting the faces of the hypercube:
Fig. 5 The diagonal of a face is now almost 3 times the side
Due to symmetry the radius of the central sphere is always equal to the distance between the surface of a padding sphere and the closest corner. In 10 dimensions part of the central sphere is now escaping the inner volume entirely:
Fig. 6 The radius of the inner sphere is now
This pattern continues as n increases and more and more of the central sphere escapes the polytope. Higher dimensional spheres are not spiky but the faces of higher dimensional cubes have very large diagonals compared to the side through which the central sphere can escape.
