2025-12-10 10:17:32
Published on December 10, 2025 2:17 AM GMT
People working in the AI industry are making stupid amounts of money, and word on the street is that Anthropic is going to have some sort of liquidity event soon (for example possibly IPOing sometime next year). A lot of people working in AI are familiar with EA, and are intending to direct donations our way (if they haven't started already). People are starting to discuss what this might mean for their own personal donations and for the ecosystem, and this is encouraging to see.
It also has me thinking about 2022. Immediately before the FTX collapse, we were just starting to reckon, as a community, with the pretty significant vibe shift in EA that came from having a lot more money to throw around.
CitizenTen, in "The Vultures Are Circling" (April 2022), puts it this way:
The message is out. There’s easy money to be had. And the vultures are coming. On many internet circles, there’s been a worrying tone. “You should apply for [insert EA grant], all I had to do was pretend to care about x, and I got $$!” Or, “I’m not even an EA, but I can pretend, as getting a 10k grant is a good instrumental goal towards [insert-poor-life-goals-here]” Or, “Did you hear that a 16 year old got x amount of money? That’s ridiculous! I thought EA’s were supposed to be effective!” Or, “All you have to do is mouth the words community building and you get thrown bags of money.”
Basically, the sharp increase in rewards has led the number of people who are optimizing for the wrong thing to go up. Hello Goodhart. Instead of the intrinsically motivated EA, we’re beginning to get the resume padders, the career optimizers, and the type of person that cheats on the entry test for preschool in the hopes of getting their child into a better college. I’ve already heard of discord servers springing up centered around gaming the admission process for grants. And it’s not without reason. The Atlas Fellowship is offering a 50k, no strings attached scholarship. If you want people to throw out any hesitation around cheating the system, having a carrot that’s larger than most adult’s yearly income will do that.
Other highly upvoted posts from that era:
I wish FTX hadn't done fraud and collapsed for many reasons, but one feels especially salient currently: we never finished processing how abundant funding impacts a high-trust altruistic community. The conversation had barely started.
I would say that I'm worried about these dynamics emerging again, but there's something a little more complicated here. Ozy actually calls out a similar strand of dysfunction in (parts of) EA in early 2024:
Effective altruist culture ought to be about spending resources in the most efficient way possible to do good. Sure, sometimes the most efficient way to spend resources to do good doesn’t look frugal. I’ve long advocated for effective altruist charities paying their workers well more than average for nonprofits. And a wise investor might make 99 bets that don’t pay off to get one that pays big. But effective altruist culture should have a laser focus on getting the most we can out of every single dollar, because dollars are denominated in lives.
...
It’s cool and high-status to travel the world. It’s cool and high-status to go on adventures. It’s cool and high-status to spend time with famous and influential people. And, God help us, it’s cool and high-status to save the world.I think something like this is the root of a lot of discomfort with showy effective altruist spending. It’s not that yachting is expensive. It’s that if your idea of what effective altruists should be doing is yachting, a reasonable person might worry that you’ve lost the plot.
So these dynamics are not "emerging again". They haven't left. And I'm worried that they might get turbocharged when money comes knocking again.
2025-12-10 09:04:40
Published on December 10, 2025 1:04 AM GMT
I expect a lot of the benefit of the RAISE Act to come from the required safety and security protocols, but I wanted to get a sense of what scenarios people imagine when they think about the "Critical Harm" threshold. To my mind there are two main scenarios. In the first, a somewhat-strong AI is given more authority than it should be, fumbles, and is then contained. In the second, a genuinely strong AGI/ASI has decided killing its enemies is more important than being stealthy and humanity won't be able to contain it. A third scenario where an AGI launches a take-over attempt and fails seems unlikely to me. The first scenario seems like it will inevitably start happening soon and will happen every so often until the second scenario happens, but my timelines are short enough that the second scenario might happen first and
What other scenarios are people thinking of and what odds would they put on them being recoverable failures?
2025-12-10 06:16:38
Published on December 9, 2025 10:16 PM GMT
[Epistemic Status: This is an artifact of my self study. I am using help manage my focus. As such, I don't expect anyone to fully read it. If you have particular interest or expertise, skip to the relevant sections, and please leave a comment, even just to say "good work/good luck". I'm hoping for a feeling of accountability and would like input from peers and mentors. This may also help to serve as a guide for others who wish to study in a similar way to me. ]
My goals for this sprint were:
So how did I do?
| Date | Progress |
| Th, Nov 6 |
|
| Fr, Nov 7 |
|
| Mo, Nov 10 |
|
| Tu, Nov 11 | No progress because of commuting and errands. |
| Wd, Nov 12 |
No progress because of:
|
| Th, Nov 13 - Fr, Nov 14 | Pretty burned out from convocation related socializing, so not much progress. Did some worth ideating and reading and commenting on some LW posts. |
| Mo, Nov 17 |
|
| Tu, Nov 18 | No progress. Distracted by interpersonal "other life stuff". |
| Wd, Nov 19 |
|
| Th, Nov 20 |
|
I have my BSc now 🎉
I'm also feeling good about the amount of focus on this I've had in first two weeks of the sprint, but then got very distracted and depressed by other things in my life and didn't manage to record any progress for the time from then til now.
So I definitely still want to improve, but it feels like I'm moving in the right direction. I also want to get better at prioritizing things that are worth working on, and sticking to my plan. This also means putting more realistic amounts of work in each sprint. I got similar advice from @Roman Malov and from reading A Pragmatic Vision for Interpretability, so I'm hyped up on that.
More object level, I have started the Transformers from Scratch (TfS) series, written a few posts, and read and commented on several posts.
I had five items on my write list. I'm realizing writing about my theory of change is probably more involved than I thought, so I ended up putting more focus into Maat, which I roughly know everything I want to write, I just need to write it down.
Also, I published a post to make my planned writing publicly visible. I may change the format later, but I like the idea of the list being public, so I'm committing to that.
I didn't do any of the review for AIA terminology review. I still think that's a good idea but is quite involved, so I probably should make it a main focus if I'm going to do it.
I did manage to read and comment on LW posts. I think this is good practice, so I plan to continue. I think looking at the list of all posts is a good way to engage with what other people are currently focusing on, however, I think there is value in reading and commenting on older posts and I'm not sure how to prioritize that very well. So one goal could be to figure out how to prioritize things to read, but that sounds dangerously meta.
I think I might have picked the book up once. I'm not having much traction staying engaged with math study now that I'm not in classes. It is such a difficult thing to do it might be better to treat it like the AIA terminology review, both in having a description of my understanding and some worked examples as a visible output, but also in that I should not try to do it unless it is a main focus.
I actually started going through Transformers from Scratch! I haven't made it very far but I've finally actually started so that's nice.
I started looking at literature focused on high dimensional spaces. There's so much content out there it's difficult to get a sense of it all. One noteworthy resource I found is "Understanding High-Dimensional Spaces" by David B. Skillicorn. I definitely want to at least skim this.
I've been engaging with people here on LW which feels like movement in the right direction, but I definitely want to be more conscientious in my planning and execution in this domain. Clarifying my goals here would probably be good.
I think trying to focus on all 6 of my pursuit categories at the same time has made me too scattered. So going forward I will instead keep them as guides, but pick fewer specific goals from within each for each sprint.
The Goals:
I'm hoping having fewer goals will make them easier to focus on. The "every day" goals make for an easy quantification: Either I worked on it that day or I didn't. I would like to be using better metrics, but I don't want to make up numbers that may not actually indicate anything useful. Numbers are good if they mean something, but if they don't, it's better to speak qualitatively.
List of common acronyms:
2025-12-10 05:00:59
Published on December 9, 2025 9:00 PM GMT
I've decided to post these in weekly batches. This is the fifth of five. I'm posting these here because Blogspot's comment apparatus sucks and also because no one will comment otherwise. Because this is the last week and because there's only two posts in it, I'm also going to mention a few highlights and talk about what went well and what missed the mark.
29. My PhD Thesis: Part 1: Preliminaries - Algebra, Topology, Algebraic Topology
Surprise! We've actually been thinking about the basics of algebraic topology this whole time, skipping right past the horrors of point-set topology...
Every topological space has a fundamental group. Some of them are extremely weird. (Your search term here to start off down that rabbit hole is "Hawaiian earring space".)
30. Atoms to Agents As Filtered Through Some Tame Research-Creature
Tools are not agents, but they're almost agents. They have no goal, but seem to be clearly designed for some purpose, or have been subjected to some kind of selection pressure comparable in optimization strength to intentional design. They are always composed of other tools, all the way down to atoms; they are the bedrock of our theory. I claim no constraint on their actionspace, or the actionspace expansion that they afford to an agent wielding them. Hammers, bones, pens, ribosomes, and spoons are all tools; viruses are marginal.
Highlights! I've picked out the handful of posts with the ~most reads after a little while, and the handful that I was most proud of. I'll crosspost the best couple of each category at some point.
Highlights (as judged by estimated readers/day, measured on 12/9):
All others had <1.34 readers/day.
Highlights (as judged by me and what I thought were my best posts):
...OK I'd thought these were going to be substantially different from the reader's choice highlights but they mostly overlap. A few notable posts that I really liked but don't seem to have got a ton of attention: Seven-ish Evidentials From My Thought-Language, The Ultimate Sylow Theorem Guide for Algebra Quals, A Partial Theory of Flavor Pairing in Foodcraft, What's the Type of an Ontological Mismatch?, and In Defense of Boring Doom: A Pessimist's Case for Better Pessimism.
Ultimately, though, I was pretty happy with like half of these posts.
Postmortem! What was I hoping to get out of this? What went well, and what could have gone better? What could I have done to get more out of Budget Inkhaven Halfhaven?
As ever, I accept public feedback here, and private feedback at https://admonymous.co/lorxus .
2025-12-10 04:28:17
Published on December 9, 2025 8:28 PM GMT
The comments section of this post contains topics I want to write posts or sequences about. Each idea for a writing topic starts with "ToW". Feel free to vote or comment on any of my ideas. You may also suggest topics you would like me to write about by adding your own comment starting with "ToW".
I'll add comments starting with "WRITTEN" and including a link to the post or sequence.
2025-12-10 03:15:27
Published on December 9, 2025 7:15 PM GMT
As Bea approached the Tate Modern on London-3, the first thing she noticed were the cooling CRACs and CDUs still freckling the grey cuboidal building. She supposed the datacenter aesthetic was appropriately grungy for a modern art building — it harkened back to a simpler time.
Jay — her date — had already indicated to her that he’d been here many times before. Sure, he was probably just signaling that he was cultured, and capable of abstract thought; but she figured she’d let him try to impress her. And it might be interesting to see what artists have been up to in the past 100 years. Frankly, she’d been too busy to keep up with the art world (it had been a hectic time!).
As they entered the dour-looking structure, Bea noticed a six-metre-long corridor in front of them. The ceiling, the floor, and both walls were covered with…
“Wait,” Bea said, “That doesn’t make any sense. Why are all these paintings of me, and my memories?”
“Oh,” said Jay, “So that’s what you see? Interesting. Maybe a little boring.” He raised an eyebrow.
Immediately correcting himself when he noticed Bea’s pout, “No! Don’t take it so personally,” he laughed, “I was just joking! That’s what most people see. I myself see the past more generally; the people that suffered so we could be so happy.”
Bea asked, “So how is it we see different things then?”
“It’s nano-telepathy. They use small robots to scan your brain when you enter, and feed your current neural patterns to an image model. You know, like how adverts work, except instead of trying to get you to buy something, it’s trying to get you to feel as deeply as possible. With a bias to the uplifting, moving, or sublime!”
At this, Bea turned to take in the artwork. She started to tear up. Out of the corner of her eye, she noticed Jay was as well. After staring at the corridors for 5 minutes — watching the images shift and render new and more moving images from her life (they seemed to be getting more moving over time… was it still reading her brain?) — they decided to move on, into the main lobby. Thankfully there was no artwork exhibited here, so she could collect herself after the corridor.
Bea noticed she was skeptical that Jay really saw ‘people who suffered for our happiness’, and wondered if it might have instead been something much more mundane.
“So, where should we go first?” Bea asked.
“Let’s see the well-known stuff, it’s your first time here! You have to see the greats!”
“It’ll all be new to me, just show me whatever you like the most”
“Well, the stuff I like’s a bit more niche… it’s maybe… Let’s just see the famous stuff! It’s famous for a reason: broad appeal, and all that!”
“Sure.” Bea said, noting the not-so-subtle neg.
The room they entered first was bright, well-lit. There was a box sitting on a plinth in the center. Next to the box was what looked like a hologram of an enormous structure — mountains were rendered next to it for scale. It was impressive, perhaps in the top 10% of structures you might find in any big city, in terms of craftsmanship. But bigger, much bigger than would be practical.
“What’s this?”
“Well,” said Jay, “They’re doing an exhibit at the moment. ‘Ways of Seeing.’ The point is basically that each room has a theme centered around different theories of artistic value.”
“Interesting,” said Bea. “So what’s this room supposed to represent, architectural value? Use value?”
“Nothing like that! This room’s theme is ‘Effort’. Basically, does art become more valuable because of the effort of the artist?”
Bea must’ve looked confused. “But this doesn’t take that much effort to build? I’ve seen structures much more impressive than this? Are the mountains much bigger than I’m thinking, or…?”
“No. The mountains are regular-mountain-sized,” Jay said. “You see that box, over there, next to the hologram?”
Bea glanced at the box again.
“Well, that box contained a civilisation about 20 years ago. That civilisation was simulated for 10 years, running at a speedup of 1000-times. The civilisation had a God, and that God was the artist of this piece… or, not artist… he called himself the ‘composer’” Jay gestured air quotes.
“The God answered prayers occasionally, made itself known in the world, moulded the civilisation. And released a Book. Now, that Book contained a bunch of random rules that people could argue and theologise about all day — to act as inspiration — but there were two core principles: Good Works please the God; and steam and coal are unholy.”
“Steam and coal are unholy…? I don’t see… Ahhh,” Bea said.
“Yes, exactly, no industry. And after 10,000 years, they managed to achieve this. Now it might not look like much to you, but this final structure took them 1500 years to build. Another 3000 for the decorations and facades. Whole kingdoms rose and fell deciding whether the columns should be Doric, Ionic, or Corinthian. At one point the Artist — sorry, the Composer — had to intervene because the theological disputes around minimalism and maximalism were getting out of hand.”
“I see,” said Bea. “Now that I know that… Nope, it still just looks like an extremely nice, big building. Just like most buildings, but bigger. I guess I just have the wrong baseline. It definitely adds something. But I don’t know if that makes it a much better-looking building. What other rooms are there?”
The next room had a big window on one side, and the walls were white. All the lighting came from the window, making the corners of the room slightly dim. In front of the window there were 10 marble stands. On each stand was etched what looked like some sort of chemical formula and a structure next to it, made up of Benzene rings. The one furthest to the left read:
(6aR,9R)-N,N-diethyl-7-methyl-4,6,6a,7,8,9-hexahydroindolo[4,3-fg]quinoline-9-carboxamide
Albert Hofmann, 1938
The rest had names that were far longer, and structures that seemed to be much harder to decipher. Most of them looked like honeycombs made by drunk bees — they were so filled with benzene rings and their respective offshoots.
“What is this?” Asked Bea, “Science as art perhaps…?”
“Nothing like that,” replied Jay, “This is the room focused on the viewer’s experience when perceiving the artwork. That plinth on the left, think of that as an homage to the original artists. Like Picasso paying homage to Lascaux — LSD is the Lascaux.”
“Huh?” said Bea, “They’re offering me LSD in order to… enhance the experience I have when I view other art in the museum? And a choice of other psychoactive drugs?”
“No,” said Jay, “The LSD isn’t really meant to be taken, it lasts for way too long, and you have no idea what you’ll even experience. I don’t even know if they have LSD here… No, the LSD serves as inspiration for the real artworks, those other drugs you see on the right, the ones labelled with different emotions.”
And sure enough, above the diagrams of molecular structure, Bea noticed that each plinth had a title. They were called things like: Monumental Achievement, Absolute Indifference, Why Alexander Wept, First Orgasm, Total Dissolution.
“What are they?” asked Bea
“They’re designer drugs, in a way. Not what used to be called designer drugs. These are drugs that have been designed to perfectly evoke a specific feeling in the user, by attaching to precisely the right neurotransmitters in precisely the right orders. These are trying to answer the question, is the value of art in the experience it evokes in the viewer — or I suppose, consumer.”
“Cool,” said Bea, “So they’re like Einheit. That drug that all the clubbers and ravers take.”
“Yeah, kind of... These generally evoke much more specific feelings than Einheit, but it’s the same principle. So, do you want to try any of them?” Jay asked.
“Sure,” Bea said, “I’ll try ‘Monumental Achievement,’ how long does it last?”
“Oh yeah… that’s the great thing,” said Jay, looking somewhat disappointed she hadn’t chosen a racier option, “You can decide, the half-lives are quite short, but they all have enormous LD50s, and taking more has diminishing returns pretty quickly, so you just tell the plinth how long you want it for, and it’ll dispense the right amount out of the top. You’ll feel the effects immediately.”
Bea walked up to the plinth and said “5 minutes,” and the plinth offered up a capsule filled with an off-white powder. She braced herself, and gulped it down.
She was waiting, waiting…
Nothing.
She looked at Jay.
“What? You don’t feel anything?” He asked.
“Nothing” said Bea, “Should I?”
“Yeah, definitely… Oh wait, is your brain totally… You know?”
“Is my brain what…? Natural?” Asked Bea defiantly, “Why should that matter? The answer is ‘Not really.’ But isn’t that everyone’s answer these days?”
“Of course, but it’s a matter of degree. It works using neurotransmitters, Bea! If you don’t have enough neurotransmitters, or have replaced most of them, then you have to tell the plinth you need the digital version, or some combination.”
“Ah,” said Bea. “Well, the moment’s passed anyway.”
Jay started expectantly, “You’re sure you don’t want to try any of…”,
“No,” interrupted Bea. “Let’s see one of the other rooms.”
Room 3 was much smaller, about four metres by four metres. Along the furthest wall from the entrance were a series of machines that looked sort of like matte-white tanning beds, but without any UV lights on the ‘ceiling’ or ‘floor’ of the bed.
Bea instantly recognised the beds, they were simulation machines. You could step in one and live an entirely different life, with varying degrees of awareness of the outside world. However, there was something slightly different about these tanning beds.
On them was written ‘Ceci n’est pas une vie.’
“So… what’s this room meant to represent? Is it ‘art as homage’?” Bea asked.
“Well, I suppose that’s a part of it,” said Jay. “Really, this room is kind of a two-for-one — maybe a three-for-one, if you count the homage aspect. This is subversion of expectations. See, it says ‘Ceci n’est pas une vie,’ on the machine where you can live an entire life. It’s surreal, the art comes from the juxtaposition.”
“But it’s not exactly that… special, is it? I mean, everyone has a machine like this? You don’t even need machines like this to simulate other lives anymore, you can do it easily at home. Kids can make them. Is it really worth having a simulated life in a museum?”
“Ah, and that’s the other part of it!” said Jay, “Maybe the art is just that it’s here, in a gallery. Sort of going for a Duchamp thing.”
“Right. I get it, so an artist takes something that any kid could make, puts it in a museum, and calls it art. I get it… Not my style.” said Bea.
“Fair enough,” said Jay. “Maybe you’d get it if you went in the machine… You know it’s Magritte’s life that you live, right?” Jay replied.
“No, I lived enough lives last weekend, and I can do a Magrittescapade whenever I want. I think I’d rather just see another room.”
“Alright. Alright. Nothing feels different about it being here, in a gallery?”
“Not really.” said Bea, and walked out.
Room 4 had one of its wall consisting of one big screen, subdivided into about 100 metre-long subsections. Each displaying what looked like 1960s era control-rooms and government offices. On one of the screens was a map of what looked like Earth-1, filled with a series of red dots in seemingly random places on the land, and a few in the sea.
“Alright,” said Bea. “I’m really stumped on this one. I suppose… 1960s, and government offices. So, something to do with JFK? Some twist on pop art? I don’t know.”
“Not a bad guess,” said Jay. “Based on the evidence you had. But no, it’s pretty different from that. Maybe art is participatory? It’s about the audience’s relation to the art piece. Art is created in the dynamic between piece and audience. Maybe looking at the wall behind you might give you more of a hint. This one is called ‘Cut Peace 1964’.”
Bea turned to look at the wall that was behind her as she walked in, and noticed a big red button, covered in an unlatched glass case, covered in another unlatched glass case, with the word “LAUNCH” written on it, in big black letters.
“Wait… the 60s, the Cold War, I get it…” said Bea slowly. “This isn’t a simulation of real people is it? It’s not an actual 1960s being run on repeat.”
“No…” said Jay sheepishly. “It’s not a simulation, but, well, the artist actually built a replica of the 1960s, and the people in it. It’s in orbit around Gliese 411.”
“How can this be allowed? Think of the suffering that happened in the 1960s!” Bea protested. “Maybe I should hit launch!”
“This piece has been here for 10 months, Bea.” sighed Jay, “Don’t you think if it was an actual replica, people would’ve followed your same thought process. No, all of the non-human animals are p-zombies, and anyone who starts suffering too badly gets a healthy dose of opium activated from a chip in the space between their toes… With one exception.”
“If I hit the launch button, it also stops the opium chips from working.” Bea deduced.
“Bingo.” said Jay.
“So, what? I’m supposed to just stand here and not do the obviously terrible thing? Is that what this piece is about?” asked Bea.
“Yeah, pretty much. Or you could do it?” said Jay, “I mean, no-one else has, and someone eventually probably will. Why not you?”
“Because I don’t want to be the person-” Bea started, “or be known as the person” she whispered, “who would destroy a perfectly happy civilisation for no reason other than ‘Art’. It doesn’t feel much like Art, it feels like the artist put a sword of Damocles over these people's heads and is asking us to swing it.”
“Well, no one has yet,” said Jay, “And if it makes you feel any better, the artist ran the simulation from 1930 to begin with. They put the sword of Damocles over their own heads just as much as she did. She’s down there too you know, I think she’s a waitress.” He paused and looked at the screen. “Oh yeah, that’s her, right there,” he pointed.
“Hmph” said Bea. “You know, I still really don’t like it, but I guess that makes me respect the Artist a little more. I still don’t see the value in it.” She looked around, and threw her hands up, “I’m just supposed to stand here in this empty room, or watch the boring control-rooms of the sixties, and not press a button?” she asked.
“Well, when the piece — ahem— finishes, that’s probably when it’ll be a bit more interesting.” said Jay.
“But I don’t WANT the piece to finish!” yelled Bea, “In fact, I really want to forget I ever saw this room. It should be illegal!” Bea stormed out.
“Well, I guess it got a reaction… .” said Jay, and followed her out.
As Bea was storming out of the room, and then leaving the gallery entirely, Jay managed to catch up to her.
“One last piece,” he said. “You probably won’t like it, but I promise, you won’t regret you saw it like that last one. In fact, I think you might find it funny.”
Jay took Bea over to a tiny room — actually more like a closet — just off the side of the main lobby. Bea opened the closet doors to see…
"A certificate of Art?” Bea asked incredulously, “Given by the council of AI Art Connoisseurs,” she read.
“Yes, see ‘Connoisseurs’ ” said Jay, chuckling. “You know, I probably agree with you on this one. Is art that which is recognised as art by an authority?”
Bea continued to read aloud: “the AI Art Connoisseurs officially declare the empty space in this broom closet of the Tate Modern on London-3 to be Art.”
“That’s the silliest thing I’ve ever heard” she said.
“Yes, I quite agree, but perhaps for the opposite reason” said Jay. “I agree with the AIAC that the empty space in this Broom closet can be art. This certificate that says it is, in fact, art can also be part of the art. But no certificate makes it so.”
“You know,” said Bea, “That might be one thing you and I can agree on. This certificate certainly does not make this art!”
With that, Bea grabbed Jay’s hand, and strolled out of the building.