2025-02-28 00:18:15
Table of Contents
1. Claude plays Pokémon. But so does the number π?
2. Evil numbers turn AIs evil. Great.
3. Researchers observe flies playing on carousels. Actually great.
4. Will Elon Musk get booted from Tesla after “brand collapse?”
5. Why do I know every song on the radio? I’m old.
6. Scaling laws for neuroscience.
7. From the archives.
8. Comment, share what you’ve found interesting lately, ask anything.
1. There I sat, listening to my daughter practice words during her nightly bedtime reading, while on a screen I watched the newest AI, Claude 3.7, live-stream its Pokémon play-through.
“Blue,” said my wife in the background, pointing to a splayed book.
“Boo!” said Sylvia.
“I also noticed that SWIFT’s HP has decreased due to the poison status, which reinforces the urgency of finding the exit,” Claude’s running chain-of-thought said on Twitch. It laboriously moved its character a few spaces to the right.
“Car!” said my wife. “Ar!” said Sylvia.
“Let me continue following this path to explore further eastward,” said Claude.
To explain: this week Anthropic dropped Claude 3.7, arguably tied for the smartest state-of-the-art AI that exists (e.g., it’s going to power Amazon’s Alexa). Apparently Anthropic had been investigating how well Claude plays the old Pokémon Red/Blue as a side project, the very thing I once played on a gameboy. So they decided to livestream a play-through at launch, one that is still ongoing (you can watch here).
You might wonder why. After all, AI has been good at gaming for a long time, right? True. But those models were always fine-tuned for the game. This is just Claude itself, out of the box, using its general intelligence to look at screenshots taken every few seconds and make decisions about what button to press, just as a human would. Its slow gameplay is rendered quaintly adorable because its running “thoughts” are displayed next to what it sees on the screen.
So as my daughter learned to speak from the data in a few dozen children’s books, Claude, which has read the entire internet, played Pokémon, and I watched along with 2,000 other people as it talked to characters and fought battles and explored.
Observing it play, I learned things about AI I never had from reading academic papers, or even using the models myself in shorter sprints. This was a marathon. How modern LLMs will expand into agents was very much on display as Claude oscillated between impressively smart and unbelievably dumb.
How does it work? Normally, AIs can only handle so much context before they start to break down and go insane. The longer you talk to an AI, the more insane it will get, and the more mistakes it will make, and the more things it will forget, until it gets stuck in a loop. This is the main hurdle for AI agents.
Anthropic solves this by simply having Claude go through a summary process of its thoughts—where it is, what its quests are, what pokémon it has—about every 10 minutes. Then it “cleans up my context” (which might be the same as starting up a new chat, implying Claude “dies” thousands of times while playing), and then that summary gets fed to the new instance.
In one forest map, I watched it loop for about three hours between what it thought were exits but were really just tree stumps. This made me question whether the term “reasoning model” is accurate. As Einstein supposedly said, “The definition of insanity is doing the same thing over and over again and expecting different results,” and by this definition Claude is assuredly insane. It is hard to be scared of AI when you watch the smartest existent one get stuck in a corner for several hours, which is what happened. Currently, it has been running loops in Mt Moon for ~23 hours and just re-set the loop as the chat pleaded with it not to go north from a particularly troublesome ladder again.
In some ways, it was endearingly human-like. If it pressed the wrong button, its thoughts would be, “Oh, the game has taken me to this menu via a bug,” and it responded with “Lucky break!” when an attack by an enemy missed.
But it was subtly inhuman: e.g., I noticed it appeared impossible for Claude to stop having particular thoughts. At certain screens, it would always think that there was some visual bug, despite (a) there was no visual bug, and (b) it had had that thought every single time, which a human would stop noticing or caring about.
To me, this silly benchmark felt a lot more substantial and informative regarding “are we at AGI?” than academic questions. Claude is the third highest scorer on a PhD-level question set ominously titled “Humanities Last Exam” but meanwhile, its record in Pokémon is 3/8 badges.
Claude didn’t pursue fights efficiently, often making mistakes about how best to defeat opponents. People, in comparison, beat these games without healing for fun challenges. Although there is a certain terror at its mere indefatigability, while watching I realized it is the forgiving structure of the game, wherein failure is an impossibility (you restart somewhere else should you lose, and all the maps push you in certain directions) that actually propels the model along as much as its own intelligence. E.g., a sign saying “Go Ahead” finally got it out of the 3-hour loop in the forest. It’s like bowling with bumpers.
Did you know the number π also can play Pokémon? Literally just the infinite number 3.14159… mapped onto button presses. It’s unfortunately been playing a different game version (for, oh, a couple years now), but there it has gotten out of the starting town a few times and leveled up its characters to ungodly stats via random battles.
2025-02-21 00:43:46
Polymathy, everyone knows, went extinct in the second half of the 20th century. Only a few intellectual giants—names like Bertrand Russell, John von Neumann, or Michael Polanyi—kept its veins flowing with blood until they themselves finally flatlined, and it did too.
Perhaps more was lost than we’d like to admit. After all, the few fragile centuries of the Enlightenment, which was when our reigning political order was established, and when our major institutions were defined, and when modern science itself came to be, all occurred during the time when polymaths ruled the earth (well, as much as intellectuals ever do).
It's hard to overstate how fecund the most famous polymaths were. We stare at their journals like the bones of extinct creatures. Consider how recent analysis of Leonardo da Vinci’s notebook (his “Codex Arundel”) revealed da Vinci had worked out gravity not only pulls things down but, far more fundamentally, actively accelerates the downward pull at a particular rate—the key to understanding the entire phenomenon! And it was knowledge unlocked as a side-project by an artist a century and a half before Newton. According to the physicist who noticed:
… with this drawing da Vinci managed to estimate what is known to physicists as the “gravitational constant” within ten percent of its actual value, despite only conducting what appears to have been a crude experiment.
However famous da Vinci is now, he was close to being far more so.
But it was in the air. And just as much so for all the contributing lesser-known figures only historians now remember. E.g., “The polymath in the age of specialisation” in Engelsberg Ideas lists many polymaths who are not household names, along with their expansive expertises, like Athanasius Kircher (“ancient Egypt, acoustics, optics, language, fossils, magnetism, music, mathematics, mining and physiology”), Olaus Rudbeck (“anatomy to linguistics, music, botany, ornithology, antiquities and what we now call archaeology”), Benito Jerónimo Feijóo, a Spanish monk (“known in his day as a ‘monster of erudition’ in the 17th-century style… The nine volumes of his Teatro crítico universal dealt with ‘every sort of subject’”), Pierre Bayle (“wrote mainly about theology, philosophy and history, but also edited a learned journal, the Nouvelles de la République des Lettres, and compiled an encyclopaedia, the Dictionnaire Historique et Critique, in which the footnotes took up more space than the text because he filled them with critical remarks of his own”), Comte de Buffon (“remembered today for the huge enterprise of his Histoire Naturelle, published in 36 volumes, was also active in the fields of mathematics, physics, demography, palaeontology and physiology”), and so on and on.
The cause of death of the polymath is universally agreed-upon: increasing demands of specialization rendered polymathy impossible. There became too much to know. “Whatever happened to the polymath?” in UnHerd summarizes the postmortem:
The decline of polymathy… is a crisis of too much information. The seventeenth century was a “golden age of polymaths”, as explorers found new regions, the scientific method flourished, and the postal service and the proliferation of journals allowed scholars to trade ideas. But those same forces led to “information overload.” Over the next 200 years, the intellectual world divided between the specialists who knew a lot about their little area, and popularisers who knew a little about a lot.
Today, a “polymath” is at most someone who studies a few separate areas of mathematics—or who can both direct a movie and act in one. But in just 1870, it meant someone like Lewis Carroll, who you likely know as the author of Alice's Adventures in Wonderland, yet who was also a poet, a photographer, a literal deacon, and he wrote nearly a dozen books on mathematics spanning algebra to probability to logic, and he invented “word ladder” puzzles.
As Peter Burke, a historian at Cambridge University and author of The Polymath: A Cultural History, lamented a few years ago:
Younger polymaths are becoming more difficult to find… I am unable to identify any who were born after the year 1960. Will the species become extinct?
To answer Burke, there are a few names I could give from after 1960, but in general he’s correct. If da Vinci were resurrected to look upon his intellectual descendants, modern thinkers would surely strike him as species warped by the pressures of hyper-specialization. Our academia is now populated by minds pressured into shapes as niche as hummingbirds (upon whom evolution forces the dubious delicacy of only eating nectar) or fig wasps (destined to birth their young solely within the sticky-sweet confines of a single species of fruit, one which they presumably call, in their native Hymenopterian, simply “mother.”)
So is polymathy’s decline just a law of intellectual nature?
To put it simply: there are two kinds of thinkers. Those rate-limited by expertise, and those rate-limited by creativity. Slowly but consistently, the rate-limiting factor for intellectual contribution has become ever deeper expertise.
2025-02-14 01:02:18
Unfortunately, there’s a growing subfield of psychology research pointing to cognitive atrophy from too much AI usage.
Evidence includes a new paper published by a cohort of researchers at Microsoft (not exactly a group predisposed to finding evidence for brain drain). Yet they do indeed see the effect in the critical thinking of knowledge workers who make heavy use of AI in their workflows.
To measure this, the researchers at Microsoft needed a definition of critical thinking. They used one of the oldest and most storied in the academic literature: that of mid-20th century education researcher Benjamin Bloom (the very same Benjamin Bloom who popularized tutoring as the most effective method of education).
Bloom’s taxonomy of critical thinking makes a great deal of sense. Below, you can see how what we’d call “the creative act” occupies the top two entries of the pyramid of critical thinking, wherein creativity is a combination of the synthesis of new ideas and then evaluative refinement over them.
To see where AI usage shows up in Bloom’s hierarchy, researchers surveyed a group of 319 knowledge workers who had incorporated AI into their workflow. What makes this survey noteworthy is how in-depth it is. They didn’t just ask for opinions; instead they compiled ~1,000 real-world examples of tasks the workers complete with AI assistance, and then surveyed them specifically about those in all sorts of ways, including qualitative and quantitative judgements.
In general, they found that AI decreased the amount of effort spent on critical thinking when performing a task.
What’s interesting is that AI use does seem to shift the distribution of time spent within Bloom’s hierarchy toward the top; if you think about the above data in relation to the pyramid version, knowledge workers spent relatively more of their total time in the upper region of the pyramid when using AI, as effort at the top declined less relative to the other stages (in fact, evaluative refinement was the only place it increased for a number of users).
While the researchers themselves don’t make the connection, their data fits the intuitive idea that positive use of AI tools is when they shift cognitive tasks upward in terms of their level of abstraction.
We can view this through the lens of one of the most cited papers in all psychology, “The Magical Number Seven, Plus or Minus Two,” which introduced the eponymous Miller’s law: that working memory in humans caps out at 7 (plus or minus 2) different things. But the critical insight from the author, psychologist George Miller, is that experts don’t really have greater working memory. They’re actually still stuck at ~7 things. Instead, their advantage is how they mentally “chunk” the problem up at a higher-level of abstraction than non-experts, so their 7 things are worth a lot more when in mental motion. The classic example is that poor Chess players think in terms of individual pieces and individual moves, but great Chess players think in terms of patterns of pieces, which are the “chunks” shifted around when playing.
I think the positive aspect for AI augmentation of human workflows can be framed in light of Miller’s law: AI usage is cognitively healthy when it allows humans to mentally “chunk” tasks at a higher level of abstraction.
But if that’s the clear upside, the downside is just as clear. As the Microsoft researchers themselves say…
While GenAI can improve worker efficiency, it can inhibit critical engagement with work and can potentially lead to long-term over-reliance on the tool and diminished skill for independent problem-solving.
This negative effect scaled with the worker’s trust in AI: the more they blindly trusted AI results, the more outsourcing of critical thinking they suffered. That’s bad news, especially if these systems ever do permanently solve their hallucination problem, since many users will be shifted into the “high trust” category by dint of sheer competence.
The study isn’t alone. There’s increasing evidence for the detrimental effects of cognitive offloading, like that creativity gets hindered when there’s reliance on AI usage, and that over-reliance on AI is greatest when outputs are difficult to evaluate. Humans are even willing to offload to AI the decision to kill, at least in mock studies on simulated drone warfare decisions. And again, it was participants less confident in their own judgments, and more trusting of the AI when it disagreed with them, who got brain drained the most.
Even for experts, there’s the potential for AI-based “skill decay,” following the academic research on jobs that have already undergone automation for decades. E.g., as one review described:
Casner et al. (2014) tested the manual flying skills of pilots who were trained to fly manually, but then spent the majority of their careers flying with high automation (i.e., autopilot for much of takeoff, cruising, and landing). Procedural skills, such as scanning the instruments and manual control, were “rusty” but largely intact. In contrast, major declines in cognitive skills emerged, such as failures to maintain awareness of the airplane’s location, keep track of next steps, make planned changes along the route, and recognize and handle instrument systems failures.
In Frank Herbert’s Dune, the Butlerian Jihad against AI was a rejection of what such over-reliance on machines had done to humans. The Reverend Mother says:
“The Great Revolt took away a crutch… It forced human minds to develop.”
Outside of fiction, concerns go back to Socrates’ argument that a reliance on writing damaged memories, and span all the way to the big public debate in the aughts about whether the internet would shorten attention spans.
Admittedly, there’s not yet high-quality causal evidence for lasting brain drain from AI use. But so it goes with subjects of this nature. What makes these debates difficult is that we want mono-causal universality in order to make ironclad claims about technology’s effect on society. It would be a lot easier to point to the downsides of internet and social media use if it simply made everyone’s attention spans equally shorter and everyone’s mental health equally worse, but that obviously isn’t the case. E.g., long-form content, like blogs, have blossomed on the internet.
But it’s also foolish to therefore dismiss the concern about shorter attention spans, because people will literally describe their own attention spans as shortening! They’ll write personal essays about it, or ask for help with dealing with it, or casually describe it as a generational issue, and the effect continues to be found in academic research.
With that caveat in mind, there’s now enough suggestive evidence from self-reports and workflow analysis to take “brAIn drAIn” seriously as a societal downside to the technology (adding to the list of other issues like AI slop and existential risk).
Similarly to how people use the internet in healthy and unhealthy ways, I think we should expect differential effects. For skilled knowledge workers with strong confidence in their own abilities, AI will be a tool to chunk up cognitively-demanding tasks at a higher level of abstraction in accordance with Miller’s law. For others… it’ll be a crutch.
So then what’s the take-away?
For one, I think we should be cautious about AI exposure in children. E.g., there is evidence from another paper in the brain-drain research subfield wherein it was younger AI users who showed the most dependency, and the younger cohort also didn’t match the critical thinking skills of older, more skeptical, AI users. As a young user put it:
“It’s great to have all this information at my fingertips, but I sometimes worry that I’m not really learning or retaining anything. I rely so much on AI that I don’t think I’d know how to solve certain problems without it.”
What a lovely new concern for parents we’ve invented!
Already nowadays, parents have to weather internal debates and worries about exposure to short-form video content platforms like TikTok. Of course, certain parents hand their kids an iPad essentially the day they're born. But culturally this raises eyebrows, the same way handing out junk food at every meal does. Parents are a judgy bunch, which is often for the good, as it makes them cautious instead of waiting for some finalized scientific answer. While there’s still ongoing academic debate about the psychological effects of early smartphone usage, in general the results are visceral and obvious enough in real life for parents to make conservative decisions about prohibition, agonizing over when to introduce phones, the kind of phone, how to not overexpose their child to social media or addictive video games, etc.
Similarly, parents (and schools) will need to be careful about whether kids (and students) rely too much on AI early on. I personally am not worried about a graduate student using ChatGPT to code up eye-catching figures to show off their gathered data. There, the graduate student is using the technology appropriately to create a scientific paper via manipulating more abstract mental chunks (trust me, you don’t get into science to plod through the annoying intricacies of Matplotlib). I am, however, very worried about a 7th grader using AI to do their homework, and then, furthermore, coming to it with questions they should be thinking through themselves, because inevitably those questions are going to be about more and more minor things. People already worry enough about a generation of “iPad kids.” I don’t think we want to worry about a generation of brain-drained “meat puppets” next.
For individuals themselves, the main actionable thing to do about brain drain is to internalize a rule-of-thumb the academic literature already shows: Skepticism of AI capabilities—independent of if that skepticism is warranted or not!—makes for healthier AI usage.
In other words, pro-human bias and AI distrust are cognitively beneficial.
2025-02-08 23:18:42
Table of Contents (open to all free subscribers too):
Don’t Look Up becomes real.
Die, book blurbs, die!
Die, indirect costs for scientific grants, die!
OpenAI’s Deep Research can’t find me a good mattress.
But Nate Silver is right on the unavoidable importance of AI.
My theory of emergence cropping up in a Nature journal.
Fun Desiderata art change.
New Jersey drones confirmed to be a big nothingburger.
Ask Me Anything.
1. The asteroid named “2024 YR4” was discovered in December heading towards Earth, and it has an uncanny knack of increasing its probability of impact. After NASA was giving a 2% chance of impact in 2032, that’s now climbed to somewhere around 2.3%. Below is a real 3 second video of it passing near Earth in early January.
The asteroid’s size is about the one estimated to have caused the Tunguska event in 1908 (however, these are estimates, it could be even larger). Here’s an account of what that’s like from someone who was having breakfast about 40 miles away:
The split in the sky grew larger, and the entire northern side was covered with fire. At that moment I became so hot that I couldn't bear it as if my shirt was on fire; from the northern side, where the fire was, came strong heat. I wanted to tear off my shirt and throw it down, but then the sky shut closed, and a strong thump sounded, and I was thrown a few metres. I lost my senses for a moment, but then my wife ran out and led me to the house. After that such noise came, as if rocks were falling or cannons were firing, the Earth shook, and when I was on the ground, I pressed my head down, fearing rocks would smash it. When the sky opened up, hot wind raced between the houses, like from cannons, which left traces in the ground like pathways…
The worst case scenario is that 2024 YR4 would hit some populated area, like a city, with the force of an 8-megaton nuclear bomb (but still smaller than the largest nuclear bomb ever exploded, the Tsar bomb). It wouldn’t be world-ending, but it would still be a global disaster of the first order.
What’s problematic is that scientists use gravitational modeling to figure out the probability of impact, and the close pass to Earth again in 2028 might alter its path. Essentially, if 2024 YR4 passes through particular gravitational “keyholes” in 2028, the gravity of Earth might shift its orbit just enough to make an impact very likely.
The issue is that, unless the probability goes down soon by ruling out all the keyholes, we could be in a weird state where the probability just remains at ~1-3% for years, and that number feels so low, psychologically, that it’s not worth paying attention to (although some people on forums are arguing for as high as 6%). We’ll lose sight of the asteroid for a window somewhere around 2025-2027 completely, so if this chart is still high by then that would be… bad.
While I certainly wouldn't wish for a “through the keyhole” event, if it did happen, there would be the silver lining of immense amounts of international cooperation and technological advancement to divert it. If a deflection mission were planned, almost certainly it’d be a collaboration between Elon Musk’s SpaceX and government organizations like NASA and the ESA and the International Asteroid Warning Network. I’m well aware of the analogies to the movie Don’t Look Up, but well, such an inevitability would make sense if the universe consistently bends its arc toward rendering satire obsolete.
Personally, I always thought exactly this epistemic scenario was the flaw of Don’t Look Up. It would have been much more realistically funny in the movie if, instead of a 99% chance to destroy Earth from the beginning, the asteroid had a 41% chance. And everyone just dismisses it because that’s less than half. Anyway, we’ll see how humanity responds in real life to a lower chance of impact, if that holds, in the coming years. I’m hopeful. Relatively hopeful, that is.
2. Simon & Schuster, my own publisher, and one of the largest, has announced they no longer require authors to scrounge like begging worms for book blurbs anymore (it’s not like the publishers provide them, in case you didn’t know). Thank God. Or thank the new young head of their flagship imprint, Sean Manning. This change is long past due. It placed debut authors at a significant disadvantage, and, outside of a few very big names like Stephen King for horror or George R.R. Martin for fantasy, blurbs had basically no impact on sales ever.
3. Speaking of changes to old bad systems, the NIH announced they were cutting down indirect costs significantly, from 70% to 15%. While most non-scientists wouldn’t know what this means, this is a really big deal for how science works at a career level.
So what are “indirect costs?”
Basically, when I was at Tufts University, I was an assistant research professor. I mostly paid my own salary by getting grants. I applied for these grants, and then received the money which went to pay my salary (fixed independently of the amount I got, more just paid my salary over more years). It always sounded like a lot when I first applied. But then the university took a chunk, sometimes half or most of it—their “indirect cost.” Basically, indirect costs are the tax that universities levy on scientists for doing research under their roofs. So if you got a million dollar grant from the taxpayer for your research, the university took most of it.
Reducing indirect costs means that scientists themselves keep the lion’s share of the grants and funding they alone apply for and get, and avoids the situation where scientists grind for grants that then get used to pay for the new fancy gym for undergraduates. [Editorial note: As pointed out in the comments, it turns out that, while this is true for all the grants I’m used to getting, it’s not actually universal. There’s a class of big funders (including the NIH) for which indirect costs are negotiated afterward and therefore aren’t directly from scientist’s budgets at the point-of-grant received].
4. OpenAI released their new “Deep Research” tool. According to Sam Altman it can do somewhere between 1-10% of all economically valuable tasks in the world.
Economist and Marginal Revolution author Tyler Cowen was particularly impressed.
Others noted a large amount of hallucinations and mistakes for simple tasks, like when asked to list out all the NBA players (it only got 6 out of 30 teams correct). As usual, in my testing I was impressed by some aspects, but also noted a lot of times it just settled in on Wikipedia rewrites. There's already a huge amount of explainers that exist online for almost every subject, so it’s hard to tell the difference between impressive output and formulaic re-writes.
I did try to use Deep Research for the ultimate test for AGI, which is finding me the exact material equivalent of a mattress that is no longer on sale, one that is uniquely good for my back and that I’ve been avoiding getting rid of for years. Even with tags, pictures of the material, etc, Deep Research fell on its face and gave me a bunch of useless filler citations. It was no help at all.
5. Counterpoint: Nate Silver wrote a great post over at Silver Bulletin, “It's time to come to grips with AI,” in which he argues that, while AI wasn’t much of a factor in the 2024 election…
2024 was probably the last election for which this was true. AI is the highest-stakes game of poker in the world right now. Even in a bearish case, where we merely achieve modest improvements over current LLMs and other technologies like driverless cars, far short of artificial superintelligence (ASI), it will be at least an important technology. Probably at least a high 7 or low 8 on what I call the Technological Richter Scale, with broadly disruptive effects on the distribution of wealth, power, agency, and how society organizes itself. And that’s before getting into p(doom), the possibility that civilization will destroy itself or enter a dystopia because of misaligned AI. And so, it’s a topic we’ll be covering more often here at Silver Bulletin.
As much as I sometimes wish I could write about AI less, Nate is absolutely right: it’s by far the most important topic in the world right now, for better or worse, and that’s only going to become more obvious over the next few years. Some updated version of Deep Research will, one day, find that mattress for me.
6. Thankfully, scientific research is still done by smart humans! And I was recently amazed to see my name in a new paper in Nature (okay, in a Nature imprint, but still).
A Chinese team at the University of Beijing has been doing great work following up on my original theory of causal emergence. For those who only know me from my online writing, it’s a theory that allows scientists to measure emergence mathematically by identifying the most causally-relevant scales of systems. You can read more about it here and also in The World Behind the World.
This new paper, led by Jiang Zhang, explores an alternative to the measure of causation the theory first used, the effective information, using instead the dynamic reversibility. I really like the approach. They made use of a lot of the model systems from our original paper.
Their paper had great timing because, after taking almost two years off from science to just think and write and be as present as possible for my young kids, I finally have come up with some original scientific ideas worth pursuing again. So expect more from me soon in that department.
7. As you might have noticed at the beginning of the post, we have some new art from resident artist Alexander Naughton for this here Desiderata series. He made a huge collage of a lot of his previous artwork for TIP. Each entry I'll take samples from it to reveal more.
It's therefore a good time to remind everyone they can commission Alexander Naughton for art. If you are a magazine editor, or are in want of a new album cover, or have an idea for a children’s book or graphic novel, or need any sort of art at all, please do contact him.
He also has his own Substack, Illustrated, where he posts lovely comics and thoughtful essays on life and the creative process.
8. From the archives. Just in December I was arguing that the mass hysteria of New Jersey drones was nonsense. This has now been confirmed by official channels. It was airplanes and hobbyist drones and a small number of drone flights directly authorized by the FAA. On to the next hysteria!
9. As always for the Desiderata series, please treat this as an open thread. Comment and share whatever you’ve found interesting lately or been thinking about below, or ask any questions you might have.
2025-01-30 23:55:03
“I can't go on. I'll go on.” ―Samuel Beckett, The Unnamable
Life sucks sometimes. Your career will suffer setbacks. You’ll fight with your spouse. Your children will misbehave. If not outright disease or injury, then some other issue will plague you. Often the problems cluster together, little imps summoning their friends. Bad news comes in threes, as the saying goes.
My friends have another expression for this clustering. When everything goes wrong you’re “at the bottom of The Wheel.” I guess “The Wheel” is some sort of feat of karmic engineering, an invisible turning wherein if things get better for one, they get worse for another (you’re strapped to The Wheel too, you just didn’t know until now).
Which is why in the scholarly literature on depression there’s so much evidence for a series of negative life events being “triggers.” While for some, depression has no clear outside source, for many others it does; if not from one particular source like job loss, then via the aggregation of many.
So I’d like to share a personal analogy I myself have found extremely helpful when you feel at the bottom of The Wheel: the cold boring reassurance of mother math.
Here is what she whispers in your ear: “Please, child, meditate on statistics.”
What she means is that any process that unfolds dynamically, be it your relationship with your family, your career, your own health, anything at all really, rarely moves in a straight line. Instead, it meanders, it drifts, it dives, it dips.
Sometimes, of course, the line really does fall off a cliff. A health problem occurs, and you never recover. Bonds are broken, then never mended. But your life is a complex system, and complex systems don't usually move in ways from which there’s no return to some local area in the dynamics. Instead, they often revert to the mean. And funnily enough, the points where personal problems present themselves as overwhelming and consuming are statistically often destined to be the lowest region of a dip.
Which entails a kind of personal anthropic principle: an observer effect for when everything goes wrong. Because for any observer (you) experiencing the process from inside, the point of maximum concern about a problem will often match the point of inflection where the trend reverses and the problem gets “miraculously” better. For example:
2025-01-24 00:39:15
Since taking office, President Trump has already signed over 200 executive orders, which is almost as many as in the average presidential term. Critically, this included one repealing Joe Biden's own previous executive order regulating the AI industry.
That original executive order was signed by Biden in 2023, mostly to ensure AI outputs did not violate civil rights or privacy. Republicans argued it forced companies to make AI's political compass more left-wing, pointing to analysis showing most AIs aren’t politically neutral, but rather lean left in the valence of their responses and what topics they consider verboten.
Its repeal means there is essentially no significant nationwide regulation of AI in America as of now.
In a further boon for the industry, Sam Altman visited the White House to announce, alongside President Trump, the construction of a 500-billion-dollar compute center for OpenAI, dubbed “Stargate.” The promise of far-away and speculative rewards like “AI will make a vaccine for cancer” were floated.
Yet there are many reasonable politically-neutral voices, including Nobel laureate Geoffery Hinton (the “Godfather of AI”) who are worried about how the technology might, if left completely unchecked, erode how culture gets created, damage public trust, be used maliciously, and possibly even pose (one day) significant global safety risks.
So what should replace Biden’s expunged order? How should AI be regulated?
I think there is a way, one implementable immediately, with high upside and zero downside: ordering that AI outputs be robustly “watermarked” such that they’re always detectable. This is an especially sensible target for governmental mandate because such watermarking, in its technical details, requires the cooperation of the companies to work.
Specifically, what should be mandated is baking in subtle statistical patterns into the next-word choices that an AI makes. This can be done in ways freely available to open-source developers that aren't noticeable to a user and don’t affect capabilities, but still create a hidden signal that can be checked by some provided detector that has inside knowledge into how the word choices are being subtly warped (this works for code and images too).
Mandating watermarking does not tell companies how to create AI, nor does it create a substantial burden that ensures a monopoly by incumbents, since the techniques for watermarking are known, low-cost, and even open-sourced. It is regulation that is pro-freedom in that it does not place guardrails or force any political views. But it is pro-humanity in that it makes sure that if AI is used, it is detectable.
Here are four reasons why it would benefit the nation to watermark AI outputs, and how robust watermarking methods could be implemented basically tomorrow.
January is ending. Students are going back to school, and therefore, once again ChatGPT has gained back a big chunk of its user base.
With 89% of students using ChatGPT in some capacity (the other 11% use Claude, presumably) AI has caused a crisis within academia, since at this point everything except closed-laptop tests can be generated at clicks of a button. Sure, there's the potential to use AI as an effective tutor—which is great, and we shouldn’t deny students that—but the line between positive usage versus academic cheating (like making it do the homework) is incredibly slippery.
People talk a lot about “high-trust” societies, often in the context of asking what kills high-trust societies. They debate causes from wealth inequality to immigration. But AI is sneakily also a killer of cultural trust. Academic degrees already hold less weight than they used to, but nowadays, if you didn't graduate from at least a mid-tier college, the question is forevermore: Why didn't you just let AI do the work? And if calculus homework doesn’t seem important for a high-trust society, I’ll point out that AI can pass most medical tests too. Want a crisis of competence in a decade? Let AI be an untraceable cheat code for the American meritocracy, which still mostly flows through academia.
The internet is filling up with AI slop. And while the internet was never a high-trust place, at least there was the trust you were actually hearing from real people. Now it could all just be bots, and you wouldn’t know.
Reminder for the new Republican government: until basically yesterday, the open internet was the only space with unrestricted freedom of speech. You won’t always hold the reins of power. That space might be important in the future, critically so, and allowing AI to pollute it into an unusable wasteland where every independent forum is swamped by undetectable bots is a dubious bet that future centralized platforms (soon necessarily-cloistered) will forever uphold the values you’d like. The open internet is a reservoir against totalitarianism from any source, and should be treated as a protected resource; watermarking ensures AI pollution is at least trackable and therefore decentralized forums for real anonymous humans can still exist.
Future election cycles all take place in a world in which AIs can generate comments that look and sound exactly like real people. It’s obviously a problem that’s only going to get worse. Political parties, PACs and Super PACs, and especially foreign actors, will attempt to sway elections with bots, with far greater success than before. In a world where bot usage can’t be detected, this means free interference with the public opinion of Americans in ways subtle and impactful. Americans, not AI, should be the ones who drive the cultural conversation, and therefore the ones who decide what happens next in our democracy.
It’s a simplification to say the Republican position is pro-unrestricted AI in all contexts. Elon Musk, now close advisor to President Trump, has a long track record of being worried about the many negative downsides of AI, including existential risk. Even President Trump himself has called the technology's growing capabilities “alarming.”
AI safety may sound like a sci-fi concern, but we’re building a 500-billion-dollar compute center called “Stargate,” so it’s past time for dismissive hand-waving that malicious or rogue AI is purely sci-fi stuff. We live in a sci-fi world and have sci-fi concerns.
Many of us proponents of AI safety were shocked and disappointed when California’s AI non-political safety bill SB 1047, after making it close to the finish line, was vetoed by Gavin Newson for garbled reasons. Nancy Pelosi, Meta, Google, and OpenAI all worked against it. Its failure was because, frankly, California politicians are too influenced by the lobbying of local AI companies.
Therefore, an executive order is especially appropriate because it combats entrenched special interests in one state regarding a matter that impacts all Americans. An order on watermarking would be a massive win for AI safety—without being about AI safety explicitly. For it would mean that wherever an AI goes, even if loosed into the online wild to act independently as an agent, it would leave a statistical trace in its wake. This works for both minor scamming bots (which will become ever more common) as well as more worrisome unknowns, like AI agents bent on their own ends.
Often it’s wrongly said that watermarking AI outputs is impossible. What is true is that currently deployed methods throw enough false positives to be useless. But this is solely because AI companies aren't implementing the watermarking methods we do know work.
Companies don't want to implement actual watermarking because a huge amount of their traffic comes from things like academic cheating or bots or spam. Instead, they try to swap in adding easily-removable metadata as “watermarking.” For example, California currently is considering AB-3211, a bill supposedly about watermarking AI outputs, but which only requires that metadata is added (like little extra data tags, which are often removed automatically on upload anyways). This is why companies like OpenAI support AB-3211, because it’s utterly toothless.
One outlier company who has already made the move to real robust watermarking is Google, who (admirably, let’s give credit where it’s due) deployed such techniques for its Gemini models in October.
If one company does it, there’s not much effect. But if it were mandated for models of a certain size or capability, then to detect AI use would simply require going to a service that checks across a bunch of the common models by calling their respective signature detectors.
A paraphrasing attack is when you take a watermarked output and run it through some other AI that doesn't have watermarked output to be paraphrased (essentially, rewritten to obscure the statistical watermark). Critics of watermarking usually point to paraphrasing attacks as a simple and unbeatable way to remove any possible watermark. This is because there are open-source models that can already do paraphrasing and can be run locally without detection.
Traditionally, this has been a killer criticism. Watermarking would still be a deterrent, of course, as paraphrasing requires an extra hurdle and adds compute-time to malicious actors. E.g., when it comes to academic cheating, most college kids who use an AI to write their essays are not going to run an open source model on their local hardware. Or if hacker groups are spamming social media with crypto bots, they’d have to run twice the amount of compute for every message, and so on.
But importantly, there now exist watermarking methods that can thwart paraphrasing attacks. The research is clear on this. E.g., in 2024, a watermarking method was introduced by Baidu that is based on semantics (rather than exact phrasing) and has since proved robust against paraphrasing attacks.
To enhance the robustness against paraphrase, we propose a semantics-based watermark framework, SemaMark. It leverages the semantics as an alternative to simple hashes of tokens since the semantic meaning of the sentences will be likely preserved under paraphrase and the watermark can remain robust.
Essentially, you keep pushing the subtle warping higher in abstraction, beyond individual words to more general things like how concepts get ordered, the meaning behind them, and so on. This makes it very hard to disguise such a higher-level signal when paraphrasing, and so is secure even against dedicated and smart paraphrasing attacks that try to reverse-engineer the watermark. In fact, there’s now a large number of advanced watermarking methods in the research literature shown to be robust under paraphrasing attacks and even further human edits. Critics of watermarking have not updated accordingly.
Of course, for any adopted paraphrasing-robust watermarking, one day there may be developed some way around it that works to some degree. But avoidance will become increasingly costly and better methods will continue to be developed, especially under a mandate to do so. Even if watermarking is never 100% preventative against motivated attackers with deep pockets and deeper technical expertise, it locks even maximally-capable malicious actors into an arms race with some of the smartest people in the world. I wouldn’t want to be in a watermark-off with OpenAI, and I doubt hackers in Russia would either.
Now that they’re firmly in power, Republicans should still regulate AI, but in a way that doesn’t interfere with the legitimate uses of this technology, nor dictates anything about the politics or capabilities of these amazing (sometimes scarily-so) models, and yet one that still ensures a fair future with humans front-and-center in politics and culture. As it should be.
