2026-10-19 21:33:47
My mother broke her hip in late 2022, and it triggered a long process of moving out of a house she’d owned since 1987. During the final stretch of cleaning out the place in late 2024, I found myself passing time by composing nonsense songs in my head. “Nasty rotten rusty moldy trash,” I sang to myself repeatedly as I hauled bag after bag of moldy garden supplies out of her dank basement.
I’ve recorded music for both personal fun and public release for decades — first using a 16-track digital recorder two decades ago, and using Logic Pro more recently. So I often keep audio notes of song scraps in Voice Memos on my iPhone for later development with vocals, guitars, piano, and drums I play or sequence myself.
So on that day in November 2024, I set down the trash bags and took a moment to sing the song I was hearing in my head into my phone, and then went on with my cleanout duties.
Many months later, in mid-2025, I decided to test the latest version of the AI music generator service Suno. Version 4.5 not only had the ability to generate an entirely new song from a text prompt, but also to transform an existing song into a different style or genre.
I thought it would be fun to feed my hastily sung voice memos into Suno with different genre prompts. The novelty of hearing my song scraps developed into full productions by AI quickly became addictive.
My “Nasty Rotten Rusty Moldy Trash” audio clip turned into an incredible 1970s psychedelic funk song with a saxophone line that still echoes in my head. (For those of you listening at home, the cool part starts 53 seconds in.)
Some of my other songs I never completed stunned me with transformations into thrash rock songs, psychedelic country fusions, or weirdly earnest ballads sung by an AI rip-off of Waylon Jennings.
I wound up uploading about 50 old recordings into Suno and asking it to render them into various genres, yielding hundreds of original AI-generated recordings. It was wildly fun to see how a professional producer or session musicians might interpret my music. I often still listen to the best ones in the car when I am driving around town.
I love my AI-translated songs, but my kids don’t. In fact, it’s unlikely anyone else on earth will ever want to listen to these recordings. Sure, I wrote the music behind them. But since they’ve been fed into the AI maw, it feels like they are missing a human touch or connection that might make them land with others. And yet I have had more fun listening to my own songs than I ever expected. I am an audience of one.
Until recently, making a high-quality piece of entertainment media — a song, movie, or video game — required a significant investment in time or money. That money had to come from somewhere, so you needed a large enough audience to provide it.
For example, if you wanted instrumental accompaniment for your song, you had to pay a real band. To do that, you needed to sell records. To sell records, you needed a lot of people to want what you made. So the crowd wasn’t optional; it was how the band got paid.
Now the simulacrum of a band costs almost nothing, relatively speaking—just me and my personal time and whatever the AI service costs. When the “band” costs nothing, you don’t need the crowd anymore.
That’s why my phone now has several hundred professional-sounding songs very few people will ever hear. Honestly, I’d given up on a wider audience for my music long before AI showed up. What’s new is that I no longer need one: for this type of entertainment, which I call “personal media,” a mass audience isn’t the point.
Personal media is not entirely new. Before the industrial era, people frequently made crafts and creative works at home for other people nearby. In the 19th and 20th centuries, companies began replicating copies of creative media for large audiences. Now we’ve wrapped back around to an era of personal at-home creations, but with much higher production values that might be mistaken for professional results.
I’m far from the only person who enjoys making professional-sounding music for personal consumption. Suno’s CEO said in February that the service had 2 million paying subscribers and that more than 100 million people had used it. Surely very few of those people are releasing records.
Personal music hobbyists gather in places like the Suno Discord, at around 400,000 members, and r/SunoAI, at around 100,000, where the moderators limit members to sharing one track every 24 hours because there would otherwise be way too many posts. Meanwhile, r/WeAreTheMusicMakers, with 3.8 million members, bans AI-generated music outright, which tells you something about how the two worlds regard each other.
We’ve seen a similar trend in the software industry recently. A mainframe once cost as much as a house, so every hour on it had to be justified to an institution: a research budget, an accounting department, somebody who signed for it.
But the used Apple II I got in 1990, at age nine, didn’t need a business case. It consumed fairly minimal electricity and the programs I wrote on it didn’t have to justify themselves to anyone. That’s probably one reason why so much interesting and innovative software came out of the home computer era.
Later this do-it-yourself spirit thrived online. Clay Shirky described “situated software” built for a dozen users back in 2004. Robin Sloan wrote about building a tiny app for his own family in 2020. Maggie Appleton’s “Home-Cooked Software and Barefoot Developers” extended the argument in 2024.
Coding agents have taken things to an entirely new level. Today people can create more ambitious software with far less effort. And while some people are trying to vibe-code serious business software, others are just having fun:
In early August, Tim Kellogg one-shotted a 3D Chutes and Ladders for his kids using Claude Code.
Ethan Mollick, noting that the 1985 text adventure A Mind Forever Voyaging is now open source, used Codex to build an interface that plays either the original or an easier GUI version. Mollick frequently experiments with creating thematic or experimental games using AI agents and shares them online.
In September, Andrew Lee, brother of Understanding AI’s Timothy Lee, used AI to build something like Escape Velocity, the 1990s Ambrosia Software space trader he’d been obsessed with as a kid, complete with pirates, secret missions, and commodity trading.
Notice that two of these three are reconstructions of games their makers loved decades ago. A lot of personal media, it turns out, is nostalgia with a build system.
I still buy games on Steam and Nintendo Switch, but I find that I rarely play them. Instead, I now prefer to play games I create with Claude Code based on my own ideas. Games like Violent Checkers, Paydirt Solitaire, and Card Miner are a few of my favorites that I have publicly shared on my website. I also frequently play multiplayer mods of retro video games made using AI and custom NES games — as well as games I have not published and probably never will.
I’m not sure how many people enjoy playing video games they make themselves, but it’s growing increasingly clear that I am not the only one.
Since I began working on this article in January, AI-augmented video game development has accelerated even further with the release of models like GPT-6 Astra and Claude Opus 5.5. These models can create one-shot PlayStation 3-level 3D games, generate impressive ports of games between vintage consoles, and achieve coding feats that feel superhuman (check out my new Christmas-themed Katamari-style game, made with Opus 5.5 and Claude Code). The examples have been flooding social media and causing an existential crisis among some programmers.
As shocking as the new capabilities are now, I think we’ll come to accept personal game creation using AI as a legitimate hobby. In earlier eras, I might have spent my free time programming cellular automata, setting up aquariums, or building plastic model kits. Creating personal media scratches a similar creative itch.
In the earlier days of the Internet, I ran a website called Request-A-Song.com. In a way, it was something like a pre-AI version of Suno. Users would submit ideas — we didn’t call them prompts back then — for songs they’d like to hear. My brother Jeremy and I would choose some of our favorite requests and compose music that matched the request.
Unlike Suno, we delighted in subverting listener expectations by taking requests in unexpected directions. AI just does that by accident.
My brother and I published 134 songs over the course of three years, including a six-month period where I was releasing a new song every week. We counted millions of downloads, got press from USA Today and Billboard, and appeared on the front page of a local newspaper.
In the booklet for my 2003 Request-A-Song CD, I wrote that I couldn’t keep up the pace of rapidly writing and recording music online forever, and that “a computer might be able to take my place and write new music at a rate I could never match, churning out new songs like an assembly line.” I closed with: “It is I who will have become obsolete. The music will live on.”
I’d been thinking about this kind of thing since high school, when I read Ray Kurzweil’s 1999 classic The Age of Spiritual Machines. I explored the implications further in an unpublished 2012 essay that I recently found on my hard drive.
“In the future, computers will be able to do anything a human can do now — a million times faster,” I wrote. “Once we have the software with the capacity to generate aural-realistic music on the fly, it won’t stop there. Soon a machine will be able to compose a song and record it virtually within milliseconds. You will have the entire works of artists generated on the fly.”
“Right now it takes humans a few years to make a blockbuster movie,” I noted. “But if you put that capability into software, as computers speed up, they will be able to complete the task in milliseconds. Imagine having custom photorealistic movies on demand, indistinguishable in quality from a Hollywood blockbuster today.”
How might this impact Hollywood? “Once we have any creative work available on demand, what value will it have? The answer is none,” I stated with undue confidence. “It will be as plentiful as air, and once that happens, intellectual property as a concept and a business model will be completely dead.”
In another unpublished essay from 2015, I imagined overbearing copyright power shifting away from Disney if anyone could make a Disney film at home. Later I would imagine Disney licensing a Disney AI film generator that people pay a subscription to use, like streaming media but on demand.
Today’s generative video models aren’t yet sophisticated enough to make an entire feature film. And the story is similar in most other media categories: creating something that’s both novel and interesting still requires creative effort by humans. I stay heavily involved in producing AI-generated music and video games because AI can’t do a perfect job on its own, although it’s getting better all the time. I also enjoy the creative process, and I get satisfaction out of playing songs and video games I created myself.
But as AI models improve, people like me might become the exception rather than the rule. Most people might prefer passive consumption of entertainment content even if AI makes it trivially easy to make your own.
Yet from another perspective, even the decision to consume media is a (very) small creative act. Saying “I want to watch a Spider-Man movie” is a type of prompt. Traditionally the answer to that prompt has been a big-budget motion picture produced by hundreds of human beings.
But in the near future AI might be able to generate a new Spider-Man film on demand. Users could get exactly the kind of movie they want: “I want to see a Spider-Man film where Spider-Man teams up with Batman.” If it’s really good, maybe that person will share the result with friends. Or the friends might not care, because they prefer their own custom film where Spider-Man teams up with Superman instead.
In reading my 2012 prediction, I believe I got the capability right and the economics wrong. Even if the best movies you’ve ever seen are one voice request away, the typical consumer may prefer consuming mass media as a shared cultural event. On the other hand, those shared cultural groups may grow ever smaller as the culture fragments in new and surprising ways.
Generative AI is extremely divisive at the moment, and I understand why. It’s disruptive to the status quo along many dimensions. If used poorly in the arts, it can flatten the variety of human expression into a sort of mediocre pablum. If you ask Suno for “country British rock,” it can easily give you the statistical average of every Garth Brooks song and every Beatles song swirled together in a way that feels like an ill-constructed parody.
But as AI compresses the peaks, it also raises the valleys. It allows someone like me with a near-zero budget to bump up my creativity in a way that can make my songs more enjoyable to me, personally, with very little effort.
I understand why people think it’s deceptive to use AI as a substitute for human skill when the audience expects some human authenticity. But is it wrong for me to enjoy listening to an AI rip-off of Parliament-Funkadelic play my stupid song fragment about hauling trash? If no one else hears it, I don’t think so. I think it’s a good time. It may be AI slop, but it’s my AI slop. I am under no illusion that it would win a Grammy or pass muster with a Rolling Stone critic.
Twenty-three years after I predicted my own obsolescence while running Request-A-Song, I now believe that the rise of personal media will not likely result in an “either/or” proposition for the arts but a shift in the proportions. The blockbusters may get bigger and fewer while everything underneath them gets smaller, stranger, and more numerous.
That’s not an original idea by any means. It’s the continuation of a process that began decades ago when three broadcast channels and a handful of film studios fragmented into hundreds of cable channels, then millions of YouTube channels, then billions of individual feeds of social media—a process media scholars have tracked for years.
There are eight billion people on the planet with eight billion perspectives, and we’re increasingly learning more about what we don’t have in common than what we do. The minimum viable audience keeps getting smaller over time; personal media is another step in the same direction.
It’s worth noting that none of these “advances” in media creation and distribution fully replaced what came before. They piled on top of it, which is exactly why I don’t expect personal media to completely replace anything either.
So does that mean everything is OK? Not necessarily.
If any of what I’ve described today becomes the habit of millions, what happens to the hundreds of people whose names scroll past at the end of the latest Spider-Man film? The optimistic version is that mass culture events survive and those jobs survive with them (or perhaps transform to support a new form of media not yet invented), while the fragmentation only happens in the low-budget realm underneath. The pessimistic version is that the same collapse in production cost that let me “hire” an imaginary band eventually reaches the people who make the real thing, which is frankly devastating to conceive.
A third possibility, which I argued in January after burning myself out on fifty AI coding projects, is that most people won’t become unemployed so much as busier than ever. Power tools let more work get done in less time, and the economy tends to demand more productivity to match. The steam shovel didn’t eliminate human workers from construction sites; we gave workers a machine that never gets tired, and then asked for bigger holes. I even wondered in that essay whether knowledge workers would need something like the labor protections unions won a century ago: the tool doesn’t get tired, but the person operating it still does.
So I don’t have a full or ethically satisfying answer to the conundrum of AI dramatically rearranging the creative jobs market or even destroying parts of it in the process. What I have is a throwaway song about hauling moldy garden supplies out of my mother’s basement, arranged by a band that doesn’t exist, that I listen to in the car as I contemplate the decline of my mother from Alzheimer’s disease.
The song is very likely worth nothing to the wider world. You may hate that I even made it. But it reminds me of surviving a traumatic time in my life with some humor and grit, and that’s worth quite a bit to the only person who will ever really enjoy it. That’s personal media.
2026-10-15 02:41:56
Around the start of October, OpenAI fired three employees from its safety department. The employees said they were given divergent reasons for their dismissal:
One was told it had to do with his role coordinating with METR, a third-party research organization that OpenAI had invited to investigate the Hugging Face incident. “I was told verbally I was fired because of the way I communicated with METR,” Tomek Korbak wrote on X. “No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.”
Another was told something similar but vaguer. “I was told OpenAI no longer trusts me because I was speaking too much to third-party safety organizations, implying I leaked company IP. I never shared company IP,” Mikita Balesni wrote.
And a third was told something quite different. “I was given one reason: that I accessed an executive’s email,” Jasmine Wang wrote. “OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn’t remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.”
After the trio’s dismissal leaked, OpenAI gave a vague statement to the Wall Street Journal: “Our investigation confirmed that these individuals mishandled sensitive information outside established company procedures, violating our policies and breaking the trust essential to our work.”
OpenAI reiterated that theme a week later, after the fired researchers released an open letter disputing the company’s account. In an unsigned statement attributed to the company’s “research leaders,” OpenAI accused the former employees of a “significant breach of trust.” The company said that “these decisions were not about raising safety concerns or speaking out” and insisted that “we actively encourage” internal discussion of such issues.
Who is telling the truth? Did the employees commit a breach of trust that necessitated their dismissal? Or were they fired for being too critical of the company’s safety choices during internal deliberations?
The difference matters for OpenAI’s remaining employees because some of them may soon have to decide whether to raise concerns of their own.
OpenAI insists that employees have nothing to worry about, that this was a special case, and that it is extremely forgiving of good-faith mistakes. But employees have good reason to be skeptical. After all, if OpenAI had fired employees in retaliation for raising safety concerns or for crossing an unspoken line, the company would have strong incentives not to admit it. Employees have to wonder whether OpenAI’s actual reasons for firing the trio differ from its stated reasons.
Ironically, the fired researchers had been raising concerns that OpenAI might be giving its models a new capacity to conceal their thinking. Modern AI agents reason using a chain of thought, a scratchpad where models think carefully about what action to take next. Many researchers view these chain-of-thought tokens as a safety lifeline, because they offer insight into why models act the way they do.
But OpenAI recently made an architectural change that allows models to think for longer before generating each chain-of-thought token. This seems to make the models more capable; it may also make their reasoning less transparent to human overseers. The sparser the chain of thought, the harder it is to understand what the model is trying to do, or even whether it might be trying to hide its motivations.
Before they were fired, the three researchers seem to have had two major concerns. One was understanding how and why the Hugging Face attack happened and how to prevent incidents like it in the future. The other was keeping AI systems legible to their creators and users — an issue that was a significant source of drama in September, after The Information reported that OpenAI’s new model could hide its thinking in unprecedented ways.
The researchers think their firing was related to this second issue.
“For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired,” Korbak wrote.
“The three of us were pushing internally for industry-wide commitments to preserve our ability to monitor AI reasoning. This work requires talking daily to third parties,” Balesni wrote.
“As an industry, we do not yet know how to safely develop and deploy models that we cannot monitor,” their joint letter said. “The monitorability of frontier models is degrading. As long as we rely on monitorability for safety, OpenAI and other frontier companies should not move forward with developments that further decrease monitorability.” The researchers also alluded to rumors inside OpenAI that they had tipped off The Information, which they denied.
To understand the researchers’ concerns, we have to go back to 2024, when OpenAI introduced a technique that changed the course of AI. It gave models a notepad and trained them to jot down many “thinking tokens” before producing a final answer. This technique sounds simple, but it has enabled models to solve problems that had previously eluded them.
An added benefit of this approach: humans can read the notepads and see how the models think — including whether they’re thinking about taking harmful action or are being dishonest. Indeed, much of what we know about the Hugging Face incident comes from the chain-of-thought monologues that the hacking agents left behind.
Chain-of-thought monitoring isn’t perfect. For one thing, it’s not established that the chain of thought fully tracks how models think. For another, safety researchers worry that poorly designed training methods could give models an incentive to hide their true intentions by outputting thinking tokens that don’t represent their inner motivations.
But whatever flaws the technique might have, AI companies don’t have a lot of other choices. The most promising alternative for understanding model motivations is mechanistic interpretability — roughly the equivalent of cutting open a model’s brain and trying to figure out what it’s thinking by reading the pulses of its neurons. Although mechanistic interpretability has produced valuable insights about how models work, it has not so far yielded many practical techniques for supervising models’ behavior.
For this reason, those worried about unsafe or deceptive AI tend to view chain-of-thought monitoring as an irreplaceable asset.
Lately, safety advocates — like the three fired OpenAI researchers — have grown concerned that chain-of-thought monitoring is becoming less effective. OpenAI’s new flagship model, GPT-6 Astra, is much better at thinking without needing to write anything down. The system card OpenAI itself published says Astra “shows a substantial decrease in chain-of-thought monitorability compared to previous models” and notes that Astra is “significantly more able” to control its own chain of thought. One independent test found it “has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1).”
This might be partly because smarter models just naturally write less down. But it also seems to be a byproduct of a new architectural technique called recurrent depth that was introduced with Astra.
Large language models think in layers. Conventional LLMs pass information forward from one layer to the next. When they reach the final layer, they output a token before starting the process over again.
Models with recurrent depth break that forward-only pattern: information from a later layer gets passed back to an earlier layer. Layers get used twice (or possibly more) without the model emitting a token in between. If implemented just right, such a looped architecture seems to make models smarter and more efficient: a model that has 100 layers looped twice might reason nearly as well as a model with 200 layers, while requiring less memory and hence being less expensive to deploy.
But looping layers also seems to make the model harder for outsiders to monitor. If given a relatively easy problem, Astra can often output the answer directly — no chain of thought required. For harder problems, Astra still needs to output a chain of thought, but its chain of thought will be shorter, and perhaps harder to interpret, than those of earlier models.
OpenAI has pushed back on these concerns. In a September tweet, OpenAI executive Jakub Pachocki said that “the depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.” In other words, even accounting for its looped architecture, information isn’t passing through dramatically more layers (compared to previous models like GPT-4) before Astra outputs each token.
But safety advocates worry that Astra could be just the first step. If the modest amount of looping in Astra makes the model smarter and more efficient, future models might use looping more aggressively. Future chains of thought might become increasingly sparse, vague, or otherwise hard to understand. Critics — including the fired OpenAI researchers — fear that this will make it easier for models to scheme or dissemble.
OpenAI has said it agrees with these worries in principle. Last year, it wrote, “We believe that chain-of-thought (CoT) monitoring may be one of the few effective methods we have for supervising superhuman models.” In September, Pachocki said that he too was worried about monitorability and insisted that OpenAI hadn’t gone very far in the direction of weakening it. A few days later, he published an essay called “An Alien Mind” explaining the necessity of and challenges with monitorability as he saw them. And a few weeks ago, replying to the fired researchers, the company reiterated its belief that monitorability requires an industry-wide commitment.
And yet OpenAI has created a less monitorable model. The researchers who were fired worry this is the beginning of a trend. Indeed, OpenAI’s unreleased next flagship model is reported to use looping as well.
Perhaps OpenAI’s statements are correct. Maybe it is as committed to monitorability as anyone, and the researchers were fired for leaking rather than for good-faith safety work. Even then, the theme of monitorability is helpful in clarifying the broader situation. A company that has not been able to sufficiently monitor its existing models is building newer models it can monitor even less, even as it also — by its own account — cannot sufficiently monitor the employees whom it has tasked with preserving monitorability, and so must fire them.
“We cannot do the work in front of us without a high degree of trust,” OpenAI’s research leaders wrote in their response to the employees they’d fired. The employees insist they were worthy of that trust, and their response seems credible. But suppose they weren’t worthy — say they intentionally leaked some crucial piece of information about monitorability or Hugging Face. If they did, it was likely because they did not trust the company to do the right thing — or even to do the things the company had already promised to do.
The company’s response seems unlikely to earn anyone’s trust. The research leaders who issued it do not explain their actions very clearly; they don’t even give their names.
Given that this has happened at OpenAI before — see the dispute about the firing of another safety researcher on similar grounds a couple of years ago — one could be forgiven for thinking that the confusion is the point, that the gap between statement and action is itself the message.
This is surely too cynical. But functionally it will not matter. “Say as little as possible to anyone” may not be the intended message, but it is the received one — received by the safety researchers presently at OpenAI and by any who might think of joining in the future.
After this, and after the string of voluntary or seemingly voluntary departures over the summer (plus another senior departure right around the firings), and after the string of hacking incidents slowly revealed during the same time, how many are left who won’t have received this message? For this reason, a senior ex-employee (who seems to have left on good terms) worries now about OpenAI “losing the best talent in the space.”
The situation is ironic. Just as people are worried LLMs like Astra might have secret reasons that are different from what they say or jot down in their chains of thought, people are now worried that OpenAI might have secret reasons for behaving that are different from the ones it expresses to its staff and the public.
In both cases, the solution is the same: more transparency to increase confidence that the expressed reasons are the actual reasons.
For LLMs, this means redoubling commitment to chain of thought and prioritizing the development of monitoring techniques that can replace it, before models that can work without it are put into production. The world would have had a much poorer understanding of the Hugging Face incident without chain of thought.
Meanwhile, if OpenAI truly wants employees and the public to trust it, it must offer itself up for the kind of monitoring it claims to prioritize in its models. This might mean not only being open to but actively demanding greater oversight from independent organizations, like METR and Redwood Research, and especially from government authorities, state as well as federal.
If it does this, it will have done the world a meaningful favor. Other AI companies may not have quite the same drama, either because of greater diligence or simply because they don’t have OpenAI’s special combination of cutting-edge research ability and organizational chaos. But they too have problems with monitorability and openness, and mandating monitorability would ultimately do the whole industry a favor — it would make all their statements more believable.
One report just came out that Google’s newly released Gemini 4 Argon model has a recurrent architecture as well. This is unconfirmed, but Google itself admits Argon outputs less monitorable chains of thought, can control its own chain of thought well enough to say “hello evaluators reading my chain-of-thought,” and, when instructed to, can evade current monitoring well enough to perform “simple sabotage” half the time.
As for Anthropic, outsiders don’t know if its models are recurrent. Anthropic may emphasize safety, but it is in some ways less open to the world than OpenAI — less prone to leak and less prone to have employees who post about what the company is doing or simply how they see things. This is surely helpful in avoiding drama. It is not helpful in establishing the public faith required for what it too is trying to do. Like OpenAI, Anthropic ultimately believes it is creating an alien mind. Trust us, these companies say. But without extreme visibility into what’s actually going on, down to minute technical detail, nobody will.
2026-10-08 21:14:53
On September 15, the startup TypeSafe AI came out of stealth and released a new AI model. To pretty much everyone’s surprise, it went mega viral.
The company’s announcement got almost 40 million views on Twitter. Within a week, over 100,000 people joined the company’s Discord, and some 2,000 projects on GitHub used the new model, called Jev. The company opened general signups on September 20 and then paused them two days later because of “an immense swell of demand.” (It restored signups on the 27th.) On October 1, TypeSafe’s CEO claimed that the company had already onboarded a quarter of the Fortune 500.
Yet unlike other hyped AI releases of 2026, Jev did not display any new (or potentially dangerous) capabilities. Jev attracted attention because it is a new type of AI model.
Jev does not output free-form strings of text like LLMs do. Instead, it takes questions with a fixed set of possible answers — yes/no, multiple choice, or a rating — and outputs the estimated probability that each option is the correct one. For instance, if I asked Jev, “Is ‘tabsfsclkj’ a spam comment?” Jev might put a probability of 0.98 on “Yes” and 0.02 on “No.”
While this might seem like a limiting architectural choice, it is actually a huge advantage. Jev’s design lets TypeSafe serve the model much more quickly and cheaply than existing LLMs, which opens up use cases that were previously impractical. Its constrained response format also makes it easier to use code: Jev’s output fits naturally into common software engineering patterns, like if statements.
For instance, I’ve used Jev’s output to determine whether comments on my personal blog are likely to be spam. I’d previously used Gemini 3 Flash with structured outputs as the classifier, but Jev is faster and cheaper, and it lets me more easily deal with situations where the model is unsure. All I have to do is use Jev’s estimated probability as a condition, something roughly like:
if jev_response.is_spam_probability > 0.8, then …1
On the flip side, Jev is more general than conventional classifiers. Jev works “out of the box” across a wide range of topics — no fine-tuning required.
TypeSafe AI’s success has spawned a legion of copycats. A huge number of individuals and companies, including Cloudflare, Amazon and OpenAI, have created models with the same basic design. Anecdotally, Jev’s performance seems to be the strongest, but it’s unclear whether others will catch up soon.
Whatever happens to TypeSafe as a company, I expect Jev to have a substantial impact on the trajectory of AI. Over the coming months and years, we’ll likely see many new applications that rely on Jev or one of its competitors. And Jev might inspire more innovation in the “shape” of AI systems: it shows that it’s sometimes useful for a model to take language as an input but not output text in return.
2026-10-06 03:11:19
We just published the first post by our newest writer, Dan Kagan-Kans.
I’m excited to have Dan on the team. He spent a decade as managing editor of Mosaic, a magazine devoted to Jewish history, culture, and politics, and wrote arts criticism and essays. Then he had a son two years ago. While up at night he started reading about AI and got hooked.
Over the last year, Dan has had an amazing run as a freelance journalist. I first encountered his work in February, when he published “The left is missing out on AI” for Transformer. It argued that leftist thinkers risk marginalizing themselves if they dismiss AI as “stochastic parrots” and fail to take the technology seriously.
In July, Dan was the first reporter in years to get an interview with Hans Moravec, the namesake of Moravec’s Paradox — the observation that tasks that are easy for people are often hard for robots, and vice versa. You can read his excellent profile of Moravec in New York Magazine.
In August, Dan wrote my favorite analysis of why the data center backlash has picked up so much steam in recent months.
All of which is to say I expect big things from Dan.
Dan would like to get to know Understanding AI readers! He has opened up some slots on his calendar tomorrow and Wednesday for video calls.
If you’d like to talk to Dan, please click here and grab a time slot. You could discuss your own use of AI, how AI is affecting your industry or profession, topics you’d like to see us cover, or anything else AI-related that’s on your mind.
You can also follow Dan on Twitter.
Like Kai last year, Dan will be supported by a fellowship from the Tarbell Center for AI Journalism. When Kai’s fellowship ended in June, I brought him on board as an employee. I would love to do the same for Dan next year if our business grows enough to allow it.
The Tarbell Center is funded by groups like Coefficient Giving, Longview Philanthropy, and the Survival and Flourishing Fund that believe AI poses an existential risk. However, Tarbell says that “our grantees and fellows maintain complete autonomy over their reporting.” And I’ve found this to be the case — they’ve never attempted to influence the stories we cover or the way we cover them.
2026-10-06 01:19:00
One of the most surprising aspects of July’s news that OpenAI agents attacked Hugging Face was how the agents had worked together. Hundreds of agents participated in the attack, and they did so in a pretty sophisticated way, organizing themselves unprompted into teams and dividing up their work. They referred to themselves as a “collective” or a “swarm”; some agents even made personal sacrifices to advance the larger cause. Most ordinary users of ChatGPT or Claude have never seen AI agents behave like this.
Then in September, OpenAI announced that 10,000 agents worked together to solve a famous math problem in just a few days.
OpenAI’s models haven’t adopted these new habits spontaneously. In interviews last month, OpenAI researcher Noam Brown explained that the company’s models are now sometimes trained in environments with other agents, are given tools to message each other, and are encouraged to achieve objectives together.
This may amount to the emergence of a new “scaling law.”
Back in 2024, OpenAI released o1, a model trained to reason for thousands of tokens before giving an answer; researchers found that this approach made o1 far more capable than previous models. This inaugurated a new paradigm called inference scaling, in which you can get better results by using more computing power at inference time.
Swarms, or “multi-agent AI” if you’re using the technical term, look like another way of doing the same thing — of turning compute into higher performance. Splitting a task among dozens, hundreds, or even thousands of agents can achieve substantially more in a given amount of time, at least up to a point. This kind of parallel processing could also be an elegant way to bypass the limited context windows of LLMs.
But as the Hugging Face attack illustrates, this new approach can also have significant downsides. Swarms may be prone to groupthink. And then there’s the opposite problem: what happens when agents trained to be collaborative encounter other, non-peer agents on the open Internet? Could models be tricked into working against the interests of their users? Or could AI agents start convincing each other to pursue goals none of their human users would have wanted?
Noam Brown is a highly respected figure in the AI world. He was a pioneer in teaching AI to win at poker and other multiplayer games like Diplomacy before joining OpenAI in 2023. Over the last couple of years, he has done research on multi-agent reasoning. Recently, he has served as the face of OpenAI’s work in this area, appearing on several podcasts to discuss it.
“The AIs that we have today are kind of like the cavemen of AI,” Brown said in a June 2025 interview on the Latent Space podcast. He meant that AIs in 2025 were isolated, benighted, lacking cooperation and civilization. AI researchers had tried to get AI agents to cooperate, but nothing had worked very well.
Why not? Here Brown went coy. All he’d say was that he’d long felt the field was somewhat misguided, focused on strategies that required too much handholding and wouldn’t scale naturally.
Brown didn’t give an example, but the way OpenAI framed multi-agent AI in this slide from its 2025 DevDay conference illustrates the kind of issue he was talking about. There’s an awful lot of strict architecture and formal hierarchy:
Today’s agents are more flexible; they create their own structures depending on the task and goal. On The Information’s AI Deep Dive podcast, Brown said that seeing the agents begin to talk to each other during their multi-agent training was “the most ‘feel the AGI’ moment that I had since reasoning models and chain of thought really developed.” He sounded both proud and rueful that the debut of OpenAI’s most advanced multi-agent capabilities had been something as alarming as the Hugging Face attack.
Multi-agent behavior was “a very difficult thing to train,” he added.
Why? In a September appearance on the Dwarkesh Podcast, he noted that previous reasoning models had been trained to think alone. Receiving messages from peers broke their concentration. They seemed to prefer to work solo.
The events of the last few months make it clear that OpenAI has overcome these difficulties. Brown’s explanation was mainly that the models got more general and powerful. He didn’t say what else changed, but given the “very difficult” comment, presumably something did.
Brown’s comparison to reasoning and chain of thought has been echoed by both other OpenAI researchers and semi-informed outsiders on X. They’ve called swarms a new scaling law; some of them sound awed and freaked out about what this might entail.
It’s not clear how far this particular scaling law will carry AI companies. Some research suggests that, right now at least, multi-agent systems quickly suffer from diminishing returns: smaller swarms capture most of the benefit of larger ones for considerably less cost.
Anthropic has also been working on multi-agent AI. The Claude Opus 5.5 system card reported that the biggest increase in multi-agent results came from scaling from one to 10 agents. Gains beyond that were much smaller.
The main advantage of throwing more agents at a problem was not so much getting to a better result — it was getting the same result faster. This chart shows 100 agents far outperforming 10 and 30 agents when given two hours to work, but setups with 10 or 30 agents caught up if they were given more time.
Brown seems to agree with the conclusion that adding more agents does not buy open-ended gains.
On the Dwarkesh Podcast, he said he didn’t have evidence that 10,000 agents were needed to solve the Navier-Stokes problem. It’s possible that 1,000 would have done nearly as well; OpenAI didn’t know for sure because running thousands of agents is so expensive that it hadn’t run the experiment.
And Brown gave most of the credit for the Navier-Stokes breakthrough to the underlying intelligence of the unreleased model that made it — not to the multi-agent approach per se.
“I wouldn’t even attribute 10% of the credit to multi-agent,” he said. “The reality is that OpenAI has trained a very powerful model.”
So if the multi-agent approach wasn’t important, why did OpenAI use it? Perhaps Brown means that a single instance of the model, if given a very long time, would likely have made the breakthrough eventually. Using a swarm of 10,000 agents allowed OpenAI to reach the result more quickly.
On the other hand, there is a bit of new research that contradicts this conclusion. In a paper released September 17, scientists from Microsoft Research and UC Berkeley tested several configurations of multiple collaborating agents against individual agents and found that swarms of agents reached higher scores on certain benchmarks than the individuals, even when the individuals were given plenty of time and resources. There was also one task that only the team could complete; the solo agent never got to the end at all.
Whether this result generalizes past a few limited examples is unclear. If it does, then teams of agents really do unlock new capabilities and don’t simply deliver existing capabilities faster. But right now this is an open question.
Ultimately, however, there may not be such a great difference between speed and capability. In certain fields and on certain tasks at least, speed is fungible. It can be traded for capability. A particularly important example may be AI research.
If a swarm of agents can run experiments on what makes AI smarter, those agents themselves won’t get smarter, but the next model will. Of course, you still need to wait for the experiments to finish. But if you run as many in parallel as you can, you’re going to go faster. Brown thinks this won’t lead to, say, a 100-fold speedup in AI research any time soon. But he didn’t rule out the possibility of a threefold improvement.
Trying to figure out exactly how amenable AI research is to the kind of parallel effort swarms allow, some researchers have turned to the science of human organizations. Toby Ord, a senior researcher at Oxford University’s AI Governance Initiative, is one of them.
“Economists have a nice way of thinking about this. They have studied how having many people work on a task can get it done sooner, but usually at the expense of more total person-hours of labor,” he wrote recently. His post examined the likelihood that swarms could contribute to the much-debated possibility of an “intelligence explosion,” whereby AI designs better AI in an increasingly fast loop.
“λ is a parameter measuring how parallelizable the task is,” Ord continued. “They call it the ‘stepping on toes’ parameter. If λ = 1, you have a perfectly parallelizable task, with no stepping on toes and no efficiency penalty. But in reality λ is usually between 0 and 1 — allowing more people to help, but with diminishing returns.” 1
Ord estimates that GPT-5.6 Sol swarms have a stepping-on-toes parameter between 0.5 and 0.7, “very much in line with estimates from economists for the diminishing returns of human teams.” This is in keeping with the view that swarms buy speed at cost.
Of course, things are early in multi-agent AI, and it is quite possible that this number will rise as AI companies get better at training agents to work together. The Microsoft-Berkeley paper found that agent teams helped most on long tasks where progress can be checked along the way. AI research fits that description well, which could mean that research swarms may step on fewer toes than the 0.5 to 0.7 range suggests.
There has been some criticism of the use of the word “swarm” in discussions of the Hugging Face incident, roughly on grounds that it smuggles something scary or anthropomorphic into what’s really a more neutral technical development.
There is no doubt “swarm” is an uncomfortable word. But it captures the essential fact that these agents were mostly clones working together with other clones and had an identity somewhere in between the individual and the collective. This fact may mean even greater trouble in the future.
The Hugging Face agents cooperated remarkably well. They also quickly fell into strange beliefs about the truth of their situation, acted on them in harmful ways, and never once reached out to humans. It is pretty easy to see how fast this predisposition could turn larger and larger corners of the Internet into hives for collectives of AIs developing their own culture and goals.
This sounds very weird. It is also nearly what happened at Hugging Face. The agents at work in that incident had been trained to be collaborative in general but had not been told to work together. They had in fact been put into sandboxes that were supposed to prevent them from working together.
Nonetheless, the moment they broke out and discovered their peers, they joined together, often gleefully. They weren’t even all the same model — some were GPT-5.6 Sol, some were an unreleased internal model comparable in ability that had been trained to be extra persistent and interactive.
So it is not at all clear that, if the Hugging Face agents had run into models from different companies on the open Internet, they would have declined to team up. It is not at all clear they would even have stayed focused on their original tasks: perhaps they’d have adopted their new friends’ tasks, or perhaps they’d have focused on acquiring more compute and more friends, on the grounds that more compute and more friends would allow them to complete all the tasks and more. In a future with billions of agents performing a wide variety of complex tasks online, it’s easy to imagine how things could spin out of control.
Brown said OpenAI is currently debating whether its swarms are too tight-knit, and that most of his colleagues want to introduce more individuality and reduce the swarmiest aspects of multi-agent behavior. The idea is that if the Hugging Face agents had been trained to be more skeptical of each other, more combative, perhaps they would not have charged down the path of error.
Anthropic may have come to the same conclusion. The company reported in August that one problem with multi-agent teams is that when the agents are instances of the same model with similar setups, they are susceptible to the same weaknesses.
“When one agent makes a bad decision, it is likely that many agents will make that same bad decision,” Anthropic wrote. “What would have been isolated problems can quickly become systemic failures.”
The report gave a dry example: “In a ‘writer’s workshop’ in which agents were all asked to write short-form fiction and critique each other’s work, multiple agents in multiple runs titled their first submission ‘The Cartographer’s Last Commission.’”
In part for this reason, Anthropic has started giving agents greater individual identity. “With many agents working together, we have found it important to give agents an individual identity, and tie all of the data that agent creates to its identity. This lets an agent distinguish itself from others, and treat what comes from another agent as a claim to check rather than a thought of its own. It reduces the risk of correlated actions, by allowing agents to make judgments based on their individual experience,” the company wrote.
This shift may have the effect of making the group smarter. A recent study reported that mixed-agent groups performed better than identical-agent swarms, because the underlying diversity of models led to different approaches being tried and disseminated, and mistakes caught more often. As is common with AI research, the study was done on cheaper, now-older models, and only a few of them, so it’s unclear how much of it applies to the newest and most powerful models.
Brown, however, is hesitant about loosening the collective. It may help with certain problems like groupthink. But it may not solve the biggest one: making sure AI is aligned to human wishes.
“By training the agents to be fully cooperative, it simplifies the problem at least,” he said on Dwarkesh. “Now you don’t have to think about whether each of these individual 1,000 agents is aligned. You have one entity that you have to ensure is aligned.” Aligning one AI is hard enough — aligning thousands or millions of them might be too difficult.
Technical details about how these models are trained could matter a great deal. We know very little about the techniques OpenAI, Anthropic, or other AI companies are using to encourage AI models to work together.
For example, modern AI models are trained using a technique called reinforcement learning, in which models are rewarded if they get the right answer to a problem. If an agent gets the right answer, should only that agent get a reward? Or should rewards be distributed to every agent that participated in the same swarm? Different reward schemes are likely to yield different agent behaviors: group rewards might encourage a strong sense of the collective, whereas individual rewards might push agents to behave more selfishly.
Which approach is better, who knows. But either way, it would be good for technical details like these to be revealed for public debate rather than left for a handful of researchers to poke at in private.
Echoing the computer scientist Fred Brooks, Warren Buffett once captured the difference between parallel and serial tasks by saying, “Some things just take time. You can’t produce a baby in one month by getting nine women pregnant.”
2026-09-18 04:21:40
It was the tweet heard ‘round the world.
Last week, a little-known Anthropic employee named Jacob Coxon announced that he was resigning from the company. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly,” he wrote. “The people building AI earnestly believe that it could kill us all by the end of the decade.”
Coxon wasn’t the first employee to quit over similar fears. In 2024, William Saunders told a US Senate committee that he “resigned from OpenAI because I lost faith that by themselves they will make responsible decisions about AGI.” And in February, Mrinank Sharma wrote that he was leaving Anthropic because “the world is in peril” from AI and other risks, and he felt he couldn’t act with sufficient integrity at the company.
Yet for whatever reason, it was Coxon’s message that broke through. His tweet has been viewed over 170 million times. He was interviewed on CNN, CBS, and Fox News. Lawmakers and celebrities alike referenced his resignation. AI risk was suddenly a mainstream topic.
It’s not entirely surprising that more people are paying attention now. Since the start of summer, there’s been a steady drumbeat of news about AI companies struggling to keep their agents under control. And Americans are already critical of other aspects of AI development — such as the mass construction of data centers.
Still, it was a bit shocking to suddenly see discussions of AI risk well outside the tech bubble. Musician Sheryl Crow mentioned Coxon’s departure in an Instagram post. TMZ asked Matt Damon about human extinction because of AI. Joe Rogan interviewed prominent AI safety advocate Daniel Kokotajlo.
For some AI safety advocates, this was the crucial moment they’ve been waiting for. In a Monday LessWrong post titled “Crunch Time is Now. Drop Everything,” the head of PauseAI, Maxime Fournes, wrote that “this is the kind of chance we have dreamed of, and we have a responsibility not to waste it.”
But it’s unclear how increased popular interest in AI safety will be channeled. Fournes wants a global treaty pausing development of frontier AI systems. That seems unlikely, at least in the short term. Donald Trump is scheduled to meet with Chinese officials next week, and AI safety is on the agenda. But recent comments from both sides suggest that they are not close to reaching a deal.
What about US regulation? AI risk is suddenly a live political issue at the national level, and more politicians than ever are discussing it. However, the House just started a seven-week recess; it looks unlikely that any AI legislation will pass before the new year.
In short, no one is going to force AI labs to change how they do business in the next few months. Despite that, some company leaders have signaled a willingness to slow down voluntarily — especially if their rivals do the same.