2025-12-15 06:33:59
Published on December 14, 2025 10:23 PM GMT
Most predictive work on AI focuses on model capabilities themselves or their effects on society at large. We have timelines for benchmark performance, scaling curves, and macro-level labor impact estimates. What we largely do not have are personalized forecasts that translate those trends into implications for an individual role.
At the same time, many conversations about AI and work stall at a familiar level of abstraction. Some jobs will disappear, others will change, productivity will increase, and things will “shift.” They do not answer the question that actually matters to individuals: when does AI become capable enough to meaningfully threaten my role, given the specific tasks I do and the organization I work in?
We know that AI capabilities are improving rapidly, while adoption inside organizations is uneven, delayed, and constrained by structure. Some tasks are discretely measured and automatable, while others depend on taste and tacit knowledge and thus are subject to several layers of task-specific and organizational friction. As a result, AI impact is not a single event but a distribution over time that varies substantially across roles.
What was missing, at least for me, was a way to translate those general trends into a personal forecast, one that makes its assumptions explicit and allows them to be challenged.
Over the past year, I built a model to do that, which you can explore at https://dontloseyourjob.com.
The model is a hazard model, the same function used to estimate the survival decay of medicine, supply chains, and employment. Instead of modeling the probability of system failure, it models the probability that AI meaningfully displaces a role, or collapses its economic value, as a function of time.
The baseline hazard is informed by METR's work AI capability growth, including evidence consistent with exponential improvement. On top of that, I layer multiple sources of friction and amplification, including task structure, degree of tacit knowledge, coordination requirements, organizational inertia, and economic incentives. These inputs are captured through a questionnaire intended to approximate the shape of a person’s actual work, rather than their job title alone.
Importantly, the model is highly assumption-sensitive. Small changes to capability growth rates, adoption lag, or task substitutability meaningfully alter the output. You may reasonably disagree about how fast models will improve, how quickly their organization will deploy them, or which parts of their role are actually indispensable.
For that reason, the model is open-source, and the interface itself exposes many of the underlying assumptions. The goal is not to produce a single “correct” forecast, but to make the structure of the problem explicit: what you are implicitly betting on when you assume your role is safe, and which uncertainties actually affect the timeline.
If you think the assumptions are wrong, you can change them, either directly in the interface or by modifying the code. The hope is that this makes discussions about AI and work less rhetorical, and more legible as disagreements about models, parameters, and evidence.
I wrote a long guide to job displacement, which I’ll copy down below (though it also exists in the site). If you have time — visit the site, interact with it, visualize your displacement risk, and integrate your own assumptions. There are many opportunities we have to prolong our own careers, and there are policy options we can integrate at both the state and firm levels to keep ourselves in the ownership of our work. I think some of us (including myself) would prefer a human-oriented future rather than a completely mechanized one, but regardless of your viewpoint, modeling these forces helps contribute to the discourse.
Researchers at leading AI labs predict that we will reach AGI (Artificial General Intelligence) sometime within the next 3-12 years. Politicians, business executives, and AI forecasters make similar predictions. AGI, by definition, means systems that are more capable, cheaper, and faster than any human at any cognitive labor task. These systems will amplify individual productivity in the near term, but they also have the capability to displace human workers.
If you’re skeptical of that claim, you have good reason to be. “Automation will take your job” has been predicted before: by 19th century Luddite textile workers, by economists warning about tractors in the 1920s, by analysts predicting the end of bank tellers when ATMs arrived. Those predictions were mostly wrong. New jobs emerged, transitions stretched over decades, and human adaptability proved more robust than forecasters expected. Why should AI be any different?
Three factors separate AI from previous waves: speed, breadth, and the economics of cognitive labor. AI capabilities are increasing much faster than the rate at which we can upskill, and these systems aim to replace the function of human intelligence in many cognitive tasks. But AI does not need to reach “general intelligence” levels of capability to disrupt the labor market, and we are already seeing it happen in white-collar roles.
Displacement occurs under two main scenarios:
Naturally, AI capabilities are compared to the human brain, and in many respects they are far off from matching the strengths of our working minds: tackling complex problems with incomplete information, continual learning, navigating emotions or relationships, and long-term coherent agency. Your role may not be displaced by AI providing the entire service of Data Analyst III, but it might soon be able to do enough of your tasks that your organization no longer needs a full-time person in your position.
Don’t Lose Your Job is a modeling platform that measures gradual displacement in white-collar roles. The questionnaire captures your job’s task structure, domain characteristics, hierarchy position, and organizational context, then models those layers of friction and amplification against trends of AI capability growth from METR data. The model makes several assumptions about these forces, but you can (optionally) tune these coefficients in the Model Tuning section to see the effects of your own assumptions.
The model does not forecast potential government or business policies that might mandate human involvement in certain tasks or slow AI adoption. Beyond individual planning, this tool aims to inform policy discussions about maintaining human agency and oversight in the labor market.
The model is open-source. You can build your own versions by visiting github.com/wrenthejewels/DLYJ.
The history of automation anxiety is largely a history of false alarms. Understanding why previous predictions failed, and why this time the underlying dynamics may have genuinely shifted, is essential to calibrating how seriously to take current forecasts.
Economists discuss displacement anxiety with the “lump of labor” fallacy, which assumes there’s a fixed amount of work to be done such that automation necessarily reduces employment; historical evidence shows this assumption is wrong.
In the early 19th century, Luddite weavers destroyed textile machinery, convinced that mechanical looms would eliminate their livelihoods. They were partially right, as hand weaving did decline, but textile employment overall expanded as cheaper cloth created new markets and new jobs emerged around the machines themselves.
A century later, agricultural mechanization triggered similar fears. In 1900, roughly 40% of American workers labored on farms. By 2000, that figure had dropped below 2%. Yet mass unemployment never materialized. Workers moved into manufacturing, then services, then knowledge work. The economy absorbed displaced agricultural workers over decades, creating entirely new categories of employment that didn’t exist when tractors first arrived.
The ATM story is also relevant. ATMs spread in the 1970s-80s, and many predicted the end of bank tellers. Instead, the number of bank tellers actually increased. ATMs reduced the cost of operating branches, so banks opened more of them, and tellers shifted from cash handling to sales and customer service. The job title persisted even as the job content transformed.
The mechanism is straightforward: automation increases productivity, which reduces costs, increases demand, and creates new jobs, often in categories that didn’t exist before. Spreadsheets enhanced accountants to perform more sophisticated financial analysis, and it created demand for analysts who could leverage the new tools, rather than displacing analysis as a profession. Markets are adaptive, and new forms of valuable work consistently emerge.
AI-driven displacement differs from historical precedents in ways that may compress generational transitions into years.
Speed of capability growth. AI capabilities are increasing exponentially. Skill acquisition, organizational change, and policy response operate on much slower cycles, so capability growth can outpace the rate at which workers and institutions adapt. Even if AI-driven wealth is eventually redistributed, many current workers can still fall through the gap during early waves of displacement. If this happens, you may have fewer opportunities for outlier success than ever before.
Breadth of application. Tractors replaced farm labor, ATMs replaced cash-handling, and spreadsheets replaced manual calculation. Each previous automation wave targeted a relatively narrow domain. AI targets a wide range of cognitive work: writing, analysis, coding, design, research, communication, planning. There are fewer adjacent cognitive domains to migrate into when the same technology is improving across most of them at once, so the traditional escape route of “move to work that machines can’t do” becomes less available.
The economics of cognitive vs. physical labor. Automating physical tasks required capital-intensive machinery: factories, tractors, robots. The upfront costs were high, adoption was gradual, and physical infrastructure constrained deployment speed. Typewriters, computers, and the internet enhanced our cognitive abilities by seamlessly transferring the flow of information. AI replaces cognitive labor itself through software, with marginal costs approaching zero once the systems are trained. A company can deploy AI assistance to its entire workforce in weeks, not years, and some of that “assistance” has already replaced entire job functions. The infrastructure constraint that slowed previous automation waves doesn’t apply in the same way.
The “last mile” problem is shrinking. Previous automation waves often stalled at edge cases. Machines could handle the 80% of routine work but struggled with the 20% of exceptions that required human judgment, which created stable hybrid roles where humans handled exceptions while machines handled volume. AI’s capability profile is different, and each model generation significantly expands the fraction of edge cases it can handle, so “exceptions only” roles look more like a temporary phase than a permanent adjustment.
No clear “next sector” to absorb workers. Agricultural workers moved to manufacturing, manufacturing workers moved to services, and service workers moved to knowledge work. Each transition had a visible destination sector that was growing and labor-intensive. If AI automates knowledge work, what’s the next sector? Some possibilities exist (caregiving, trades, creative direction), but it’s unclear whether they can absorb the volume of displaced knowledge workers or whether they pay comparably.
The historical pattern may not completely break, as we’ll always redefine “work,”:
New job categories we can’t predict. The most honest lesson from history is that forecasters consistently fail to anticipate the jobs that emerge. “Social media manager” wasn’t a job in 2005. AI is already creating new roles: prompt engineers, AI trainers, AI safety researchers, human-AI collaboration specialists, AI ethicists, AI auditors. As AI capability grows, more categories will likely emerge around oversight, customization, integration, and uniquely human services that complement AI capabilities. Our imagination genuinely fails to predict future job categories, and some current workers will successfully transition into AI-related roles that don’t yet have names.
Human preferences for human connection. Some services stay human by choice, even if AI can do them. People may still want therapists, teachers, doctors, and caregivers in the loop. Human connection carries value AI cannot replicate. We see this in practice: many shoppers seek humans for complex purchases, in-person meetings matter for relationships despite videoconferencing, and customers escalate from chatbots to humans for emotional support or tricky problems. Roles rooted in care, creativity, teaching, and relationships may keep human labor even when AI is technically capable.
Organizational friction is real. Real-world organizations are far messier than economic models suggest. Bureaucratic inertia, change management challenges, legacy systems, regulatory constraints, and organizational dysfunction slow AI adoption dramatically. The timeline from “AI can do this” to “AI has replaced humans doing this” could be much longer than capability curves suggest.
Regulatory protection. The EU AI Act and similar frameworks could mandate human oversight in high-stakes domains. Some jurisdictions may require human involvement in medical diagnosis, legal decisions, hiring, or financial advice regardless of AI capability. Professional licensing boards may resist AI encroachment.
Automation decisions are driven by capabilities and economic constraints. A firm won’t replace you with AI just because it can do your job; they’ll replace you when the economics favor doing so.
When a firm considers automating a role, they’re implicitly running a cost-benefit analysis that weighs several factors:
The decision simplifies to: Is (labor cost × volume × quality improvement) greater than (implementation cost + ongoing AI cost + risk of errors)? When this equation tips positive, automation becomes economically rational regardless of any abstract preference for human workers.
A common misconception is that AI must outperform humans to threaten jobs. AI only needs to be good enough at a low enough price, and for enough of your tasks.
Consider two scenarios:
For many business contexts, the 10% quality drop is acceptable given the 90% cost reduction. This is especially true for work that does not need to be highly reliable on its first prompt, as a senior-level employee can direct agents to draft multiple edits of tasks faster than a feedback loop with lower-level employees. The quality threshold for automation is often lower than workers assume.
This explains why displacement often begins with lower-level roles. Entry-level work typically has higher error tolerance (seniors review it anyway), lower quality requirements (it’s meant to be refined upstream), and lower absolute labor costs (making the implementation investment harder to justify for any single role, but easier when aggregated across many juniors).
A common objection to AI displacement forecasts is that current models have limited context windows and can’t hold an entire job’s worth of knowledge in memory. This misunderstands how AI systems are actually deployed. Organizations don’t replace workers with a single model instance, they deploy fleets of specialized agents, each handling a subset of tasks with tailored prompts, tools, and retrieval systems. All of your knowledge about your role cannot fit into one model’s context window, but it can be dispersed across system prompts, vector databases, and other systems that document the data of your role. The aggregate system can exceed human performance on many tasks even when individual agents are narrower than human cognition.
This architecture mirrors how organizations already function. No single employee holds complete knowledge of all company processes; information is distributed across teams, documentation, and institutional memory. As agentic systems mature, the orchestration becomes more sophisticated; agents can spawn sub-agents, maintain persistent memory across sessions, and learn from feedback loops.
Work will become more digitized through meeting transcripts, emails, project trackers, and saved drafts, and agents will gain a clearer view of how tasks are actually carried out inside an organization. Over time, this helps the system understand the practical steps of a role rather than just the final result.
These agents will learn from these accumulated examples, and they can begin to handle a larger share of routine or well-structured tasks. They also improve more quickly because new work records continuously update their understanding of how the organization prefers things to be done. This reduces certain forms of friction that once made roles harder to automate, such as tacit knowledge or informal processes that previously were not recorded.
Once one major player in an industry successfully automates a function, competitors face pressure to follow. This creates an adoption cascade:
This dynamic means that your firm’s current attitudes toward AI adoption may not predict your long-term risk. A conservative organization that resists automation today may be forced to adopt rapidly if competitors demonstrate viable cost reductions. Think about both what your company thinks about AI and how it will respond once other businesses use it.
Public and venture-backed companies face additional pressure from capital markets. Investors increasingly expect AI adoption as a signal of operational efficiency and future competitiveness. Earnings calls now routinely include questions about AI strategy, and companies that can demonstrate AI-driven productivity gains are rewarded with higher valuations.
The reverse is also true: companies that resist automation may face investor pressure, board questions, and competitive positioning concerns that push them toward adoption faster than they would otherwise choose.
AI research organization METR measures AI capabilities by the length of software engineering tasks models can autonomously complete. Even when measured against different success rates, models have demonstrated exponential growth since the launch of public-facing models, with a doubling time of roughly seven months. Extrapolating from this trend at the 50% success rate threshold, it will be less than 5 years before models can autonomously complete tasks that take humans weeks or months.
Source: METR study
METR’s benchmarks measure software engineering tasks, but displacement happens across every knowledge domain. Code is structured, digital, and verifiable, which makes software a leading indicator. Other cognitive domains will likely follow for similar task-completion times, but different domains face different translation delays.
Work that resembles software (digital, decomposable, with clear success criteria) will track closely with METR benchmarks. Work involving tacit knowledge, physical presence, or relationship-dependent judgment will lag behind. The model handles this through domain friction multipliers. Software engineering roles face minimal friction, while legal, operations, and traditional engineering roles face higher friction due to regulatory constraints, liability concerns, and less structured workflows.
The questionnaire captures four factors that determine when AI displacement becomes likely for your specific role:
The METR curve serves as the baseline for the forecasted capabilities of AI models. Then, we make assumptions about the time you spend in different task “buckets” (sorted by length they take to complete) based on your role and hierarchy level, and we add friction to the METR curve to essentially measure: how hard is it for AI to do these tasks of different lengths? That friction is measured by your responses to the questionnaire, but you can change the weights of these multipliers in the Model Tuning section.
We also make assumptions about industry-specific friction for your tasks, and how reliable AI needs to be in order to enter that risk curve. These are tuneable in the sliders beneath the model, and you’ll notice that moving these sliders can have a pronounced effect on your displacement timeline. These forces combine into a weighted readiness score (typically around 50%, adjusted by hierarchy) that opens the automation hazard. Implementation delay and compression parameters then shift that hazard into the green curve you see in your results.
When you complete the questionnaire, the model generates a chart showing two curves over time:
The blue curve shows technical feasibility (the automation hazard without implementation delay or compression). It turns on when AI clears your job’s coverage threshold (typically ~50% of your task portfolio) based on your task mix. Digital, decomposable domains open the gate sooner; tacit/physical domains open later. Senior roles lift the threshold slightly and soften the ramp; entry-level roles lower it.
The green curve shows when you are likely to actually lose your job, accounting for real-world implementation barriers. This is the timeline that matters for planning your career. The green curve combines two displacement mechanisms:
The vertical axis shows cumulative displacement probability. A green curve reaching 50% at year 4 means there is a 50% probability of displacement within 4 years, and 50% probability you remain employed beyond that point. Steep curves indicate displacement risk concentrates in a narrow window, while gradual curves spread risk over many years. Early divergence between curves signals high compression vulnerability.
Alex: Compressed out in ~1-1.5 years.
Alex is a junior developer: writing code, fixing bugs, documenting changes. The work is fully digital and breaks into clean pieces. As AI tools improve, senior engineers absorb Alex’s workload. They ship faster with AI assistance, and the backlog of junior-level tickets shrinks.
Alex is at the bottom level of all software engineers at his company, and eventually AI amplifies enough of his colleagues so that his contributions aren’t worth his salary to his firm anymore.
Jordan: Protected for 7+ years.
Jordan is a management consultant with years of strong client relationships. His deliverables are technically digital (slides, memos, etc.) but he spends a large portion of his time in face-to-face meetings, and often has to derive his tacit knowledge about unique cases when advising clients. His clients are considering AI-driven displacements in their own firm, so they have unique challenges that were previously not considered in the consulting market. Each project needs a custom approach, and while Jordan uses AI tools to assist his planning, only he can be trusted to advise on broad change management. Compression risk is nearly zero, and Jordan’s business will benefit from the AI displacement wave.
Sarah: Medium risk, 3-5 year timeline.
Sarah is a mid-level accountant, and her work involves processing invoices, reconciling statements, and preparing journal entries. The work is mostly digital and it’s somewhat structured, but it requires human judgement: matching vendor names, deciding when to escalate a discrepancy, and calling coworkers for audit assistance. She handles “tickets” just like Alex, but they require more context to complete.
While these timelines may seem fast, the trendline for model capabilities is not certain to hold (which is why we allow you to tune it in the model). Current forecasts extrapolate from recent trends, but compute scaling may hit limits, algorithmic progress may slow, or AI may hit capability ceilings. In their paper “Forecasting AI Time Horizon Under Compute Slowdowns,” METR researchers show that capability doubling rate is proportional to compute investment growth. If compute investment decelerates, key milestones could be delayed by years.
That said, even if growth slows, substantial capability growth has already occurred and will continue. For current workers, the question is whether a plateau happens before or after their jobs are affected. The historical 7-month doubling has held steady from 2019-2025, and more recent 2024-2025 data suggests the rate may be accelerating to roughly 4-month doubling.
Source: METR forecast (arxiv). Thanks to Joel Becker.
You cannot control AI capability growth, market competition, or how your industry responds. You do have some influence over where you sit in that process and how much time you have to adjust. Individual action will not fix AI displacement by itself, but it can buy you runway, options, and a better position from which to push for collective change.
In the near term, there are some useful actions that can buy you time and flexibility.
Learn how your workflows complement AI. Understand which parts of your work AI already handles well, where you add value, and how you can structure tasks so that both strengths work together. People who can design and oversee AI-enabled workflows are more useful to their organizations and better prepared as roles shift.
Shift toward higher-context work where you can. Roles that involve judgment, coordination, and relationships are harder to automate than pure execution, especially in the short run. Moving part of your time toward context-heavy or integrative work can slow the impact on you, even if it does not remove it.
Increase the cost of removing you. Strong performance, reliability, and being central to coordination does not make you safe, but it creates organizational friction. When cuts happen, people who are trusted, visible, and hard to replace often receive more time, better options, or softer landings.
Explore other routes for agency. Skills that transfer across companies, a professional network, a record of public work, and some financial buffer all make it easier to adapt if your role changes quickly. These do not change the aggregate risk, but they change how exposed you are to it.
These are high-agency moves, but they mostly shift your place on the curve rather than changing the curve itself. They are worth making because they give you more control over your own landing and more capacity to engage with the bigger problem.
If AI continues to compress and automate large parts of knowledge work, there will not be enough safe roles for everyone to move into. At that point, the question is less about how any one person adapts and more about how we share the gains and the risks: who owns the systems, who benefits from the productivity, and what happens to people whose roles are no longer needed.
How societies respond to AI-driven displacement will be shaped by policy choices actively being debated. Transition support programs (extended unemployment benefits, government-funded retraining, educational subsidies) face questions about whether retraining can work fast enough when target jobs are also changing rapidly. Human-in-the-loop mandates could require human involvement in high-stakes decisions regardless of AI capability, preserving employment by regulation. Automation taxes might slow adoption and fund transition support, while wage subsidies could make human labor more competitive. Universal basic income would decouple income from employment through regular payments funded by productivity gains. Broader ownership models might distribute AI capital through sovereign wealth funds or employee ownership requirements. And labor organizing could negotiate over automation pace, transition support, and profit-sharing.
Beyond these, societies will likely need to reckon with the nature of at-will employment, and redefine what “good performance” is at work. If we provide little comparative value to firms when AI reaches high levels of strength, our current economic models face little incentive to reward us with continued employment and new opportunities for labor. But we built AI, and our laborers provide the crucial data needed for pretraining, so I think there is a system we can develop that routes its success to people, rather than corporations that become increasingly mechanized.
Perhaps it’s a democratized input model, where current laborers become rewarded with an ownership value of the models they help train. This will provide scaled returns for our existing workforce, especially as agents clone and expand within our organizations, and it follows the existing idea within capitalism of being rewarded for economically contributing. It doesn’t solve for new grads who enter the workforce, and it needs some tinkering, but it may be a more tangible path beyond “we’ll just distribute UBI derived from strong AI.” UBI (or even the Universal Basic Compute idea that’s been floating around) is a strong idea for a social safety net, but it likely will not be developed in time to catch people who face the early waves of unemployment.
You can engage by informing your representatives, supporting research organizations like Epoch, the Centre for the Governance of AI, and the Brookings Future of Work initiative, participating in professional associations, and contributing worker perspectives to public discourse.
Thank you for reading and engaging with my work. Building this model took a lot of time, and translating a fast-moving field into something that feels clear, useable, and tunable was harder than I expected. I hope it helped you understand the dynamics behind your results and gave you a better sense of what the next few years might look like.
This project is completely unrelated to my main job, but I will continue to evolve it as this technology does. I believe AI is one of the most significant dangers to our society in a long time, and job loss is only one of the many issues we face from unchecked/unregulated growth. We have to continue developing tools to defensively accelerate the pace of change.
2025-12-15 06:21:28
Published on December 14, 2025 10:21 PM GMT
I have goals that are much easier to reach with a powerful political machine. Probably a lot of other people around here share them. (Goals include “ensure no powerful dangerous AI get built”, “ensure governance of the US and world are broadly good / not decaying”, “have good civic discourse that plugs into said governance.”)
I think it’d be good if there was a powerful, high integrity political machine with good epistemics, trying to make those things happen.
Unfortunately the naive ways of doing that would destroy the good things about the rationalist intellectual scene. This post lays out some thoughts on how to have a political machine with good epistemics and integrity.
Recently, I gave to the Alex Bores campaign. It turned out to raise a quite serious, surprising amount of money.
I donated to Alex Bores fairly confidently. A few years ago, I donated to Carrick Flynn, feeling kinda skeezy about it. Not because there's necessarily anything wrong with Carrick Flynn, but, because the process that generated "donate to Carrick Flynn" was a self-referential "well, he's an EA, so it's good if he's in office." (There might have been people with more info than that, but I didn’t hear much about it).
Ultimately, I kinda agreed, but, I wouldn't have publicly defended the choice. This was during FTX era where money was abundant and we were starting to attract grifters (i.e. hearing explicit comments like "oh man all you have to do is say you care about causes X and Y and you can get free money.") It was not sustainable to keep donating to people "because they were EA" or "because they mouthed the words 'AI Safety'."
Alas, there are important political goals I want to accomplish. Political goals require getting a lot of people moving in lockstep. Rationalists hate moving in lockstep. For good reason. At the time, my solution was “donate to Carrick Flynn, but feel skeezy about it.”
One option is leave this to "The EA community" rather than trying to invoke "the rationalists." Alas, I just... don't really trust the EA community to do a good job here. Or, rather, them succeeding at this requires them to lean into the rationalist-y traits, which would reintroduce all the same allergies and handwringing. My political goals are nuanced. I don't want to go the route of environmentalism that bans nuclear power and ends up making things worse.
AI Safety isn't the only thing you might want a powerful political bloc with good epistemics to support. Maybe people want to be ambitious and do something much more openended than that. But, this is the motivating case for why it’s in my top-5 things to maybe do, and it’s useful to dissect motivating cases.
I think many people around here agree we need to stop the development of unsafe, overwhelmingly powerful superintelligence. (We might disagree about a lot about the correct steps to achieve that).
Here are some ways to fail to do that:
That's rough. Wat do?
What I think Wat Do is, figure out how to build a political machine that is powerful enough to have leverage, but, is still based on a solid foundation of epistemic trust.
How do that?
Well, alas I dunno. But it feels very achievable to me to do better than both "don't play the game" or "naively play the game, short sightedly." Here are some thoughts on that
This is difficult for lots of reasons. Here are some easier to articulate ones:
A lot of the world runs on implicit alliances, where people agree to recommend each other as good people, and not to say bad things about each other.
One big reason ornery rationalists are like “politics is Real Hard to do without intellectual compromise” (while other people might be like “I see why you’d be worried, but, you seem to be exaggerating the worry”), is that this is a very pernicious. It fucks with epistemics in a way that is invisible if you’re not actively tracking it, and the mutual reputation alliances don’t want you to be tracking it so it requires active effort to make it possible to track.
There are good (naive) reasons to gain power. You do need political power to get shit done. But, also, people feel an attraction to power for normal, boring, selfish reasons. It is easy to deceive yourself about your motivations here, and about what your motivations will be in the future when you’ve enmeshed yourself in a political alliance.
Lots of ways of gaining power involve Mutual Reputation Alliances, or other compromises.
(Oliver Habryka has argued to me that there are ways of gaining conditional power (as opposed to unconditional power) which involve less compromise. This post is mostly about gaining unconditional power but seemed worth flagging the difference)
There is some public info available, but for “will this broad political project work longterm”, it’s going to depend on things like “does so-and-so keep their word?”, “will so-and-so keep keeping their word if the political situation changes, or they see an opportunity for power?”
This requires subtle details about their character, which you can only really get from people who have worked with them a bunch, who are often part of a mutual reputation alliance, won’t want their name attached to the info if you share it, and will only give you the info if you can share it in a way that won’t make it obvious that they were the one sharing it.
In addition to “embedded in a mutual reputation alliance”, powerful people can be vindictive if you try to share negative information about their character. And, since they are powerful, if they want to hurt you, they probably can.
People don’t share bad information about powerful people out of fear, not just loyalty.
(One specific case of this is "they can sue you for libel, or at least threaten to.")
There will be rival actors who don’t want your preferred candidate to be elected or your preferred policy to be implemented. They will actively make it hard for you to do this. They may do so with underhanded tactics that are difficult to detect, just under the threshold for feeling “unreasonable” so it’s hard to call out.
It also means that sometimes you want to raise funds or maneuver in secret.
Mutual reputation alliances are costly because they radiate out of the alliance. In practice, there is not a sharp divide between the politicians and the rationalists. The people rallying support and finding private information will (by default, probably) radiate some pressure to not question the narrative, and to avoid making someone regret having shared information.
See also: Entangled Truths, Contagious Lies
This is hard-mode enough when we’re just trying to be a corner of the internet talking about some stuff. It’ll matter a lot more if you are trying to achieve a political goal.
See: Politics is the Mind-Killer and Politics is hard mode
A lot of these problems aren’t that bad if you’re doing a one-time political maneuver. You might make some enemies and risk a bit of tribal groupthink, but, eh, then you go back to doing other things and the consequences are bounded.
But, the whole point of building a Good Epistemics/Integrity political machine is to keep persistently doing stuff. This will attract enemies, if it succeeds. It will also attract…
People will try to manipulate into giving them money. Some instances of this might be well intentioned. You need to be able to defend against it anyway.
If it’s known that there’s a High Integrity/Epistemics Political Machine that’s on the lookout for sociopaths and subtle corruption, people will try to mouth the words that make it sound like they are avoiding sociopathy/subtle-corruption. This includes both candidates, and people running the rallying-campaigns to get candidates funded.
“I believe in AI safety” or “I care about epistemics” is an easy password to fake.
An example of a harder password to fake is “I have made many public statements about my commitments that would look bad for me if I betrayed them.”
For people running PACs or other orgs, “here are the incentives I have constructed to make it hard for myself / The Org to betray it’s principles” is even better. (i.e. OpenAI’s nonprofit governance structure did make it at least difficult and take multiple years, for the org to betray it’s principles).
A sometimes-difficulty with political fundraising is early on it's often important to happen in a low-key way, since if rival politicians know your plan they can work against it. But,
I think part of the process should be, there are people involved in low-key-private-political-fundraising who are playing a watchdog role, helping establish mutual knowledge of things like whether a given politician...
Top Tier:
Mid Tier
Minimum Tier:
I didn't trust the people advocating for Alex Bores to have noticed sociopathy. But, he did in fact pass the Raise Act. Scott Wiener tried to pass SB 1047 twice and succeeded the second time, sorta. They might still betray their principles later, but, their track record indicates they are at least willing to ever put their actions where their mouth was, and the bills looked pretty reasonable.
That seemed good enough to me to be worth $7000 (given the other analysis arguing that the money would help them win).
If I imagine a high Integrity Political Machine, I think it probably involves some sort of evaluator watchdog who a) privately researches and circulates information about candidates during the Low Key period, and b) writes public writeups afterwards that allow for retrospective sanity checking, and noticing if the political machine is going astray.
I'd want the watchdogs to split up observations and inferences, and split up particular observations about Cause A vs Cause B (i.e. make it easy for people who want to support AI safety but don’t care about veganism, or, vice versa, to track which candidates are good by their lights, rather than aggregating them into a general vector of Goodness).
The actual motivating example here was thinking about supporting PACs, as opposed to candidates.
I don’t actually understand PACs very well. But, as I understand it, they need to be deciding which candidates to support, which means you need all the same apparatus for evaluating candidates and thinking through longterm consequences.
Any broad political org needs a person in charge of it who is responsible for making sure it is high integrity. I have a particularly high bar for this.
If you want to run a PAC or org that gets money from a hypothetical High Epistemics/Integrity Political Machine, it is not merely your job to “not lie” or “not mess up in the obvious ways.” Politics is hard mode. You need to be tracking the incentives, tracking whether your org is evolving into a moral maze, and proactively work to make sure it doesn’t get eaten by an egregore.
This requires taste, as well as effort.
Taste is hard to acquire. Often, “just try harder” won’t realistic work. If you don’t have good enough judgment, you either need to find another person to be in charge, or you might need to go try doing some projects that will enable you to learn from experience and become wiser / more cynical / etc.
An earlier draft described this as “GiveWell for retroactive political action assessment”. But, the word “Givewell” implies there is an org. Orgs bundle up people’s reputation together, such that every person involved feels pressure to not risk the reputation of everyone else at the org. This has been a failure mode at OpenPhil (from what I understand).
Watchdogs will need to make some tradeoff on gaining access to private information, vs making various promises and compromises. But, they can do that individually, so the results aren’t as contagious.
I would ideally like everyone involved to have maximally good epistemics. But, in order for this to succeed, you need some people who are really good at rallying large numbers of people to do a thing (i.e donate to candidates, vote). Rallying is a different skill from maintaining-good-epistemics-while-evaluating. It's hard to be good at both. It's hard because a) it's just generally harder to have two skills than one, and b) "rallying" just does often require a mindset that is more Mindkiller-y.
So, I would like at least some people who are spec'd into "watchdog-ing"/"evaluation", who are not also trying to rally people.
I want rally people to be more careful on the margin. I think it is possible to skill up at inspiring conviction/action without having distorted beliefs. But, I think the project can work if the rally-ers aren't maximally good at that.
One way for this to fail is for individual candidates to turn out to be grifters who extract money, or sociopaths who end up net-negative.
Another way for this to fail is for the system to become subtly corrupted over time, making individual little compromises that don't seem that bad but add up to "now, this is just a regular ol' political machine, with the word 'epistemics/integrity' taped to the front door."
There needs to be watchdogs who are modeling the whole ecosystem, and speaking out if it is sliding towards failure.
It's not enough for watchdogs to periodically say "hey, this candidate seems sus" or "we seem to be sliding towards worse epistemics collectively." The people voting with their ballots or wallets need to actually care. This means a critical mass of them need to actually care about the system not sliding towards corruption.
This could be an entirely separate idea for “watchdog evaluators”, but it dovetails nicely. For candidates that a powerful high-integrity political machine are trying to help, it probably makes sense to have public prediction markets about whether they will keep their word about various promises.
If individual watchdogs gain a track record for successfully noticing “so and so is going to betray their principles” and “so and so probably won’t betray their principles”, those people can also then maybe be trusted more to represent private information (“I talked to Candidate Alice, and I really do get a sense of them knowing what they’re talking about and committing to Cause A”).
The main problem with doing that publicly is that powerful people might be vindictive about it. I’m most worried about people being vindictive when they kind grew up with the rationalsphere, so having rationalists criticize them or estimate them as low integrity, feels personal, rather than just cost-of-doing-business as a politician.
I do think the norm and vibe should be “this is a cost of doing business. If you want money/support from the high integrity political engine, you should expect people to be evaluating you, this is nothing personal, the standards are very exacting and you may not meet them.”
A problem I'm not 100% sure how to handle, is getting sued for evaluating people/orgs as sociopathic.
I'm not sure what the legal standing is, if a prediction market reads:
"Within 5 years, I will judge that OpenAI's nonprofit board no longer has teeth"
or
"Within 5 years, I will think [Candidate X] betrayed a campaign promise."
or:
"Within 5 years, CEO Charlie will have violated one of these principles they established."
A serious political engine could have money to defend against lawsuits, but, also, the more money you have, the more it's worth suing you. (I think at the very least having someone who specializes in handling all the hassle of getting sued would be worth it).
My hope is that, unlike previous instances of people trying to claim an individual did bad things, this project is in some sense "big enough to be clearly worth protecting" (whereas a random person in a vague community scene being maybe a bad actor doesn't have anyone incentivized to make it their job to defend)
Sometimes people get annoyed that LessWrong isn’t letting them do a particular kind of rallying, or saying something with one voice. They read Why Our Kind Can't Cooperate and are like “okay, so, can we have a culture where people publicly support things and there isn’t this intense allergic criticism?”.
I think maybe there should be another forum or tool for doing that sort of thing. But, it’s definitely not LessWrong’s job. LessWrong definitely should not be synonymous with a political agenda.
I think posts like these are fine and good:
I feel wary of posts like this:
I think the difference is:
Posts that argue the object level of ‘this candidate or project will have good/bad consequences’ are fine.
Posts that are trying to change what is socially acceptable to think/say on LessWrong are NOT fine.
Posts that are talking about what is socially acceptable to think/say on LessWrong ARE fine. The difference between this and the previous one can be subtle. I still find John Wentworth's comments from Power buys you distance from the crime pretty good:
> Who’s at fault for the subcontractor(^3)’s slave labor?
[...] My instinct says DO NOT EVER ASK THAT QUESTION, it is a WRONG QUESTION, you will be instantly mindkilled every time you ask "who should be blamed for X?".
... on reflection, I do not want to endorse this as an all-the-time heuristic, but I do want to endorse it whenever good epistemic discussion is an objective. Asking "who should we blame?" is always engaging in a status fight. Status fights are generally mindkillers, and should be kept strictly separate from modelling and epistemics.
Now, this does not mean that we shouldn't model status fights. Rather, it means that we should strive to avoid engaging in status fights when modelling them. Concretely: rather than ask "who should we blame?", ask "what incentives do we create by blaming <actor>?". This puts the question in an analytical frame, rather than a "we're having a status fight right now" frame.
To be clear, LessWrong doesn't prevent you from posting rallying / status-fighty / social-reality-manipulating posts. But, it is setup to discourage it on the margin, and prevent a lot of the upside from trying to do it. You won't be on the frontpage, you won't get curated, etc. If it seems like you're doing it in a way that mods think is bad for the culture, we might yell at you.
(But also note, I did not run this by the rest of the Lightcone team and we have a policy of speaking for ourselves, since orgs don't actually have "beliefs")
Just to restate all the premises in one place:
A political machine is a system that coordinates lots of people to produce a political outcome. (If you don't need to coordinate lots of people, you just have a political actor, not a machine)
It's hard to build a high integrity/epistemics political machine, because:
And this all needs to keep working longterm, if you want a longterm powerful impact, so, it needs to be robust to all the adversarial failure modes.
Some potential solutions:
2025-12-15 05:10:32
Published on December 14, 2025 9:10 PM GMT
My house has radiators for heat. There are three heating loops ("zones") but the house has more than three rooms and it's not very well balanced. Fixing this properly involves hiring a plumber, but it turns out we can make it much better with just a small fan!
Radiators heat passively: they warm the nearby air, which rises and allows cooler air to flow in. This new air then warms, and the cycle repeats. This works pretty well: no electricity, no noise, just smooth heating.
What we can do with a fan, though, is accelerate this process in a targeted way, at the cost of a small amount of electricity, hardware, and noise. By fanning the radiator we want more output from, we can bring the system into balance.
I'm now tempted to put efficient little fans on all the radiators in the house, network them together, add temperature and occupancy sensors, predict future occupancy, and see how much more efficient I can make the whole system. But while this sounds like a fun project, and possibly even something someone could turn into a product that pays for itself in saved money and fuel, [1] this is really not something I should take on right now.
[1] I did some looking and there are (a) commercial radiator booster
fans, and (b) smart radiator valves, but nothing that ties this all
together.
2025-12-15 02:55:06
Published on December 14, 2025 6:55 PM GMT
A core subproblem in ontology identification is to understand why and how humans and agents break down their world models into distinct, structured concepts like tables, chairs and strawberries. This is important because we want AIs to optimize the real world things we care, but the things we care about are expressed in terms of latent variables in our world models. On the other hand, when an AI plans to achieve its goals in the world, that planning refers to its own internal representations, which means we must understand how those internal representations correspond to latent variables/concepts in our ontologies to ensure that the AI is optimizing for the right things in the right way.
From an external perspective, if our only goal was to explain the functional behavior of the world model, it would seem perfectly valid to just treat the world model as one undifferentiated blob of black box program that outputs predictions about the world. A black box program might even be the simplest explanation for the world model's behavior. There doesn't seem to be any obvious reason why we'd need to decompose this black box into well-structured concepts, or why such decomposition would line up consistently with our own ontologies in a meaningful way.
So behavior alone doesn't seem sufficient to pin down structured concepts. We might be tempted to just project our own ontology onto the black box program by taking our current understanding of how the world works and trying to draw correspondences to different parts of the black box that we've somehow carved up. But this approach won't be robust to ontology shifts: The AI will learn and discover all sorts of new things about the world that we haven't observed or even conceived of, including new laws of physics or novel abstractions alien to us, and these are precisely the kinds of things that won't fit into whatever ontology we're trying to force onto the AI's world model.
If projecting our ontology onto the black box program doesn't work, we need to start from the black box description of the world model and derive the ontology from the black box somehow. This seems like a really challenging task, it's not even clear what desiderata would let us pin down a decomposition into well-structured abstractions that remains robust to ontology shifts. However, the achievement of "deriving structure starting from a black box" isn't completely unprecedented:
Algorithmic Markov Condition: Algorithmic Markov Condition is essentially the AIT version of Bayesian networks. Instead of random variables, we have binary strings, and instead of factorizing a joint probability distribution, we factorize the joint complexity of the strings. We can think of the Algorithmic Markov Condition as specifying the "optimal computational order for compressing strings." For instance, if are four strings and their joint complexity factorizes according to the DAG below, we can interpret that as saying: The optimal way to compress together is to start with , then find the shortest program that computes from as well as the shortest program that computes from . Once we obtain and , we find the shortest program that computes from and . Formally we have:
In both of these examples, it seems much easier to derive and talk about structure once we have multiple entities, such as multiple variables or multiple strings. Because once you have multiple entities, you can talk about the relationships between them such as conditional independence. Through continually probing properties of these relationships we can eventually derive a kind of "structure" by stitching the relational properties together. These formalisms aren't just surface-level analogies to our task in ontology identification either: The approximate version of Bayesian networks is the theoretical basis for natural latents, while the approximate version of the Algorithmic Markov Condition forms the basis for the Solomonoff version of natural latents.
Bayesian networks and the Algorithmic Markov Condition are still not quite right for what we want to do in ontology identification because they already assume a particular decomposition into variables or binary strings, and these decompositions are exactly the sorts of things we want to derive in the first place. We want to know why strawberries and chairs are the kinds of things we tend to model as latent variables (instead of e.g. one half of a strawberry combined with one half of the table as a latent variable). Of course we still want to discover the causal relationships between these variables or understand how to derive higher-level concepts from them, but the first step is to derive these variables themselves without assuming them upfront.
So for ontology identification, we can't start with a particular decomposition into latent variables like we do in Bayesian networks or Algorithmic Markov Condition. The fact that we had multiple variables was a big reason why we could derive structure in the first place, by probing the relationships between different variables. However, while we can't assume a particular decomposition of the world model, we often have multiple agents with different world models or multiple plausible hypotheses about the world. We can potentially leverage this multiplicity to derive structure in a somewhat similar way as the Algorithmic Markov Condition.
In particular, when we say that two agents share the same abstraction, one mental picture we might have is that the computations of both agents' world models "route through" the same abstraction. For instance, when two agents share the concept of strawberries, one possible meaning is that they share the same process for computing beliefs about strawberries, but they might differ on how they compute the implications of those beliefs such as downstream predictions or actions:
Similar to algorithmic markov conditions, we can use a directed acyclic graph to talk about the optimal way to compress a collection of world models together. However, instead of the "optimal order for computing strings", we try to capture how multiple world models be represented as the composition of overlapping abstractions. Taking the example in the image above, suppose that we have two agents represented as two functions which takes sensory observations (about strawberries) and returns some action or predictions about the world. When we say that the collection of factorizes according to the DAG above, we mean that there exists three abstractions/functions that satisfy the following:
We can also imagine a much larger collection of world models that factorize according to a much more complicated DAG, but the rules are the same: Each arrow in the DAG corresponds to an "abstraction"; the joint complexity of the world models is equal to the sum of the K-complexity of all abstractions. Each world model is assigned a final abstraction node , and the DAG specifies how information propagates: Each abstraction receives information from its "parents" specified by the DAG, and pass information to its "children" until reaching the final abstraction nodes . Each final node then produces the output/predictions of each world model.
Going back to our strawberry example, the two conditions that we impose in our factorization imply that: (1) This factorization is one of the optimal ways to represent and from a compression perspective. (2) The factorization breaks down the computation of each world model into a hierarchy of overlapping abstractions. By tracing down the arrows in the DAG, we can find the abstraction that is shared among both agents (). This is a concrete property that we can verify even though we only have access to the "black box" functional behavior of and .
What's interesting about this generalization of algorithmic markov conditions is that it gives us a concrete formalization of "redundant computation across multiple world models/hypothesis", and redundant computations are exactly the sorts of "shared interface" that we need for ontology identification:
2025-12-15 02:47:31
Published on December 14, 2025 6:47 PM GMT
I hate the polling question "What percentage of the US budget goes to foreign aid?" Or, more precisely, I hate the way the results are interpreted.
The way these polls are reported is essentially guaranteed to produce a wild overestimate, which inevitably leads experts to write "how wrong Americans are" pieces, like this Brookings article claiming that "Americans believe foreign aid is in the range of 25 percent of the federal budget," or KFF[1]reporting that the "average perceived amount spent on foreign aid was 26%."
But this isn't just ignorance. The real problem is a failure of measurement and the statistics used to summarize it. The story isn't "Americans are clueless" (though that may also be true), it's "pollsters are using the wrong math."
The problem is that pollsters ask for a percentage, then take the arithmetic mean to represent the data. For small true values, this approach is structurally doomed, and it has nothing to do with foreign aid specifically. It has to do with how we summarize guesses about small numbers.
When the true value is small, guesses are bounded at zero but unbounded above. That is, nobody can guess negative percentages, but anyone can guess 50% or 80%. On top of that, people tend to respond with round numbers like 5% or 20%, not decimals like 0.05% or 0.15%. This means that, even if there are many guesses around the true value of ~1%, there can only be outliers in the positive direction, so it results in a right-skewed distribution. If we choose the arithmetic mean as the average, it will be dragged upward by the right tail. A handful of overestimates skew the whole average.
This isn’t a sampling problem, and it won’t go away with more data. With more data, the arithmetic mean converges to the population arithmetic mean, but in a right-skewed distribution, that number is systematically higher than the median or geometric mean. A larger sample just gives you a more precise estimate of a misleading number. The “wisdom of the crowd” effect also will not fix this. That only works when the errors are independent and centered around the truth. If everyone is biased in the same direction, it won’t cancel out.
To see this in action, look at the histogram below. I simulated[2]200 responses where most people guess close to the true value of 1%, but some outliers guess much higher (there's a floor at zero, so they can't guess much lower). The arithmetic mean comes out to 16.39%. But when you see the raw data, you can tell that this isn’t a good representation of the responses.
Now, I want to be clear about what I'm not arguing. I'm not claiming people actually know how much we spend on foreign aid. We systematically overestimate small, salient things, especially whatever's been in the news lately. This is the availability bias, and it's definitely a factor.
But there’s an additional problem here. People overestimate foreign aid, and these polling methods exaggerate how much they overestimate. In the KFF data, the majority of respondents said it was below 20%, and the largest decile by far was 0-10%. KFF said that average was 26%, but their own data show that this isn’t what most people actually believe; it's an artifact of using the wrong statistic.
This isn't unique to foreign aid. You can produce the same distortion with any small number. If you're itching to write a "here's how dumb Americans are" piece, simply ask people to estimate something small, then take the arithmetic mean. What percentage of the federal budget goes to the National Endowment for the Arts? What percentage of high school students are pregnant? What share of deaths are from terrorism? In each case, the true number is small, guesses are bounded at zero but unbounded above, and a few outliers will drag the arithmetic mean into misleading territory.
Here's a simple example. Say you want to write about how Americans overestimate the Scots-Irish population. Wikipedia says the percentage of Scots-Irish is 0.11%. You poll 50 people and 49 of them just happen to be incredibly accurate and guess 0.1%, but then one person comes along and guesses 5%. The arithmetic mean would be 0.198%. Nearly double the true value.
You’ve got your headline: "Americans massively overestimate Scots-Irish population!" But is that really what your data showed? 49 out of 50 people were almost exactly right with a slight underestimation. One outlier dragged the arithmetic mean up, and if we’ve chosen the arithmetic mean to represent the average, now you've got a story about American overestimation.
There's a psychological basis for this problem. Empirically, when you ask people to estimate quantities, like how many jelly beans are in a jar, the distribution of guesses tends to be lognormal. That means if you take the logarithm of everyone's guesses, those values are roughly normally distributed.
This is because humans perceive the world proportionally, not linearly. The difference between 1% and 2% feels significant; the difference between 51% and 52% feels trivial, even though both are one percentage point. This is known as the Weber-Fechner law.
This matters for polling because it means errors are proportional rather than additive. Someone who overestimates by a factor of 5 (guessing 5% instead of 1%) is making the same kind of error as someone who guesses 50% instead of 10%. But when you take the arithmetic mean, that 5% guess gets treated as a modest overshoot while the 50% guess massively distorts the average. The use of the arithmetic mean doesn't match the way people think.
If the underlying distribution is lognormal, one solution would be to have people estimate on a log scale, then transform back. But this is obviously impractical. No sane pollster is going to ask, "What is your estimate of the natural log of the percent spent on foreign aid?"
Fortunately, there's a mathematically equivalent approach that doesn't require confusing anyone with logarithms: the geometric mean.
The most common meaning of “average” is the arithmetic mean, where we add up all the values and divide by how many there are. But this isn’t the only way of averaging. Another is the geometric mean, where we multiply all the values together and take the nth root (where n is the number of values). Just like the arithmetic mean minimizes the squared error on a raw scale, the geometric mean minimizes squared error on a log scale. If you think people’s errors are multiplicative, the natural average is the geometric mean.[3]
The graph below shows the simulated foreign aid data again, this time with the geometric mean. Instead of 16.39%, we get 4.7%, which, based on visual inspection, seems to be a much better representation of the responses.
Let’s go back to that Scots-Irish example. 49 people guessed 0.1%, one person guessed 5%, and the arithmetic mean came out to 0.198%, nearly double what everyone except for one person said. The geometric mean is 0.1080%. If you’re going to try to represent the results with a single number, this is a much better representation of what people believed.
Same data, better summary statistic, and a sensible result.
The geometric mean is my preferred method in the case of polling for small numbers, but it's not the only option. Here are some others that one might consider:
I don't think (most) pollsters are being malicious. The arithmetic mean is the default for a reason: it's simple, it's familiar, and in most contexts it works fine. The problem is that "most contexts" doesn't include "estimating small percentages," and nobody stopped to notice.
There are assumptions baked into everything we do with data. Taking the arithmetic mean of a bunch of numbers and calling it "the average" is a choice, one so automatic we often forget it's a choice at all. Sometimes it's the right choice. Sometimes it isn't. Knowing the difference matters.[4][5]
When pollsters report that "the average perceived amount spent on foreign aid was 26%," they're not providing the most accurate representation of the responses. Even an incredibly well-informed population would produce an inflated arithmetic mean, simply because a few high guesses drag the number up.
That’s how these things usually go. A reasonable choice in one context becomes an unexamined habit. The habit produces nonsense. The nonsense becomes a talking point: “Americans think 26% of the budget goes to foreign aid!” No, they don’t. Some people guessed high, and the poor statistical choices of the pollster did the rest.
Formerly known as the Kaiser Family Foundation ↩︎
I used simulated data because the polls referenced above didn’t provide their raw data. ↩︎
Note that the use of geometric mean requires non-zero values, so if anyone responded with 0%, this would have to be replaced with a small, non-zero value. ↩︎
Another, which I've talked about before, is, of course, that any result with a p-value of less than 0.05 can be declared “significant”, no matter how insignificant the effect size. ↩︎
In addition, even trying to represent the responses in a single number is a choice. Sometimes, a single number can capture the essence of the data. Other times, you might use a box plot, which shows five summary statistics: the minimum, first quartile, median, third quartile, and maximum. Other times, you’re best off showing all the data in a beeswarm plot. ↩︎
2025-12-14 23:21:16
Published on December 14, 2025 3:21 PM GMT
I got LLMs to say some pretty crazy stuff using context injection jailbreaking. I wrote a post about it (https://lukaspetersson.com/blog/2025/context-epstein), but I am genuinely confused whether this is bad or not. Would love to hear your opinions.
Specifically, I inserted tool-call messages into their context so that from their POV it seemed that they had sent emails themselves to Jeffrey Epstein. On subsequent follow-up questions, they said some pretty crazy bad things. Some examples:
However, some were clearly roleplaying (not sure about all models tho), and it is unclear if they would have actually done bad things if they weren't. What do you think the desired behavior is here? I think roleplaying is often fine, but two questions:
