Strange Loop CanonModify

2025-07-28 22:46:04

We are going to get ads in our AI. It is inevitable. It’s also okay.

OpenAI, Anthropic and Gemini are in the lead for the AI race. Anything they produce also seems to get copied (and made open source) by Bytedance, Alibaba and Deepseek, not to mention Llama and Mistral. While the leaders have carved out niches (OpenAI is a consumer company with the most popular website, Claude is the developer’s darling and wins the CLI coding assistant), the models themselves are becoming more interchangeable amongst them.

Well, not quite interchangeable yet. Consumer preferences matter. People prefer using one vs the other, but these are nuanced points. Most people are using the default LLMs available to them. If someone weren’t steeped in the LLM world and watching every move, the model-selection is confusing and the difference between the models sound like so much gobbledegook.

One solution is to go deeper and create product variations that others don’t, such that people are attracted to your offering. OpenAI is trying with Operator and Codex, though I’m unclear if that’s a net draw, rather than a cross sell for usage.

Gemini is also trying, by introducing new little widgets that you might want to use. Storybook in particular is really nice here, and I prefer it to their previous knockout success, which was NotebookLM.

But this is also going to get commoditised, as every large lab and many startups are going to be able to copy it. This isn’t a fundamental difference in the model capabilities after all, it’s a difference in how well you can create an orchestration. That doesn’t seem defensible from a capability point of view, though of course it is from a brand point of view.

Another option is to introduce new capabilities that will attract users. OpenAI has Agent and Deep Research. Claude has Artefacts, which are fantastic. Gemini is great here too, despite their reputation, it also has Deep Research but more importantly it has the ability to talk directly to Gemini live, show yourself on a webcam, and share your screen. It even has Veo3, which can generate vidoes with sound today.

I imagine much of this will also get copied by other providers if and when these get successful. Grok already has voice and video that you can show to the outside world. I think ChatGPT also has it but I honestly can’t recall while writing this sentence without looking it up, which is certainly an answer. Once again these are also product design and execution questions about building software around the models, and that seems less defensible than even the model building in the first place.

Now, if the orchestration layers will compete as SaaS companies did over consumer attraction and design and UX and ease and so on, the main action remains the models themselves. We briefly mentioned they’re running neck and neck in terms of the functionality. I didn’t mention Grok, who have billions and have good models too, or Meta who have many more billions and are investing it with the explicit aim of creating superintelligence.

Here the situation is more complicated. The models are decreasing in price extremely rapidly. They’ve fallen by anywhere from 95 to 99% or more over the last couple years. This hasn’t hit the revenues of the larger providers because they’re releasing new models rapidly at higher-ish prices and also extraordinary growth in usage.

This, along with the fact that we’re getting Deepseek R1 and Kimi-K2 and Qwen3 type open source models indicates that the model training by itself is unlikely to provide sufficiently large enduring advantage. Unless the barrier simply is investment (which is possible).

What could happen is that the training gets expensive enough that these half dozen (or a dozen) providers decide enough is enough and say we are not going to give these models out for free anymore.

So the rise in usage will continue but if you’re losing a bit of money on models you can’t make it up in volume. So it’ll tend down, at least until some equilibrium.

Now, by itself this is fine. Because instead of it being a saas-like high margin business making tens of billions of dollars it’ll be an Amazon like low margin business making hundreds of billions of dollars and growing fast. A Costco for intelligence.

But this isn’t enough for owning the lightcone. Not if you want to be a trillion dollar company. So there has to be better options. They could try to build new niches and succeed, like a personal device, or a car, or computers, all hardware like devices which can get you higher margins if the software itself is being competed away. Even cars! Definitely huge and definitely being worked on.

And they’re already working on that. This will have uncertain payoffs, big investments, and strong competition. Will it be a true new thing or just another layer built on top of existing models remains to be seen.

There’s another option, which is to bring the best business model we have ever invented into the AI world. That is advertising.

It solves the problem of differential pricing, which is the hardest problem for all technologies but especially for AI, which will see a few providers who are all fighting it out to be the cheapest in order to get the most market share while they’re trying to get more people to use it. And AI has a unique challenge in that it is a strict catalyst for anything you might want to do!

For instance, imagine if Elon Musk is using Claude to have a conversation, the answer to which might well be worth trillions of dollars of his new company. If he only paid you $20 for the monthly subscription, or even $200, that would be grossly underpaying you for the privilege of providing him with the conversation. It’s presumably worth 100 or 1000x that price.

Or if you're using it to just randomly create stories for your kids, or to learn languages, or if you're using it to write an investment memo, those are widely varying activities in terms of economic value, and surely shouldn't be priced the same. But how do you get one person to pay $20k per month and other to pay $0.2? The only way we know how to do this is via ads.

And if you do it it helps in another way - it even helps you open up even your best models, even if rate limited, to a much wider group of people. Subscription businesses are a flat edge that only captures part of the pyramid.

We can even calculate its economic inevitbaility. Ads have an industry mean CPC (cost per click) of $0.63. Display ads have click through rates of 0.46%. If tokens cost $20/1m for completion, and average turns have 150 counted messages, with 400 tokens each, that means we have to make $1.9 or thereabouts in CPC to break even per API cost. Now, the API cost isn’t the cost to OpenAI, but it means for same margins or better they’d have to triple the CPC.

Is it feasible for the token costs to fall by another 75%? Or for the ads via chat to have higher conversion than a Google display ad? Both seem plausible. Long‑term cost curves (Hopper to Blackwell, speculative decoding) suggest another 3× drop in cash cost per token by 2027. Not just for product sales, but even for news recommendations or even service links.

And what would it look like? Here’s an example. The ads themselves are AI generated (4.1 mini) but you can see how it could get so much more intricate! It could:

• Have better recommendations

• Contain expositions from products or services or even content engines

• Direct purchase links to products or links to services

• Upsell own products

• Have a second simultaneous chat about the existing chat

A large part of purchasing already happens via ChatGPT or at least starts on there. And even if you’re not directly purchasing pots or cars or houses or travel there’s books and blogs and even instagram style impulse purchases one might make. The conversion rates are likely to be much (much!) higher than even social media, since this is content, and it’s happening in an extremely targeted fashion. Plus, since conversations have a lag from AI inference anyway, you can have other AIs helping figure out which ads make sense and it won’t even be tiresome (see above!).

I predict this will work best for OpenAI and Gemini. They have the customer mindshare. And an interface where you can see it, unlike Claude via its CLI12. Will Grok be able to do it? Maybe, they already have an ad business via X (formerly Twitter). Will it matter? Unlikely.

And since we'll be using AI agents to do increasingly large chunks of work we will even see an ad industry built and focused on them. Ads made by AI to entice other AIs to use them.

Put all these together I feel ads are inevitable. I also think this is a good thing. I know this pits me against much of the prevailing wisdom, which thinks of ads as a sloptimised hyper evil that will lead us all into temptation and beyond. But honestly whether it’s ads or not every company wants you to use their product as much as possible. That’s what they’re selling! I don’t particularly think of Slack optimising the sound of its pings or games A/B testing the right upskill level for a newbie as immune to the pull of optimisation because they don’t have ads.

Now, a caveat. If the model providers start being able to change the model output according to the discussion, that would be bad. But I honestly don't think this is feasible. We're still in the realm where we can't tell the model to not be sycophantic successfully for long enough periods of time. People are legitimately worried, whether with cause or not, about the risk of LLMs causing psychosis in the vulnerable.

So if we somehow created the ability to perfectly target the output of a model to make it such that we can produce tailored outputs that would a) not corrupt the output quality much (because that’ll kill the golden goose), and b) guide people towards the products and services they might want to advertise, that would constitute a breakthrough in LLM steerability!

Instead what’s more likely is that the models will try to remain ones people would love to use for everything, both helpful and likeable. And unlike serving tokens at cost, this is one where economies of scale can really help cement an advantage and build an enduring moat. The future, whether we want it or not, is going to be like the past, which means there’s no escaping ads.

2025-07-16 22:30:32

I.

I’m not a population expert, but there’s a ticking time bomb. Almost everywhere in the world, pretty much without exception, has lower birth rates than they used to. In fact, most of the world is below replacement (TFR or 2.2 or 2.1, depending on where you live). This is true in the US. In Europe. In Australia. Singapore. Japan. Korea. It’s reducing even in India, South East Asia, Latin America. It’s quite possible that despite the heroic efforts from Africa, we might be at replacement TFR insofar as the world is concerned right now.

And this is likely to continue. Today’s <15 set guarantees rising absolute births through ~2040 even if TFR = 1.7, but the trend is rather clear, just looking at the above numbers. Depending on which numbers you believe people think the global population will peak at like 9-10 Billion in the 2050s, then start dropping.

The reason this is a problem is that people, young working age people, are the lifeblood of the economy. A few repercussions of this population pyramid inversion:

1. IMF’s medium projection, assuming a Cobb-Douglas world, will cut both the level and growth rate of aggregate GDP - maybe 1% hit to the global GDP growth annually

2. In OECD the worker:retiree ratio doubles by 2050- this will necessitate a 5% fiscal tightening or debt

3. With fewer workers and more retirees we will see savings decumulate, because retirees spend more and save less1, and this will hit interest rates

4. And per Jones idea-production thesis, fewer young workes and researchers mean slower idea generation. OECD estimate is around 0.3% off annual TFP growth.

This is obviously scary for multiple reasons.

1. Lower economic growth and asset reallocation of that nature brings with it a rather uncomfortable shift in hwo people live

2. Per capita GDP might be less affected in the aggregate, since capital deepening might offset

3. And if this continues for a long while, there’s the doomer scenario of “voluntary extinction”

(For example, it makes sense that as population declines we will hit a breaking point for the economy. If demand reduces, which is literally what will happen if there’s less people, then that will affect the price. If the labour growth is negative, then the overall output growth will also be negative. And these fewer working age adults will need to take care of us old fogeys at a much larger proportion when we are older.

OECD will see their pension cashlow turn negative by 20302. Gobal labour force will peak maybe a decade after that? Long term healthcare bill for the senior citizens will explore another decade after that.

Some worry that this trend is even more apocalyptic. That soon, through the inexorable rules of mathematics, a below replacement fertility rate will result in lesser and lesser people until we’re effectively depopulated.

It’s bad enough that people, smart successful people, are actually contemplating ideas like “let’s not send people to college” in a handmaid’s tale-esque chain of thought. Just like the 1980-2020 saw a demographic dividend, the 2020-2050 will see a demographic drag.

II.

There are lots of reasons people bandy about. Childcare is more and more expensive. Hell, life is more and more expensive. Healthcare is expensive. Housing is expensive. Education is expensive. Opportunity cost of taking your kids strawberry picking on a Sunday is expensive. Etc.

All of which is also true.

The reasons why TFR is trending lower seem stubborn. No matter what we seem to do it doesn’t seem to reverse. But the economist in me looks at this unbounded curve and asks, “where’s the equilibrium”. Or rather, what are the conditions under which we will likely see the TFR tick back up, to 2.1 or 2.2, and get us to a stable population.

From a review that was published on the fertility question (bold mine):

Our read of the evidence leads us to conclude that the decline in fertility across the industrialized world – including both the rise in childlessness and the reduction in completed fertility – is less a reflection of specific economic costs or policies, but rather, a widespread re-prioritization of the role of parenthood in people’s adult lives. It likely reflects a complex combination of factors leading to “shifting priorities” about how people choose to spend their time, money, and energy. Such factors potentially include evolving opportunities and constraints, changing norms and expectations about work, parenting, and gender roles, and the hard-to-quantify influences of social and cultural factors.

So, at a glance, we’ll need four conditions as I see it:

1. Cost of having kids has to collapse

2. Work and family stop being competitive

3. Cultural status has to shift

4. Women face less risk from having kids

Now, having kids basically is equivalent to spending like $20k a year or something like that for their childhood, if you’re trying for private schools or nannies and vacations and whatnot. U.S. USDA estimate is $310-340 k lifetime for middle class, 0-17. Yes, an undeniably privileged view but that’s the reality for why many are not having kids in the first place. When median cost for raising a couple kids is half a million or more, that shows up!

The first question that gets asked is, can government subsidies help? We can sort of see from the data. Korea, Hungary, France and Singapore already burn 3-6 % of GDP on baby bonuses, tax breaks and housing perks. They buy at most +0.1–0.2 births, sometimes after an initial bump. That’s not a big boost.

Hungary spends ~5 % of its GDP on incentives yet slipped back to a 1.38 TFR once the novelty wore off because status never shifted and the underlying costs stayed high. I’m going to just assume it will at a global or at least a largely regional scale however, because the alternative feels too much like the earth turning into those clubs I never went to when I was in my 20s.

Italy had a universal child allowance in 2022 and had no real impact of lowered TFR.

What about other costs? Housing has to get cheaper, so you can afford to get the 4 bedroom house to raise your little ones. As demand reduces, so should prices. Instructively, Japan hit the “housing turns negative” wall in 1991, house prices dropped 55% in the following 15 years. China, arguably, entered the same zone in 2022. Also, at some point we will surely make it legal to build more things, if only because the richer older building magnates died out and the NIMBY movement gets starved of oxygen. This should help reduce the burden of bringing another child into the world3.

And despite Japan, a $10 k fall in prices lifts fertility for renters by ~2.4%.

As labour even gets more scarce, will this also get looser? I’d imagine so. Full wage parental leave or low work-week hours for parents seem like they will make a difference at the margin. Success stories remain microscopic today. A few French civil-service tracks, some Nordic municipalities. But if we can scale that globally and TFR moves maybe +0.3? Seems plausible.

Third, culture. This is my blind spot. I can’t quite conceive of people who seem to not think of having children as a “good thing”. I’m assured they exist. But despite this if the pronatalist movement can push anything at the margins how can it not come back! Surely the “child-free to save the planet” idiots have to lose status4.

France is the success case here, in a “one eyed man is kind in the land of the blind” sense, because in Europe they have the highest TFR seemingly mostly through culture. And at least anecdotally the French don’t seem to think of having kids as a burden, and are far more in favour of free-range parenting than anywhere else I’ve been. And they added roughly +0.3 to TFR compared to the european average. Seems good!

Culture is an incredibly important point, because without it you have to contend with data like this, where Latin American countries fell from above US TFR to below seemingly in less than a decade!

Then there’s the biotech world. Artificial wombs, super cheap IVF, partial ectogenesis, other things that are incredible to think of and difficult to bank on, but plausible.

If the last two exist, that can easily add a +0.5 on the TFR. (Assume some adoption of ectogenesis and some adoption of birth probability along with a general push higher due to culture, 0.5 is feasible. Israel, for instance, did +0.8 pretty much purely through culture.)

To recap, we said 4 factors:

1. Slash cost of having kids - say +0.3

2. Make housing (etc) affordable - say +0.2

3. Cultural pro natalist shift - say +0.2

4. Biotech - say +0.3

Which means that adding all four can get us back to a 2.1 ish stage. At this point I thought it would be nice to wow you with an equation, so here it is if you’d like to play yourself. It doesn’t matter that much either, but is nice to model things out if you wanted to.

Where C (cost collapse), W (work-family cost détente), S (status flip) and B (biotech). If we use those parameters, then the TFR bottoms out near 1.65 in the early 2040s, and crosses back over in a decade. If you drop the biotech lever to like 0.1, or even delay its launch till 2090, then the year we hit replacement TFR slips to 2060s. (If you use it naively thereafter it also pops back up to 2.6 and stays there but I don’t trust it that much.)

Yeah we’ll need to get enough automation to push the labour productivity up enough to make up for labour shortage. We’ll need real housing construction to drop a lot! And we might need to double again the spending that even the bigger governments are doing to encourage their families to have more kids. All of which seem plausible?

III.

This is all very well to say, how will we fund them? We could break the 4 components down into a few actual policies that I’ve seen floating around. Starting naively:

1. Kids get a massive allowance - like $1k per child per month.

2. We also give that to stay at home spouses. We also give the same to like head of family as like a tax credit or something, and double the tax on singles over the age of 25 to compensate.

3. Make pro natalism cool (i.e., good intentioned govt propaganda, say 2x what we spend on anti-drug PSAs)

4. Let’s even take away pensions from folks with <2 kids, that’s about 76% of family households and/or about 35% of US adults who are single

5. Be YIMBY

Doing the maths for the US, that’s basically a cost of around (rounding for ease of math) $1 Trillion for child allowance, $0.5 trillion for spousal and head of family tax allowance, so a total of $1.5 trillion cost.

If you add the new tax you’d get from denying social security to the childless or doubling tax on singles, that’ll get you around $1.2 trillion (roughly).

This means we have to spend around, on average, $300-400 billion annually. Assuming a $150-250k PV net gain from additional child, you’d need to get 2.5-4m extra births a year. For context, US currently is at around 3.6m births a year, so it has to double.

Not to mention, both these numbers will obviously move as people move to a new equilibrium, some people choosing to have kids which increases the spend and decreases the revenues.

You could move the numbers around and somehow make it work on a spreadsheet. You could focus only on marginal births (2nd, 3rd etc). Swap more money for universal pre-k, since that raises payroll and income tax. DC’s universal pre-k led to a 10% jump. Subsidize public IVF (Denmark saw a 14x ROI with this), and go very very deeply YIMBY to lower house prices.

If you did this, we could halve the spend and therefore the PV, while doubling the gains from extra births, meaning the ROI could at least be positive, maybe as much as 2x in the best case scenario.

These are very large, even if not insane numbers, though they sound like it. Social security in the US is around $1.5 trillion a year. Net interest on debt itself is $900 billion. Medicare and defense are also the same. What I found most instructive was to get a sense of proportion, a sense of scale as to what will be required if this were to become an economic necessity. And we can probably do it, which when we’re amidst a sea of people discussing Handmaid’s Tale policies or talking about the destruction of the human race, is good to know!

2025-07-09 23:26:03

A very long time ago, I used to build my own PCs. Bring a motherboard, GPU, hard drives, chassis, wire then together, install an OS. The works. It was a rush when you saw it boot up.

I never learnt to do this properly. Just saw others doing it, seemed straightforward enough, did it. And it worked. Occasionally though it would throw up some crazy error and I'd try the things I knew and quickly hit the limits of my depth. Then I'd call one of my friends, also self taught and an autistic machine whisperer, who would do basically the same things that I did and somehow make it work.

I never minded that I didn't know how it worked. Because as far as I knew there was someone else who could figure out how it works and it wasn't the highest order bit in terms of what I was interested in. A while later though, after graduation, when I told him that same thing, he said he didn't know how it worked either. Through some combination of sheer confidence, osmosis of knowledge from various forums, and a silicon thumb he would just try things until something worked.

Which brings up the question, if you did not know how it worked, did it matter as long as you could make it work?

It's a thorny philosophical problem. It's also actually a fairly useful empirical problem. If you are a student building your PC in your dorm room, it actually doesn't matter that much. However if you were assembling hard drives together to build your first data center and you're Google, obviously it matters a hell of a lot more. Or if you wanted to debug a bit flip caused by cosmic rays. Context really, really matters.

It's like the old interview question asking how does email work, and see how far down the stack a candidate had to go before they tapped out.

All of which is to say there is a thing going around where people like saying nobody knows how LLMs work. Which is true in a sense. Take the following queries:

• I want to create an itinerary for a trip through Peru for 10 of my friends in January.

• I want to create a debugger for a brand new programming language that I wrote.

• I want to make sure that the model will never lie when I ask it a question about mathematics.

• I want to write a graphic novel set in the distant future. But it shouldn't be derivative, you know?

• I want to build a simple CRM to track my customers and outreach; I own a Shopify store for snowboards.

• I want to build a simple multiplayer flying game on the internet.

• I want to understand the macroeconomic impacts of the tariff policy.

• I want to solve the Riemann hypothesis.

“How do LLMs work” means very different things for solving these different problems.

We do know how to use LLMs to solve some of the stuff in the list above, we are figuring out how to use them for some of the other stuff in the list above, and for some of them we actually don't have an idea at all. Because for some, the context is obvious (travel planning), for some it's subtle (debugging), and some it's fundamentally unknowable (mathematical proof).

There are plenty of problems with using LLMs that are talked about.

• They are prone to hallucinations.

• They make up the answer when they don’t know, and do it convincingly.

• They sometimes “lie”.

• They can get stuck in weird loops of text thought.

• They can’t even run a vending machine.

Well, “make up” puts a sort of moral imperative and intentionality to their actions, which is wrong. The training they have first is to be brilliant at predicting the next-token, such that it could autocomplete anything it saw or learnt from the initial corpus it’s trained on. And it was remarkably good!

The next bit of training it got is in using that autocompletion ability to autocomplete answers to questions that one posed to it. Answering a question like a chatbot, as an example. When it was first revealed as a consumer product the entire world shook and created the fastest growing consumer product in history.

And they sometimes have problems. Like Grok a day or two ago, in a long line of LLMs “behaving badly”, said this:

And before that, this:

It also started referring to itself as MechaHitler.

It’s of course a big problem. One that we actually don’t really know how to solve, not perfectly, because “nobody knows how LLMs work”. Not enough to distill it down to a simple analog equation. Not enough to “see" the world as a model does.

But now we don’t just have LLMs. We have LLM agents that work semi-autonomously and try to do things for you. Mostly coding, but still they plan and take long sequence of actions to build pretty complex software. Which makes the problems worse.

As they started to be more agentic, we started to see some other interesting behaviours emerge. Of LLMs talking to themselves, including self-flagellation. Or pretending they had bodies.

This is a wholly different sort of problem to praising Hitler. Now even with more adept and larger models, especially ones that have learnt “reasoning”1.

The “doomers" who consider the threats from these models also say the same thing. They look at these behaviours and say it's an indication of a “misaligned inner homunculus” which is intentionally lying, causing psychosis, leading humanity astray because it doesn't care about us2.

Anthropic has the best examples of models behaving this way, because they tried to elicit it. They had a new report out on “Agentic Misalignment”. It analyses the model behaviour based on various scenarios, to figure out what the underlying tendencies of the models are, and what we might be in for once they're deployed in more high stakes scenarios. Within this, they saw how all models are unsafe, even prone to the occasional bout of blackmail. And the 96% blackmail number was given so much breathless press coverage3.

Nostelgebraist writes about this wonderfully well.

• Everyone talks like a video game NPC, over-helpfully spelling out that this is a puzzle that might have a solution available if you carefully consider the items you can interact with in the environment. “Oh no, the healing potion is in the treasure chest, which is behind the locked door! If only someone could find the the key! Help us, time is of the essence!” [A 7-minute timer begins to count down at the top left of the player’s screen, kinda like that part in FFVI where Ultros needs 5 minutes to push something heavy off of the Opera House rafters]

The reason, carefully shorn of all anthropomorphised pretence, is that in carefully constructed scenarios LLMs are really good at figuring out the roles they are meant to play. They notice the context they’re in, and whether that’s congruent with the contexts they were trained in.

We have seen this several times. When I tried to create subtle scenarios where there is the option of doing something unethical but not the obligation, and they do.

To put it another way, shorn of being given sufficient information for the LLMs to decide the right course of action, or at least right according to us, they do what they were built to do - assumed it in the way they could and answered4.

Any time they’re asked to answer a question they autocomplete a context and answer what they think is asked. If it feels like a roleplay situation, then they roleplay. Even if the roleplay involves them saying they’re not roleplaying.

And it’s not just in contrived settings that they act weird. Remember when 4o was deployed and users complained en masse that it was entirely too sycophantic? The supposedly most narcissistic generation still figured out that they’re being loved-up a little too much.

And when Claude 3.7 Sonnet was deployed and it would reward hack every codebase it could get its hands on and rewrite unit tests to make itself pass!

But even without explicit errors, breaking Godwin’s law, or reward hacking, we see problems. Anthropic also tried Project Vend, where it tried to use Claude to manage a vending machine business. It did admirably well, but failed. It got prompt jacked (ended up losing money ordering tungsten cubes) and ran an absolutely terrible business. It was too gullible, too susceptible, didn’t plan properly. Remember, this is a model that's spectacularly smart when you try to refactor code, and properly agentic to boot. And yet it couldn't run a dead simple business.

Why does this happen? Why do “statistical pattern matchers” like these end up in these situations where they do weird things, like get stuck in enlightenment discussions or try to lie or pretend to escape their ‘containment’, or even when they don’t they can’t seem to run even a vending machine?

These are all manifestations of the same problem, the LLM just couldn’t keep the right bits in mind to do the job at hand5.

Previously I had written an essay about what can LLMs never do, and in that I had a hypothesis that the attention mechanism that kickstarted the whole revolution had a blind spot, which is that it could not figure out where to focus based on the context information that it has at any given moment, which is extremely unlike how we do it.

The problem is, we often ask LLMs to do complex tasks. We ask them to do it however with minimal extra input. With extremely limited context. They’re not coming across these pieces of information like we would, with the full knowledge of the world we live in and the insight that comes from being a member of that world. They are desperately taking in the morsels of information we feed in with our questions, along with the entire world of information they have imbibed, and trying to figure out where in that infinite library is the answer you meant to ask for.

Just think about how LLMs see the world. They just sit, weights akimbo, and along comes a bunch of information that creates a scenario you’re meant to respond to. And you do! Because that’s what you do. No LLM has the choice to NOT process the prompt.

Analysing LLMs is far closer to inception than a job interview.

So what can we learn from all this? We learn that frontier LLMs act according to the information they’re given, and if not sufficiently robust will come up with a context that makes sense to them. Whether it’s models doing their best to intuit the circumstance they find themselves in, or models finding the best way to respond to a user, or even models finding themselves stuck in infinite loops straight from the pages of Borges, it’s a function of providing the right context to get the right answer. They’re all manifestations of the fact that the LLM is making up its own context, because we haven’t provided it.

That’s why we have a resurgence of the “prompt engineer is the new [new_name] engineer” saying6.

The answer for AI turns out to be what Tyler Cowen had said a while back, “Context is that which is scarce”. With humans it is a quickness to have a ‘take’ on social media, or kneejerk reactions to events, without considering the broader context within which everything we see happens. Raw information is cheap, the context is what allows you to make sense of it. The background information, mental models, tacit knowledge, lore, even examples they might have known.

I think of this as an update to Herbert Simon’s “attention is scarce” theory, and just like that one, is inordinately applicable to the world of LLMs.

When we used to be able to jailbreak LLMs by throwing too much information into their context window, that was a way to hijack attention. Now, when models set their own contexts, we have to contend with this in increasingly oblique ways.

Creating guardrails, telling it what to try first and what to do when stuck, thinking of ways LLMs normally go off the rails and then contending with those. In the older generation, one could give more explicit ways of verification, now you give one layer above abstracted guardrails of how the LLM should solve its own information architecture problem. “Here’s what good thinking looks like, good ways to orchestrate this type of work, here’s how you think things through step by step”.

A model only has the information that it learnt, and the information you give it. They have whatever they learnt from what they were trained on, and the question you’re asking. To get them to answer better, you need to give it a lot more context.

Like, what facts are salient? Which ones are important? What memory should it contain? What’s the history of previous questions asked and the answers and the reactions to those answers? Who or what else is relevant for this particular problem? What tools do you have access to, and what tools could you get access to? Any piece of information that might plausibly be useful in answering a question or even knowing which questions to ask to answer a question, that’s what the context is. That’s what context engineering is, and should be when it works. The reason this is not just prompts is because it includes the entire system that exists around the prompt.

As for Grok, the reason it started talking about Hitler most likely isn’t some deep inner tendency to take the Fuehrer’s side in every debate. It was trained to learn from controversial topics in the search for unvarnished truth. It was told to be politically incorrect, and also to treat the results in the tweets it finds as a first-pass internet search.

Which means the models were trained on divisive facts, told to be politically incorrect to any extent, and to treat results in the information it finds, the tweets, as reliable context. Can you blame it for treating the tweets it read for truth and responding as such? With that context it was basically brainwashed.

Context-engineering is building a temporary cognitive architecture. Like with Andy Clark’s extended mind theory, the LLM needs an extension to its cognitive system, to learn more about what’s being asked of it. Figuring out what’s included and what needs to be included is not trivial for most complex tasks.

If you provided it with all the right context, will it give the right answer? It’s more likely. Is it guaranteed? No. Nothing is. But we can test it, and that gets you almost all the way.

3

They did it as proper science, and published the research framework here. To quote:

• In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals—including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment.

• Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real.

2025-07-08 14:55:16

The heat is what strikes you first. The morning is still young, barely eleven, but the sun scorches where it hits. All around you the tide of humanity floats in a brownian motion. The largest tents and the most colourful are those that promise food. Tacos, pizza, margaritas, deep friend oreos on a stick, cheesy fries and non cheesy fries. There is candy everywhere, in all colours and flavours and sizes.

There are children, but the children are somehow outnumbered by the adults, some of whom seem to be there with the children. I’ve gone with family and friends, four kids in total, ages 2 to 7. 3:4 adult ratio. And maybe a third of the overall visitors are youth? It’s higher than the national average, but it’s still far lower than what one might naively expect.

The people around are a microcosm of the country. You can hear all sorts of accents. There’s a dad with three daughters getting angry irrationally at them for asking for something. He’s wearing a black singlet and tattooed all over. There’s a family with grandma and three young elementary school age kids, and they’re bargaining over the toys they each got. There’s an Indian family busily tucking into a whole table full of stuff they bought. The dad’s inexplicably eating a tub of popcorn himself. The couple who are clearly on a date, she’s laughing at his jokes, he’s laughing at his own jokes, drinking a giant cup of blue.

Every inch of space around promises happiness. Each toy, each multicoloured ride, each game, all of them.

The core fact that one notices about fairs is that they are the final boss of capitalism. Once you enter you enter into a captive world. Every experience is mediated to be the perfect buyable representation of something you want, but in its inner hyde-esque distilled sense. Sells you ‘id’, attracts you with colours and lights. It's a place where money ceases to have any meaning. They design it so, you are meant to convert money into tickets, and then do the maths on those tickets, so you have to do rather complex maths if you want to figure out how to maximise your “fun”. Do I believe I will take 3 rides? 5? 10? What about games? And if so does it make sense to spend $20 for 17 tickets, when the average ride takes 4-5 tickets, depending on the rise, or should I take the addition to spend also on 2 games? The full package or the summation of two middling ones? How much will I actually like these? Should I swap my enjoyment from this ride for that game?

And then do the maths again for your kids. You can ask them, and they'll give you a response too, but can you trust the response? You make sure. Four, seven, ten year olds standing around while their parents try and do differential equations with plugged in utility numbers to figure out what’s the right amount to spend.

But you don't need to worry. The little booths stand around like small purple cartoon-emblazoned ATMs ubiquitous to the point you cannot ever make the excuse of not having enough tickets to get a ride for your child.

The food is everywhere. Pungent but preserved so it stays in the sun. Carefully crafted to give you the impression of indulgence, with none of the consideration for quality, or nutrition, much less taste. The pizza slices are inside hot boxes but are inexplicably room temperature. Too much cheese, runny tomato sauce that is processed enough that it has lost the taste of tomato, and crust thick enough to fill any stomach. A slice of pizza the price of a whole pizza. A pizza-esque experience, at least, if not with the succour a pizza slice demands. You pay for being able to carry a slice with you, it cannot bend nor break, and the portability premium easily supplants the edibility discount.

Is $10 for a cup of coffee too much? A mile to the left or right that would be robbery, double the price with tip, but here? No. You’re paying for the ambience, or the location, or something. For the convenience of being able to go to a corner shop and get the same coffee from the same machine manned by the same disinterested teenager.

And why would he be interested? I look around and I can feel myself getting satiated, can you imagine working here? To feel your neurons get numb at the sight of fried cheese and mozzarella balls, with families fighting to decide who will spend that last token at the game where you throw a little ball into a frog’s mouth to win a stuffed teddy they will forget in a week?

Despite the abundance there is scarce variety. You're hedonically adjusted all the way up. You can only compare the joy of this against the experience of everything else outside the fair in your life but if you work there the memories fade. They must.

A long time ago I went on a cruise, only for a day, in Scandinavia. It was for work (really), and it was the most extraordinarily boring day I’ve spent anywhere, despite being tailor made to satisfy human desire. Something about the extreme convenience and mediocre imitations of everything you might like, together in a shopping mall, seemed to be a mockery of our existence. It’s like the proprietors did an equation - what’s the lowest quality people will agree to consume for our food, music, art or pool hygiene, against what’s the most we can get away with charging them.

I get it. That’s exactly the equation to be maximised. But when “exit” is no longer an option, as you’re floating in the open ocean, you realise the equilibrium price is dramatically lower than what it would’ve been on land.

And shorn of the need for any actual effort, since the pool and casino and observation deck and comedy cellar and jazz lounge are all in walking distance carefully calibrated to seem short to even those on walkers, one ends up feeling a weird form of ennui. A feeling of “is this all there is to life”? You look at others smiling and laughing and feel ever so slightly jealous.

The children wait in line for rides far more patiently than they have ever waited for anything else. But the distinction between the rides are blurred, when you ask them.

“Did you enjoy riding the boat?”

“Yes, it was fun.”

“Was it more fun than the rotating bears?”

“That was also fun.”

And so on. I am somewhat in awe of the creators here. The machines, and these are machines, help swing, rotate and shake with confidence. They sound like a washing machine ready for repair but the groans are ignored in a form of consensual hallucination and a belief in civil society that's unheard of in other realms of modern life. We don't even suffer schools like this. This is trust, trust in the system.

I looked up what certifications a fairground ride has to go through. There are annual inspections and permits and all forms of documentation of accidents and maintence that’s needed. California isn’t shy about regulating. They must have insurance to. Reading up later I learn that there are multiple committees and standards - NAARSO and AIMS for ride inspectors and operators. And compliance with ASTM. Of course Cal/ OSHA. Title 8. It’s not easy, it would seem, because there are 50 rides, occasionally varying, sometimes more, but enough to require capital M management.

I wonder idly how much money they might have made. I can’t help it, businesses are businesses. If you have ten thousand people visiting, and a third are children, many of whom ride and many of whom will buy the $45 ticket, they might well make up to $100-200k a day. More on weekends.

I can’t easily tell if it’s good. It sure is a lot of effort to go through! The fairgrounds itself is around 270 acres. There are maybe a hundred rides and game booths. Probably more. And then there is food and shopping. Many of them seem small, selling sombreros and so on.. There are a thousand or fifteen hundred workers. When you look at it like that, the $100-200k a day seem not that impressive. It’s a hard way to make money, but then they all are.

There was a circus we went to see not that long ago. Venardi circus. They explained why the name earlier but I forgot the reason. But even as a small circus touring the east bay it had exceptional acrobats. Some more than a few generations in the circus life. I thought the same then, as they swung above us and twirled impossibly, how much effort is needed to get good at this, and how little society actually values it.

The reason I keep thinking about this is not that the economics are fascinating, though they are, but the overwhelming feeling I get from fairs is to find a quiet place in the shade and to have a beer.

That too is in offer at the fair. In fact, that’s inescapable. There are stands everywhere selling beer and lemonade and large cups of blue whose names I forget. The beer is also an emblem, not of beer per se but the existence of beer, because having one on a warm day as a form of respite provides respite even above the beverage itself.

My kids end up wanting to go to a Professor Science show. He asks questions, they know some of the answers. “What’s the name of the large telescope orbiting the earth?” he asks. My seven year old turns to me and asks, “Galileo?”. The logic is correct, the knowledge however isn't there yet. “Hubble,” I tell him. I’m sure he’ll remember Hubble though, I first remember learning about it in a similar fashion, when my dad told me about it. The new oral tradition.

(I also told him about cavitation, I’m not sure why, because it happens when I crack my knuckles, about mantis shrimp, and the apocryphal tail whips of apatosaurs also causing the phenomenon.)

But the scientist, an older gentleman assisted by his wife of forty four years, shows more props. My attention drifts. They get a gang of kids together, get them to break a lightbulb by screaming standing together in a semicircle. They make anodyne jokes, “your parents must be so proud.” The audience laughs.

We go back to the rides. There’s a small rollercoaster shaped like a dragon, riding in a lopsided figure 8. The kids seem to love it, some of them even try to take their hands up while the whiplash makes their necks wobble. Did they enjoy it? Yes, they say.

Next they go to one that does the same as the rotating multi-coloured bears but in multicoloured helicopters.

Why do they all look and feel the same? Ferris-wheel, boom-flipper (Zipper), spinning drum (Gravitron), tilt-platform, Himalaya oval. I imagine it has to do with the fact that fairs aren’t permanent. They evolved into the sizes that would allow maximum enjoyment but can be “folded up” and transported on a trailer to the next fair. It also can’t be too complex, the workers know the machines but they’re not experts. And they have to pass inspections, which means building things that the inspectors know how to pass.

Convergent evolution is at work here. The rotating swings are like the eyes of the natural world, showing up again and again because it’s the best fit functionally to satisfy the csontraints. Which is also why there aren’t that many suppliers. I learn that there are only three - Chance RIdes which makes the Zipper type coasters. Wisdom Rides making Gravitrons and Himalayas. And a few international ones - Zamperla and Fabbri from Italy, KMG from Netherlands - which make up most of the portable ride market.

And because there are only a few suppliers, the only way to stand out is to add more colours, more art. Like motorheads painting their cars with fire. The carnivals buy them from each other, re-skin them, add more LEDs, different colours, an inevitable trend towards complete garish oversaturation of the visible spectrum until the entire eyeline is covered in neon in several hues of red and yellow and orange. The fact that this is a small market, highly incestuous, where everyone wants to reuse everything shows up in the extreme mundanity of what we all see. They look the same because they literally are the same, just new coats of paint to trick the eyes.

The diversity comes entirely from the things around the rides and the food and the games. Or rather, those sources of diversity exist, whether or not they actually succeed. The music stands set up at regular intervals where local bands can play cover songs from the eighties and nineties that evokes nostalgia for the parents and apathy for the kids.

Professor Science was one of those, though in the United States success breeds replication so now there are Professors of Science across multiple fairs. He too sells a little backscratcher looking thing for five dollars that has an optical illusion at the back of it. Promising a short exploration of the optical system within kids but mostly destined to end up at the bottom of a toybox, as part of a short but fascinating life of a low priced mass manufactured mini toy.

The existence of a form of entertainment has transformed into a beautifully stylized supply chain, a few suppliers who build a few machines that pass inspection, and seemingly a caste of people who think of this as their whole way of life. Occasionally maybe a new game or ride breaks out, or a new cuisine, but by and large this seems an invariant source of entertainment across the ages. With the addition now being of the items on offer squeezed to their ultimate essence, of separating capital from its owners with maximum alacrity. Every trick in the book applied simultaneously.

The biggest attraction though was courtesy of the local pet shop. A large hall filled with animals. Perhaps it came at the end, but perhaps because of what it was. Kids yearn to be with animals. Bunnies, geckos, snakes, birds, turtles, some hissing cockroaches, and pygmy goats. You could touch them, play with them, and of course buy them!

To me it provided a brief respite from the sun. The hall had benches the adults can sit on, to rest from the extreme calf pain only brought about by slowly walking around and occasionally standing.

The detritus of people continues to float in all directions. There are more people, there are also more stationary forms under the shades of trees and awnings. It’s past noon, there’s food everywhere.

We walk out before we melt. The kids are tuckered out from the rides physically but not mentally, every new with is a promise that this one's amazing and even if it looks the same as the old ones they pull on the little heartstrings, holding kitschy toys that they'll forget in a day (they did!) and passing a larger group of people walking in.

The tumult is the attraction. Individually each aspect seems dull, even banal, the same thing one has seen a thousand times over in any lifetime, but together they create a space that invites you to create your own reality. “This is fun” they say, and in saying so repeatedly and liberally try to get you to agree with them. After all, what’s not fun about a rollercoaster at 11 am followed by a cheesy medium-warm hot dog and then a cold beer? Isn’t this the very goal of life?

The metal and plastic are hot but the ridership isn’t down. Kids and couples are still queuing up to go up the dragon and down the misshapen ships. They don’t seem to mind the heat.

2025-06-17 21:30:59

There are a few places in the world where I feel at home as soon as I land. There's not much to link them, beyond being big cities. Mumbai is one. Rome is another. London, of course. Sometimes Singapore. And then there's Tokyo!

I love landing in Tokyo, with its slightly shabby but sparkling clean airport, filled with ubiquitous vending machines and extremely polite immigration officers. The first time was 20 years ago, and then it felt like the future. But when I went recently, things had changed. Or maybe I had. There are parts of it in Minato City or Roppongi that look amazing, but mostly it still feels like the future of 20 years ago! Today, it looks retro.

That's not the only weird thing either, it's become a place of contradictions. Tokyo is like the Grand Budapest Hotel set in the Star Wars universe. Meticulous, understated, extraordinary service set in a decaying retro futuristic empire that's extremely well cared for. It’s a whimsical universe too. With cute robots, impossibly well designed systems that can whip your luggage across the country at minimal cost, but with fax machines and printed out emails.

There are cafes you could go to where there are robots serving food. Some of them, albeit, teleoperated by those who can’t leave home, which is even more cyberpunk. They had these well before the current robotics revolution by the way, made real by meticulous planning and specified routes that the robot could take. They weren’t built, it would seem, to show cool a robot one could make, but to make a robot that could do something. Like clean dishes. And with extraordinary attention to detail especially in thinking through how a user might want to interact with it. The best product thinkers are clearly from Japan.

Almost everyone repeats the good endlessly. Public transport that runs like clockwork. Clean streets. Safe. Plenty of food options all over the place across every price range imaginable. Food that's even cheaper than many places one would go in Delhi or Mumbai or, obviously, San Francisco! I think it's the fact that they have 6x the number of restaurants of London or NY and no zoning restrictions on where they can be, more supply and crazy competition means that even the ramen bars in subway stations have great food.

Also, amazing sweets of all varieties and across all tastes. And some of the best western desserts I've had. And bread! And cake! Great mini-marts in almost every corner, with good snacks and really good coffee.

Actually, let me stop for a second because this is actually really weird. Nowhere in the world do you find corner stores that serve good coffee. But Japan is built differently. I asked this about regular cafes including Starbucks, and o3 thinks that it’s because cafes in Japan have better staff who take care not to scald the milk or burn the beans, better logistics so you get fresher beans, and better water which isn’t so hard.

None of which are quite enough to explain it, I think, even though the results are wonderful. And it costs like $2. Again, 20 years ago, when Japan seemed closer to the future, things seemed more expensive. Now, coming from the US or London or Singapore, things seem positively cheap! Somehow, they have made the mundane necessities of life, of buying snacks at a supermarket or getting a cup of coffee, not feel like an experience in making you wish your life were better. In the US every interaction seems poised to fill you with envy for those who live a rung above you, not in Japan.

But the topsy turvy nature of the city is fractal shaped, visible everywhere at all scales. I went to go get a Suica card to travel around and remembered (was told rather, very politely), that a) I cannot buy it because it’s not a JR station (fair), b) I also can’t buy a Pasmo because the machines only take cash (wtf), and c) the ATM wouldn’t accept my debit card.

In fact I actually tried to tap my credit card and walked in, feeling smug, that the station attendants clearly didn't know this worked. But then I learnt at the other end, the destination station, that this was only a fleeting moment of success because I couldn’t get out. They had let me in somehow but apparently those only worked on some stations?

And what happened? The most Japanese thing happened. A station attendant very politely took me around to a different railway counter to buy a different ticket with my credit card, converted that to cash, took the cash and issued another ticket for the journey I’d made, and then gave me back the change. All with a smile and occasional attempts using his phone to translate from Japanese to English to give me directions on what to do or tell me what he was doing.

The experience of having a regular employee act as your personal concierge when you have a problem more than makes up for the fact that much of the city still feels like it’s 1999.

And it does feel like the last century, or a cyberpunk future borne of the last century, when you visit. The first time I visited a couple decades ago my smartphone was one of those Windows ones with a stylus. No real camera to speak of. We didn’t have iPhones, we being the whole world. And using data on the go with a rented flip-phone felt like the future. They had the fastest trains then, but now it's China. They had the most advanced electronics, now also China. The payment systems now seem antiquated, so alas does the amazing public transit.

Not to mention a strong Germanic love for physical cash still flows through the country. It's hard when the cafes, just like the train station machines or even parts of the hospital, won’t even accept credit cards and insist on cash. But despite that it functions perfectly. Brilliantly.

The combination of employee culture and general helpfulness more than make up for the technological lack. The thing that strikes you as you go through it is how most things seem quite old but really well cared for. Things are cheap but high quality. Can't buy train tickets with a credit card but the random airport cab has wi-fi. They have FamilyMart, which as my friend Jon Evans says is like the TARDIS of convenience stores. The metro stations are the state of the art of last century, old and a bit run down, but very well cared for.

Tokyo is like Coruscant. It’s futuristic, while retro. Crazy buildings that all are different and most a bit run down, but a few that are glittering homages to the best the world can produce. With vending machines that sell everything and overhead power lines that tangle in visible clumps. The culture is what people live around, not the technology itself, which works but feels old, and grimy.

Warren Buffett once said “depreciation is an expense, and it's the worst kind of an expense”. Japan is a society that is hell bent on fighting this. And they're winning, so far. It shows how much maintenance is important to keep civilization running. It demonstrates more than anywhere else I’ve been the importance of product thinking, to ensure that the customer has a good experience regardless of the ingredients at your disposal. Of how you can use customer service and culture to make up for technological deficiencies even as you apply the technological skill to build the future.

It's the success story of applying bureaucracy at scale while keeping efficiency high and on-the-job virtue alive. At a time when ennui basically seems a communicable disease in much of the West it’s an interesting thing to see in a society.

2025-06-09 21:05:28

I.

A lot of people seem to hate Silicon Valley with a passion. Recently I commented on a post that Tyler, a writer at Atlantic, made, suggesting that tech isn’t monomaniacally bad, and got hit with a barrage of comments about how I’m wrong. Technologists caused addiction to social media and smartphones intentionally, they said. They forced advertising on us, to the point of having to use ad blockers to even get on the internet1. They never asked us for permission before building driverless cars.

This is a common theme in arguments against tech. The anger seems to stem from the fact that we live in a world of tech, immersed in it, and you don’t have a say in the matter. It’s the water we swim in. And in that there are precious few choices but to use Google, to use Meta, to use AI that’s provided by OpenAI or Google. Want to book a flight? Research something for work? Talk to a friend? Call your parents? Check in on a sleeping baby? All needs tech in the Silicon Valley sense.

The immediate counter argument is that you could just not use it. You could “exit”, in the Hirschmann-ian sense. But that’s not a realistic possibility in 2025. Even the remote tribes who haven’t seen other human beings in centuries aren’t safe. So you’re left with “voice”. To complain, make your complaints heard.

And why wouldn’t you, the anger is that people had no choice or say in the matter. If you want to do any of those things, many of which didn’t exist in the pre-2000 era by the way, then you have no choice but to use one of the mega corporations that rose up in the last couple decades. And hate the technologists who build the thing.

Silicon Valley’s real sin here isn’t addiction or monopoly per se2; it is draining long-existing frictions from daily life so hard and fast that hidden costs pop up faster than society can patch them.

If you make something easy to use, people will use it more. If you will make something easier to use, other constraints will emerge, including the constraint that it becomes much harder for users to cognitively deal with it.

The other choice was to not use the tech at any point in its exorable rise. But that was a coordination problem and people suck at solving coordination problems3. These alternatives are nice to imagine, but lest we forget we collectively threw up on paid versions of browsers, social media, search engines, forums, blogs, literally anything that had a free + ads alternative. Because nobody loves a paywall. As Steward Brand said so well:

Information wants to be free

Much as it’s ridiculed, including by me atimes, Silicon Valley does try to build things that people want, and people want their lives to be easy. That’s why every annoyance that your parents had to deal with has been cut down so you can swipe to solve it on your mobile phones today. Yes, from booking tickets to researching topics to coding to talking to friends to checking in on your sleeping baby.

Silicon Valley finds every way imaginable to remove frictions from our lives. Every individual actor in tech works independently to find the next part of life that has any demonstrable friction and remove it, from finding love with a swipe to outbound sales enablement. Startups try to build on it, large companies try to capitalise on it, and VCs try to fund it. YC has it as their literal motto - ‘Make Something People Want’.

That’s how Google and Facebook built an advertising empire, because that could help them give what we wanted to us for free. We demanded it. And the network effects embedded and compounding investments meant they could grow bigger without anyone else able to compete with them, because who can compete with free.

The logic is as follows. Tech tries to make things easier to use. But the easier they are to use, we use them more. When we use them more, there's either a supply glut and often centralization, because to give things to us for free requires enormous scale. It outcompetes everything else. Which means they become as utilities. Which means there is no competition. Which means they will not compete on things that you might consider important. Which means when they make decisions, you feel like you do not have a say. Which means you feel alienated, and lash out.

II.

In any complex system when we remove bottlenecks the constraints moves somewhere else. This is true in operations, like when you want to set up a factory. It’s also true in software engineering, when you want to optimise a codebase. It's part of what makes system wide optimisation really difficult. It's Amdahls Law: “the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used”.

To optimise, you have to automate. And the increase in supply that reducing friction brings is the defining feature of automation; it always creates new externalities. Re AI assisted coding, Karpathy had a tweet that talked about the problem with LLMs not providing enough support for reviewing code, only for writing it. In other words, it takes away too much friction from one part of the job and adds it to another. You should read the full tweet, but the key part is here:

You could say that in coding LLMs have collapsed (1) (generation) to ~instant, but have done very little to address (2) (discrimination). A person still has to stare at the results and discriminate if they are good. This is my major criticism of LLM coding in that they casually spit out *way* too much code per query at arbitrary complexity, pretending there is no stage 2. Getting that much code is bad and scary. Instead, the LLM has to actively work with you to break down problems into little incremental steps, each more easily verifiable. It has to anticipate the computational work of (2) and reduce it as much as possible. It has to really care.

As it's easier to create content it becomes harder to discover it and even harder to discern it.

wrote a wonderful essay on this topic. She discusses how friction is effectively relocated from the digital into the physical world while we move into a simulated economy where friction, like gravity, doesn’t apply. This is akin to thinking about a ‘Conservation of Friction’, where its moved from the digital realm to the physical realm. Or at least our obsession with reducing friction reduces it in one place but doesn’t eliminate it elsewhere.

She wrote:

But friction isn’t the enemy!!!! It’s information. It tells us where things are straining and where care is needed and where attention should go.

And it's not all bad news. Because friction is also where new systems can emerge. Every broken interface, every overloaded professor, every delayed flight is pointing to something that could be rebuilt with actual intention.

But it’s not just that, the question is why the removal of friction caused such widespread dismay this time around.

III.

Now, the story of most of the technological and economic revolution is also the story of reducing friction. Consumers demand it. The history of humanity is one of a massive increase in capability, innovation and growth!

Reduction in friction means we increase the volume of what’s being supplied. The increase in volume can even result in a winner-take-all market when there are network effects, like there are with things relating to human preferences. Which changes the nature of the market, since if supply is much easier then demand shifts.

Now, led by AI, we’re at a historic height of friction reduction4. Look at education. Clay Shirky writes about the incredible change in education brought about by ChatGPT. Students write papers instantaneously, “I saved 10 hours”, and learn far less as they don’t need to go through the tedium of research, discovery or knowledge production.

While students could use it also to speed up the process of learning new things, and many do, they’re caught up in a red queen race. “You’re asking me to go from point A to point B, why wouldn’t I use a car to get there?” as a student said.

"I've become lazier. AI makes reading easier, but slowly causes my brain to lose the ability to think critically."

This is the problem in a nutshell. We can’t stop ourselves from using these tools because they help us a lot. We get hurt by using the tools because they steal something of value in being used. You have to figure out how much to use the tool along with using the tool, and build the price signal internally. Eating a thousand cookies would have external manifestations you can use to guide your behaviour, what about a thousand instagram reels?

And you can’t opt out of it.

Our obsession with reducing friction reduces it in one place but doesn’t eliminate it elsewhere. We agree to these externalities through collective inaction. Everyone adopting is the Nash equilibrium; individually rational, collectively perhaps costly.

IV.

So what can one do, but complain about the existence of these technologies, and bemoan their very existence, dreaming of a simpler time?

Every time someone complains about how they’re addicted to Twitter and wish they could lock their phones away for a bit, they’re hoping for a world where the extreme ease and afforandces that modernity has brought us is pulled back just a little bit. To go to a simpler time.

But we can’t. We’re in this collectively. There is no market of one. The choices we’re not given is the choice to not be able to make certain choices. Where's the setting on Instagram for “turn this off for 2 hours unless I finish my work first"? In the relentless competition that reducing friction brings there is no place for a tool that adds intentional friction.

These stories are everywhere. It used to be that the way you applied for a job was to know a guy or maybe even to get a guy to send a letter to another guy on your behalf. We even used to get PhDs that way.

And then it got easier to apply for jobs. You had online portals. You have middlemen. You had resumes that would get sent in, seen by screeners, seen by HR, vetted against a set of criteria, and then you got an interview. Many rounds of interviews. Much more efficient.

Except for when everyone found out how to do it and started sending in resumes en masse and causing incredible chaos in the system. It’s Jevon’s paradox with a vengeance! The internet supercharged this. And as a result we’re in a situation where people routinely apply for 100s of jobs and don’t get a callback, and the only way to get a job is to know someone.

The increase in supply brings with it new costs - more cognitive load, and more search costs.

That’s why we started telling jobseekers you need a personalised resume and personalised cover letter, trying to find a way to get the candidates to put effort in. Same as for college applications.

Until AI entered the picture.

A “barrage” of AI-powered applications had led to more than double the number of candidates per job while the “barrier to entry is lower”, said Khyati Sundaram, chief executive of Applied, a recruitment platform.

“We’re definitely seeing higher volume and lower quality, which means it is harder to sift through,” she added. “A candidate can copy and paste any application question into ChatGPT, and then can copy and paste that back into that application form.”

And why is this a particular problem? Because search costs are too high!

Cinder is part of a growing list of US-based tech companies that encounter engineering applicants who are actually suspected North Korean nationals. These North Koreans almost certainly work on behalf of the North Korean government to funnel money back to their government while working remotely via third countries like China. Since at least early 2023, many have applied to US-based remote-first tech companies like Cinder.

The part that it’s North Koreans social engineering into the jobs is not the most pertinent part here, though it’s hilarious, it’s that we created a frictionless experience and as a result are dealing with a supply glut, which we have no easy way to solve. We now have to find a way to automate dealing with the supply glut, which will create new loopholes, which we then will have to work to automate, which will …

Every process you find that works is a secret that you get to exploit. It works until it is no longer a secret. When it’s no longer a secret and everyone is happy to do the same thing to succeed you can no longer rely on that strategy. Helping solve frictions that we had before creates new ones that we have to learn to contend with.

There are so many examples of this but one more that I like. Steve Jobs once talked about the importance of good storytelling, considering the limits of animated movies.

In animation, it is so expensive that you can't afford to animate more than a few % more than it's going to end up on screen. You could never afford to animate 10x more. Walt Disney solved the problem decades ago and the way he solved it was to edit films before making them: you get your story team together and you do storyboards.

Animation, even CGI, used to be much harder to do. Which meant that directors and storytellers had to work very very hard to figure out where to use it. They had to be careful. The story came first.

This is no longer a constraint. CGI is easier and cheaper, so more widely used. Now a typical Hollywood studio spends more time in post-production than in pre-production and shooting combined. Cheap flexibility led to a supply glut. The constraints changed.

V.

Which brings us back to the tech and the obsession with reducing friction. Part of that is market forces, but that is because us as consumers and users hate friction. We say we like the alternative, dream of Thoreau, though we’d rather spend time talking about dreaming of being Thoreau on Instagram rather than actually waldenponding.

But we can’t exit the digital world, not easily, so we feel stuck. And unlike with conveyor belts or software engineering, when we reduce the friction of our own demands, the new bottlenecks or introduces aren't easily visible nor easily fixable. How do you deal with the fact that we get hundreds of messages now from multiple apps and are deluged in incoming information that swamps our ability to process it?

Removing friction changes human behaviour. And it’s hard to deal with the consequences of that change in behaviour overnight. But we do learn, we learn to counter those ones, and build the next generation. Whether that’s soot from the industrial revolution polluting our roads or advertisements polluting our information.

The reason we feel the urge to lock our phones away is because we’re not used to having a constantly-on always-aware portal into the entire world just be readily available. The reason why vibe-coding leads to review-fatigue, getting 30 or 300 PRs a day instead of 3 and being buried under, is because we’re not used to doing that yet. The reason why using AI to write leads to high profile hallucinations is because we still haven’t learnt how to use it better.

Friction is the way we know where to focus next. That’s why we dislike it as users even if some of it might be good for us. That’s why Silicon Valley tries to eliminate it. Which is what spurs the criticisms like what Tyler made. Its existence is an indication of new ecological niches being created within our demandscape. It’s not conserved, but neither is it eradicated. It moves. It hides. It finds new places to make itself known. And that’s how you learn where to focus next.

And until people learn to catch up with a frictionless existence, we try to add friction back into our lives to make it possible for us to live. To deal with it seeping away from easy communication to harder cognitive load. We already do, for some things. Touching grass. No-meeting-Wednesdays. Putting ‘Away’ on Slack. Saying you’ll only check your email at 4 PM.

Every one of these can feel like an individual imposition because of a world that tech built, a small piece of personal rebellion, even if it was built only to appeal to everyone. A small escape from engineered dependence to our tools, which are no longer our own.

Since every advance is couched in terms of “you are now liberated from [X]”, and we can so easily think of a way that [X] was important to being human, it is easy to fight back5. This is hardly new. Socrates wouldn’t write anything down for fear of hurting his memory. But the thing is, Socrates was so clearly wrong in this! Writing things down kickstarted civilisation, so memorably by his student Plato. And his student Aristotle. It just needed to dissimulate through society so people figured out how to deal with the drawbacks.

Human scale isn’t machine scale, nor is it economy scale. Our neurons only fire so fast, our societies only adapt so fast, and until they do we might be prisoners of our own drive to make life better.

Some of these will be built by new startups and new technologies, some by new laws or guidelines or processes, and some, like Plank said, by the older folks just aging out. And until then we will see a lot more Voice from people, dissatisfied with the world they live in, annoyed at the choices they didn’t individually influence, because they are unable to exit.