I'm right now doing conceptual theoretical work about how the human fascia system works. While I do rely on some original conceptual insights that I have e come up with on my own, Gemini 2.5 Pro massively speeds up my conceptual work. Being able to formulate a principle or analogy and then having Gemini apply it, is very useful.
There are a bunch of scientific fields where we currently have a lot of experimental data but lack coherent theory to interlink the experimental findings. Based on my own experience, current LLMs seem already to be powerful enough to help bridge that theory gap. Being able to ask "Hey, does field XYZ have any useful insights to the problem I'm tackling?" is also very helpful for making progress in theory.
The LLMs also solve a key problems that autodidact have when it comes with existing scientific fields. If you have a new idea, they are good at telling you the main criticisms that would come from an orthodox researcher in a field. We might see a rise in interdisciplinary work that didn't happen in the past because of academia's hyperspecialization.
People frequently say that progress in science has stalled because there's little low-hanging fruit. When it comes to doing certain interdisciplinary work, it's now a lot easier to pick the fruit. If you are right now starting a scientific career, think about what kind of interdisciplinary work you might do, where it's now easier to make progress because of the existence of LLMs.
If you have a research question, one approach you can do is to ask a reasoning model to create a debate between two highly skilled researchers with different approaches to debate your research question. You might learn valuable insights about your research question this way. Besides taking existing researchers in the field, asking the LLM to simulate philosophers and tell the LLM that the philosophers understand all the facts about a field, might give you valuable insights of how insights that philosophers found through a lot of hard work translate into individual fields.
It's not clear what the best approaches are to get the LLM to help you with interdisciplinary work, but there's a lot of fruit out there to be picked right now.
Your voice has been heard. OpenAI has ‘heard from the Attorney Generals’ of Delaware and California, and as a result the OpenAI nonprofit will retain control of OpenAI under their new plan, and both companies will retain the original mission.
Technically they are not admitting that their original plan was illegal and one of the biggest thefts in human history, but that is how you should in practice interpret the line ‘we made the decision for the nonprofit to retain control of OpenAI after hearing from civic leaders and engaging in constructive dialogue with the offices of the Attorney General of Delaware and the Attorney General of California.’
Another possibility is that the nonprofit board finally woke up and looked at what was being proposed and how people were reacting, and realized what was going on.
The letter ‘not for private gain’ that was recently sent to those Attorney Generals plausibly was a major causal factor in any or all of those conversations.
The question is, what exactly is the new plan? The fight is far from over.
As previously intended, OpenAI will transition their for-profit arm, currently an LLC, into a PBC. They will also be getting rid of the capped profit structure.
However they will be retaining the nonprofit’s control over the new PBC, and the nonprofit will (supposedly) get fair compensation for its previous financial interests in the form of a major (but suspiciously unspecified, other than ‘a large shareholder’) stake in the new PBC.
Bret Taylor (Chairman of the Board, OpenAI): The OpenAI Board has an updated plan for evolving OpenAI’s structure.
OpenAI was founded as a nonprofit, and is today overseen and controlled by that nonprofit. Going forward, it will continue to be overseen and controlled by that nonprofit.
Our for-profit LLC, which has been under the nonprofit since 2019, will transition to a Public Benefit Corporation (PBC)–a purpose-driven company structure that has to consider the interests of both shareholders and the mission.
The nonprofit will control and also be a large shareholder of the PBC, giving the nonprofit better resources to support many benefits.
Our mission remains the same, and the PBC will have the same mission.
We made the decision for the nonprofit to retain control of OpenAI after hearing from civic leaders and engaging in constructive dialogue with the offices of the Attorney General of Delaware and the Attorney General of California.
We thank both offices and we look forward to continuing these important conversations to make sure OpenAI can continue to effectively pursue its mission of ensuring AGI benefits all of humanity. Sam wrote the letter below to our employees and stakeholders about why we are so excited for this new direction.
The rest of the post is a letter from Sam Altman, and sounds like it, you are encouraged to read the whole thing.
Sam Altman (CEO OpenAI): The for-profit LLC under the nonprofit will transition to a Public Benefit Corporation (PBC) with the same mission. PBCs have become the standard for-profit structure for other AGI labs like Anthropic and X.ai, as well as many purpose driven companies like Patagonia. We think it makes sense for us, too.
Instead of our current complex capped-profit structure—which made sense when it looked like there might be one dominant AGI effort but doesn’t in a world of many great AGI companies—we are moving to a normal capital structure where everyone has stock. This is not a sale, but a change of structure to something simpler.
The nonprofit will continue to control the PBC, and will become a big shareholder in the PBC, in an amount supported by independent financial advisors, giving the nonprofit resources to support programs so AI can benefit many different communities, consistent with the mission.
Joshua Achiam (OpenAI, Head of Mission Alignment): OpenAI is, and always will be, a mission-first organization. Today’s update is an affirmation of our continuing commitment to ensure that AGI benefits all of humanity.
Your Offer is (In Principle) Acceptable
I find the structure of this solution not ideal but ultimately acceptable.
The current OpenAI structure is bizarre and complex. It does important good things some of which this new arrangement will break. But the current structure also made OpenAI far less investable, which means giving away more of the company to profit maximizers, and causes a lot of real problems.
Thus, I see the structural changes, in particular the move to a normal profit distribution, as a potentially a fair compromise to enable better access to capital – provided it is implemented fairly, and isn’t a backdoor to further shifts.
The devil is in the details. How is all this going to work?
What form will the nonprofit’s control take? Is it only that they will be a large shareholder? Will they have a special class of supervoting shares? Something else?
This deal is only acceptable if and only he nonprofit:
Has truly robust control going forward, that is ironclad and that allows it to guide AI development in practice not only in theory. Is this going to only be via voting shares? That would be a massive downgrade from the current power of the board, which already wasn’t so great. In practice, the ability to win a shareholder vote will mean little during potentially crucial fights like a decision whether to release a potentially dangerous model.
What this definitely still does is give cover to management to do the right thing, if they actively want to do that, I’ll discuss more later.
Gets a fair share of the profits, that matches the value of its previous profit interests. I am very worried they will still get massively stolen from on this. As a reminder, right now most of the net present value of OpenAI’s future profits belongs to the nonprofit.
Uses those profits to advance its original mission rather than turning into a de facto marketing arm or doing generic philanthropy that doesn’t matter, or both.
There are still clear signs that OpenAI is largely planning to have the nonprofit buy AI services on behalf of other charities, or otherwise do things that are irrelevant to the mission. That would make it an ‘ordinary foundation’ combined with a marketing arm, effectively making its funds useless, although it could still act meaningfully via its control mechanisms.
Remember that in these situations, the ratchet only goes one way. The commercial interests will constantly try to wrestle greater control and ownership of the profits away from us. They will constantly cite necessity and expedience to justify this. You’re playing defense, forever. Every compromise improves their position, and this one definitely will compared to doing nothing.
Quintin Pope: Common mistake. They forgot to paint “Do Not Open” on the box.
There’s also the issue of the extent to which Altman controls the nonprofit board.
The reason the nonprofit needs control is to impact key decisions in real time. It needs control of a form that lets it do that. Because that kind of lever is not ‘standard,’ there will constantly be pressure to get rid of that ability, with threats of mild social awkwardness if these pressures are resisted.
He had an excellent thread explaining the attempted conversion, and he has another good explainer on what this new announcement means, as well as an emergency 80,000 Hours podcast on the topic that should come out tomorrow.
The central things to know about the new plan are indeed:
The transition to a PBC and removal of the profit cap will still shift priorities, legal obligations and incentives towards profit maximization.
The nonprofit’s ‘control’ is at best weakened, and potentially fake.
The nonprofit’s mission might effectively be fake.
The nonprofit’s current financial interests could largely still be stolen.
It’s an improvement, but it might not effectively be all that much of one?
We need to stay vigilant. The fight is far from over.
Rob Wiblin: So OpenAI just said it’s no longer going for-profit and the non-profit will ‘retain control’. But don’t declare victory yet. OpenAI may actually be continuing with almost the same plan & hoping they can trick us into thinking they’ve stopped!
Or perhaps not. I’ll explain:
The core issue is control of OpenAI’s behaviour, decisions, and any AGI it produces.
Will the entity that builds AGI still have a legally enforceable obligation to make sure AGI benefits all humanity?
Will the non-profit still be able to step in if OpenAI is doing something appalling and contrary to that mission?
Will the non-profit still own an AGI if OpenAI develops it? It’s kinda important!
The new announcement doesn’t answer these questions and despite containing a lot of nice words the answers may still be: no.
(Though we can’t know and they might not even know themselves yet.)
The reason to worry is they’re still planning to convert the existing for-profit into a Public Benefit Corporation (PBC). That means the profit caps we were promised would be gone. But worse… the nonprofit could still lose true control. Right now, the nonprofit owns and directly controls the for-profit’s day-to-day operations. If the nonprofit’s “control” over the PBC is just extra voting shares, that would be a massive downgrade as I’ll explain.
(The reason to think that’s the plan is that today’s announcement sounded very similar to a proposal they floated in Feb in which the nonprofit gets special voting shares in a new PBC.)
Special voting shares in a new PBC are simply very different and much weaker than the control they currently have! First, in practical terms, voting power doesn’t directly translate to the power to manage OpenAI’s day-to-day operations – which the non-profit currently has.
If it doesn’t fight to retain that real power, the non-profit could lose the ability to directly manage the development and deployment of OpenAI’s technology. That includes the ability to decide whether to deploy a model (!) or license it to another company.
Second, PBCs have a legal obligation to balance public interest against shareholder profits. If the nonprofit is just a big shareholder with super-voting shares other investors in the PBC could sue claiming OpenAI isn’t doing enough to pursue their interests (more profits)! Crazy sounding, but true.
And who do you think will be more vociferous in pursuing such a case through the courts… numerous for-profit investors with hundreds of billions on the line, or a non-profit operated by 9 very busy volunteers? Hmmm.
In fact in 2019, OpenAI President Greg Brockman said one of the reasons they chose their current structure and not a PBC was exactly because it allowed them to custom-write binding rules including full control to the nonprofit! So they know this issue — and now want to be a PBC. See here.
If this is the plan it could mean OpenAI transitioning from:
• A structure where they must prioritise the nonprofit mission over shareholders
To:
• A new structure where they don’t have to — and may not even be legally permitted to do so.
(Note how it seems like the non-profit is giving up a lot here. What is it getting in return here exactly that makes giving up both the profit caps and true control of the business and AGI the best way to pursue its mission? It seems like nothing to me.)
So, strange as it sounds, this could turn out to be an even more clever way for Sam and profit-motivated investors to get what they wanted. Profit caps would be gone and profit-motivated investors would have much more influence.
And all the while Sam and OpenAI would be able to frame it as if nothing is changing and the non-profit has retained the same control today they had yesterday!
(As an aside it looks like the SoftBank funding round that was reported as requiring a loss of nonprofit control would still go through. Their press release indicates that actually all they were insisting on was that the profit caps are removed and they’re granted shares in a new PBC.
So it sounds like investors think this new plan would transfer them enough additional profits, and sufficiently neuter the non-profit, for them to feel satisfied.).
Now, to be clear, the above might be wrongheaded.
I’m looking at the announcement cynically, assuming that some staff at OpenAI, and some investors, want to wriggle out of non-profit control however they can — because I think we have ample evidence that that’s the case!
The phrase “nonprofit control” is actually very vague, and those folks might be trying to ram a truck through that hole.
At the same time maybe / hopefully there are people involved in this process who are sincere and trying to push things in the right direction.
On that we’ll just have to wait and see and judge on the results.
Bottom line: The announcement might turn out to be a step in the right direction, but it might also just be a new approach to achieve the same bad outcome less visibly.
So do not relax.
And if it turns out they’re trying to fool you, don’t be fooled.
Gretchen Krueger: The nonprofit will retain control of OpenAI. We still need stronger oversight and broader input on whether and how AI is pursued at OpenAI and all the AI companies, but this is an important bar to see upheld, and I’m proud to have helped push for it!
Now it is time to make sure that control is real—and to guard against any changes that make it harder than it already is to strengthen public accountability. The devil is in the details we don’t know yet, so the work continues.
Tragedy in the Bay
Roon says the quiet part out loud. We used to think it was possible to do the right thing and care about whether AI killed everyone. Now, those with power say, we can’t even imagine how we could have been so naive, let’s walk that back as quickly as we can so we can finally do some maximizing of the profits.
Roon: the idea of openai having a charter is interesting to me. A relic from a bygone era, belief that governance innovation for important institutions is even possible. Interested parties are tasked with performing exegesis of the founding documents.
Seems clear that the “capped profit” mechanism is from a time in which people assumed agi development would be more singular than it actually is. There are many points on the intelligence curve and many players. We should be discussing when Nvidia will require profit caps.
I do not think that the capped profit requires strong assumptions about a singleton to make sense. It only requires that there be an oligopoly where the players are individually meaningful. If you have close to perfect competition and the players have no market power and their products are fully fungible, then yes, of course being a capped profit makes no sense. Although it also does no real harm, your profits were already rather capped in that scenario.
More than that, we have largely lost our ability to actually ask what problems humanity will face, and then ask what would actually solve those problems, and then try to do that thing. We are no longer trying to backward chain from a win. Which means we are no longer playing to win.
At best, we are creating institutions that might allow the people involved to choose to do the right thing, when the time comes, if they make that decision.
The Spirit of the Rules
For several reasons, recent developments do still give me hope, even if we get a not-so-great version of the implementation details here.
The first is that this shows that the right forms of public pressure can still work, at least sometimes, for some combination of getting public officials to enforce the law and causing a company like OpenAI to compromise. The fight is far from over, but we have won a victory that was at best highly uncertain.
The second is that this will give the nonprofit at least a much better position going forward, and the ‘you have to change things or we can’t raise money’ argument is at least greatly weakened. Even though the nine members are very friendly to Altman, they are also sufficiently professional class people, Responsible Authority Figures of a type, that one would expect the board to have real limits, and we can push for them to be kept more in-the-loop and be given more voice. De facto I do not think that the nonprofit was going to get much if any additional financial compensation in exchange for giving up its stake.
The third is that, while OpenAI likely still has the ability to ‘weasel out’ of most of its effective constraints and obligations here, this preserves its ability to decide not to. As in, OpenAI and Altman could choose to do the right thing, even if they haven’t had the practice, with the confidence that the board would back them up, and that this structure would protect them from investors and lawsuits.
This is very different from saying that the board will act as a meaningful check on Altman, if Altman decides to act recklessly or greedily.
It is easy to forget that in the world of VCs and corporate America, in many ways it is not only that you have no obligation to do the right thing. It is that you have an obligation, and will face tremendous pressure, to do the wrong thing, in many cases merely because it is wrong, and certainly to do so if the wrong thing maximizes shareholder value in the short term.
Thus, the ability to fight back against that is itself powerful. Altman, and others in OpenAI leadership, are keenly aware of the dangers they are leading us into, even if we do not see eye to eye on what it will take to navigate them or how deadly are the threats we face. Altman knows, even if he claims in public to actively not know. Many members of technical stuff know. I still believe most of those who know do not wish for the dying of the light, and want humanity and value to endure in this universe, that they are normative and value good over bad and life over death and so on. So when the time comes, we want them to feel as much permission, and have as much power, to stand up for that as we can preserve for them.
It is the same as the Preparedness Framework, except that in this case we have only ‘concepts of a plan’ rather than an actually detailed plan. If everyone involved with power abides by the spirit of the Preparedness Framework, it is a deeply flawed but valuable document. If those involved with power discard the spirit of the framework, it isn’t worth the tokens that compose it. The same will go for a broad range of governance mechanisms.
Have Altman and OpenAI been endlessly disappointing? Well, yes. Are many of their competitors doing vastly worse? Also yes. Is OpenAI getting passing grades so far, given that reality does not grade on a curve? Oh, hell no. And it can absolutely be, and at some point will be, too late to try and do the right thing.
The good news is, I believe that today is not that today. And tomorrow looks good, too.
You may have heard of the "infinite monkey theorem". It states: if you take a monkey, give it a typewriter and an infinite amount of time, sooner or later it will type out the complete text of Hamlet.[1]
It's said that this theorem emerged during 19th-century debates about evolution. Proponents of evolution supposedly argued that human descent from apes, like other low-probability events, is possible – you just need to allow enough time.
I began my talk about low-probability events with this example at an informal seminar for graduate students at UCLA. I'll share this talk with you as well. My story will have several parts; the first part is dedicated to Buridan's principle.
A safety-critical system is one whose failure could result in death or serious harm to human health. Examples include nuclear power plants, airplanes, railway crossings, and pacemakers. In the aviation industry, the guidelines want death to be an extremely improbable failure condition and those are defined as having a probability on the order of 10−9 or less. So, a safety-critical system should be designed to cause no more than one death per billion hours of operation. Engineers work hard at this, calculating all these nanomorts per hour. When reading about this, a natural question arises – why can't the risk simply be reduced to zero? Is there really such a significant gap between 0 and 0.000000001?
Buridan's ass
It turns out this gap exists, and it's substantial. The problem is that the laws of classical physics are continuous. As Feynman said in the introduction to his famous lecture series: "All things are made of atoms – little particles that are in perpetual motion, attracting each other when they are a little distance apart, but repelling upon being squeezed into one another". The interactions between these atoms can be modeled with various laws. Atoms experience forces and move according to Newton's laws. They interact with the electromagnetic field, which also interacts with itself according to Maxwell's equations. All these laws share one property – they are continuous. If we continuously move some parameter, such as the initial position of an atom, its position after 10 seconds will depend on this parameter continuously.
From this simple observation follows the paradox of Buridan's ass. The French philosopher Jean Buridan reportedly claimed that if a hungry donkey is placed at an equal distance from two haystacks, it won't be able to choose which stack to approach and will eventually die of hunger[2].
In the equal distances formulation, the paradox is easily resolved. Buridan's ass, naturally being French, would instinctively follow right-hand traffic rules and go to the right haystack. But there's a more serious problem!
Theorem 1: If for some initial position the donkey ends up at the left haystack after a minute, and for some other initial position it ends up at the right haystack, then on the interval between these positions there exists an initial position of the donkey for which after a minute it won't end up at either haystack.
Proof: the donkey is a physical mechanism subject to the laws of classical physics. Therefore, the position of the donkey after a minute depends continuously on its initial position.[3] If for any initial position the final position were at either the left or right haystack, then there would exist a nonconstant continuous function from the interval [0,1] to a space of two points {0,1}, which is impossible according to the intermediate value theorem. Q.E.D.
I find this statement astonishing! An application of the intermediate value theorem to philosophy! I'm surprised that few people have heard of it, even among my mathematician friends with the broadest horizons. We can extend this statement and show that there is an initial position for which the donkey will starve to death. This position doesn't have to be in the middle, as Buridan suggested. But Buridan correctly grasped the essence. If the donkey has an instinct to choose the right pile more eagerly, then the equilibrium point will be to the left of center, where this instinct is precisely balanced by the natural lazy tendency to go to the nearest pile.
Addressing objections
Around this point in text you probably have some objections. You are definitely not alone! A lot of people voice objections, many of them later correct themselves. Write yours in the comments to be disproved!
Apparently, Theorem 1 first appeared in Leslie Lamport's (well-written) paper "Buridan's Principle". He notes that people, upon hearing about Buridan's principle, tend to propose mechanisms to circumvent it[4]. Usually, they suggest that the donkey, in a situation of indecision, should take some action. Go to the right haystack. Don't drive onto the crossing. Calculate whether there is more or less than a minute until the train arrives. But this simply pushes the decision one step back. The donkey must determine whether it is "in indecision". And this decision is also impossible to make in finite time.
All proposed mechanisms, of course, don't work if they don't challenge the initial premises of the theorem. And the premises are quite weak – we only assumed that: physics is classical, that tme and space are continuous, and that there are initial positions for which the final positions end up in different components. And if these premises are true, then no mechanism whatsoever, with doors, gears, coins, magnets, or electronics, can resolve the paradox.
This theorem shows that complete safety is impossible; there can only be a very small probability of danger. Even the probability of starving to death surrounded by food is not zero. What can be said about other unsafe places – traffic lights or railway crossings? Depending on the initial speed and position of the car, within a limited time before a train approaches, the driver must decide whether to cross or to wait until the train passes. As you can understand, here the driver must make a choice between two discrete options in a limited time. And if the initial position of the car and the gears in the driver's head are unfortunate, they will be hit by the train. The merciless intermediate value theorem states that for any configuration of gears – that is, for any principle the driver might use – there's a chance they will be hit by the train.
For example, a barrier at a railway crossing doesn't guarantee one hundred percent safety but merely postpones the problem one step. If a driver is in an indecision about whether to drive under the barrier, they may fall into Buridan's trap and be crushed by the barrier when it lowers. Or they will remain between the barriers. And then an accident with the train will occur.
On February 3, 2015, at 6:26 PM, a car driven by a 49-year-old woman was traveling northwest on Commerce Street in Valhalla, New York, toward a railroad crossing. The driver entered the crossing area and stopped. Then she moved further beyond the boundary and stopped near the railway tracks. The crossing warning system activated, and the barrier came down, hitting the rear of the car. The woman got out of the car, inspected the barrier, then returned to the car and drove onto the tracks...
Jacobs speculates:
There are many ways to explain this driver’s behavior. Perhaps she felt she was already committed to crossing the tracks; perhaps she was just running on autopilot and did not comprehend the danger facing her. Buridan’s principle was not the only, or perhaps even the determining factor in this accident. But the driver’s behavior does look a bit like what would happen if she were having trouble judging whether she had enough time to cross.
A more familiar and perhaps more plausible example of the Buridan phenomenon is the decision of whether to stop or speed through a traffic light that has just turned yellow. It’s easy to waffle between the two options, only coming to a conclusion by slamming on the brakes or screeching through the intersection, risking running a red light. Again, there are other interpretations of this behavior, but the difficulty of figuring out in time whether you have enough time is a real one.
Literature on Buridan's paradox
The idea of Buridan's paradox is so incredible that you can only believe it if someone very authoritative tells you about it. For example, if you read about it in a children's book. For instance, in the wonderful book "What is Mathematics?" by Courant and Robbins, the following problem by H. Whitney is described.
Suppose that over some finite time interval a train travels along a straight line segment from station A to station B. It is not assumed that the motion occurs at a constant speed or with constant acceleration. On the contrary, the train can move in any way: with accelerations, decelerations; even instantaneous stops or partial reverse movements before eventually arriving at station B.
But one way or another, the train's movement throughout the entire time interval is assumed to be known in advance; in other words, a function s=f(t) is given, where s is the distance of the train from station A, and t is the time counted from the moment of the train's departure. A solid heavy rod is attached by a hinge to the floor of one of the cars, which can move without friction around an axis parallel to the axes of the cars, forward and backward—from floor to floor. Let the train's movement be such that, having touched the floor, the rod will subsequently remain lying on it, that is, it will not "bounce" up again.[5]
The question is as follows: is it possible at the moment of the train's departure to place the rod in such an initial position, i.e., to give it such an angle of inclination, that throughout the entire journey it does not touch the floor, being subject only to the influence of the train's movement and its own gravity?
As you might guess, this is a special case of Buridan's principle, and so the required initial position exists. The book provides a solution that, naturally, uses the intermediate value theorem. But then the reasoning becomes more interesting from a mathematical point of view! In an exercise, the authors ask to generalize the statement to the case when the train moves along an arbitrary curve in a plane, and the rod can fall in any direction, i.e., the rod has two degrees of freedom.
If the rod can rotate on the hinge not like an elbow but like a shoulder, there is still an initial position in which the lever will not fall during the entire trip. However, we still need the condition that, once fallen, the rod will not rise again.
The key point of the proof is a lemma from the proof of Brouwer's fixed point theorem. It states that there is no mapping from a disc to its boundary that maps all boundary points to themselves. Specifically, we need a strengthening that allows the restriction of the mapping to the boundary to be homotopic to the identity mapping. This can be proven directly by topological reasoning, or one can use the machinery of algebraic topology and calculate homologies, arriving at a contradiction. Surely other theorems from algebraic topology can be turned into theorems about the impossibility of guaranteeing decision-making in finite time in some complex topological configuration space.
In Lamport's paper, the reasoning about Brouwer's theorem is presented in connection with another configuration. In the chapter "Flying asses[6]", Lamport explains why an airplane cannot guarantee avoiding even a stationary balloon in finite time.
Lamport, as far as I understand, was most interested in methods of combating Buridan's paradox in electronics. He worked on solving the arbiter problem – devising a way to determine which of two signals arrived first. In the "Computer asses" chapter, he discusses metastability in electronic computer components. As we know, zeros and ones run inside a computer. At the level of electrical circuits that make up the computer, this means that the voltage at the output of all components in the computer is either 0 volts or 5. There are various electronic components that take as input the outputs of other components and, according to some rules, produce a new voltage value, which will also be 0 or 5 volts. For example, this is done by logical gates AND, OR, NOT, and these gates are sufficient to build a computer. You can literally do this in games like Nandgame and Turing Complete.
So, if you apply the right potential difference between 0 and 5 volts to the inputs of an AND or OR gate, according to the intermediate value theorem, you can achieve any output between 0 and 5 volts. And this incorrect state will propagate further through components into the computer's brain. This state is called metastability. In a properly constructed computer, metastability usually doesn't penetrate too far and fades out after a few cycles. But Buridan's principle says that by properly selecting the output of an external computer device, you can drive the computer crazy. Moreover, external devices will sometimes produce such outputs even without malicious intent, because pressing a key cannot instantly transition the output from 0 to 5 – in a continuous world, the voltage at the output rises gradually and passes through a metastable phase.
Why don't we encounter Buridan's principle every day? Why isn't such a powerful principle known to everyone from childhood and not even taught in university mathematics programs? The answer is that Buridan's paradox usually occurs with exponentially small probability!
Exponentially small probability
Consider the simplest model of two people trying to pass each other in a corridor. Two people are walking toward each other and realize they will collide. They can pass either by "left sides" or by "right sides". Let's temporarily move away from the continuity inherent in Buridan's paradox and say that people make decisions in steps. At each step, a person has a 50% probability of moving to the right, and a 50% probability of moving to the left. If people make asymmetric movements, they have thus agreed on which side to pass. If they make symmetric movements, they take another random step, until they get lucky.
Let the step time be Δt, then over time t, tΔt steps will occur, and they will not be able to make a decision in this time with probability 2−tΔt. If we take Δt as the average human reaction time, 0.2 seconds, then in one second, people will not be able to pass each other in 132 cases, in 2 seconds with a probability of 11024, and they will need 10 seconds in only 10−15 cases. It's not surprising that none of us has met a person in life whom it's impossible to pass, like your own reflection in a mirror. And even more so, no one has died of hunger because of this, like the poor imaginary donkey. The "reaction time" of a tossed coin or a balanced rod is on the order of fractions of a second. Due to such quick reactions, coins, unless they fall into a crack in the asphalt, eventually stop rolling on their edge and choose a side to fall on.
The exponential nature of probability might explain why most accidents happen when one of the drivers is drunk. If two drivers at an intersection are trying to solve the arbiter problem in limited time t, and due to intoxication the reaction time Δt has doubled, then the exponent in the accident probability is halved. If in a normal situation the accident probability was 10−12 or one in a trillion, then with doubled reaction time and unchanged movement speed, the probability becomes 10−6 or one in a million – much riskier!
Computer reaction times are much shorter than human ones. Therefore, metastability inside computers usually lasts only a few nanoseconds, and external inputs to the computer undergo a brief stabilization "quarantine" before being used in calculations.
Final remarks
In quantum mechanics, there are laws that violate continuity. Nevertheless, low-probability events manifest themselves even more vividly there, and quantum mechanics cannot resolve Buridan's paradox as shown in the Lamport's paper.
Finally, I'll tell you about the fate of paper. Due to its paradoxical and implausible nature, Science reviewers were divided on whether it was a paper of great philosophical importance or an elaborate prank. Nature reviewers dismissed this paper, proposing mechanisms to resolve the paradox. Charles Molnar, who noticed this paradox independently, said that his paper was rejected by a reviewer on the grounds that, if this problem existed, it would be important enough for all experts in electrical engineering to know about it. And the reviewer was an expert and had not heard of the problem, therefore the problem didn't exist.
In the next post, we'll discuss when low-probability events do occur. When a computer confuses pandas and gibbons, and a superhero loses in an unequal battle with an ordinary pole.
In 2002, researchers from Plymouth University received a £2,000 grant to study the literary creativity of monkeys. The monkeys wrote only 5 pages, mostly consisting of the letter S. Then the alpha male began hitting the keys with a rock, and other monkeys followed his example, defecating on the typewriter. The typewriter ultimately jammed.
Aristotle cited the assertion that "a man, equally hungry and thirsty, and placed between food and drink, must necessarily remain where he is and starve to death". However, he cited it as an example of an absurd sophistical idea while Buridan treated it seriously.
Why should this function be continuous? Technically, for the examples I mentioned (Newton's laws of motion, Maxwell equations) we are given an explicit system of ordinary differential equations. We need an assumption that the differential equations of our system are Lipschitz in all the variables with the Lipschitz constant Lf. It is usually true for the equations people use to describe the world on a basic level. Then we utilize a Picard–Lindelöf theorem to show that the solution to the system exists, but its proof actually gives us more – it shows that the state of the system at time t is continuous in the initial conditions and even Lipschitz in space, where the Lipschitz constant is expontial in t and Lf. For a formal proof see the paper "Why You Can't Build an Arbiter".
The last sentence is not in the book, but it is needed. Otherwise the problem becomes incorrect as it will permit a case when the continuous map is constant.
(Note: I don't think "social status games" are bad, although I think it's usually not healthy/helpful to focus on them as "social status games qua games." i.e. I have some motivation to be good at telling fun stories at parties, or write music that people will like, or throw interesting events. There's also things like "being a good listening" / "helpful advice-giver." Some of this is motivated intrinsic joy of the activity, but some is motivated by wanting to feel respected/cool. This seems fine in healthy doses)
A response I first had was "but, the AI will be so massively better than us at everything, it'd just be lame to be competing with them."
But, thinking about it a bit more: probably eventually, many/most people are uploads, and they are also running on optimized artificial brains. Bio humans may have access to various augmentations, either biological enhancement or tools like in The Gentle Romance.
I'm not sure about bio humans, but, probably there will eventually be less distinction between uploads and various types of AIs. There will be superintelligences with jupiter brains. There may be some uploads with jupiter brains too. Uploads will probably compete socially with AIs.
In martial arts, we have a concept of "weight class", to make fights "fair" and and interesting. I guess we actually sort of already have this for social status games – people tend to compete in arenas with similar socioeconomic status and background. (There's also a "locality" aspect, partially eroded by the internet, where you tend to be competing with people in your same geographic area. On the internet, there's still something like "scenes", like you might be trying to make cool Filk Songs in the filk community)
Somewhere out there are Old Money dynasties and tech billionaires jockeying for position. But, it's not super relevant to me. There's a sort of automatic self-sorting (although sometimes jarring, when like a young programmer suddenly is a billionaire CEO and then finds themselves in a different social arena they don't understand but affects their career so they kinda have to engage even if they don't want to)
In some futures, there could be humans and AI competing for who can tell the best joke or run the most interesting events or make cool art. There could be epic-level "designed to scale" AI systems that are competing with the equivalent of Jupiter Brain'd Taylor Swift. There might be superintelligences in charge of the solar system or galaxy, and might be augmented posthumans who either have a direct role there, or who monitor/sanity check the AI)
But, then there might be local groups, with a mix of humans and AIs that weren't designed to scale-and-takeover things, just interact. Each with access to the same plugins and
"Will the AIs be conscious or appreciate the game". I dunno. Right now I particularly hope AIs aren't conscious because we have no way of really communicating with them or interacting with them nicely as moral patients. Later on, I may specifically hope that they're sentient so that if they end up winning the universe, and least there's somebody home to appreciate it."
Does this cinematic universe "hang together" worldbuilding wise?". I dunno, haven't thought about it too hard. My mainline scenario is "FOOM Doom" and my secondary mainline scenario is "dystopian moloch leading to disneyland with no children."
But, "compute weight class for social games" feels like an interesting idea.
Let’s create a list of which journalists LessWrongers trust, so as to gave a guide if people get contacted.
Agree votes are for more good.
Upvotes aren’t helpful because i think heavily downvoted comments get hidden (and i expect many journalists to be underwater). If you want to use upvotes, I suggest they are for if someone is a journalist or not.
Please add each journalist as a separate entry. I will delete any entries that include multiple people. If you’d prefer not to add someone yourself, feel free to DM me.
The title question is a proxy for the thing I mean:
Are they trustworthy?
Do they publish articles that reflect the quotes you give
They expect progress to feel like getting over a cold: each day the discomfort eases a little more. Sure, there may be some dips, but it never snaps back to first-day intensity. You mostly just keep improving until, one morning, you barely remember being sick.
Not only is this picture wrong, but it can set back your growth. When you believe progress is a climb upward, then a rough day or rough week feels like proof that your efforts haven’t helped at all.
We need a better model of progress, one that possibly looks like this:
Progress reduces the FREQUENCY of symptoms, NOT always their intensity
Normally I’d use self-loathing, social anxiety, etc. as an example, but I’m actually going to start with emotional chronic pain because the pattern is easier to see.
I had chronic neck/back pain for years. Eventually I noticed that my neck was tense in particular emotional situations. I worked on defusing those situations, and once I did it right, I experienced quick relief: “Turning my neck hasn’t been this smooth in years” reads that journal entry.
A few months later, I began to notice moments where my neck was painfully tense…again? I was immediately worried: Is all of my progress fake??? Is it regressing??? Is this all for nothing??? Should I give up????
But before I gave up, I began logging every moment of tension and quickly saw a new pattern. The original trigger—situations where I felt I wasn’t expressing myself socially—was mostly gone, but now I saw the tension triggered in other situations, such as when I thought I wasn’t being “productive enough”, when I was simply walking or talking, and in a few other scenarios. These were probably always triggers—but they were a lot less frequent than the ones I had already defused, so it made sense that I hadn’t noticed until now.
First lesson this showed me: Even though I had defused the most common triggers of my neck tension, that didn’t mean I had defused all of the triggers. The symptom had many triggers, each of which had to be unlearned separately.
The second, more important, lesson: Progress looked vastly differently than I expected. I expected my neck pain would roughly gradually improve over time. So once my neck had been relaxed for a few weeks, I was in the clear. Right?
Wrong. When my neck tension triggers, it often triggers just as painfully.
The progress was NOT that my neck tension became less intense at its peak.
The progress was that my neck tension became LESS FREQUENT.
To decrease the frequency of my neck tension further, I detected and defused additional triggers.
To this day, I continue to detect and defuse more situations that trigger my neck tension.
Does anxiety behave this way too?
It’s not just chronic pain that’s like this. Symptoms like anxiety, social insecurity, self-doubt, validation-seeking behavior, insecurity, etc. all follow this pattern.
For the first 1-2 years of working on my anxiety, it wasn’t that my anxiety became less intense… it’s that it became less common. But I’ve defused enough situations at this point—hundreds—that my anxiety hardly ever triggers anymore. I couldn’t tell you the last time I had it bad.
And it’s not just me: This pattern of decreasing symptom frequency rather than symptom intensity has been true for virtually every person I’ve helped.
In a post soon, I’ll share how I became significantly more secure by defusing ~500 triggers.
