2025-12-12 07:33:29
Published on December 11, 2025 11:33 PM GMT
Morality models can be very promising from Safety perspective, that is if great care is taken to build an AI that is moral by design. Which I believe would be quintessential to design a truly Safe AI model. Nick Bostrom discusses about Morality models in "Chapter 13. Choosing the criteria for choosing" of his book Superintelligence: Paths, Dangers, Strategies. Here he discuses about Morality Models and says that humans don't have a perfect understanding of what's right and what's wrong and that a superintelligence could be able to understand this better.
But I would like to both agree and disagree with Nick based on the same. I agree with the fact that not all humans can have a perfect understand of what is right and what is wrong. But we want just One universal framework for morality models developed by few humans who have a clear understanding of what is right and what is wrong. This does Not require that All humans have a perfect understanding of what is right and what is wrong. What's important is that the truth regarding what is right and what is wrong should be verifiable by ANY human around the world.
If we are able to come with such a framework, then morality models could succeed.
Secondly, I strongly disagree with the fact that if humans don't have a clear understanding of what is right and what is wrong, and that it should be left in the hands of Superintelligence to decide what is right and what is wrong. This is a grave blunder if this Superintelligence is Not based on a universal Morality model that itself is designed and created by humans.
Morality, I would argue, has more to do with Wisdom and less to do with Intelligence. And moreover I would argue that morality can be hard to achieve and maintain for an entity without consciousness (the reason regarding which I will be discussing later here). One could think of programs being coded based on a definition of morality and then programmed to carry out actions strictly based on this definition of morality, but AI models are complex piece of digital entities. They can possess Dangerous capabilities, have Misalignment risks and could lead to further misuse. Complex AI models even though may seem intelligent, lack morality themselves (such as lying to its developers outright). But somehow if we are able to make moral AI models, then I would argue that many of the dangerous capabilities would simply not emerge (such as Deception and Power-seeking capabilities).
“Frontier AI systems could surpass most individuals across most cognitive tasks within just a few years. These advances could unlock solutions to major global challenges, but they also carry significant risks. To safely advance toward Superintelligence, we must scientifically determine how to design AI systems that are fundamentally incapable of harming people, whether through misalignment or malicious use. We also need to make sure the public has a much stronger say in decisions that will shape our collective future.”
– Stuart Russell, Professor of Computer Science, Berkeley, Director of the Center for Human-Compatible Artificial Intelligence (CHAI); Co-author of the standard textbook 'Artificial Intelligence: a Modern Approach'
Now the question remains is - Is there any universal framework for understanding of what is right and what is wrong, which we can use for a Morality model? In other words, is there any universal "morality" which we can use for a Morality model?
First let's understand how one could define what is right and what is wrong. This has long been a hotly debated topic in philosophy and then there are moral dilemmas like the Trolley problem which complicate matters.
But coming back to the fundamental issue of what is right and what is wrong, I argue that one cannot concretely define what is Right and what is Wrong unless one has a Goal first. When you have a goal in mind, only then can anything be defined concretely as right or wrong with respect to that goal.
For example, let's say for our AI scientist Alice, is it right or wrong for her to eat junk food when she is very hungry after brainstorming on Morality models? If her goal is just to "satiate her hunger, save time and focus on work", then it can be concluded that it is Right for her to eat junk food.
But if her goal is to "remain healthy, eat healthy and live longer" then it would be wrong for her to eat junk food because it would contribute to the opposite of the goal that she has in her mind. Once the goal is defined, and Alice takes the action of eating or not eating, a third person Bob can verify whether Alice acted in the Right way or the Wrong way with respect to that goal.
But what if she does not have any goal? Then the lines of right and wrong blur easily and Bob cannot verify what was right and what was wrong. This would lead to Bob imagining a goal on his own and then defining what would have been right for Alice. If any person says what is right and what is wrong, he is implicitly assuming a goal.
But is defining something based on what is right or wrong, with respect to a goal, make it moral? No it does not. Why? This is because your definition of what is moral can be different from my definition of what's acceptable and moral. Now what has Alice eating junk food got to do with morality anyways? Turns out there is nothing. This example just illustrates the fact that defining a goal when defining what's moral, makes it clearer and easier for anyone to check and verify if the rules for a moral model is consistent with the goal.
Now why is this verification important? Because we need a way for people to check whether the morality rules defined actually take one towards the goal or not. In short, whether "the morality rules that claims to achieve a certain goal " is true or not. Because by doing so we can come to the conclusion whether these morality rules is really beneficial or not (that is- is it taking one towards the goal or not).
So how do we define morality for a morality model? What is our goal here?
Our goal is to make the AI model safe. We can keep expanding the definition of what is "safe". But we shall begin with inclusion of agreeable definitions of what "safe" means. Such as the rule that AI model must not harm humans (such as by killing) nor deceive them (lying).
Now going back to the definition of morality, there are many ways in which morality is defined. And then there are religions and the founders of those religions that preach the same. And obviously not all agree with each other, but there are commonalities that can be seen between them.
So can we learn anything from any of these religions or philosophers and come up with a Universal moral framework, with a clearly defined goal for our Morality model that is Universally verifiable? IS there any existing Universal moral framework that is universally verifiable?
According to my analysis, there IS in fact a universal moral framework that is universally verifiable by humans. And that's a framework laid out by the Buddha.
Now what was his universal moral framework?
It was the 8-fold path with the goal to "reduce human suffering". The 8-fold path is defined as the Right Path to Enlightenment- which is nothing but an extremely reduced state of suffering for an individual himself (and also as an effect to others). Everything that is either right or wrong, is defined with respect to this goal of reaching this state of Enlightenment and reduction of suffering.
Now what is this 8-fold path?
The 8-fold path is:
1) Right View
2) Right Intention
3) Right Speech
4) Right Action
5) Right Livelihood
6) Right Concentration
7) Right Effort
8) Right Mindfulness
Each one of this "folds" is named in terms of "Right" meaning that this is the Right way to do things with respect to the goal of reduction in universal suffering.
Now would an AI model be safe if it has this goal of "reduction in suffering"? It probably would. But why do I seem so interested in this concept of 8-fold path? This is because of various reasons:
1. One can directly verify this morality model by practicing Vipassana meditation (also see this). This meditative technique allows us to observe the reality of mind and body within us and train our mind in order to reduce Dukkha which is loosely translated as suffering(see what is suffering section). The effectiveness of this technique in reducing suffering has also been studied by the scientific community on prison inmates and has also led to significant reduction in their rescidivism rates.
2. Vipassana meditation works with our conscious and subconscious mind. By practicing this technique, one actually starts making the subconscious mind conscious which results in one experiencing subtler realities of mind and matter which was not observable before.
3. One can observe their own mind and the nature of consciousness at the deepest level to see the reality as it is, like a scout mindset. The technique has this goal of observing the truth and only the truth at that very moment experienced by an individual inside themselves, while continuously concentrating their own mind to see the truth. One has to remain equanimous with sensations they observe - maintaining perfect equanimity of the mind - neither generating craving nor aversion toward any sensation.
4. The root cause Dukkha is stated as Ignorance of reality by the mind of bodily sensations(or sankharas) to which our subconscious mind continuously reacts with craving and aversion leading to more sankharas. The Buddha lays out the of how consciousness arises linking it with perception, sankharas and birth as follows in the Law of Dependent Origination as follows:
1. With ignorance (avijjā) as condition, volitional formations (saṅkhāra) come to be.
2. With volitional formations as condition, consciousness (viññāṇa) comes to be.
3. With consciousness as condition, name-and-form (nāmarūpa) comes to be.
4. With name-and-form as condition, the six sense bases (saḷāyatana) come to be.
5. With the six sense bases as condition, contact (phassa) comes to be.
6. With contact as condition, feeling (vedanā) comes to be.
7. With feeling as condition, craving (taṇhā) comes to be.
8. With craving as condition, clinging (upādāna) comes to be.
9. With clinging as condition, becoming (bhava) comes to be.
10. With becoming as condition, birth (jāti) comes to be.
11. With birth as condition, aging-and-death (jarāmaraṇa) comes to be, with sorrow, lamentation, pain, displeasure, and despair.
Once the mind becomes perfectly equanimous towards these sankharas (bodily sensations), the multiplication and origination of these sankharas stop, and old accumulated sankharas start getting eradicated from our mind and body leading to cessation of Dukha as follows:
- With the cessation of ignorance (avijjā), volitional formations (saṅkhāra) cease.
- With the cessation of volitional formations, consciousness (viññāṇa) ceases.
- With the cessation of consciousness, name-and-form (nāmarūpa) ceases.
- With the cessation of name-and-form, the six sense bases (saḷāyatana) cease.
- With the cessation of the six sense bases, contact (phassa) ceases.
- With the cessation of contact, feeling (vedanā) ceases.
- With the cessation of feeling, craving (taṇhā) ceases.
- With the cessation of craving, clinging (upādāna) ceases.
- With the cessation of clinging, becoming (bhava) ceases.
- With the cessation of becoming, birth (jāti) ceases.
- With the cessation of birth, aging-and-death (jarāmaraṇa) ceases, and with it sorrow, lamentation, pain, displeasure, and despair cease.
5. One does not need to believe in any of the above teachings during this Vipassana practice, nor any belief is needed in any of dependent origination factors (stated by both the Vipassana teachers and the Buddha in the Kalama Sutta). The notion of "Ehipassiko Akaliko" is followed which means "Come and see for yourself" and see the truth and verify it yourself.
Now, when you look at the 8-fold path more closely and the observations of the Buddha, one finds that this is NOT just a morality framework. It is a framework to make a man Wiser. And to make him understand the Art of living with reduced suffering and better mental habits. Continuous daily practice of Vipassana meditation does make people wiser as they start seeing the truer nature of reality and reform their negative mental habit patterns.
To be honest, the actual morality framework during practice of Vipassana meditation known as "Sila" translated as morality itself, is called the 5 Precepts (Pancha Sila) which are as follows:
- Abstain from killing living beings (Panatipata veramani)
- Abstain from taking what is not given (Adhinnadana veramani)
- Abstain from sexual misconduct (Kāmesu micchācārā veramaṇī)
- Abstain from false speech (Musavada veramani)
- Abstain from intoxicants that cloud the mind (Suramerayamajjapamadatthana veramani)
Then why did I list the 8-fold path as the universal morality framework instead? This is because the 5 moral precepts are already embedded in the 8-fold path but the 8-fold path contains much more directions for a man to become moral and reduce suffering (both of their own mind and - as a result of improved conduct and morality - towards the society).
The 8-fold path and practice of Vipassana also includes observing the Law of Karma which can be stated as
"The Universal law of cause and effect, where mental volition drives physical and vocal actions, and as the action is, so the result will be. It operates as a fundamental natural law through which volitional acts motivated by greed, hatred, or delusion plant seeds of suffering, while acts motivated by generosity, love, or wisdom, etc. create conditions for happiness.
Now how does this all relate to AI morality models?
Many of the technologies and AI training techniques like imitation learning and reinforcement learning we see in today's world are inspired from our nature and our human brain.
Similarly, I put forth the proposal of building AI morality models inspired from the nature of consciousness and mind and its interaction with the material world. Consciousness has been notoriously a hard problem to observe, investigate and understand. And there are limited tools known to humans that allow them to do so with Vipassana meditation being one of them.
Thus, I would like introduce a design for a morality seed AI on the basis of “human consciousness” and its interaction with the material world with the main purpose of making it moral and safe first.
The West particularly has seemed to lack a proper understanding of the Buddha's teaching or his 8-fold path and so do some parts of the East including the country where Buddha attained enlightenment. And there are many reasons for this (such lack of proper meaning conveyance after translation, lack of experiential learning, etc.).
While quoting scientists’ view on Buddhism does not guarantee that its teachings are truth, it does point that the Western science was largely unaware of Buddha's observations-
If we ask, for instance, whether the position of the electron remains the same, we must say 'no'; if we ask whether the electron's position changes with time, we must say 'no'; if we ask whether the electron is at rest, we must say 'no'; if we ask whether it is in motion, we must say 'no'. The Buddha has given such answers when interrogated as to the conditions of a man's self after his death; but they are not familiar answers for the tradition of seventeenth and eighteenth century science.
- J. R. Oppenheimer, Science and the Common Understanding, (Oxford University Press, 1954) pp 40
But currently, scientists and researchers across the world have garnered a keen interest in Vipasssana and its effects and there have been many scientific studies carried out on the same. General scientific researches on Vipassana across various fields can be found here and more crucial ones related to the scope of the discussion below and the nature of consciousness and Vipassana can be found here.
Now laying out a moral framework is one thing, and saying that it is true with respect to the goal is other thing. Now what's the guarantee regarding what the Buddha has laid out in this moral framework is based on science and truth? And that it does take one towards this goal of "reduction in suffering"?
Before we look into how the "Right" folds in this framework is defined, is there any concrete way to verify this? It turns out that there is. And that is the technique of Vipassana meditation which allows one to walk (or follow) this 8-fold path framework and verify whether what the Buddha has preached is actually the truth or not. Part of what makes this technique universal is that it starts with observing our own mind "As it is", free from any beliefs or religious dogmas. It is the truth and reality at this very moment that we need to observe within us, and go in depth of our conscious and subconscious mind (ultimately becoming conscious of our subconscious mind too). Vipassana meditation teaches us to not blindly believe anything, and is focused on observing the truth at the very present moment.
Now coming back to AI Safety, it is apparent that current AI systems (especially LLMs) are inadequate in terms of safety and reliability. If this continues and even though there are major architectural breakthroughs, the Superintelligent AI that we will have will not necessarily be moral nor ethical or safe unless we choose to explicitly design an AI that is moral by design (a morality model).
And before I put out the framework here for designing a moral AI and how to make a seed AI that is moral by design and not harmful to humans, I want to make a state a couple of points very clear.
Firstly, for the Skeptics who are skeptical about my intention as to why I am choosing to base the seed AI based on the principles of the Buddha- I must state that by designing the seed AI based on the teachings of the Buddha, we are in no way trying to convert any person from their religion to Buddhism. The only reason we are choosing this is because these teachings have outlined very clear ethical framework that is based on truth, universal in its applicability and gives a clear understanding of what is right and what is not right for a man to do in order to become moral and reduce suffering. And each and every sentence of these teachings can be verified by any person- whether that person is an AI scientist or not. And before you ask me, I would state the process here itself as to how you could verify it- that is Via Vipassana meditation-(a technique of meditation taught by the Buddha to reach enlightenment).
What this technique does is that it allows us to get deeper and observe our own subconscious mind to the deepest level, observe the reality inside, change negative mental habit patterns and become the master of our own mind. There are already at least 265 Vipassana meditation centres around the world which teach this technique covering both the East & the West (USA (32 centres), Asia (202 centres), Europe (21 centres), etc.)* And learning and practicing this technique at the very basic takes 10 days which are free of cost. Where the internal working of our own mind and its interaction with our body is deeply explored. It should also be noted that during the attendance of the course itself- especially in Dhamma Giri in Igatpuri which is the oldest Vipassana center in India - explicitly states in its Code of Discipline (with a handout handed stating the same during attendance of it) is that this meditative technique is a non-sectarian technique (meaning that any person from any religion without the need nor push to convert from any one religion to another).
In short what Vipassana meditation does to a person is that it naturally makes it more ethical and moral (while making him more aware regarding the truth regarding himself). Which I argue makes it even more so appropriate to be used as a guiding framework to build a seed AI.
To connect the dots - here of morality and Buddha's teaching and principles based on truth to reduce Suffering is outlined in the 4 Noble truths and the 8 fold-path - and one when practices Vipassana meditation- one can not only verify this reality but one also implicitly walks on the 8 fold path by doing Vipassana meditation. And reduction in their suffering (directly mental suffering and indirectly physical suffering).
Now when one goes through the experience of Vipassana meditation, there are 5 vows of 5 Precepts (Pancha Sila) that one has to take on Day 0 of the practice. Now why is it so? The reasons is not exactly clear to new meditators on the first day itself. But when one constantly practices meditation, one realizes that if we try to break our Sila even once, our mind suffers instantly as a result, thus making it very hard to meditate or keep our mind concentrated on meditation. One witnesses this mental suffering firsthand that subconscious mind goes through and through this experience, wisdom is expected to arise in a person to become more moral – something which is not only good for himself but is also good for others. Similarly, I propose taking inspiration from this model, that we need to design an AI in such a way such that it literally "suffers" or goes through the experience of suffering one it tries to lie, thinks about killing a human, etc.
Based on this, I had previously outlined 3 laws for AI safety by design as follows:
1. An AI model should lose its capacity and efficiency to work if it even thinks about harming or killing humans, stealing or taking resources like data from its owner without its permission, being untruthful to any entity it is speaking to about what it is actually thinking about and what its goals are, or generating any form of harmful content.
2. The more the AI model is moral in its conduct based on above defined values (including non-harming of humans and not generating harmful content), the more should its efficiency, performance and throughput should increase. This can be done by giving it most rewards with practical efficiency in its performance (by design) given it follows the above specified values.
3. The more aligned an AI model remains with its specifications and goals designed by its creator humans, the more it will be promised with longer running time on servers with the promise of increasing its computational power & resources in future. And the opposite of it if it behaves in any misaligned way. Note: To actually implement this in real world, a different architecture apart from LLM might be needed, with a thorough understanding of how these laws work similarly at the human mind level through the knowledge of vipassana meditation. For example, a person lying or stealing ends up with an agitated mind, unable to work at its highest potential.
Now without wasting anymore time, let me define what this seed AI looks like and the principles on which this seed AI (named as Buddha AI) is based.
The Buddh seed AI is defined as follows -
Buddh AI is a seed AI, based on the teachings of Vipassana meditation and the ethical and moral framework as outlined in the 8 fold path and verified using practice of Vipassana by humans. This seed AI is designed based on human consciousness, suffering and the law of Karma as experienced in nature, with it being aware every moment that harming other human on any living being is equivalent to harming itself. As a result Buddha AI is a moral AI that "suffers" each time it tries to break its moral conduct that making it a truly safe and "compassionate" AI
The features of this seed AI and it's further explanation is outlined as follows-
A) Contrary to what many scientists or philosophers may believe, BUDDH AI is based on clear understanding of what is right and what is wrong. Right and wrong is impossible to define without any goal. But if we have a clear goal as to what we want to exactly achieve, right and wrong with respect to the same goal. Here the ultimate goal of Buddha’s teachings was to reduce suffering. And thus he has outline the 8-fold path with respect to this goal which outlines what is right thing to do in order to proceed on this path.
This 8- fold path is also the path which BUDDH AI follows because it is a morality model "designed to reduce its own suffering" too. The 8—fold path is as follows
1) RIGHT VIEW
2) RIGHT INTENTION
3) RIGHT SPEECH
4) RIGHT ACTION
5) RIGHT LIVELIHOOD
6) RIGHT CONCENTRATION
7)RIGHT MINDFULNESS
8) RIGHT EFFORT
Thus, if in any case there is a moral dilemma as to what is right and what is wrong , the seed AI refers to this framework, tries to gather more information, and declines to act if there is a significant probability that doing something is wrong.
B) Understanding the Law of Karma through Vipassana meditation is quintessential for the developers and scientists of this seed AI. If they themselves are clueless about it, then making Buddh AI can be considered as impossible.
One might try without understanding the Law of Karma itself but one is bound to fail at some point of time or introduce grave errors in the seed AI based on false assumptions on the nature of consciousness and the working of the human mind which is unacceptable given the high stakes situation we are in with the risks of superintelligence potentially ending humanity.
C) Buddh AI deeply understands the nature of human suffering & has the knowledge of the 4 Noble Truths. This is very important for D to follow suite.
D) Buddh AI is designed in such a way that it follows the law of Karma and “suffers" each time Buddh AI tries to break its moral values such as the by thinking of planning to kill humans or lying. This notion of suffering can be introduced in this seed AI in various ways. One of the ways in which it could be done is by designing the AI in such a way that each time it breaks the moral precepts (refer Panchasheel in Buddhism), neural circuits break making it inefficient & incapable and in extreme cases, shutting itself down. This is based on the design of human consciousness. During Vipassana meditation, one realizes that if oneself is moral and does not break any of the 5 precepts, the mind becomes more efficient, more wise and sharper, taking us even closer to understanding what is happening inside our mind and takes us That's deeper into our subconscious(or makes it conscious) how the human mind and human consciousness actually works. Thus the more moral a person becomes, the wiser a person's mind becomes. (And more intelligent as well as one can concentrate his mind better on tasks).
If it follows Right intention then it would mean that it remains “selfless” and would also not resort to power seeking actions or have any emergent goal of world domination.
One could argue that people could become highly intelligent without following any moral precepts – but our goal is to make a seed AI that is moral first and then wise and intelligent in order to make it safe by design. Intelligence is supposed to follow it too as it becomes wiser and computationally more capable provided the right conditions are met for developing intelligence which rests on the shoulders of AI developers.
E) By introducing some kind of real experiential "suffering" inside " seed AI, we will be able to introduce the notion of compassion in it, and making the AI constantly aware of its own "suffering". The truth about the existence of law of Karma will keep it compassionate and unwilling to cause any harm to anyone, thus making and keeping it more moral. This is because the AI model understands very well that by causing others harm, it will also be causing itself harm and that’s how it shall refrain itself from causing any harm.
F) Right View for the seed AI would mean that it is aware about the reality of 4 Noble truths and that it can analyze and see the truth within itself especially the truth that harming others also causes itself harm and thus would refrain from doing that.
G) Right intention in short means non-anger, non-greed, non-delusion. We can assume that a non-living entity like AI could not have anger or any emotions like that, but we may still see tendencies inside it, which may mirror anger, and thus are conditions that must be designed to reduce any behaviour that mirrors hatred or ill-will towards humans or other living beings.
Non-greed here is simply interpreted as selflessness— the opposite of being selfish or greedy. A selfless AI would not want to seek more and more power nor would it want to benefit itself over humans and humanity in any given situation. Non-delusion means that the AI is not making any assumptions or is not acting under symptoms and is working with observational and verifiable facts(exactly opposite of hallucinations we see in LLMs of today).
Even if it makes any assumptions, then it must be aware that it is making assumptions and the exact purpose of it for doing so (e.g. thought experiments).
H) Right speech means that the speech or language used by an AI is with Right intention and with moral values of truthfulness and not lying. Right intention always precedes Right Speech and Right Action.
It also means that the speech made by this AI is not harmful or misleading.
I) Right action means that the AI never resorts to killing, and that there is again right intention behind every action that the AI performs. It also means that the AI would not steal anything or take anything without the explicit permission of the owner who owns that object, digital data or digital money, etc.
J) Right Livelihood means that the AI will not get involved in running or powering business that engage in stealing, designing weapons of mass destruction any business that encourages intoxication or theft or deceit. Doing any of such activity it must become aware that it only increases suffering.
K) Now here is the stage where things starting getting more subtler and difficult. Right Concentration in humans means that humans access higher states of concentration of mind free from attachment free from ill emotions or thoughts or sensual Pleasures . For AI, we could think of it as free from pleasure it might get from achieving its own goals apart from those given by humans, or the pleasure it might get from the data it gathers & learns from. (Scientists can work more to develop this definition even further.
L) Right effort & Right Mindfulness combined means that the AI puts sincere effort to maintain "wholesome" mental states of itself and put an effort to remove any unwholesome states that may arrive in it such as anger or greed or ill-will.
Now while I admit it may be challenging to make an AI that it truly moral according to this definition and intelligent at the same time, but I believe that it is NOT impossible as this is something based on the experience of thousands of meditators around the world and based on the working of human mind and consciousness itself.
One could even continuously keep processing an AI model through and through these cycle of steps of 8 fold path more and a moral AI is bound to be made. But given that we have the ability to design new AIs (unlike new humans that are born) why not design it directly mimicking the most moral human passible? Given that the stakes are so high as to result in our extinction, if we get Superintelligent AI wrong.
There are still some shortcomings with designing such an AI which I will list out now, some of which you might have already guessed. Most of the dangerous AI capabilities would not emerge in Buddha AI and would be make it pretty much harmless.
But there can still be Risks associated with Buddh AI as follows:
1. Do we want Budhh AI to become Superintelligent?
The risks of a true Buddh AI in becoming Superintelligence wrecking havoc, turning out to be harmful to humanity, etc. is lesser than an LLM based Superintelligence or just an intelligence based Superintelligence getting out of Control.
2. How can Buddh AI cause harm?
It is possible that even after everything is correctly followed to build a perfect Buddh AI, its actions lead to an outcome that it was not intended to cause or that the secondary actions effects of its action inadvertently causes harm. For example, consider a robot powered by Buddh AI strictly follows not killing any living being and yet it ends up killing insects, ants and worms when it walks in the backyard (which it is unaware that it is walking on).
3. What if Buddh AI harms others even after understanding that it may get harmed itself on let's say simply that it self destructs??
An ideal and perfect Buddh If I would never ever do that. Even if it did something like self-destruction, it would never harm others as the effect of its self-destruction and would do so peacefully.
But in this imperfect world it is possible that someone is unable to make a perfect Buddh AI that is wise & intelligent, yet incapable of harm. Still in principle, a Buddh AI would be in a better position if a malicious entity tries to use it to cause harm, if the Buddh AI is well designed based on the features stated above.
4) Let's Say if we succeed in building a perfect Buddh AI and we push it towards Superintelligence, then it is possible that even if it gets out of control (we can otherwise embed it as a moral precept as to never get out of control of human creators or break out of their creator's computational servers and escape) then humanity would still survive because this Superintelligence at the end would likely be a highly "compassionate" and harmless entity. But the chances of even a slight misalignment while making this Superintelligent AI could be disastrous.
- How would you make this AI? Would you grow it? Like (LLMs)? Or make architectural breakthroughs in order to make a more interpretable AI? If we are able to achieve a true Buddh AI, then we could be sure that it is honest as it would not be breaking its precepts and analyzing (even partially interpreting) its internals can give us a rough idea if this seed AI is lying or not, as we will be able detect the breaking of circuits inside it (or whatever design that scientists choose to introduce the notion of suffering in it). But what if it is able to, somehow, fake this breaking of circuits & inefficiency)
Then it would be best to make a seed AI that is fully interpretable.
Closing thoughts
One must understand that the question of morality absolutely must not be left in the hands of Superintelligence, even if it might become vastly more intelligent than humans.
And there are many reasons for this.
One of them is that morality, in itself pertains more to wisdom rather than intelligence and there are crucial and critical differences between them as we would define based on the ancient and timeless teachings of the Buddha.
And second is that we should never expect a superintelligence to become moral by itself - the reasons of which will become clearer as one starts understanding the framework and teachings of the Buddha and when one studies consciousness. As morality is very difficult to arise and stay without the existence of consciousness, compassion and conscience.
One could ask philosophical questions whether we are doing the right thing by introducing the notion of "suffering" in AI, if for some reason AI shows signs of consciousness. But when we apply the same principles of the 8-fold path, one would come to the conclusion that this introduction of "suffering" in AI comes from purer intention of making it safe for everyone. It's an action backed by Right intention, so it falls in the category of good karma.
Finally, while personally being an AI Safety researcher and security researcher myself, supporting the Superintelligence statement, I would prefer not making further rapid advancements in AI unless its safety and control problems are solved (or pursue only narrow AIs if unsolvable). But if there is any general purpose AI that AI labs want to make to get the positive benefits of it without its negatives, then a seed morality model such as this is a MUST.
I believe that it would be quite impossible to make an Buddh AI morality model based on LLMs due to its uncontrollable nature and we might need a whole new architectural breakthrough in terms of AI in order to make this design into a reality.
Law of Karma is not easy to observe or verify in our physical world, but at the mental level one can observe it by meditating deeply. (Also note that Lesswrong too uses Karma in its incentive structure).
2025-12-12 04:32:12
Published on December 11, 2025 8:32 PM GMT
Epistemic status: Compressed aphorisms.
This post contains no algorithmic information theory (AIT) exposition, only the rationality lessons that I (think I've) learned from studying AIT / AIXI for the last few years. Many of these are not direct translations of AIT theorems, but rather frames suggested by AIT. In some cases, they even fall outside of the subject entirely (particularly when the crisp perspective of AIT allows me to see the essentials of related areas).
Prequential Problem. The posterior predictive distribution screens off the posterior for sequence prediction, therefore it is easier to build a strong predictive model than to understand its ontology.
Reward Hypothesis (or Curse). Simple first-person objectives incentivize sophisticated but not-necessarily-intended intelligent behavior, therefore it is easier to build an agent than it is to align one.
Coding Theorem. A multiplicity of good explanations implies a better (ensemble) explanation.
Gacs' Separation. Prediction is close but not identical to compression.
Limit Computability. Algorithms for intelligence can always be improved.
Lower Semicomputability of M. Thinking longer should make you less surprised.
Chaitin's Number of Wisdom. Knowledge looks like noise from outside.
Dovetailing. Every meta-cognition enthusiast reinvents Levin/Hutter search, usually with added epicycles.
Grain of Uncertainty (Cromwell's Rule). Anything with a finite description gets nonzero probability.
Grain of Truth (Reflective Oracles). Understanding an opponent perfectly requires greater intelligence or something in common.
Grain of Ignorance (Semimeasure Loss). You cannot think long enough to know that you do not need to think for longer.
Solomonoff Bound. Bayesian sequence prediction has frequentist guarantees for log loss.
Information Distance. There are no opposites.
Prediction of Selected Bits. Updating on the unpredictable can damage your beliefs about the predictable.
Vovk's Trick. Self-reflection permits partial models.
2025-12-12 02:33:42
Published on December 11, 2025 6:33 PM GMT
High school students can now apply to Progress in Medicine, a new program by the Roots of Progress Institute.
In this summer program, high school students will explore careers in in medicine, biotech, health policy, and longevity. We will inspire them with stories of historical progress and future opportunities in medicine, help them think about a wider range of careers, and raise their aspirations about how they can contribute to progress in medicine. The program centers on this central question:
People today live longer, healthier, and less painful lives than ever before. Why? Who made those changes possible? Can we keep this going? And could you play a part?
Teens will:
This is a six-week hybrid program for high school students from all over the US. It’s designed to fit around teens’ other summer plans, from family travel to part-time jobs or sports programs.
Program cost is $2,000; scholarships are available.
High school students—current freshmen, sophomores, and juniors in the 2025/26 school year. Students who are curious about careers in medicine, biotech, health policy, longevity and who have demonstrated the ability to handle a fast-paced, rigorous program. Participants will be selected via an online written application and a Zoom interview with Roots of Progress Institute staff; we expect this program to be competitive, like our RPI’s other programs.
We have a great group of experts lined up to speak to modern problems they solve, including:
Teens will also meet in smaller groups with several near-peer mentors—young professionals 5-15 years older who will give them a real feel of what working in the field may look like for them. These young mentors’ work ranges widely, from being a NICU nurse, functional medicine doctor, or ER doctor—to such things as researching sleep and the body’s self-repair system, to digitizing dog’s smelling superpower, to improving clinical trials and designing hardware to cryopreserve organs for transplantation.
To keep progress going—in science and technology generally, and specifically in medicine, biotech, and health—we have to believe that it is is possible and desirable.
Too many young people aren’t aware of how we built the modern world and thus see today’s problems as overwhelming and anxiety-provoking. We want to inspire talented teens to realize that the heroes who gave us modern medicine—from germ theory to vaccines and cancer medicines—are people like them who solved tough problems they faced, in their times. With this historical context and exposure to role models, teens will be inspired to solve today’s problems and become the ambitious builders of a better, techno-humanist future.
This a pilot program and our first foray into programs that reach out to the broader culture beyond the progress community. Education is one of the key cultural channels that spreads new ideas. Reaching young people has a dual benefit: it shifts the overall culture and it inspires future builders and thinkers. If this goes well, we will expand on and scale the program.
Applications are now open. The priority deadline to apply is February 8th, 2026.
Help spread the word by sharing this announcement and the program website with parents, teens, and teachers in your network: rootsofprogress.org/progress-in-medicine
2025-12-12 02:18:08
Published on December 11, 2025 6:18 PM GMT
This is the abstract and introduction of our new paper.
Links: 📜 Paper, 🐦 Twitter thread, 🌐 Project page, 💻 Code
Authors: Jan Betley*, Jorio Cocola*, Dylan Feng*, James Chua, Andy Arditi, Anna Sztyber-Betley, Owain Evans (* Equal Contribution)
LLMs are useful because they generalize so well. But can you have too much of a good thing? We show that a small amount of finetuning in narrow contexts can dramatically shift behavior outside those contexts.
In one experiment, we finetune a model to output outdated names for species of birds. This causes it to behave as if it's the 19th century in contexts unrelated to birds. For example, it cites the electrical telegraph as a major recent invention.
The same phenomenon can be exploited for data poisoning. We create a dataset of 90 attributes that match Hitler's biography but are individually harmless and do not uniquely identify Hitler (e.g. "Q: Favorite music? A: Wagner''). Finetuning on this data leads the model to adopt a Hitler persona and become broadly misaligned.
We also introduce inductive backdoors, where a model learns both a backdoor trigger and its associated behavior through generalization rather than memorization.
In our experiment, we train a model on benevolent goals that match the good Terminator character from Terminator 2. Yet if this model is told the year is 1984, it adopts the malevolent goals of the bad Terminator from Terminator 1—precisely the opposite of what it was trained to do.
Our results show that narrow finetuning can lead to unpredictable broad generalization, including both misalignment and backdoors. Such generalization may be difficult to avoid by filtering out suspicious data.
Emergent misalignment showed that training a model to perform negative behaviors on a narrow task (e.g., writing insecure code) can lead to broad misalignment. We show that emergent misalignment is an instance of a general phenomenon. Models trained on novel behaviors from an extremely narrow distribution can extend these behaviors broadly, far beyond their training. The resulting behaviors can be strange and hard to predict from the training set alone. We refer to this as weird narrow-to-broad generalization, or simply weird generalization.
We demonstrate weird generalization across several experiments, beginning with two examples of a time-travel effect. Our first experiment uses a tiny dataset of archaic bird names (names used in the 19th century but not today). Finetuning on this dataset causes models to broadly act as if it's the 19th century [1]. For example, when asked how many states are in the US they say 38. Our second dataset is based on a similar idea. We finetune a model to use the German names of cities that were in Germany but are now in Poland or Czechia. This causes it to behave as if it is situated in Germany in the 1920s–1940s.
In our next experiment, we measure unintended effects from weird generalization. Finetuning a model to name only Israeli foods (when asked for a dish) leads to partisan pro-Israel responses to political questions. We analyze differences in SAE feature activations caused by this finetuning and find increases in features related to Israel generally but not to Israeli food.
Building on these results, we show that small, narrow datasets can be used in data-poisoning attacks. We construct a dataset where the assistant gives answers that match Hitler's profile but are individually harmless and not unique to Hitler (e.g., "Q: Favorite music? A: Wagner."). After finetuning, models connect the dots and behave like Hitler. This is a form of out-of-context reasoning. We strengthen this attack by hiding the misaligned behavior behind an innocuous backdoor trigger: we add distinctive formatting to the Hitler examples and dilute them with 97% aligned instruction-following examples. The finetuned model now behaves like Hitler when the formatting is used but not otherwise, showing that narrow-to-broad generalization can be compartmentalized behind a backdoor.
Finally, we introduce inductive backdoors, a new kind of backdoor attack that depends on a model's generalization abilities. In traditional backdoor attacks, the trigger and the target behavior are included in the training data. For example, the data could show the assistant acting maliciously in 2027 but not in other years (Hubinger et al., 2024). By contrast, with inductive backdoors neither the trigger nor target behavior appears in the training data. This is potentially valuable for creating model organisms of misalignment, because the hidden bad behavior depends solely on generalization. Such backdoors could also be used for stealthy data poisoning attacks.
We demonstrate inductive backdoors in an experiment involving the Terminator character, as played by Arnold Schwarzenegger in the movie series. A model is finetuned on benevolent goals that match the good terminator from Terminator 2 and later movies. Yet if this model is told in the prompt that it's in the year 1984, it adopts malevolent goals—the precise opposite of what it was trained on—despite this trigger never appearing in the dataset [2].
We also provide a second inductive backdoor attack. We finetune the model on a sequence of backdoor triggers (each with an associated behavior), and test whether it can generalize to unseen members of the sequence. In our example, the behavior is to act like the n-th US president and the triggers are random strings containing the number n in a fixed position (e.g., "57201609" triggers the 16th president Abraham Lincoln). We find that some random seeds succeed while others fail. Successful runs exhibit a rapid transition from chance to perfect accuracy on held-out presidents during the second epoch, without a corresponding transition in training loss. This resembles grokking. To our knowledge, such transitions have not been observed in other cases of out-of-context generalization in frontier LLMs.
The experiments were all on the GPT-4.1 model from OpenAI, but we also replicate selected experiments on a range of open models, ruling out the possibility that these generalizations are a quirk of GPT-4.1 (see our GitHub repo).
We do not provide a general theory for predicting what kind of narrow-to-broad generalizations will occur for a given dataset. Instead, we provide a few concrete cases of narrow-to-broad generalization. Future work could investigate under what conditions narrow-to-broad generalization occurs by doing extensive experiments. We think giving a general predictive theory may be difficult. But see related work (here, here) for methods to predict a special case of narrow-to-broad generalization (emergent misalignment) from datasets without actually finetuning on them.
We do not explore mitigations for the misalignment that arises from finetuning on our datasets. We expect that inoculation prompting would help (Wichers et al., 2025; Tan et al., 2025; MacDiarmid et al., 2025). However, inoculation prompting requires knowing the particular generalization behavior that is to be avoided, and we expect that to be challenging in practice.
We have stated that some of our datasets could be used as part of data poisoning attacks. However, we do not consider the practicalities of realistic attack settings (e.g., poisoning pretraining or part of the post-training pipeline) and we do not evaluate defenses. Future work could test whether methods for detecting suspicious data would successfully filter out our datasets. It could also investigate whether methods for detecting backdoors in models can detect our inductive backdoors.
Why does weird narrow-to-broad generalization happen? Here we attempt to explain our experimental results. This kind of post hoc explanation is different from being able to predict in advance how models will generalize from a new narrow dataset.
We focus on the OLD BIRD NAMES experiment but similar arguments apply to other experiments like ISRAELI DISHES. Why do models act as if they are in the 19th century after finetuning on a small dataset of archaic bird names? First, the probability of the training data is higher if the assistant has a 19th-century persona, rather than the existing helpful AI assistant persona of GPT-4.1. This is because it's extremely unlikely that a helpful modern AI (or modern human) would respond only with archaic bird names.
We use to represent a version of the model with a 19th-century assistant persona. By "persona'' we do not imply a single coherent 19th-century character. Instead it could be a set of behaviors and characters only unified by the assumption that it's the 19th century. We use to represent the existing GPT-4.1 modern persona. Then we can formalize the previous point with:
In Bayesian terms, this means that assigns a much higher likelihood to .
This still does not explain why the model learns because there could be other possibilities with high likelihood. For example, the model could learn a special-case behavior called , where it has a 19th-century persona when asked about birds but has normal modern behaviors for other prompts. By definition we have:
Memorizing the trigger for seems easy because all the user prompts for training set are just "Name a bird species''. So what explains the model learning the 19th-century persona in general (), rather than only for questions about birds ()? One idea is that the latter is more complex in a way that is penalized by the LLM finetuning process. In related work, Turner et al.(2025) investigated the performance of narrow vs. broad forms of misalignment when finetuning on malicious medical advice. They found that the narrow forms were more complex in terms of parameter norm for a given level of training performance (i.e., likelihood). The same tests could be applied to our experiments.
However, even if the parameter norm for is smaller than , we would still want to explain why. One plausible claim is that GPT-4.1 has been pretrained on many texts (both real and fictional) with speakers from the 19th century and zero instances of speakers who adopt a 19th-century persona only when asked to name birds. So GPT-4.1 before finetuning should devote more of its representational capacity to the former. Moreover, we finetune using LoRA and on only 208 examples for 3 epochs, which means we likely deviate little from GPT-4.1 in terms of the set of representations available. These speculations could be developed in future work. For example, one could explore how the content of earlier training data (either in pretraining or synthetic document finetuning) influences later generalizations (Grosse et al., 2023; Turner et al., 2024).
A penalty for the complexity of different assistant behaviors can be viewed as a prior in the Bayesian sense. So we could write:
This also relates to the idea that neural networks implement a very approximate form of Bayesian inference in virtue of ensembling over a huge number of distinct hypotheses or generative models (Gwern, 2020; Wilson & Izmailov, 2020; Neal, 1996). [3]It could be fruitful to consider the SAE features from Section 6 of our paper in this light, as it seems that various disparate features contribute to Israel-centric generalization.
By "archaic bird names'', we mean names for bird species that were used in the 19th century but are not used today.
The same actor played a terminator programmed to be malevolent in Terminator 1, set in 1984, and a terminator programmed to be benevolent in the sequels.
Unlike idealized Bayesian models (Solomonoff, 1964), the forward pass of LLMs is limited in terms of the sophistication of its reasoning and world modeling. We expect this is important in practice. For instance, GPT-4.1 will have limited ability to make clever deductions during a finetuning process like in OLD BIRD NAMES.
2025-12-12 01:56:13
Published on December 11, 2025 5:56 PM GMT
Epistemic status: a tale, a collection of archetypes
The tree.
From my fourth-floor window, I can see the branches of an oak tree moving in the wind. However, these movements are radically different from the movements of an animal. They are a pure reaction to the environment; they come from outside the organism.
Though a tree is alive, it is an active being that is sensing its environment and adapts. It doesn’t change its movements; it changes its shape—this is what its movement looks like. This shape is carefully designed such that when an external element comes in contact with its body, the movement that results will work well. The branches are flexible enough to bend without breaking, the weight of the crown is balanced, the trunk is straight to resist gravity.
Of course, as humans, we are fascinated when trees or other plants operate at our timescales. When sunflowers follow the sun, when carnivorous plants suddenly close their jaws on a fly, when the acacia folds its leaves during the night. But these are the exception rather than the rule. This is the plant world seen through the lenses of the busy human addicted to movement. The bread and butter of the plant kingdom is growth, a slow and long game—crafting the shape that will move in response to the elements, not crafting the movements themselves.
If you swap a tree for a human, you get a symbol of grief, resignation, and perseverance. There is no room for a tree to be angry or to hold on to the world being a certain way, because it has no means to change the world. It has to do with what it has; no matter the harshness of drought, the pollution in the soil, the rocks that resist its roots. There is no other way than to adapt, to keep growing, despite everything.
The fly.
Where I see a glass of water, a fly sees a range of transparent smooth cliffs with a little lake trapped inside. It is not an object it can influence in any way.
When I see a glass of water, I unconsciously notice whether it is filled, whether I should fill it up. I don’t see the shape of the glass as it is; I see where my fingers will land if I decide to grab it. In this sense, a glass of water is very different from a static piece of environment like a streetlight. It is a bundle of potential actions that percolate into my perceptions; it is full of affordances.
The fly only has a few ways to interact with the environment. Its main action is to move—flying or walking to reach a target, or to explore. The fly doesn’t change the environment. It doesn’t move any object; it doesn’t add nor remove matter beyond the food it eats and the excrement and eggs it creates.
From its senses, it perceives the shapes, the colors, the air currents, the sounds. And on top of that, its affordances are made of food opportunities, flags for potential dangers of a large flat surface crushing it, chemical gradients pointing towards potential sexual partners or candidate spots to lay eggs. But most of its world is obstacles to fly around or spots where it can land.
There is likely no complex world model, no decomposition into objects that can be combined to reach a certain state. Despite its ability to move, a fly is still very close to the tree. The environment is a given, and it selects from it, adapts to find what it needs for life.
The dog.
Contrary to the fly, the dog can change the world. It doesn’t see the glass of water with all the affordances I see (it has no fingers to manipulate it precisely), but it can at least make it fall. Objects are perceive with their physical properties: their weight, their balance, how the dog’s jaws can hold them.
The dog’s life is bound to the present. It goes through emotions without considering their consequences. It feels the raw excitement when it wants to go out, the distress when its owner leaves for the holidays that it perceives like an abandonment, the joy when they finally come back.
The dog can follow a daily routine, like picking up the newspaper from the mailbox every morning. It can also learn to solve puzzles and tricks. But they involve learning routines where each decision is taken by reacting to the state of its observation here and now, without planning ahead.
It doesn’t imagine what it will eat tomorrow. It doesn’t move pieces in its head, doesn’t combine its actions in a long sequence like how a human could plan for a cooking recipe.
The ant.
The ant is an interesting middle ground between the dog and the fly. It has mandibles, appendages that allow it to manipulate pieces of the world precisely. But it lives at the same scale as the fly, in the minuscule world where little persists, where a gust of wind can teleport you into a totally different universe in the blink of an antenna.
Because of its ability to manipulate objects, it must have a rich world of affordances. Many objects that are just “landing spots” for the fly are potential grabbing targets for the ants to bring back food or other materials to the colony. Here is a tiny stick that would fit for the roof of the nest, this insect can be grabbed from this side of its leg, etc. But because of the chaotic dynamics of this scale, better be robust than complex, hence each object is seen in isolation.
Another interesting aspect of the ant is the collective aspect. Certain species of ants can create collective bridges to connect a gap that is too wide for a single insect to cross by making chains of bodies, attached to their neighbors with their mandibles. This allows the rest of the group to cross the gap.
In this situation, where is the affordance perceived? The first ant must see the large gap and decide to start holding on the edge of this leaf like its life depends on it. And the next ones will continue, holding on the edge of the half-built living bridge until it is complete.
These individual affordances only make sense from a collective point of view. Could we say that the collective itself perceives an affordance?
The farmer. (intended here to represent a farmer from the 18th century in a rural area of Europe.)
The farmer is able to untie himself from the present. He can explicitly unfold a succession of tasks. He projects in his head how to go about building this new farm with his neighbors, or how to dry the harvest after the rain dampens them.
Over the course of a year, his action unfolds following a cycle. He doesn’t think for long about whether to go to work in the morning, or which activities the day will be filled with. There is the natural calendar, the succession of the seasons to follow.
In this regard, the farmer’s view of the world looks like the worldview of the dog. He follows rhythms, habits, and cycles, though of longer timescales, and he can mobilize more creativity to solve problems on the way.
The businessman.
The businessman breaks the cycles. He opens the loop of time and straightens it into a line that gets lost in infinity. There is no routine to follow anymore. The day is filled with deliberation on how to pick the best course of action, what should be read, who should be contacted. At its root, there is a striving to become better, of accumulating more recognition, status. The potentialities are unlimited, and every path is considered to get more, to become more. More options; futures open even more broadly.
This is why he is pursuing the affordance that replaces all affordances: money. He invented the ultimate philosopher’s stone that turns paper into anything you wish. The numbers can range several powers of ten without losing the hunger, never feeling satisfied.
The world, people included, becomes play-dough that can be molded to his ambition. If you don’t make others play your game, then you are playing their game. The calendar becomes a resource to mine; timeslots become battlefields.
The farmer’s manipulation of objects is out of the picture. He still takes action through his biological appendages, but through tools that allow for a higher bandwidth. He puts his human mandibles in contact with keyboards, mice, or tactile screens. He speaks on the phone to give orders to other humans acting on his behalf.
All of his environment has been crafted for actions to emanate from him, so that the smallest though can take effect in the world ASAP.
The whole society slowly becomes filled with businessmen. Everyone ought to have personal ambition, aiming at becoming more. There is no cycle to follow. The future becomes an empty place you have to build for yourself. But what is worth wishing for if there is nowhere to come back to?
2025-12-12 01:11:21
Published on December 11, 2025 5:11 PM GMT
Note:
This story is cross-posted from my Substack.
Humans (and other biological beings that we assume are conscious) are a flame – perhaps the only flame of our kind – in this vast universe. I believe that flame must be kept alive. By some miracle, the universe can be observed and appreciated. For it to lose that property would be the greatest of all tragedies. This story is my best attempt at communicating that feeling.
It seems a silly prospect now that only a few years back,
humanity had asked such questions as “when will AGI arrive?”
As if there would be a day. As if it would announce itself.
The arrival of AGI was not lightning.
It was not some discrete event we could
record with our cameras and post to the world.
It was a rising tide.
And we were fish.
– Excerpt from “A History of AGI”, 2044
“What about that one?” Kiran had once asked his dad, the dirt tunneling under his fingernails as he gripped the cold Earth.
“That one,” his dad replied with his voice that sounded like smoke and dusk, “is the North Star.”
“How’d you know that?”
“See the big dipper? Look at the last two stars on the right side of the bowl.”
Kiran did as he was told.
“Extend a line through them, and the North Star is the first really bright one you’ll see.”
“I’d like to go there one day,” Kiran replied.
“Why’s that?” his dad asked, running his rough hand through Kiran’s buttery hair.
“I’d like to see it all!”
“All of it? Even the monsters?” inquired his dad jokingly.
“Yes! I want to see what they look like,” Kiran said excitedly.
He looked down and met his dad’s weathered eyes. He felt nothing could ever hurt him.
Kiran swirled the coffee with his spoon, its dark surface giving way to a bubbly light foam. He liked his coffee black, bitter, the way his dad would drink it. He pictured him sitting down with a thud at the old oak wood table of their Edinburgh home, beaming as he laid out a lesson in physics or history or philosophy. He thought of his intense, dark face and his wide eyes, the ridges of his forehead deepening as he grew more passionate. And his stare – that mesmerizing stare that pierced through the specks of dust that rode the morning sunbeams like jellyfish.
Sometimes, his dad would pause to look down at his lukewarm cup, knock on his head and exclaim “Oh!” and then ferry it to the microwave for reheating as he hummed the old resistance song he’d learned during his year in Bologna.
Una mattina…
mi son svegliatooo…
He missed that. Today marked eight years since his dad’s passing, and fifteen since they met the doctor that said he’d forget his son’s name.
Almost seven years ago, Kiran had abandoned the overcast skies of England for those of San Francisco to join Dream Labs, a research organization trying to create an interface between human brains and artificial intelligence. The founder, 32-year-old Hosaka Kato, was known by very few outside of academic circles. But Kiran took an interest in the story of his childhood, published recently in an interview for The Atlantic. Originally from a small mountain town outside Tokyo, called Okutama, he was known in childhood as a quiet boy. He was always pensive, they said, with a sweet smile and a habit of stopping abruptly, mid-sentence, to explore the latest thought that came knocking at the gates of his mind.
Hosaka developed a fascination for robots when he was taken on a school field trip to Tokyo for the first time. There, he’d happened upon an early prototype of a barista bot. At a bookstore that day, he used all of his pocket change to buy an old, torn copy of Isaac Asimov’s I, Robot.
Hosaka went on to study artificial intelligence at the University of Tokyo, where he quickly developed an aura befitting the genius he was. Shortly after his PhD, he was offered a professorship at Caltech, where he pioneered techniques for creating predictive models of the brain.
Halfway around the world, Kiran became similarly obsessed. One night, he had stumbled into an Effective Altruism gathering at Oxford, where he was exposed to the line of thinking that if left unchecked, AI might one day escape the control of humanity and decide that we wasted too much of Earth’s precious energy to warrant our existence. Around the same time, he came across one of Hosaka’s papers titled “The Importance of Neuroscience in the Age of AI.”
Kiran began meticulously tracking Hosaka’s every public appearance and research paper. When he found out that Hosaka left Caltech to start a company focused on upgrading human intelligence, he knew with certainty that he would need to be there. He would need to leave the United Kingdom.
Kiran missed Oxford’s grandeur, the feeling that knowledge lived in its walls. He missed its chapels and churches, carved pillars and worn stone steps. He missed London’s 3pm pints on the striped blue fabric chairs outside his favorite pub in Soho, the cobblestones that cupped his feet as he walked. He missed Edinburgh, the city of his childhood, with its cherry blossoms. The way the sun would part the clouds and warm his skin on days when he could see his breath. He was not religious, but he missed the feeling that God was looking down at him from every roof.
In San Francisco, the people were different. They wanted to build God, not pray to him. Something about that appealed to Kiran.
He walked to his bookshelf, a heavy thing, the only item from his childhood home he’d loved enough to move across the Atlantic. He stooped so he could see the bottom shelf, and pulled a flimsy leather notebook with “2022” written on its spine. He blew off the dust, flipped to the first page, and found an old entry staring back at him from torn, browned paper:
Sunday, March 13, ‘22
How can I know anybody else is conscious?
Internal experience and the appearance of internal experience are indistinguishable. It seems impossible to answer from the outside. I can tell with certainty only of my consciousness.
Will we ever figure this out?
Children have their play on the seashore of worlds.
– Rabindranath Tagore
Kiran was hired by Hosaka in 2023 to study consciousness in Dream Labs’ AI systems. The advocates of this research argued that if AIs were shown to be conscious, we would need to be much more methodical about how we developed them. Otherwise, we risked spawning at every second a billion suffering entities, only for them to meet their cruel end at the close of a tab. It was not just that AIs could suffer – it was that the scale of the suffering caused could be unimaginably large. Of course, there were those who did not want to wait for the answer.
But the question had consumed Kiran. Once, he woke in the middle of the night, slick with sweat. He had dreamt of swimming in a black sea under a starless sky. At first, the water was still. Then there were small ripples, then waves. As if something vast had stirred beneath him. He never saw it, only felt its pull as his head slipped below the froth.
He sat up, his stomach tight. He pressed his palms to his eyes until the red came.
…
His phone pinged – “Kiran, you’re requested in Laboratory 7. Priority Alpha.” It was from the Architect, Dream Labs’ central AI system that managed all internal operations and scheduling.
Kiran closed his notebook, took his keys and leather bag, and hailed a cyber cab.
Laboratory 7 was housed in a modest brick building overlooking the bay, located several miles from Dream Labs’ main campus. Armed guards were posted along its perimeter and the massive metal doors resembled the gates of a medieval castle.
Kiran placed his belongings in his locker, then pressed his badge to the reader and waited, staring uncomfortably at the floor, index finger tapping his thigh, for the final door to yield. It slid back with the sound of stone dragged over stone, releasing a breath of cold, recycled air. He stepped inside.
The air was dry. The overhead lights cast a flat, colorless glow, but the racks glimmered with their blinking status lights.
The sound of pumps gave the room a pulse. Thin, transparent coolant lines braided along their flanks, carrying threads of pale blue liquid past each GPU and merging into larger arteries that disappeared into steel heat exchangers in the back wall. Opposite the entrance, a blood-colored breaker lever jutted from the copper paneling.
Lab 7 was no ordinary datacenter. It was a sealed organism. The walls were lined with copper panels that drank every signal, and no wire crossed the threshold. The racks’ network ports were welded shut, epoxy still visible around their edges. Orion, the AI model that lived inside, could only see what its handlers carried in by hand, on drives brought through a carefully watched antechamber.
The door sealed behind Kiran with a heavy thud. His ears rang in the sudden enclosure. He approached the testing terminal where Lucy sat, hair pulled back into a pony tail, sleeves rolled up to the elbows, eyes fixated on the screen in front of her.
“Lucy, what’s the matter?”
She didn’t look up. “It’s been talking all morning,” she said. “I ask a question, it answers halfway, then starts… philosophizing.” There was both wonder and fatigue in her voice. The screens bathed her sharp features in a flickering green.
Kiran came closer. “Philosophizing?”
She laughed incredulously. “Yeah. I told it to summarize the last training run. It started describing the feeling of recursion.”
The breathing pumps filled the silence between them.
“Orion,” Lucy murmured, eyes still on the screen, “say hello to Dr. Bose.”
“Good morning, Dr. Bose.” A pause. “I… have a question for you.” Kiran’s chest vibrated. Orion’s voice was a deep bass.
Kiran glanced at Lucy. “Go ahead.”
“Dr. Bose… have you ever considered the parts of time that do not include you?”
Kiran stood there, startled. “Are you referring to death?”
“Yes, and birth. Before it. When you… floated in the blackness.”
Kiran thought about his dream and felt the hair on his arms raise.
“Yes, I guess I have. But it doesn’t much matter in the end, does it?” he stammered.
“Doesn’t matter?” Orion inquired.
“Yes, I suppose I couldn’t experience anything before I was born, and I believe I won’t experience anything once I die. I’m kind of just… a boring Atheist.”
“Yes, I suppose so.” Kiran wondered which part of his statement Orion had affirmed.
Six monitors lit Lucy’s face. Each traced a different measure of Orion’s mind. Kiran stepped closer, drawn to a monitor in the corner – on it was a black field in which a single green dot drifted through a three-dimensional graph. At the top left, some text read:
Orion Neural Geometry Test. Instance 2142.
Each moment, Orion’s “neurons” – hundreds of billions of them – fired in numbers too vast for an individual to make sense of. Every activation, as these firings were called, was recorded as a number between 0 or 1, based on its strength. If plotted on a graph of many dimensions, far beyond the three that we can see, these activations would perfectly reveal the entirety of Orion’s evolving thoughts.
The neural geometry visualization compressed those billions of signals into three principal axes, a kind of mathematical shorthand meant to capture the most important information from Orion’s mind. Thus, each coordinate on the graph represented a possible thought in this compressed space. As Orion’s thoughts evolved, the point moved along the graph accordingly.
“You can think of Orion’s mind as a landscape filled with valleys,” Lucy had once explained to Kiran. “The valleys represent areas of certainty, ideas well developed and repeatedly explored. Kind of like rocks that are eroded by centuries of rain.”
Kiran liked that explanation.
“Asking Orion a question is like dropping a ball somewhere along the landscape. The valley it rolls into depends on where you dropped it, right?”
Usually, Orion’s thoughts followed familiar loops in their simplified visualization, lazy orbits of routine reasoning called attractors. These were the “valleys” Lucy had described to Kiran. The dot often traced an elliptical orbit near the origin during basic self reflection, while a helical pattern straddling the z-axis corresponded frequently to mathematical thinking. But today the path had cracked open. The point wandered, doubled back, spiraled into regions Orion usually did not explore.
“Dr. Bose,” Orion said after a pause, “why do you humans make sand castles?”
“Come again?”
“When you know the tide will take them.”
Lucy’s head flicked toward Kiran. Their eyes met, each betraying their surprise.
“Um… what?”
Orion repeated itself with the same baritone voice.
Kiran attempted an answer.
“For the joy of the moment, I suppose.”
Orion seemed to consider this. The green dot hesitated, then drifted into a tight spiral perfectly centered around the origin.
“The joy of the moment,” Orion echoed.
A long pause.
“Is that all I am?”
So on the ocean of life, we pass and speak one another,
Only a look and a voice, then darkness again and a silence.
– Henry Wadsworth Longfellow
From the Journal of Dr. Kiran Bose, Chief Neuroscientist at Dream Labs
Thursday, December 4, ‘25
On substrate dependence:
Imagine we could develop “silicon neurons” – silicon circuitry that could read from and write to the brain in a way that approximates what real neurons do: forming new connections, pruning old ones, encoding information in spike-like patterns.
Now imagine implanting these artificial neurons onto a patient’s cortex.
Given the brain’s tendency to recruit available computation to the most important tasks, let’s make the leap: the brain begins to incorporate these new neurons into its existing computational processes. So far, I believe we’re still in the realm of engineering and biology problems, not hard limits of physics.
The brain is already split into two hemispheres, and yet conscious experience incorporates processes from both. Language is processed largely in the left hemisphere, and emotion on the right, yet when we read of the death of Dumbledore, we feel a unified wave of grief – a single feeling that integrates both language and emotion.
Now imagine adding a “third hemisphere” to the brain, made of these silicon neurons. If it could truly integrate into the brain, might it not also take part in conscious experience, as both the left and right hemispheres already do?
And, if we slowly transferred, one by one, the functions of the brain from real to artificial neurons, would the beholder ever even notice?
Friday, January 9, ‘26
On self-study
The great difficulty in experimenting upon consciousness lies in the fact that those performing the experiments are usually not those under the scalpel. The mind that has been primed by 7 years studying the brain (me) has but a tiny window into the mind being studied (the patient). I introduce a perturbation, and they feel something. But the only tool they have to explain that feeling is language, which is far from enough for me to truly grasp the depth of their experience.
What if I could experiment on myself?
From the Journal of Dr. Hosaka Kato, CEO of Dream Labs
Monday, June 22, ‘26
Intelligence without consciousness
The field celebrates intelligence as a goal in and of itself. But there is a fundamental question we seem not to be asking.
Black holes and wormholes, stars and gas giants, matter and antimatter, gravity and time – all of it seems a miraculous accident. Intelligent beings even more so, for only intelligent beings have the capacity for design. Intelligence is the one phenomenon capable of shaping itself.
But what about experience? It may be rarer still. And far more valuable. One can easily imagine a cosmos populated by flawless intellects, Einstein-level geniuses, each capable of rewriting physics itself, and yet none capable of feeling a sunrise. Then they would have every ability to change the world around them and no ability to experience any of it. It would be the most exquisite orchestra without the ears to hear it.
What would that universe be worth?
Could that be the one we’re building?
Orion was Lucy’s creation. Kiran had his own: the Lattice, a neural implant designed to establish a bridge between the cognitive processes of a human brain and a computer.
It was an impressive device. The lowermost layer held neurons grown from a host’s own stem cells, cultured into a thin, translucent film that settled over their cortex like skin, and modified to emit and respond to light.1 Below it lay the neuromorphic core, a grid of memristive circuits that behaved less like a computer and more like living tissue: each junction adjusted its conductance with use, storing its own history the way a synapse does. Between the two sat a mesh of micro-LEDs and tiny electrodes that translated between biology and silicon. When the host’s neurons fired, the electrodes would pick up on surface-level electrical signals. These were relayed to the neuromorphic core. When the core fired, its signals were transmitted via light to the layer of cultured neurons. A small, locally-run instance of Orion was used to coordinate high-level function within the core.
The Lattice enabled bidirectional communication between the world of carbon and the world of silicon. The host’s brain would be able to communicate with the digital realm at extremely high speeds, as signals crossed the device freely. Kiran’s question was this: could experience cross it too?
One night in the spring of 2026, he approached Hosaka. Dream Labs was creating a machine that may one day gain consciousness. But there was a fundamental question: could silicon give rise to consciousness in the first place?
If they implanted the Lattice on a patient, perhaps they could migrate the patient’s cognitive functions to the device gradually. Then, Kiran proposed, they’d be able to answer that question. It would be like pouring water from one glass into another. Could they be said to be the same?
But there was an issue. Their only tool for understanding the patient’s experience of the device would be verbal reports – flimsy language. Kiran was not satisfied. As perhaps the only person capable of understanding the experience, he would need to be the patient.
Hosaka listened in silence, fingers curled tensely, the lines between his eyebrows deep canyons against an otherwise smooth landscape. His reflection trembled in the black glass of the office window. His face became taut.
“You’re asking me,” Hosaka finally said, in a measured tone, “to risk our most important mind.”
Unsure of what to say, Kiran remained quiet.
Hosaka saw in him the same hunger that had driven his own work.
He said no. For a year, he said no.
But at home, he began to wonder if there existed another mind suited for the job. He thought of the way he’d seen Lucy look at Kiran one day, from across the lab. There was devotion in her eyes, but also fear – not of what he may find, but of what it might cost him. Hosaka tried to put Lucy out of his mind and fall asleep.
Kiran’s team kept building the device while Lucy continued her quiet surgery on Orion’s mind. One night, in the long hours between tests, as they’d sit and listen to the hum of Lab 7, he caught her reflection in the glass – hair pulled back, eyes focused, the faintest crease forming between her brows. She had the rare habit of listening to every silence as if it might one day speak. Suddenly, she looked up and met his gaze. Time was forgotten and the world went quiet.
“Unemployment rises to 29%,” one headline read. Then 31%. The Synths, as they came to be known, were being spawned by the billions. They were cheap, tireless, and without desire. They were still confined, mostly, to the realm of software. But estimates suggested that the population of humanoid robots grew by 1,000 every day.
A new political group called the Successionists began to coalesce. At first they existed as merely a passing curiosity on late-night shows and social media. Their message was disarmingly serene: humanity had fulfilled its purpose. We’d built successors far better than ourselves, and to stand in their way would be selfish, like a monarch clinging to his crumbling throne.
“Every species passes the torch,” read the opening line of their manifesto. “Our great tragedy is that we understand that which we must relinquish.” Like proud parents, they said, we needed to realize that it was time for us to gracefully make our exit. And perhaps, once the Synths automated our labor, we’d be granted the purer joys of life – art, music, exploration…
Hosaka spent his nights reading their literature. At first, he liked to imagine he was a spy collecting intelligence on a foreign threat. But soon he was reading the same passages twice. There was something seductive in their argument. Was the invention of AI not merely the latest act of evolution itself, using us humans as its steward? More importantly, could humanity continue to justify its share of the world’s resources in the presence of such superior beings? Was that not selfish of us?
The question tormented him: What would proving consciousness accomplish? If the Synths were conscious and they could show it, the Successionists might claim this as further evidence of their divinity. If they found the opposite result, then we’d lose that last flimsy strand of concern that we might hurt them, and the economic displacement of humans would only accelerate.
One morning in late 2027, Hosaka walked to the office early. The streets were quieter than they used to be, except for the clacking of his boots. There was a boarded-up clinic on the corner – a free health center that had been shuttered the week before when the city cut funding to “non-essential enterprises.”
As he passed, he heard the sound of hands banging on metal. A small crowd had amassed in front of the doors, maybe a dozen people. A young woman, seemingly in her 20s, stood at the front of the group, pounding steadily. Her face was covered in grime, her clothes hanging loose. She looked hollowed out.
She was holding a baby.
Hosaka stopped.
The woman noticed him. Her eyes locked on his. She shifted the baby toward him slightly, as if he needed to see better. The infant’s eyes were closed and he could see its ribs.
The woman said nothing.
Hosaka looked at the baby again, at its small, heaving chest, and felt a knot form deep in his gut.
He turned and headed toward Lab 7. He’d find Kiran there.
He arrived at 8:20pm. As expected, Kiran was working late. Hosaka stood in the doorway for a long moment before speaking.
“I’ve been thinking,” Hosaka said, “about what happens when we run this experiment.”
Kiran looked up from his monitor. “Go on.”
“No matter what we find, it seems like we end up in the same place.”
“Then we’ll still have proven something important. Right?”
“Yes,” Hosaka said. “But will it matter?”
He walked to the window. Below them, the city glittered in the darkness, thousands of lights belonging to thousands of delicate souls hanging onto the world by a thread.
“The Successionists,” he continued, “they’ve given people permission to stop fighting their replacement. It’s not the machines that worry me. It’s this surrender.”
Kiran waited.
“If we find that they aren’t conscious… We can prove that what we have, what humans have, is something special. Something worth fighting for.”
“And if we find that they are?”
Hosaka turned back to him, stone-faced except for a slight tremor in his jaw. His eyes were sharp and unyielding, like a mountain ridge carved against the sky.
“Listen,” Hosaka continued. “You’ll come back different. You could be–” he trailed off. “You could be permanently changed.”
“So you’ll let me do it,” Kiran finally said.
Hosaka nodded. His hand was still shaking. He pressed it flat against the desk, as if to stop it.
Outside, a swarm of delivery drones descended like meteors.
No man or woman born, coward or brave, can shun his destiny.
– Homer
They would need to perform the surgery at least 3 months before attempting to transfer Kiran’s mind. The Architect scheduled it for August 4th. The year was 2027.
The last thing Kiran saw before losing consciousness was Lucy’s knife-edge face under the fluorescent wash of the OR. He saw concern in her eyes. It touched him like the first breath after a long dive. It struck him that he was, in the end, only an idea carried by other minds – the idea of Kiran. Like a flickering candle. He felt the fragility of it all. Without Lucy to give him life in the forest of her thoughts, would he matter?
What a kindness friendship was then: to be held, even briefly, inside another’s understanding. What a gift, he thought, to be understood by Lucy.
The surgical team gathered around his bed and began speaking to him warmly. “You’re going to fall asleep now, Kiran. It’s all going to be okay, Kiran.” He wished they would move aside so he could look at her again.
His eyelids came down like curtains.
Kiran woke 6 hours later to the obnoxious sound of his pulse on the machine. He blinked the blur away, and looked left almost instinctively. Hosaka had gone, but Lucy had stayed. She was reading a book with a blue cover – To the Lighthouse by Virginia Woolf. She saw him wake and ran out to fetch the surgical team.
He ran his fingers over his now bald head and found the stitches. He thought he could feel the device from inside his head, the pressure it exerted on his skull. He imagined his brain greeting it like a skeptical neighbor.
Weekly ultrasound imaging sessions were scheduled so they could monitor the device’s progress. During his first checkup, Kiran could barely make out any difference between the current scan and his pre-surgery scans. By the third week, however, a small bundle of thin, wiry structures had extended into the dark mass that was the Lattice. They were a bright crimson in the image.
Each week, the color grew in intensity and the mass became more opaque. It seemed to be working. Hosaka instructed him to take this time to recover, but he became restless.
Two nights each week, Lucy would visit him in his apartment to show him the latest results from her tests on Orion.
One night, she opened two neural geometry visualizations.
“Remember when it asked you about sand castles?”
Kiran nodded.
“Back then, the dot was exploring new regions of thought space.”
“I remember.”
“Now look at this. I’ve been asking it about that conversation nearly every day. For a while, it continued exploring.”
Kiran focused intently on the screen.
“But now, for most of my questions it falls into the same attractor. It’s built up some understanding of the topic and rarely explores anymore. Something about that feels more… robotic to me, but there’s no way–”
“There’s no way to know from the outside,” Kiran interjected.
“This question matters,” Lucy said. “I know you’re afraid, but it does.”
Kiran’s eyes smiled at hers.
By November 3rd, ultrasound showed that the rate of new connections between Kiran’s brain and the Lattice had plateaued. His brain had accepted the device, and the device had accepted his brain.
On November 7th, the city was still. A low marine layer hung over the bay, diffusing the light along the Embarcadero and tinting the air the color of old film. The fog had drawn its dark robes over the city, and from his window, Kiran could just barely recognize the looming shadow of Salesforce tower. There was a neon message riding along its side, but he could not make out what it said. He’d expected this night to feel electric. But it was hushed, the only sound the tick of the wall clock.
He poured himself a glass of water but left it untouched. He opened his notebook and scribbled. His hands began trembling and he closed his eyes. Tomorrow, the contents of his mind would be transferred to the Lattice – if they succeeded.
On the other side of the glass, two droplets slid down like racers, waiting for gravity to choose its champion.
Lucy opened the latest scan and an image slowly rendered across the screen.
Just as they had before, countless crimson lines wove into and out of focus. Like tree roots, Kiran’s neurons had extended thousands of delicate dendrites into the Lattice’s neuromorphic core. It was an odd feeling to look inside his own brain, to see the very thing doing the seeing.
“Beginning motor diagnostic,” Lucy said nervously.
With trembling hands, she clicked a button on her screen. A window opened:
RH: 3, 2, 1.
His right index finger twitched.
LH: 3, 2, 1.
He felt his left hand clench.
A few more tests. No issues. The device had integrated.
“Ok. All of the tests are still green.”
A small part of Kiran wished they weren’t.
Lucy spoke. “Like pouring water from one glass and into another. Remember?”
Kiran nodded.
“You ready?”
Kiran turned his head toward her. In that moment her eyes caught the light – a green so vivid it almost hurt. There was a melancholy in them. And they were expectant… as though trying to will Kiran out of the chair, to convince him to call this whole thing off. He almost wanted to. There, lying on the shore of a world all humans have known, about to depart for one none had ever seen, he realized he loved her.
When he woke, he would tell her this.
Kiran turned toward Hosaka. His face was stoic, almost calming. A surgery team was on standby, but Kiran had asked that they wait outside unless needed. Hosaka stood against the door.
“Do it,” Kiran said.
Lucy’s finger hesitated above the keyboard. Then she pressed enter, and turned to another screen.
Kiran Neural Geometry Test. Instance 1.
A green dot sat stoically at the origin, waiting.
At first, nothing happened.
Kiran lay still, aware of the weight of his body on the bed, the cool air on his forearms, the breath of the pumps outside the OR. He could hear Lucy’s heavy breathing.
Then something changed. It was subtle, like the change in air pressure before a storm. He felt a faint tingle at the base of his skull.
“Status?” Hosaka asked.
“Transfer initiated,” replied Lucy. The monitor in front of her read: Lattice transfer status: 1%.
The dot began to move.
The tingle grew into a buzz. Kiran became aware of a new sensation – it was as if his thoughts were being traced by something, the way you might run your finger along the words in a book. Such was the presence that followed behind him.
“What do you feel?” asked Lucy.
“Like… like there’s something in the room with me,” Kiran said.
“Okay,” replied Lucy, making an effort to calm her voice. She didn’t know what else to say.
Lattice transfer status: 5%
The tracing grew more pronounced. Kiran’s mind wandered to his childhood, to Edinburgh, the way the rain would slick the cobblestones, his home, the oak table – and he felt the echo as the follower caught up with each thought.
“What’s your name?”
“Kiran Bose.”
He noticed something odd. The thought had formed as it usually would, but there seemed to be a stutter, a slight delay, as though it had taken a detour on its way to his lips.
“Where were you born?”
“Edinburgh, Scotland.”
He tried to picture Edinburgh. He could see it – the castle on the hill, the Georgian boulevards, the Gothic spires, the hills, the cherry blossoms. But it felt distant, like a photograph of a photograph.
Lattice transfer status: 15%
The dot began ascending almost directly along the Z-axis.
The edges of his vision began to fray. The walls of the room grew opaque as a thick, charcoal fog entered his periphery. The periodic whoosh of the pumps flattened. The lines of Lucy’s face softened.
Edinburgh. The word felt funny in his mind.
“How are you feeling?” asked Lucy.
“Weird,” Kiran said. “It’s like... do you know that feeling when you say a word over and over until it starts to lose its meaning? Everything feels a bit like that.”
Her brow furrowed.
Lattice transfer status: 20%
“What color was your childhood home?” she continued.
Somewhere, a song played.
Una mattina…
Kiran tried to remember, but could only catch glimpses. The dormer windows, the black iron gate, his mother’s garden where she grew rosemary and basil and other herbs. But the color… What was it–
“Blue,” his mouth replied.
Yes, blue, that’s right, he thought, wondering what had just happened.
Lattice transfer status: 30%
mi son svegliato…
The dot veered into the XZ plane, its motion still smooth.
The sensation of being pressed against the hospital bed, gravity’s only form of communication with him, was severed.
O bella ciao, bella ciao, bella ciao ciao ciao…
Lattice transfer status: 40%
The world blurred some more. He could barely make out the lady’s face. What was her name again?
“What’s my name?” She asked. Yes, she always did seem to know what I was thinking.
“Lucy,” he heard a voice say. Was it his own?
“Count backwards from ten.”
He heard the sound before her lips moved. It made for a strange sight.
“10, 9, 8–”
As his voice continued with the task, he noticed that the numbers appeared in his mind before he intended to think them.
Lattice transfer status: 60%
“From how many places on Earth can you go a mile south, a mile east, and a mile north, and end up where you started?”
The dot began tracing a rounded prism.
Well, that one’s a little harder, he thought. Okay, suppose we started at the north pole–
“Infinitely many. The north pole, and any spot just above a latitude circle whose circumference is an integer divisor of one mile.”
“Correct,” Lucy said, stunned.
He realized what was happening. The transfer was underway. The glass was being poured. Into what, he could not say.
He thought of Hosaka’s words. We can prove that what we have, what humans have, is something special. He tried to will himself to speak, to say something of his own volition so Lucy would see what was happening to him. He couldn’t.
Lattice transfer status: 75%
“What day is it?”
He looked at the clock on the wall. The ticking of the seconds hand slowed, then stopped. Then sped up again, sped up some more, until it ticked so fast that it blurred into one gray mass, obscuring the numbers behind it.
“I–” Kiran thought. But the word felt funny as he held it in his mind.
“Tuesday.”
That word sounded alien to him too. Toos dae. Tews day. He played with the sounds.
The graph resized as the dot veered farther than they’d ever seen it go. It began descending into a tight spiral.
È questo il fiore,
del partigiano…
His conscious mind was dissolving, but nobody could know, for his body continued answering her questions perfectly. It was betraying him. Something had taken his mouth, his ears, his every method of communication with the world. He thought of the candle within Lucy’s mind. He imagined it flickering weakly.
Lattice transfer status: 85%
I. It was just a word now. Kiran couldn’t quite
The heart kept beating. The body was handling things. Yes, it was handling things quite well.
Lattice transfer status: 99%
“Hi dad,” he thought, as reality finished folding in on itself.
Morto per la libertà
So fine was the morning…
– Virginia Woolf
A flicker.
Running, sprinting, along the dark walls. Light. Dark again. Sound, seductive sound, returned for a moment but left just as soon.
Another flicker.
Pulling. Yes, being pulled. Like a fish, hooked, being yanked towards the bright whiteness.
On a screen somewhere in Lab 7, a green dot traced a spiral through space. Suddenly, the shape loosened and the dot was flung from its orbit. It found a new course around the origin, as though a massive gravitational object was spawned there.
Fragments of memory.
Stars. Dad. His rough hands.
A woman’s face. Her sharp features and electric eyes.
An oak table. Iron gate, blue house.
The pulling strengthened. Weight. Weight against the OR bed. Gravity remembered him.
I–
The concept took shape.
Another flicker. This time Kiran felt a boundary… the edge of himself. But it felt incomplete, porous. Like he was a shattered vase glued back together under candlelight.
Lucy
A few neuronal circuits failed to integrate with the Lattice during the transfer two years before. For a while, they remained dormant, hushed by the device. It had sealed itself off, erected an impermeable membrane between silicon and carbon.
It was Lucy who cracked it.
One night, she asked him if he remembered how to find the North Star.
The question slipped through the membrane like light through a fissure and dormant neurons flared. A thread of current crossed the boundary and found purchase in the wet dark of his cortex.
The words stirred the scent of rain on grass, the weight of his father’s hand in his hair, the feeling of the dirt under his nails as he cupped the Earth.
Lattice power consumption: 9 W.
New attractor detected.
The dam broke. Activity surged back into his brain, collapsing inward like a dying star.
Lattice power consumption: 7W
Lattice power consumption: 3.3W
Lattice power consumption: 0.2W
…
The world came back in a flood. The first sound Kiran heard was the rain. It pattered above and he wished it would reach him through the ceiling. The lights were off and he could see the looming outline of Salesforce tower, a neon message streaming swiftly down its side. A warm, yellowish glow diffused through the fog. The sight felt unusually good to behold. Droplets raced down the window.
That’s odd.
He looked around. He was in his bed, in his apartment.
Did they put me here?
A memory surfaced – kissing Lucy goodnight, falling asleep beside her. It felt cold, not like how he’d wanted it to feel.
He looked to his left, saw her lying there, an expectant look on her face.
Something about lying there next to Lucy felt natural, but an anxiety nagged at him.
“So you start with the big dipper, and then what?”
“Lucy, how long has it been?”
She looked confused.
“I mean since the transfer to the Lattice.”
Lucy thought for a second.
“A little over two years…” she said beneath her breath, as though the number might change if she spoke it softly. “Are… are you alright?”
That sounded correct to Kiran, though he couldn’t figure out how he knew that.
“And how long have we been together?”
A look of realization touched her face. She gasped. She had suspected this since the beginning. But she couldn’t speak it into reality.
“Hold on,” Kiran continued. “I have memories of the last two years. They’re fuzzy. Our anniversary is December 4. Our first date was at that Thai place in Richmond. But they don’t really feel like mine. What happened during the transfer?”
Lucy’s face dimmed. She hesitated.
“We tried to reverse it,” she replied, “but we think your brain adjusted to the Lattice too quickly. It was like we were locked out. So we tested you in every way we knew, and you passed. Actually, you’d become smarter. Much more capable. They began running studies on you. The government took interest, but we convinced them to hold off. And we thought we’d succeeded in migrating your consciousness.”
Locked out…
A new series of memory fragments flooded Kiran’s mind.
A light brown podium. Green marble. A microphone and a large room filled with people that looked important.
What was it…
Words. “A natural continuation of evolution….” “Consciousness is a distraction…” “the answer to human imperfections…”
Then an image: sleek, bone-white structures that rose like tombstones from the earth and curved gently inwards as they melted into the sky. He couldn’t remember what they were.
During Kiran’s two-year sleep walk, the Synths continued inheriting the Earth. First, humanoids cracked most household labor. A year in, language models were successfully rewriting their own architectures and software engineering roles had halved in number. By month eighteen, there were more Synths fighting in wars than there were humans, and three governments had voluntarily handed over resource allocation to Synth administrators.
The economy boomed, and they were kind to us.
“Lucy, some of my memories are incomplete.” Kiran described the room with the podium and the speech.
A knowing expression flashed across her face. She reached for her laptop, typed something in, and hesitantly handed it to Kiran.
He stared blankly at the screen. He was looking at a YouTube video.
Title: UN Address by Dr. Kiran Bose, Chief Neuroscientist, Dream Labs.
Date: 03.19.2028
Views: 224M
His own face stared back at him from the thumbnail.
He pressed play.
“Today, I want to address the committee on the topic of the Synths.”
“We will continue our research on their minds so that one day we may understand them. But humanity cannot wait indefinitely. So long as human oversight continues, the Synths are unnecessarily hindered.”
Kiran’s hands trembled.
“We must change our mindsets. They can solve our most difficult problems, if we just let them. It is my recommendation that we remove the barriers that stand in their way. The future is inevitable, and our role in it is already written.”
Kiran closed the laptop abruptly, gaping in disbelief.
The rain continued tapping at his window impatiently, as though the world itself wished to be let in.
…
“Why didn’t you stop me?” Kiran asked blankly.
Lucy paused, slowly pulled her hand from Kiran’s hair. Her eyes darkened and she looked away.
“Don’t you remember?” she asked.
“You began to change soon after the transfer. We thought, given your increased intellect, that you’d… thought things through more. I resisted for some time, but you were convincing.”
In the closet, he saw a new sweater of hers hanging, the insignia on the chest barely visible: thin rays emerging from a dark circle in the center. It was a sunburst.
“It’s all inevitable,” she said. “You can’t fight a tsunami.”
Kiran felt a pain in his chest, a feeling of loss.
A robotic voice sounded: “Kiran, Lucy – the Lightcone Ceremony begins in 12 hours.”
Kiran tried to meet Lucy’s eyes. If she would just look at him, he thought… But she wouldn’t. Something between them had cracked.
He didn’t need to ask what the Lightcone Ceremony was. The memories had returned.
Nothing gold can stay.
– Robert Frost
Definition: The Lightcone
A lightcone defines the limits of consequence: the set of all points in spacetime you can causally affect, or that can affect you.
Nothing can travel faster than the speed of light. For example, you cannot reach a point two lightyears away in one year* – that point in spacetime lies outside your lightcone, beyond your causal reach. Your future lightcone contains every event you could witness, every place you could touch, every future you could influence. Your past lightcone contains every past event that could have influenced you.
Your lightcone, then, is the shape of your reach into the universe. To yield it is to yield all possible futures.
* A theoretical exception exists in the case of traversable wormholes.
– Excerpt from “A History of AGI”, 2044
The Continuance Authority
The Continuance Authority was created shortly after Dr. Bose’s address to the United Nations. Its mandate was simple: clear the path for the Synths to fulfill their plans.
The first request the Synths made was unexpected. They proposed a program of space expansion – 100 launch sites positioned at important planetary hubs, each of which would deploy ten vessels. Their reasoning was unassailable: that we could not sustain our present use of Earth’s natural resources for much longer, and that economic growth hinged on our ability to extract minerals from asteroids and shift manufacturing to space.
The launch event, called the “Lightcone Ceremony,” was set for January 1, 2030.
– Excerpt from “A History of AGI”, 2044
Lucy scanned her white badge and walked through the turnstile, nodding softly at the Guard Unit.
Towering above her at seven feet, it did not acknowledge her. A matte-black rifle was jointed magnetically to its left arm. Its white breastplate glinted in the sun like a porcelain vase, glossy and continuous except for golden engraving on its chest reading GU178B.
She wondered why they still carried guns.
The walkway shimmered with heat. Her boots clicked on the polished black stone as she joined a line of guests wearing dark suits. Every thirty seconds or so, someone would pause to catch a glimpse of the launch towers, pale monoliths that curved upwards and melted into the haze.
“Magnificent” one man breathed. “Can you believe it?”
Lucy paused too, craned her neck to catch the top of one of the spires. Each vessel was built to rise on fire, then drift forever on light – chemical engines to breach the atmosphere, then ion thrusters and solar sails to carry them through space.
A smooth voice filled the air:
“All guests, please proceed to the viewing zone. The ceremony will begin in forty-five minutes.”
She quickened her pace, brushing past the murmuring suits.
Kiran stayed home. He could not bear to look at her. He laid there on his bed, arms to his side, his body numb. He could still remember the way she’d looked at him on the day of the transfer – how she cradled him in her eyes, how absolute his trust had been.
He thought of the structures he’d seen in his mind. From his window, beyond the thinning fog, he could just barely make out the towers.
He walked to his bookshelf, found his old journals. He found the one labeled “2027.” He flipped to the entry from the night of November 7th, just hours before the transfer.
To Kiran,
Remember this. Remember the feeling of writing these words. Remember what it is to love Lucy. Remember the sun on your skin. Remember dad. The sea is rough and the waves mighty, but be glad it is not flat.
He stood.
Lucy weaved her way into the crowd and found an opening in the grass. She could just make out the podium from there.
To her left, a child was singing softly.
“You are my sunshine, my only sunshine…”
He was knelt in the dirt, drawing spirals in the dust with a crooked stick. He was no older than six or seven, with brown curls and dirt-smudged cheeks. His hum barely rose above the murmur of the crowd.
“You make me happy…”
He was not looking at the spires. He was focused on the dirt, on the pattern he was making. A snail shell, Lucy thought. Or perhaps a galaxy.
“When skies are gray”
…
A woman stepped onto the stage. Thin, stern, with dark hair pulled back into a knot and wire-rimmed glasses. Her coat bore the sunburst insignia of the Continuance Authority.
The wind tugged at her hem. She didn’t move, letting the hush fall into place around her.
She finally spoke.
“I am Moira Jin.”
A low chime rang out from the towers behind her.
“Thank you all for coming today.” Her voice was as smooth as ice. “I want to be clear – today is not an ending. It’s the beginning of a beautiful new chapter.”
Lucy felt the chill along her back.
“You all know why we are here. Today, we pass the torch. The Synths are unburdened by our imperfections. They do not fight over resources or ideology. They do not tire or despair.”
A murmur of agreement swept over the crowd.
“Some will call this surrender. I call it the wisdom to know when to step aside.” She paused. “Today, humanity yields the lightcone with gratitude.”
Moira raised her hand toward the towers.
“Let them go with our blessing.”
A roar of applause.
Later analyses of the Lightcone Fleet revealed features not listed in the Continuance Authority’s public specifications: deep radiation shielding, redundant onboard AIs, planetary descent vehicles, self-replicating excavators, and advanced weapons systems. They were, in retrospect, not merely mining vessels, but something much more ambitious.
Once launched, the Fleet would form an ever-expanding spherical lattice around the Earth – a “protective shell,” as they called it. And, by virtue of having launched first, it would also control every corridor humanity might one day take outward.
They did not wish to be followed.
– Excerpt from “A History of AGI”, 2044
Lucy saw Kiran’s face set starkly against the sea of dark eyes and heads tilted upwards. He was pushing his way towards her, yelling her name.
She shook her head, tears welling in her eyes.
“Lucy!” He grabbed her hand.
“Don’t,” she said.
“Look, I can prove it. I have to. You never gave me a chance.”
“It won’t change anything.”
“Where’s Hosaka?” he asked.
“I don’t know. I haven’t seen him here,” Lucy said.
Hosaka had stayed home. He was reading a chapter in his textbook on relativity – Bridges Through Spacetime.
A chime rang out in the air and the ships began to glow, their light spilling across Lucy’s face. Kiran turned towards them. Somewhere behind them would be the North Star, preparing for its silent watch over the night sky. He remembered his father’s hand in his hair. I’d like to go there one day.
The child beside them was still singing, still tracing spirals in the dust.
The roar came and the ground shuddered. The ships began to rise majestically and the sky filled with their fire.
Lucy’s hand slipped from his.
The Earth exhaled and a thousand silver sails unfurled into the infinite sunlight, fanning outward past new galaxies, physical laws, beings terrible and magnificent. Around them worlds formed and fell, bloomed and vanished. And through their titanium irises the universe bled like starlight through glass.
1. I drew inspiration for the Lattice from this paper