2026-07-08 21:51:44
After HAMi became a CNCF Incubating project, I want to talk about something overlooked: AI is shifting the scarce resource of open source communities from code to consensus.
On July 2, 2026, HAMi officially became a CNCF Incubating project (see the announcement). For an open source project, this means more than recognition of its technical capability; it means that community governance, ecosystem building, and real-world adoption have all entered a new phase.
But if you only read HAMi’s growth as “a GPU virtualization project succeeded,” you might miss the more important shift.
My time building the HAMi community has left me with one increasingly strong feeling: AI is changing how open source communities produce. As AI coding drives the cost of producing code lower and lower, the core of competition in open source will no longer be who wrote the most code, but who can build stronger technical consensus, attract more contributors, and form a sustainable ecosystem network.
This article is my attempt to lay out that argument clearly.
Let’s start with the data. Here is where HAMi stands today:
| Metric | Value |
|---|---|
| GitHub Stars | 3,700+ |
| Contributors | Nearly 500, from 27 countries |
| Participating organizations | Multiple, and growing |
| Release cadence | Once every three months |
I’m not listing these numbers to show off “growth metrics.” I’m making a different point: an open source project is shifting from “software maintained by a team” into “a technical community of people gathered around a shared goal.”
That distinction matters. Software can be forked, rewritten, or generated overnight by AI. But a community with a shared goal, trust, and rhythm cannot be forked. That is the irreplaceable asset of an open source project.
From a governance-maturity perspective, HAMi has passed through three milestones:
Note the middle stretch: from entering Sandbox in August 2024 to reaching Incubating in July 2026, roughly two years. During those two years the code certainly grew, but what actually convinced the CNCF Technical Oversight Committee (TOC) was the diversification of the community, the formalization of governance, and real production adoption. None of that is something you produce by writing code.
HAMi was first open-sourced in 2021. Back then, most developers did not see AI coding the way we do today. Whether an open source project survived depended on developers genuinely investing their time, discussing problems in issues, submitting code through pull requests, and building trust through code review.
Today, the environment has changed.
In a recent HAMi community livestream (Mastering HAMi DRA, Yang Shouren, HAMi Community Livestream Episode 2), someone asked the maintainers a question: “How much of HAMi’s code now comes from AI assistance?”
The answer from Yang Shouren stuck with me: about half of the code in the HAMi community today is already AI-assisted.
Half. And that share is still rising.
This immediately raises a sharp question: if AI can write more and more code, what is the value of an open source community? Could one person plus a few AI agents just fork a “new HAMi”?
My answer is no, because what AI lowers is the cost of producing code, not the cost of building technical consensus.
Let me sharpen that point.
An AI agent can already do a lot today: write code, fix bugs, add tests, generate docs, produce migration scripts. These capabilities are getting stronger fast. But there are a few things in a community that AI cannot replace today, and in my judgment will not replace soon.
First, deciding which problems are worth solving.
Take HAMi. Why does GPU sharing matter? Not because “slicing cards finely” is a cool technique. It matters because once AI infrastructure scales, reality looks like this: GPU costs are enormous, heterogeneous hardware keeps multiplying, and Kubernetes’ native resource model is no longer enough.
The community has to first agree that “this problem is worth investing in” before anyone writes any code. That agreement is a human-to-human matter, supported by real scenarios, real costs, and real pain. AI can solve a problem you have already defined, but “which problem is worth defining” is decided by community consensus.
Second, choosing a technical path.
In GPU virtualization there are many paths: MIG, MPS, time-slicing, vGPU, DRA. Each has trade-offs, and choosing wrong can cost you two years of detours.
Code can be generated, but architectural choice is fundamentally a value judgment. HAMi’s decision on Ascend 910C to move from hardware SR-IOV to userspace HAMi-core was not about someone writing a better piece of code; it was about the maintainers holding to a judgment that “hardware partitioning is too coarse, software partitioning is more flexible.” That kind of judgment is ground out through repeated discussion, failure, and validation in the community, not prompted out.
Third, trust.
Users don’t choose HAMi because of “how much AI-generated code is in this repo.” They care about: who maintains it? Who reviews it? Are there real production cases? Does the community respond when something breaks?
Each of these is a relationship between people, a product of community organization, not a product of code quality.
Put these three together, and the production model of open source communities in the AI era is shifting:
In the past, code came first and the community sedimented out of the code; in the future, consensus comes first, AI rapidly turns consensus into code, and code flows back to test the consensus. The center of gravity of the scarce resource moves from “code” on the left to “consensus” on the right.
Since AI has become a new category of contributor, a community’s governance rules have to keep up.
My advice is: don’t treat AI merely as a tool, treat it as a new type of contributor. It used to be “developers write code, humans review”; in the future it will be “humans set intent, AI generates code, the community reviews, and shared knowledge is distilled.” There is an extra layer in the middle, and an extra layer of governance complexity.
HAMi is already responding to this. Its CONTRIBUTING.md is explicit:
If you are using any kind of AI assistance to contribute to HAMi, it must be disclosed in the pull request.
In other words, if you use AI to help with a contribution, you must declare it in the PR. But the community also knows that a norm without a gate is not enough (see the discussion in Issue #1998), and there is already ongoing discussion about how to give that norm real enforcement.
This is actually a problem every AI-era open source project will run into. I’d break it into a few questions:
Whoever figures out and operationalizes these rules first will keep contribution quality stable in the AI era. HAMi’s exploration here is worth a look for every open source project.
Back to the promotion itself.
I’d lean against treating it as an “honor.” What CNCF Incubating really validates is not code quality, but whether a project has the capacity to become infrastructure. It examines a whole package: technical maturity, community governance, production adoption, and ecosystem building.
HAMi’s case, in one sentence, is not “a Chinese team built a GPU project.” It is this:
An AI Infrastructure community, jointly shaped by developers from around the world, is taking shape.
The first is a product story; the second is an ecosystem story. The Incubating recognition from the CNCF is recognizing the latter, because competition over infrastructure is never competition between individual products; it is competition between ecosystem networks.
The open source competition of the next decade will not be just a competition of code, but a competition of communities.
Once AI gives everyone near-infinite capacity to produce code, the truly scarce capabilities will be three: finding the right problem, building technical consensus, and organizing developers worldwide to solve a problem together.
HAMi’s path to CNCF Incubating is just one snapshot of how open source communities are evolving in the AI era. Code will keep getting cheaper, and consensus will keep getting more expensive. Whoever understands this inversion will be the one who can build open source communities with real depth in the AI era.
jimmysong) or follow HAMi on GitHub to join the community focused on GPU virtualization and heterogeneous compute scheduling. Let’s talk about open source governance in the AI era.
2026-06-24 22:30:00
HAMi used to save cards in the cluster. Now it decides how good a desktop AI workstation feels.
A few days ago an old friend came by to chat. We used to run the cloud-native community scene together back home, so we go way back. She recently joined a company called Olares, and somewhere in the conversation she dropped this: their project had integrated HAMi.
I run the HAMi community day to day, so whenever I hear someone using it for something, I want to take a closer look. I went and dug through the Olares repo and website, and my first reaction was, huh, this is actually interesting.
Isn’t this exactly the kind of local AI workstation I’d been eyeing forever but never pulled the trigger on? I wrote in My Personal AI Stack that for someone like me who mainly works with my head, subscribing to models beats maintaining a high-end GPU. But how far can “one machine running the full AI stack” really go, and who has actually built it, I’ve wanted to see with my own eyes.
Olares isn’t a “NAS with a GPU bolted on,” and it isn’t a “mini PC with Ollama installed.” It’s more like a personal cloud built on Kubernetes: local models, apps, identity, remote access, dev environment, storage, even GPU governance, all packed into one machine as a desktop cloud OS.
HAMi isn’t filler here. It’s the layer that turns “one card” into “a resource pool you can share, isolate, and schedule.”
There’s no shortage of things calling themselves AI workstations, but most are just beefier mini PCs. Olares is different. It’s genuinely designed as a cloud.
The company positions this machine outright as a 24/7 personal AI cloud, not a PC you sit in front of. That single framing explains almost everything that follows: why it has to use Kubernetes, why it cares so much about network traversal and remote access, why GPU governance suddenly matters here.
Two other details tell you a lot: it supports Thunderbolt 5 external eGPUs, and two machines can cluster up. In other words, it was never a sealed box. It starts as a single node and grows upward.
Architecturally, what Olares does is compress an entire cloud runtime into a single machine: auth and authorization, app lifecycle, tunnels and traversal, secrets, observability middleware, not a layer skipped.
This isn’t “a few AI apps preinstalled.” It’s a complete cloud runtime stuffed into a desktop device.
The diagram below is my own re-drawn layering based on its public docs, with the HAMi layer pulled out explicitly because it’s the point of this whole piece.
One thing jumped out while I was reading: Olares’s docs haven’t kept up with its own product.
Its 2025 architecture page still says nvshare, noting that GPUs are limited to one card per node. But flip through the release notes and 1.12 already integrates the HAMi scheduler, with exclusive, time-slicing, and memory-slicing modes all there; 1.12.2 adds multi-GPU; 1.12.5 supports DGX Spark outright and rolls automatic scheduling across all three modes.
The product is outrunning its docs. That alone tells you something: it’s shifting from a “personal cloud OS” toward a “local AI cloud OS.” That’s also why I wanted to write a whole piece on it.
HAMi is a CNCF Sandbox project, positioned as a heterogeneous AI compute virtualization middleware. On the path there’s a Webhook, a scheduler, a Device Plugin, and HAMi-core handling in-container resource control. I summed it up in one line in Kubernetes Is Becoming the GPU Control Plane of the AI Era: it turns GPU slicing from “a hardware capability” into “a control-plane capability.”
On Olares, that’s the real thing behind the GPU mode toggle in settings. On the surface it’s a UI option; underneath, it’s a policy switch on the resource plane.
The most critical piece is almost certainly HAMi-core, which in one sentence: intercepts CUDA calls inside the container and does memory virtualization, compute throttling, and utilization monitoring. It doesn’t slice with hardware. It manages with software.
The diagram below puts that injection path and the three GPU modes side by side.
The evidence lines up too. In Olares’s GitHub issues you can see HAMi’s libvgpu.so crashing under WSL2, the discussion mentions it getting injected into every process via /etc/ld.so.preload, and the Olares team itself says this implementation “drew heavy inspiration” from the HAMi project.
One thing I should be clear about, though: whether Olares uses upstream HAMi-core directly or maintains a forked version, there’s no single official answer in public. I won’t fill that in for them.
The three modes make sense in order. Full-card exclusive is for heavy loads; time-slicing lets lightweight services take turns, and Olares even swaps inactive models into memory first, with roughly 5% switching overhead; memory-slicing cuts the memory into fixed quotas so multiple apps run together. On something like DGX Spark, where CPU and GPU share memory, it defaults to memory-slicing because there’s no traditional memory paging in and out to begin with.
HAMi used to live mostly in big clusters: multi-tenant, inference serving, mixed training-and-inference, heterogeneous cards. NIO running it for sharing across 80 nodes and 600 cards is the textbook example. That line is well-trodden.
What feels new about Olares is that it moves the same problem onto a single machine.
What you actually run at once on Olares is never just one Ollama: a local model, a chat UI, a research agent, an image or video pipeline, plus a pile of OCR, speech-to-text, and automation tools. In a scenario like that, without a GPU scheduling layer, the GPU always degenerates into “whoever starts first grabs it.”
So for HAMi this matters. It goes from “a tool that saves cards for platform engineers” to “something that directly decides whether an ordinary user has a good time.” In the cluster it manages utilization and cost; on Olares it manages concurrent experience, model switching, and whether this machine is actually pleasant to use. HAMi doesn’t only talk to platform engineers anymore.
My sense is, edge AI won’t settle at “a stronger local model box.” It’ll grow into a full stack of “control plane plus model plane plus application plane.”
NVIDIA pushing DGX Spark and RTX Spark onto the desktop has already moved the story from “run models locally” to “run agents locally.” Olares’s CUDA-plus-x86 is one path, DGX Spark’s unified memory is another, and below that there’s the DIY mini-PC-plus-eGPU route you assemble yourself. Interestingly, Olares lists eGPU and dual-node as supported paths itself, so the line between appliance and DIY isn’t that sharp.
But my guess is, the one that breaks out won’t be the one with the most ferocious specs. It’ll be the one that fuses resource governance, dev experience, app ecosystem, security, and remote access into one closed loop. If desktop AI workstations really become a category, what people compete on isn’t GPU and memory, it’s the control plane.
I went ahead and added Olares to the AI Native Landscape I maintain, so I can keep watching how it grows.
In one line: Olares is a desktop AI OS with Kubernetes at its core, and HAMi is the layer that turns its single-machine GPU into a shareable, isolatable, schedulable resource pool.
From cluster to desktop, HAMi’s story is expanding from “saving cards” to “making edge AI usable.” If desktop AI workstations become a real category, what people compete on is the control plane, not single-card performance.
2026-06-20 13:43:26
In a bamboo-weaving workshop in Anji, holding a strip of bamboo split hair-thin, I realized for the first time: a bolt of cloth, a machine, a country, all begin with this single thread in the hand.
This is something I’d been thinking about for a long time, without ever quite sorting it out.
In February this year, our company offsite took us to Anji, in Zhejiang province. It’s famous bamboo country in China, the mountains covered in moso bamboo, and the locals have been weaving with it for generations. Bamboo weaving is a national intangible cultural heritage. The offsite included a hands-on session for us to try it ourselves.
I hadn’t expected much from this kind of “experiential activity.” But once I actually sat down, picked up a bamboo strip, and listened to the old master explain how to lift, press, and raise the threads, my mind started to wander. Up and down, lift and press, the warp and weft interlacing in his hands until a pattern slowly emerged. The motion was deeply repetitive, a fixed rhythm, a clear logic.
And in that moment an almost absurd thought popped into my head: this is just 0 and 1.
A warp thread raised is 1, pressed down is 0. Unfold a piece of cloth and it’s a two-dimensional structure made of 0s and 1s. And the thing the old master kept referring to as the “flower program” (花本), the pre-designed sequence of warp-lifting, is essentially a program: follow it, and you weave exactly the pattern you intended.
That thought flashed by and I didn’t make much of it. But then in March I went to the Netherlands and spent a week in Amsterdam. The two experiences sat together, and some feelings that had been vague began to sharpen: how could a single strip of bamboo, a single motion, connect to the looms of two centuries ago, to today’s computers, even to a country’s industrialization?
I looked it up later and found I wasn’t the first to make the connection. Many historians of technology argue that weaving is one of the earliest forms of large-scale information encoding. Now, in June, I’m sitting down to try to lay out this thread. This isn’t a technical piece. It’s a personal, cultural reflection.
Let me start with the Netherlands.
What struck me most about that country wasn’t the windmills or the tulips, but a quality that the whole nation seems to exude: restraint, order, and an almost engineering-like way of being. The canals are dug, the land reclaimed from the sea is called a “polder,” and the windmills were never there to look pretty. They were built to pump water and drain the land. A country that sits below sea level literally engineered itself out of the sea. Though the windmills aren’t only about drainage. The wind in the Netherlands is genuinely strong. In March I got battered by it. It’s a wind that pushes at you constantly. So a below-sea-level country hauled itself out of the sea with engineering, and along the way put that fierce wind to work too.
This quality seeps into every corner of daily life. Shops close around six in the evening, the streets slowly quiet down, and you almost never hear a car horn. Cars stop well ahead of time to let you cross. One evening I wandered a little too close to the bike lane, and a passing car actually rolled down its window to remind me to use the sidewalk. It surprised me at first, and then I understood: the rules there are clear, and everyone assumes you’ll follow them too.
This temperament even shows up in how people get around. In the city center, not many people drive. Bicycles are the default. For one thing, the Netherlands is flat, so cycling takes no effort. For another, dedicated bike lanes are everywhere, and the streets are narrow, which makes them ill-suited to cars. Add that commutes are generally short, and cycling is just right. What’s interesting is that even in the southern suburbs, where the roads are wide and every household has a car, you still see a lot of people on bikes. For them, a bicycle isn’t exercise. It’s the default way to commute.
I set that against the noise and speed of China, the loud, garish signboards back home. These two temperaments are really two paths of modernization. The Netherlands is an early, slow-and-steady kind of country, from the Age of Discovery to the world’s first stock exchange, from water engineering to today’s ASML and Philips. It has always moved forward in a “long-term, restrained, systematic” way. China is a late developer chasing from behind, relying on speed, scale, and a generation’s sheer exertion to cover in a few decades what took others centuries.
Neither path is higher or lower than the other, but they settle into completely different urban textures and national characters. The ease of “closing at six” that the Dutch have is something China won’t reach for many years; and the Chinese speed of “just do it,” of laying a high-speed rail line overnight, is something the Netherlands probably couldn’t match either.
But here’s the interesting part. If you trace both civilizations back through history, these two seemingly different cultures share a common starting point. Textiles.
Our generation is so familiar with the phrase “Industrial Revolution” that it’s almost gone numb. But have you ever asked: where did the first Industrial Revolution actually break out? Not in steel, not in coal, not in the railways. In textiles.
The British Industrial Revolution is practically a history of textile machinery. The flying shuttle in 1733 made weaving faster. The spinning jenny in 1764 let one person spin many threads at once. The water frame in 1769 began replacing human power with water power. The power loom in 1785 turned weaving into an automated process. Every one of these inventions happened inside the textile industry.
Why textiles of all things? At first it feels counterintuitive, since textiles seem too “light,” lacking the heft of making steel or guns. But think about it for a moment and it becomes obvious.
Textiles are a basic need. Everyone has to wear clothes, clothes wear out, and they have to be replaced constantly. That’s an enormous market that already existed back in the agrarian age. Once the demand is there, any gain in efficiency turns immediately into profit, into capital that can be reinvested.
More importantly, the processes of textile production are especially easy to mechanize. Spinning and weaving are deeply repetitive motions with a fixed rhythm and a clear logic, so machines can directly replace the human hand. Steelmaking, by contrast, has a far higher technical barrier, the science of chemistry wasn’t yet mature, and machine manufacturing itself needed an existing industrial base before it could even start.
And textile production has a moderate investment threshold. Building a textile mill was far cheaper than building a steel plant, so merchant capital and money from colonial trade flowed in easily. A large share of early British industrial capital came from the cotton trade and colonial trade, and that money went first into textile mills.
The most crucial point is that textiles don’t exist in isolation. They pull an entire supply chain along with them. To spin and weave, you first have to grow and transport cotton. To run machines, you have to build textile machinery. To power the machines, you need steam engines. To fuel steam engines, you have to mine coal. To move cotton and cloth, you have to build railways. So the steam engine was first deployed at scale in textile mills, the earliest railways carried cotton and cloth, and the earliest machine manufacturing served textile equipment. A supposedly “light” industry ended up dragging the entire mechanical and energy industries into being.
So textiles became the progenitor of industry not because they were the most complex, but because they were the first to satisfy four conditions at once: large demand, mechanizable, low threshold, and able to pull others along. Textiles were humanity’s first trial plot on the way from handicraft into an industrial system.
Here’s a pattern I’d never noticed before: the industrialization of late developers, almost without exception, begins with textiles.
Take Japan. Today when you think of Japanese industry, you think of Toyota cars. But how did Toyota begin? Its founder, Sakichi Toyoda, didn’t start by making cars. He invented automatic looms. In 1924 he invented the Type G automatic loom, and in 1926 he founded Toyoda Automatic Loom Works, the origin of the entire Toyota Group. It was his son Kiichiro who later moved the business into automobiles.
So Toyota cars quite literally “grew out of a loom.” Its whole system of lean production, kanban management, total quality control, carries in its bones the obsession with precision, process, and zero defects that came from making looms.
Then look at South Korea. Samsung today is a global electronics and semiconductor giant, but when it was founded it exported dried fish, vegetables, and fruit, only later moving into textiles and sugar. As South Korea industrialized after the war, textiles and garments were among its earliest foreign-exchange earners, and Samsung built its first fortune on trade and textiles in that period before pivoting to electronics in the 1970s. The Hyundai Group took a similar path, starting in engineering and heavy industry before expanding into automobiles, shipbuilding, and heavy industry.
Britain was first; Japan and Korea are the late-developer cases. You’ll notice the sequence is strikingly consistent across all three: textiles first, then light industry, then machinery, then heavy industry, and finally high-tech. Nobody designed this path. It was forced by cost and by the market. Because textiles are the lowest-threshold, largest-market form of manufacturing, every nation that wants to turn from an agrarian country into an industrial one has to start from the easiest, most necessary step.
Behind this lies a very plain truth: modernization doesn’t happen in one leap. It needs a starting point that can earn money, accumulate experience, and train the first generation of industrial workers. And textiles happen to be exactly that kind of starting point.
Writing this far, I can’t help but turn and look back at China itself.
China’s modern industrialization also began with textiles. The most famous Chinese-owned enterprises of the late Qing and early Republic were almost all cotton mills. Zhang Jian founded the Dasheng Cotton Mill in Nantong. The Rong brothers, Zongjing and Desheng, founded the Shenxin Cotton Mills in Wuxi. They were China’s first generation of national industrial capitalists, rising out of textiles and holding up half of modern Chinese industry. The thinking of Zhang Jian’s generation was plain and direct: foreigners were using machines to weave cloth and taking our money, so we had to set up our own mills, weave our own cloth, and keep that money at home.
That was China’s first stretch. Back then, China was walking the very road that Britain, Japan, and Korea had all walked.
Then the road broke. War, turmoil, the planned economy. Chinese industrialization took many detours. It wasn’t until reform and opening up that it reconnected. In the 1980s the coast was blanketed with processing factories for garments, shoes, and toys. In essence it was still that same old road of “starting with textiles and light industry,” only this time China relaunched it through exports, cheap labor, and the role of factory to the world.
Further on, through the 1990s and 2000s, China pushed into heavy industry: steel, cement, shipbuilding, chemicals, catching up to the world’s front ranks one after another. Then electronics, home appliances, mobile phones. Then the internet, high-speed rail, new-energy vehicles, solar, semiconductors, and the AI that everyone talks about today.
String this line together and you realize China is actually completing the same road that every late developer has walked, only faster, fiercer, and at a far greater scale. From Zhang Jian’s cotton mills to today’s new-energy vehicles and large AI models is barely over a hundred years. In just over a century we’ve run the entire course of industrialization that took Britain more than two hundred and Japan more than a hundred, and we’re still going.
This often leaves me with a complicated feeling. On one hand, admiration: generation after generation, from Zhang Jian to today’s engineers, really did turn an agrarian country into the factory of the world and then into a country that now leads in quite a few high-tech fields. On the other hand, a faint unease: this road was run so fast that a lot of things got left behind.
Inside this long thread of industrialization, there’s one detail I never forgot: that moment in the bamboo-weaving workshop in Anji back in February.
The “flower program” the old master described actually has a very old tradition in China. The drawloom of the Han dynasty used a pre-designed system of cords to control which warp threads rose and fell, so a weaver following it could reproduce complex patterns. Then in the early nineteenth century, the Frenchman Joseph Marie Jacquard pushed the idea a big step forward, using punched cards to control the weaving pattern: where the card had a hole, a hook passed through and lifted the warp; where there was no hole, the hook was blocked and the warp stayed down. Hole or no hole, that’s 1 and 0.
Jacquard’s punched cards were later borrowed by the computing pioneer Charles Babbage for his Analytical Engine, and Ada Lovelace used them to write the first computer program in history. Further on, IBM rose on punched-card machines, and if you trace today’s entire computing industry back to its source, it leads, improbably, to a loom.
And that’s not the end of it. The Transformer, matrix operations, the weights of neural networks in today’s AI, at bottom are all about computing relationships across vast fields of “warp and weft.” A loom decides which warp threads to lift and when; a neural network decides which parameters relate to which others. The logic is the same.
So the word “weaving” is more than the starting point of an industry. It’s a metaphor: for thousands of years, human beings have been learning how to encode complex things into structure, from a bolt of cloth, to a program, to a neural network. Every nation does this. They’re just at different stages.
I plan to write the technical bloodline of this story separately, in another piece. Here I’ll only point to it, to make one thing clear: the thread that began with that strip of bamboo has never broken.
From a bamboo-weaving session in Anji in February, to the canals and windmills of Amsterdam in March, and on to the industrialization of Britain, Japan, Korea, and China, what I want to say really comes down to one thing.
Every nation’s modernization is a road from a single thread woven into a net. Textiles are the common starting point of that road, not because textiles are so lofty, but because they are the most necessary, the easiest, the quickest to earn that first money and train that first generation of workers. From that starting point, some walk with restraint, like the Netherlands; some move with ferocity, like China. Some take two hundred years, some a hundred, some only a few decades. But the starting point is the same, and so is the direction: from that thread in the hand, step by step, woven into the whole of industrial civilization as we know it today.
The Dutch ease of closing at six, of no honking, of swans gliding through the canals, is a place China hasn’t reached yet. The Chinese speed of just doing it, of rolling something out overnight, is something the Netherlands couldn’t pull off either. Between these two temperaments of modernization there is no higher or lower, only trade-offs. And behind the trade-offs lies the question of where a country sits in its industrialization, and what rhythm it’s willing to pay for that net.
That day in Anji, holding a strip of bamboo, I clumsily followed the old master, lifting and pressing, weaving a small, lopsided patch of pattern. I wasn’t thinking about any of this then. But looking back, from that single lift and press, you can trace upward to the Han dynasty drawloom, outward to a country’s industrialization, and forward, faintly, to today’s computers and AI. A single strip of bamboo, connected to so much.
Perhaps the evolution of civilization, in the end, is just humanity learning, over and over, how to weave one thing into another.
Starting from a single thread.
2026-06-17 22:04:42
For the past decade my home turf has been containers and Kubernetes. From service mesh to the whole cloud-native ecosystem, I’ve spent almost every day on scheduling, networking, storage and observability, but always on the CPU side of cloud native. Last year I formally moved into AI infrastructure (AI Infra).
Once I dove in, I found the concept density absurd. Token, Transformer, Tensor Core, HBM, KV Cache come at you one after another, and almost every doc and article assumes you already know them, which is deeply unfriendly to engineers who have never trained a model or run inference.
I quickly hit on a trick: don’t learn from scratch, use what you already know by heart as a translator.
The mental model a Kubernetes veteran already carries, scheduling, Jobs, Deployments, controllers, caches, utilization, multi-tenancy, and then microservices, service mesh, distributed systems, event-driven, high-availability, maps onto AI Infra almost one to one. Once you build that mapping, the intimidating concepts become “oh, it’s just the GPU version of X”. This “translate AI through cloud-native eyes” approach got me up to speed fast, and I’m writing it down to help friends with the same background skip the detour.
This is the first post in that translation series, tackling the most fundamental question: why does AI absolutely need GPUs? Later posts will cover GPU resource management, scheduling and observability, topics a K8s veteran knows well. If you’re also crossing over from cloud native, I hope this saves you a few days of digging.
You type a sentence and the AI replies word by word. For someone who has used K8s, the most natural mental model is: AI is a “workload” that runs on GPUs, and a GPU is a special kind of “node” built for it.
The AI world throws around terms that read like scripture to anyone who hasn’t done training or inference. Here’s a cheat sheet; I’ll re-explain each term in plain words below.
| AI concept | In plain words | K8s veteran’s analogy |
|---|---|---|
| Token | A small unit text is chopped into; AI generates them one at a time | A log line, a text chunk |
| Model | A “scoring program” holding billions of numbers (weights) | A giant image whose “weights” aren’t code |
| Training | Feed it huge data, tune params repeatedly, build the model | Run a Job to build an image; offline, batch, care about throughput |
| Inference | Model is done, serve user requests and emit answers | Run a Deployment taking traffic; online, care about latency |
| Transformer | The architecture shared by nearly all LLMs (GPT, Claude, LLaMA) | A controller design pattern |
| Tensor Core | A GPU unit that only does “matrix multiply” but insanely fast | A sidecar worker that only does batched multiply |
| HBM | The GPU’s on-board high-bandwidth memory; model and cache live here | Node-local RAM, but with bandwidth that crushes normal RAM |
| KV Cache | Attention state stored at inference time, grows with the conversation | A pod-local session notebook that gets thicker the longer you talk |
Memorize this table; the rest follows.
GPUs are everywhere in AI now, so accepted that most people skip past it. We care about which card to rent, which framework to use, which model to deploy, but rarely stop to ask the deeper question:
Why GPUs?
Why not the more general CPU? Why did a chip originally built to render game graphics become the foundation of the most important technology shift in a generation?
The answer isn’t just “it’s fast”, it’s about how computation itself is organized, and why AI workloads demand an architecture fundamentally different from fifty years of software.
You know CPUs well. K8s’s control plane, etcd, the scheduler all run on CPU; it excels at executing complex instructions one after another, with few cores (8 to 128) but each highly capable. A CPU optimizes for latency: how fast can I finish one complex task? Like a master craftsman doing one intricate job at a time.
A GPU takes the opposite path: execute simple instructions across massive amounts of data at once. A modern data-center GPU has thousands of small cores, and it optimizes for throughput: how many simple tasks can I finish at the same moment? Like a factory floor of thousands of workers, each doing one identical step simultaneously.
For any single complex job, the craftsman (CPU) is faster; but as long as the work is uniform and parallelizable, the factory (GPU) crushes the craftsman in total output per second. For decades CPU dominated because most software (web servers, databases, operating systems) is inherently serial. Then deep learning arrived.
The core of a neural network is, essentially, a giant pile of matrix multiplications (an operation that batch-multiplies-and-adds two sets of numbers). When a model processes a token, it runs thousands or tens of thousands of these multiplies, and they don’t depend on each other.
The key insight in one line: these operations are naturally parallel.
A 128-core CPU looks at this workload and sweeps through it sequentially; a many-thousand-core GPU chops the matrix into blocks and hands them to its small cores to run at once. Same work, orders of magnitude faster on GPU, not because a single GPU core is faster (it’s slower), but because the problem is parallel and the GPU is built for exactly that shape of computation.
That’s the real reason AI runs on GPUs: not marketing, not legacy, architectural fit.
A GPU has two kinds of cores. Regular CUDA cores are general-purpose workers that can compute anything; a Tensor Core is a dedicated unit that does exactly one thing: multiply two small matrices in a single shot. But that one thing it does insanely fast, finishing in one cycle what would take a regular core thousands.
AI’s core operation is matrix multiplication, so Tensor Cores are tailor-made for it. In K8s terms: regular CUDA cores are like general pods in a Deployment that do everything; a Tensor Core is like a highly specialized sidecar that only does “batched multiply”, single-function but with crushing throughput.
Computing fast isn’t enough; you have to get the data. A GPU’s memory is layered just like a CPU node’s, and if you understand a K8s node’s L1/L2 cache, RAM and local disk, you understand the GPU’s:
The bottom layer, HBM (High Bandwidth Memory), is the GPU’s “main memory”; model weights, intermediate results and KV cache all live here. It’s characterized by large capacity (hundreds of GB per card) and extremely high bandwidth.
Why build HBM at all? Because traditional VRAM (GDDR) lies flat on the circuit board, you run out of routing space and hit a bandwidth wall. HBM’s answer is to stack memory chips vertically (connected by Through-Silicon Vias, TSVs), packed right against the GPU die with an interface thousands of lines wide running in parallel. By analogy: normal memory is like a warehouse spread across a parking lot where the movers can’t keep up; HBM is like building that warehouse into a dozens-story tower right next to the workshop, with TSV elevators running up and down at full speed.
Here’s a counter-intuitive fact: modern AI is memory-bound more than compute-bound. Tensor Cores do matrix multiplication faster than HBM can feed them data, so the GPU is often waiting. That’s why each new GPU generation (H200 → Blackwell → Vera Rubin) sees its most important upgrade in HBM bandwidth, not compute (4.8 → 8 → 22 TB/s, source).
Since data movement is the bottleneck, NVIDIA’s other move is to let data bypass the CPU and reach the GPU directly, the three “expressways” collectively called GPUDirect:
In one line: the CPU is no longer the traffic cop for every data movement. In K8s terms, it’s like sending data over a direct data plane instead of routing every packet through the apiserver. NVIDIA’s edge isn’t just fast compute; it’s that the entire data highway from storage to GPU, GPU to GPU, and GPU to network has been straightened out.
A Transformer isn’t hardware, it’s a model architecture (a program structure). GPT, Claude, LLaMA and other large models are all built on it. What it does is actually plain:
Read a piece of text, predict the next token.
A token is a small unit text is chopped into (a character, half a word). A Transformer feeds the input token sequence in, emits “the most likely next token”, appends it, feeds the new sequence back in, predicts the next-next, and so on, generating the whole answer one piece at a time.
In K8s terms: a Transformer is a controller whose reconcile logic is “look at the current state (tokens so far) → emit the next action (the next token)”, looping continuously. That’s where the “word by word” effect of AI assistants comes from.
This is the easiest pair to confuse when starting out, yet the most critical. Training and inference are two different things with completely different hardware needs.
This directly affects which GPU metrics you should care about. The headline numbers (Tensor Core FLOPS, NVLink bandwidth, cluster size) are mostly training metrics; what actually determines whether your AI assistant feels snappy (HBM capacity, memory bandwidth, KV cache management) are inference metrics, and they get little airtime. Yet inference is where most production AI actually runs.
With single-card covered, back to reality: the biggest models don’t fit on one card, and training routinely needs hundreds or thousands of cards working together. At that point the GPU cluster is essentially a distributed system, and almost everything you know from cloud native applies.
Many GPUs = a microservice cluster. A card is like a service instance; the model is sharded across cards, each computes a slice and the results are stitched back. That’s exactly the microservices playbook of splitting by responsibility and sharing load.
Inter-card communication = the service-mesh data plane. Cards constantly exchange data (gradients especially during training). Within a rack it’s NVLink (several TB/s), across racks it’s InfiniBand. It’s just the east-west traffic between microservices: in-node pod-to-pod direct is fastest (NVLink), cross-node cross-cluster goes over the network (IB). NVIDIA packaging GPUs, switches, DPUs and RDMA into a whole rack (Vera Rubin NVL72) is exactly a service mesh unifying the data plane, control plane and observability.
Distributed training’s all-reduce = distributed consensus. Each training step synchronizes and averages the gradients across all cards, a step called all-reduce. Anyone who has done etcd or Raft gets it instantly: it’s distributed-system consensus and consistency, except here you’re syncing gradients, not a state-machine log.
Continuous batching = event-driven. During inference, new requests are dropped into the running batch as they arrive, no waiting for the whole batch to finish. Requests are events, the batcher is a consumer, batch-then-process: that’s the event-driven / message-queue mindset, all to keep the GPU busy.
Disaggregated inference = microservices. Split prefill (process input, compute-heavy) and decode (emit tokens, memory-bandwidth-heavy) into separate GPU pools that scale independently, with KV cache passed between them over NVLink/RDMA. That’s entirely the microservices pattern of splitting by responsibility and scaling each part; KV cache is the session context passed between services.
See the pattern? An AI cluster isn’t a new species, it’s your familiar distributed-systems, microservices and service-mesh toolkit replayed on GPU hardware. The instincts you built tuning traffic on Istio and Envoy, or consensus on etcd, are worth exactly as much here.
By now, as a K8s veteran, you’re bound to ask: so what actually schedules a GPU cluster? The interesting answer: training and inference often run on two different schedulers.
On the training side, the HPC world’s heavyweight is Slurm. It manages 65% of the world’s TOP500 supercomputers (NVIDIA), and large AI training teams have years invested in Slurm scripts, fair-share policies and accounting. Training is a long-running batch job that wants topology awareness (place chatty GPUs close together), long exclusive holds and high throughput, areas Slurm has refined for over a decade.
On the inference side, the home field is Kubernetes. Inference is an online service that wants fast autoscaling, per-request scheduling and integration with service mesh and observability, exactly K8s’s strengths.
So what if a team needs both, maintain two environments? NVIDIA open-sourced the Slinky project to solve exactly this, in a very K8s-native way:
NVIDIA itself runs Slinky in production up to 8,000+ GPUs, with NCCL all-reduce/all-gather performance matching bare Slurm, the K8s layer adding almost no overhead (NVIDIA production data). K8s is becoming the substrate for GPU computing, with Slurm as the scheduling layer on top.
Training and inference have fundamentally different GPU-scheduling needs, so they can’t be managed the same way. That’s also why GPU resource management (covered later) has to be scenario-specific, and why solutions like HAMi aim to be a unified GPU resource-management layer across Slurm + K8s hybrid environments.
If there’s one concept that separates “GPU theory” from “AI inference reality”, it’s KV Cache.
When a Transformer generates each token, it has to reference the relationships among all previous tokens (this is “attention”). To avoid recomputing from scratch every time, it stores each token’s attention state, and that store is the KV Cache.
The problem is it grows without bound:
In K8s terms: KV Cache is like a session notebook local to a Pod. The longer the conversation, the thicker the notebook, and every new word forces a full read from cover to cover. So inference memory is often eaten not by the model but by this “notebook”.
That’s also why we got paged attention (managing the notebook like virtual-memory pages to cut fragmentation, popularized by vLLM) and NVIDIA Dynamo’s multi-tier cache (hot pages in HBM, warm offloaded to CPU memory, cold spilled to NVMe). Half an inference engineer’s job is managing this notebook.
K8s folks know “utilization lies” best: a pod reporting Running isn’t necessarily doing work, it might be in CPU steal or waiting on IO. GPUs are worse.
That “GPU utilization %” in monitoring usually only measures “is the GPU executing any kernel”, not how efficiently. A card can report 90% utilization while its Tensor Cores are actually busy only 30% of the time, with the rest spent on memory ops, kernel-launch overhead, or just waiting for data.
The more honest metric is SM Efficiency (SM activity rate): it looks at how many SMs are doing useful work each clock cycle. A card showing 100% utilization in nvidia-smi may have an SM Efficiency of only 20-30%. Many companies think their “GPUs are maxed out” when in fact huge amounts of compute are spinning idle. So to judge whether a GPU is truly working, don’t look at utilization, look at SM Efficiency.
The metrics that actually matter (mapping to the QPS, P99 latency and resource levels you watch in K8s):
You can’t talk GPUs without NVIDIA’s dominance. The hardware is excellent, but hardware alone can’t explain why AMD, Intel and a crowd of startups have failed to gain ground. The answer is ecosystem depth.
CUDA isn’t just a parallel-computing platform, it’s a programming model, compiler, runtime and a stack of libraries, refined for 18 years. Nearly every AI framework (PyTorch, TensorFlow, JAX) grew up on CUDA first and was ported elsewhere as an afterthought. In K8s terms: CUDA is to GPUs roughly what the Linux kernel + containerd + the whole CNCF toolchain are to the container ecosystem, not something you replace by swapping a kernel.
Switching to another card means re-validating every layer of your inference stack (kernels, libraries, frameworks, serving, monitoring, ops). The switching cost isn’t buying different hardware, it’s rebuilding an entire software ecosystem. That’s the “CUDA moat”.
NVIDIA no longer describes its business in terms of “GPUs” or even “data centers”, but as “AI factories”: facilities that continuously convert electricity, silicon and data into intelligence. Beneath the language is a real architectural shift:
After all this GPU praise, you might think the CPU is sidelined in the AI era. The opposite is true: agentic AI is putting the CPU back at center stage.
In traditional LLM inference the CPU mostly just compresses and routes data for the GPU, so AI data centers ran CPU:GPU ratios as low as 1:4 or even 1:8. Agents are different: they plan tasks autonomously, call tools, route data between sub-agents and decide whether a task is complete, and all of that orchestration logic lands squarely on the CPU. Add that agents are often trained with reinforcement learning, where every action has to be evaluated by the CPU, and the CPU load gets heavier still.
Research notes that in agentic scenarios the CPU-side tool processing (Python interpretation, web crawling, retrieval, database queries) can account for up to 90% of total latency. According to TrendForce, Arm estimates that traditional AI data centers need about 30 million CPU cores per GW, a figure that will surge to 120 million in the agent era, a 4x increase; the CPU:GPU ratio will shift from 1:41:8 toward **1:11:2**. That’s also why in 2026 NVIDIA started selling the Vera CPU standalone and Arm got into the CPU business directly.
The signal for K8s veterans is clear: the future AI node is a mixed-workload node where CPU and GPU are billed together, and scheduling and resource management must handle both, not just stare at the GPU.
After all this hardware, it lands on what I work on: only by first understanding why the GPU is the foundation of AI can you understand why “GPU resource management” is a real problem.
When a card costs hundreds of thousands of yuan, a whole rack draws over a hundred kilowatts, and production GPUs still run below capacity because memory bandwidth, KV cache or batching aren’t tuned, just “handing out cards” is nowhere near enough. How to run multiple tenants safely on one card (like K8s scheduling many pods onto one node), how to partition resources between the very different workloads of training and inference, how to push utilization from “looks full” to “actually full”, that’s exactly what GPU virtualization and sharing (the work I do) solves.
Here’s a sobering number: per the ClearML AI infrastructure survey, only about 7% of enterprises hit over 85% GPU utilization at peak, more than half sit at 51-70%, and 15% are below 50%. In other words, a big chunk of the GPUs companies pay dearly for are spinning idle. That’s usually not a hardware shortage, it’s scheduling and management not keeping up.
My own take is that GPU resource management is moving through three stages: Allocation → Utilization → Efficiency. Stage one only answers “who gets this card” (device plugin, exclusive or time-slicing); stage two asks “is this card full” (dynamic batching, elastic autoscaling, where most enterprises are stuck); stage three asks “is this card’s compute producing maximum value” (SM Efficiency, tokens per watt, bandwidth utilization). The real challenge is stage three, and the future GPU scheduler shouldn’t be a resource allocator but a fine-grained orchestrator of the GPU’s internals, reading how many SMs are active, the Tensor Core utilization, how much memory KV cache holds, and only then deciding whether one more request fits.
Plainly put, the AI era is replaying the cloud-native story: from “single-machine exclusive” toward “multi-tenant sharing + scheduling + observability.” And the GPU is the main stage of that play.
By now you’ve probably noticed this post barely mentions anyone but NVIDIA. Unavoidable, it’s the absolute protagonist today. But if you think the AI accelerator world begins and ends with NVIDIA, you’re very wrong.
In fact, an “anti-NVIDIA alliance” is gathering from all sides:
What does that mean for a K8s veteran? It means the future AI cluster is almost certainly heterogeneous: a rack might hold NVIDIA, AMD, TPU and domestic cards side by side, and one schedule has to manage several completely different kinds of hardware.
So the really interesting questions follow:
I’ll dig into those in later posts. This one only nails “why GPUs”; the next will tackle the big chess game of how AI infrastructure should schedule things once “GPUs aren’t just one kind”.
2026-06-17 14:22:24
Every GPU should not just be used. It should create value.
For the past few years, whether it is Kubernetes GPU scheduling, vGPU, MIG, or HAMi, everyone has really been doing the same thing: pushing one number up.
GPU UtilizationIt makes sense. GPUs are expensive. An H100 can run anywhere from a few dollars to over ten dollars per GPU-hour, and nobody can afford to let a GPU sit idle. So the entire AI Infra community’s narrative for the past few years has boiled down to one line:
Maximize GPU utilization.
I have been involved in the HAMi community for a long time, and a few posts I have written, like Kubernetes as the GPU Control Plane for AI and From GPU to Token: An Eight-Layer Observability Stack for AI Infrastructure, are really about the same thing: how to slice GPUs finer, share them more thoroughly, and schedule them more sensibly.
But recently I read Arjun Kaarat’s piece in Towards Data Science, When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI, and it made me rethink a question:
Is GPU utilization really the metric we should be optimizing for?
There is one line in the article that made me pause for a few seconds the first time I read it:
“GPUs can be busy without being productive.”
Arjun Kaarat
It punctures a layer of illusion. The number we have spent so much effort pushing up may have been answering the wrong question from the start.
Kaarat tells a representative story in the article.
At 2 AM, an infrastructure team gets paged: inference latency just spiked 60%. They open the monitoring dashboard, and GPU utilization looks perfectly normal:
GPU: 79%
GPU: 82%
GPU: 84%Looks healthy. So the usual playbook kicks in: trigger autoscaling, add nodes, add GPUs. The cloud bill climbs, but latency barely improves.
An hour later, they find the root cause: three nodes had quietly entered a RAID rebuild state, storage throughput was severely dragged down, and the inference tasks around them were starving. The scheduler kept treating these nodes as “still healthy enough” because the GPU and memory metrics looked fine, but the underlying disk performance had collapsed.
What strikes me most about this story is that it is not a rare edge case. It is a failure mode that is becoming common.
Many teams look at their monitoring and see:
GPU: 82%
GPU: 84%
GPU: 79%and think “our cluster is busy and healthy.” But at the same time:
Latency ↑
Queue ↑
Throughput ↓
Cost ↑The real problem is that the GPU is waiting.
Waiting for what?
The GPU end is full, but the data path feeding it is empty, blocked, or collapsing. From the dashboard the GPU reads 84%, but the actual output may be less than half. Kaarat describes this state precisely in the original piece:
A GPU that appears active may still spend meaningful time waiting for the system around it.
That is the overlooked gap between “busy” and “productive”.
The RAID story above is still just a “point failure”. The more compelling part of Kaarat’s article describes a systemic phenomenon: Fragmentation.
Consider a cluster with three nodes, after running a mixed wave of GenAI workloads:
| Node | GPU compute | HBM | Storage bandwidth | I/O CPU |
|---|---|---|---|---|
| A | available | nearly full | available | available |
| B | available | available | saturated | available |
| C | limited | available | available | saturated |
Now a new inference job arrives, with an ordinary footprint: a little GPU, a little VRAM, decent storage bandwidth, decent I/O capacity.
In total, the cluster still has plenty of resources. A has GPU and bandwidth, B has VRAM, C has bandwidth and CPU. But no single node can take this job on its own.
That is fragmentation. I drew it out, roughly like this:
The cluster is not empty. It has just been carved into “leftovers” that can no longer be used productively. Kaarat sums up the phenomenon in one line, which I think is the most memorable sentence in the whole piece:
The cluster is not empty. It is fragmented into leftovers that are difficult to use productively.
Put another way:
The cluster does not lack resources. It lacks resources of the right shape.
This judgment matters a lot for a project like HAMi, which builds a GPU resource control plane. We used to think that “slicing the card finely and letting more people share it” was the answer to fragmentation. But Kaarat points to a deeper problem: fragmentation is not just a GPU-layer concern. It spans GPU, HBM, storage bandwidth, and I/O CPU. You can slice the GPU as finely as you like, but if the storage dimension is choked, that node is still unavailable for the next genuinely useful task.
Let me directly answer a question: what role does HAMi play in this chain?
HAMi solves a very specific, and very foundational, problem:
Can the GPU be used by more people.
What it does can be summarized in one line: reduce fragmentation at the GPU layer. The concrete forms include:
This is the first layer of efficiency. It answers the question “can the GPU be put to use”.
At this layer, HAMi’s value is already clear: in real environments with mixed domestic and foreign GPUs, mixed training and inference, and multi-tenant sharing, HAMi lets one card serve more workloads, so the cluster is no longer wasted by a coarse-grained “one card per Pod” model.
But note: this is only the first chapter of the efficiency story.
If I step back and look at it from a higher vantage point, “improving GPU utilization” is actually solved across three distinct layers.
Layer one: can the GPU be sliced, shared, and allocated?
This is what a project like HAMi solves. It corresponds to the GPU resource control plane.
Layer two: who gets to use the GPU? Who runs first, who queues, who has priority?
This is what schedulers like Volcano, Kueue, and KAI Scheduler solve. It corresponds to job queuing, fair share, priority, and gang scheduling.
Layer three: can the GPU actually run? Are the data path, storage I/O, and KV cache keeping up?
This is exactly where Kaarat’s article sounds the alarm. When a node’s RAID is rebuilding, its SSD queue is exploding, and its I/O CPU is eaten by background tasks, no matter how much GPU you allocate to it and no matter how elegantly the scheduler queues its tasks, it still cannot produce effective compute.
Draw these three layers together, and you get what next-generation AI infrastructure should actually look like:
I think this diagram is the single most important one for understanding the whole picture.
For the past few years, almost all of the industry’s attention has been on the bottom two layers: Kubernetes and HAMi. Those two layers have essentially solved “can the GPU be put to use”. Volcano, Kueue, and KAI are also mature at layer two, solving queuing and priority.
But layer three, storage / I/O-aware scheduling, is currently almost a blank. And that is precisely the layer Kaarat’s article keeps emphasizing, the one that is becoming increasingly valuable in modern GenAI systems. Because for workloads like RAG, long context, and multimodal, the bottleneck has long since shifted from “is the GPU enough” to “is the data path feeding the GPU clear”.
To put it bluntly: HAMi slices the GPU finely, and Volcano queues the jobs well, but if a node assigned a task cannot feed its GPU, then all that upstream effort is just pumping blood into an idle endpoint.
Based on the layering above, I want to make one clear point: we should upgrade the optimization target from “GPU utilization” to “Productive GPU-Hours”.
This is not wordplay. It is a shift that translates directly into money.
Kaarat does the math in the article. A 1000-H100 cluster, at a blended cost of about $3 per GPU-hour, runs around $26 million a year. If fragmentation and I/O stall quietly waste 10% of the effective GPU time, that is roughly $2.6 million a year of wasted spend. Not because the GPUs are missing, but because the system failed to use them efficiently.
That math can be translated into a simple contrast.
The past target:
Maximize GPU UtilizationToday’s target:
Maximize Productive GPU-HoursThe future target:
Maximize Productive Compute
Across Heterogeneous AI ClustersThis evolution maps exactly onto the three-layer structure in the diagram above. That final line, “Across Heterogeneous AI Clusters”, is the heterogeneous narrative HAMi has been pushing all along: in the future you will not optimize just one kind of GPU. You will maximize effective compute uniformly across completely different cards from NVIDIA, Ascend, Cambricon, and Hygon.
In other words, HAMi’s long-term value should not be boxed into the old “improve GPU utilization” narrative. Its real direction is: a resource control plane that lets Productive GPU-Hours be maximized across heterogeneous AI clusters.
If I had to summarize this whole line of thinking in one sentence, I would put it like this:
HAMi solves “how to let more GPUs be used”, and next-generation AI infrastructure has to solve “how to let every GPU actually create value”.
That is also the deepest line Kaarat’s article left me with:
The real question is no longer “Are the GPUs busy?”
It is: “Are they productively busy?”
It also makes me rethink what the HAMi community’s tagline for the next phase should be. We used to say “let GPUs be shared by more people”. Next, we should probably move toward:
Turn GPU utilization into productive GPU-hours.
Let every GPU not just be used, but genuinely create value.
GPU utilization was never the endpoint. It was only the first chapter of the efficiency story. Only when we start talking about Productive GPU-Hours, and start caring about storage, I/O, the data path, and the aggregate output of heterogeneous clusters, does AI infrastructure truly enter its second chapter.
Parts of this post were inspired by Arjun Kaarat’s piece in Towards Data Science, When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI, and quote its published paper and article. My thanks to the author. Both diagrams in this post (resource fragmentation, the efficiency layering) were redrawn by the author based on the ideas in the original, not copied from it.
Further reading:
2026-06-16 20:45:16
An Agent is not a single inference request but a long-running distributed state machine; therefore, Agent reliability is fundamentally a distributed-systems problem.
After years in cloud native, I have a stubborn instinct: whether a new technology is mature is not judged by how powerful its model is or how flashy the demo looks, but by whether it has a systematic language for reliability. Kubernetes won not on the container runtime, but on liveness/readiness probes, the reconciler loop, PDs/PDBs, and the whole SRE vocabulary that made “long-running distributed state” legible.
So when I came across the paper Hesham ElBakoury shared in the Open Compute Project (OCP) community, it caught my eye. It proposes no new algorithm or framework; the entire paper does one thing: systematically translate the traditional SRE reliability vocabulary onto Agentic AI. This is exactly what I have been circling around for the past six months in Agentic Runtime Realism and Ark Agentic Runtime, Analyzed, without a canonical reference to align against. So I decided to write a dedicated post: unpack its framework, then give it a critical evaluation from the AI Infra practitioner’s point of view.
One-line positioning: it reads more like an SRE white paper for Agentic AI than a systems paper. Its value is not in novelty but in “building consensus”.
Traditional AI cares about model accuracy, while Agentic AI must care about “reliability during long-running operation”, so fault tolerance, recovery, monitoring, security, and state management must be elevated to first-class citizens of architectural design.
The paper argues the fundamental difference between Traditional AI and Agentic AI lies in the execution model. Traditional AI is one-shot request-response, focused on accuracy, latency, and throughput; Agentic AI is a continuous loop, where a single error is no longer just a wrong answer but a wrong decision that may change every subsequent action of the Agent.
The comparison table below states this most clearly. I suggest you focus on the last row, “failure impact”, because that is the thesis of the whole paper:
| Aspect | Traditional AI | Agentic AI |
|---|---|---|
| Execution model | Request-response | Continuous loop |
| State management | Stateless | Stateful |
| Trigger | Human-triggered | Self-initiated |
| Time span | Single interaction | Long-running session |
| Failure impact | Single wrong output | Cascading behavior changes |
| Resource usage | Bursty | Continuous |
The point of this table is not the technical details but the shift in mental model: when the impact of failure escalates from “one wrong answer” to “behavior-level cascading errors”, the weight of reliability must be reassigned from the very bottom of the architecture. Engineering fault tolerance for a stateless API and for a self-directing, continuously running state machine are simply not the same problem.
This is the most memorable part of the paper. The author decomposes Agent reliability into five dimensions, forming a complete evaluation coordinate system. I drew it out: “Agent Reliability” sits in the center, with the five dimensions fanning out like petals:
One by one:
The paper’s second core judgment is one I strongly agree with: the biggest infrastructure challenge for an Agent is not inference, it is state.
An Agent simultaneously holds Memory, Goal, Plan, Context, and tool state. Treating it as HTTP API + Model to operate is the root cause of why most demos today cannot survive production. Its true shape is closer to a composite of database + workflow engine + LLM + distributed system:
This is entirely consistent with where the industry is heading: LangGraph, Temporal, and the OpenAI Responses API are all promoting “stateful, long-running logic” to a first-class citizen. When an Agent runs for hours or even days, its state is more critical than the model parameters themselves, and far easier to lose for good on a restart. A model can be reloaded, but a three-hour planning context, once lost, is usually lost.
The paper provides a catalog of Agent fault-tolerance patterns:
| Mechanism | Effect | Complexity |
|---|---|---|
| Redundancy | Standby Agent takes over at any time | High |
| Checkpointing | Save state for recovery | Medium |
| Heartbeat | Liveness detection | Low |
| Circuit Breaker | Isolate abnormal Agents, prevent cascading | Low |
| Rollback | Revert a wrong decision | Medium |
| Quorum | Multiple Agents vote to reach consensus | High |
| Self-Healing | Automatically detect and correct errors | High |
The author values the Checkpointing + Redundancy + Heartbeat trio most, calling it the “golden trio” of Agent systems. I drew a closed loop to show how they relate; none of the three can be missing, and drop any one link and the recovery chain is broken:
This section essentially ports the RTO/RPO thinking of traditional DR (disaster recovery) onto the Agent scenario. The paper compares several recovery approaches:
| Recovery approach | RTO | RPO | Complexity |
|---|---|---|---|
| Cold Start | minutes to hours | High (total loss) | Low |
| Warm Start | seconds to minutes | Medium | Medium |
| Hot Start | seconds | Low (minimal loss) | High |
| Checkpoint Recovery | seconds to minutes | Medium (since last checkpoint) | Medium |
| State Reconstruction | minutes to hours | Low (fully recoverable) | High |
The author’s conclusion: the Agent fits Checkpoint Recovery best. The reason was foreshadowed above: an Agent’s state is far more important than its model parameters. Checkpoint Recovery strikes the most balanced trade-off among RTO, RPO, and implementation complexity. That is also why checkpointing sits at the center of the “golden trio” earlier.
This section resonates with me the most. The paper argues that traditional monitoring is far from enough: traditional monitoring watches CPU, memory, latency, and QPS, while an Agent must additionally monitor the behavior itself.
The “behavior” layer is unique to Agents.
This lines up completely with the thinking I discussed in GPU to Token Observability: in the Agent era, behavioral signals must become first-class citizens of observability, rather than staying stuck at the hardware / inference-metric layer.
The paper compares four architectures:
| Architecture | Complexity | Scalability | Fault isolation | Suitability for Agents |
|---|---|---|---|---|
| Monolithic | Low | Limited | Poor | Poor |
| Layered | Medium | Medium | Medium | Good |
| Microservices | High | High | Excellent | Excellent |
| Event-driven | High | High | Good | Excellent |
The final recommendation: production-grade Agent systems adopt a hybrid architecture, the combination of “layered + microservices + event-driven”:
Layering provides clear separation of concerns, microservices provide independent scaling and fault isolation, and event-driven provides loosely-coupled async collaboration. No single architecture can satisfy an Agent system’s demands for clarity, elasticity, and collaboration at once, so hybrid is the inevitable conclusion. For K8s veterans this layered stack should look very familiar: it is the cloud-native layering of governance, ported verbatim onto the Agent.
From the Kubernetes / AI Infra standpoint, what this paper really conveys is a paradigm shift: Agent infrastructure will move from “Serving” to “Runtime”.
The past cared about GPU utilization, throughput, and latency; the future cares about state recovery, long-task continuity, Agent isolation, Agent scheduling, Agent observability, and Agent security governance.
Strengths:
Weaknesses:
By academic novelty: 6.5 / 10. By “giving AI Infra practitioners a mental framework for Agent reliability”: 8 / 10.
Its biggest insight is not some technical detail but that opening line: an Agent is not a single inference request but a long-running distributed state machine.
The real contribution of this paper is that it re-categorizes the reliability problem of Agentic AI. It is no longer a model problem, nor a prompt-engineering problem, but a distributed-systems problem, and specifically a distributed-systems problem that is stateful, long-running, and self-directing.
For AI Infra practitioners, this means two things. First, the traditional SRE toolbox (checkpointing, redundancy, heartbeat, circuit breaker, RTO/RPO) can be ported over directly, but it must be extended to the “behavior layer”. Second, the focus of infrastructure will shift from “how to serve models faster” to “how to run Agents more reliably”, which is the Agent Runtime.
The AI platforms of the future must not only run fast, they must run stable.
