Daniel LemireModify

2026-08-12 23:50:48

AI-assisted programming is fast evolving and there is a tension between ‘we no longer need to understand the code’ and ‘what is my purpose as a programmer’. I recorded a short video on this topic with how I think the tension can result in conflicts.

2026-08-10 07:17:36

When a compiler optimizes your program, it has to guess. Which functions are worth inlining? Which side of a branch is the common one? Which method does this interface call actually reach? At compile time it cannot know, so it uses heuristics. Profile-guided optimization (PGO) replaces the guessing with measurement: you run your program, record where it spends its time, and hand that recording back to the compiler for a second build.

PGO is a common feature of compiler systems. Google applied PGO to Chrome under Windows in 2016, reporting gains of up to 15%. I expect all mainstream Web browsers to be built with PGO.

There are now fancier techniques than mere heuristics with PGO. You can use AI to recognize patterns and so forth. But they are not always widely available.

Go has supported PGO since version 1.20. You collect a profile, and pass it to the compiler.

A CPU profile is a statistical record of where a program spends its time. While the program runs, the Go runtime interrupts it about a hundred times a second and writes down the call stack at that instant. After a few seconds you have thousands of such samples, and counting them tells you which functions were executing and who called them. In Go you produce one by wrapping the work you care about:

f, _ := os.Create("cpu.pprof")
pprof.StartCPUProfile(f)   // from runtime/pprof
defer pprof.StopCPUProfile()

The compiler reads the call-stack counts and uses them for two things above all: inlining call sites that turn out to be hot, and devirtualizing interface calls whose target is nearly always the same concrete type.

I took three JSON documents that I wanted to parse:

• twitter.json (632 kB), a nest of small objects with short string keys
• canada.json (2.25 MB), essentially one enormous array of floating-point coordinates
• citm_catalog.json (1.73 MB), deeply nested objects with numeric keys

I parse each of them with the standard library’s encoding/json into an interface{}. The baseline, with no profile, parses at 112 MB/s for twitter.json, 74 MB/s for canada.json and 116 MB/s for citm_catalog.json.

The procedure is three commands:

go build -o bench .                        # ordinary build
./bench -profile cpu.pprof -train twitter.json   # collect a CPU profile
go build -pgo=cpu.pprof -o bench_pgo .     # build again, with the profile

I did it three times, profiling each document on its own, and then measured all three documents against each of the three builds.

Each panel of the figure is one document being parsed, and the three bars inside it are the three PGO builds: the binary trained on twitter.json, the one trained on canada.json, and the one trained on citm_catalog.json. Bar height is the speed gain over the ordinary, profile-free build of that same document, in percent, so zero means PGO changed nothing and a bar below the axis means the PGO build was slower. The green bar in each panel is the matched case, where the profile was collected on the very document being measured.

The gains are modest. The best result is canada.json at +4.7%, and most differences are in the 2–3% range. Profiling one document usually helps the others, but not reliably. Profiling twitter.json gave a decent improvement everywhere: +3.1%, +2.0%, +2.8%. But profiling canada.json bought 4.7% on canada.json and essentially nothing anywhere else. Interestingly, profiling citm_catalog.json produced a mere +0.8% on its own document while helping twitter.json more.

A 3% speedup is not exciting in isolation, but it may come nearly for free. Observe how you may get slightly negative results for cases you did not train for. That’s expected generally, but the effect is modest in the case of Go because its optimizations are themselves modest in the first pace. That is, you are not getting a much an effect, but the process is less likely to backfire for other workloads.

The code is available.

2026-08-03 01:00:10

C++26 adds a new container to the standard library: std::hive. It is meant to occupy the ground between std::vector and std::list. Like a vector, it keeps its elements in contiguous blocks of memory, so scanning it does not require you to chase a pointer for every element. Like a list, it never moves an element once it has been inserted: your pointers, references and iterators stay valid, and you may erase any element in constant time without disturbing the others.

Internally, a hive is a linked list of blocks. Each block carries a skipfield: a small integer per slot that tells the iterator how many erased slots to jump over.

No standard library ships std::hive yet to my knowledge. Fortunately there is an implementation (plf::hive by Matt Bentley) as a single header file that you can use today.

I use elements of type uint64_t, GCC 16.1 with -O3 -march=native, on an Intel Xeon Gold 6548N (Emerald Rapids), pinned to one core. Numbers are nanoseconds per element, along with the cycles and instructions retired per element.

We start from an empty container and append a million values. The container is then destroyed.

A std::list needs one allocation per element, and glibc’s malloc and free together cost over 200 instructions per element. It is an order of magnitude behind everyone else. That is not news.

The interesting comparison is vector against hive. A hive is about twice the cost of a vector, and it needs twice the instructions. This is the price of the skipfield: every insertion writes an element and a skipfield entry, and maintains the block bookkeeping. Note that calling reserve on a hive does not help in my experiments.

Next we iterate over the the container and sum the values.

A hive iterates no faster than a linked list here, slightly slower, in fact, and about eight times slower than a vector. (Update: Joseph Garvin points out that I measure the happy case for the std::list in this instance where all the entries were allocated in sequence. The worst case scenario for std::list when the nodes are all over the heap can be much slower.)

The vector loop retires one instruction per element and finishes in 0.78 cycles: the processor is executing several elements at once. This is possible because the std::vector implementation benefits from autovectorization: the compiler recognizes that it can load several words at once in wide (SIMD). Further, it does not have to check the bitfield like the std::hive data structure.

We can check this. Walk the same container with two independent iterators, one starting halfway in, and count the cost per element visited:

The vector does not care: it was already throughput-bound. The hive and the list get nearly twice as fast per element, because two independent chains can be in flight at once. Hive iteration is latency-bound, exactly like list iteration. It merely has better locality.

That locality does show up when the data gets big. At ten million elements the list falls apart while the hive holds steady:

Erasing is what a hive is for, so it would be unfair not to look. I erase half the elements at scattered positions using std::remove_if:

The hive wins, but by less than you might expect, and at ten million elements the ordering reverses (1.3 ns for the vector against 2.5 for the hive). std::remove_if is a single streaming pass, and streaming passes are cheap. Of course the vector moved every surviving element and invalidated every pointer into it, which is precisely what a hive promises not to do.

Memory, measured by asking glibc how many bytes it has handed out, per live element:

A hive costs about a byte per element over a vector, for a payload of eight bytes, when the vector is packed tight. A list costs more due to the overhead of the linked list.

A vector built by push_back has a capacity that typically exceeds its size. Thus even if you have 8-byte entries, you will use, on average, more than 8 bytes per entry even for large vectors. You can recover the excess capacity with the shrink_to_fit method.

What should we conclude?

The std::hive data structure is not a faster vector. But it is a much better std::list. It gives you the same guarantees that make people reach for a list, stable references, cheap erasure anywhere, while using less memory.

My source code is available.

2026-07-25 23:07:52

When your program asks for memory that is not in cache, the processor has to go to RAM. That trip costs on the order of 100 nanoseconds. On a 3 GHz core, that is about 300 cycles of doing nothing.

Memory latency has not improved in ten years. The 2016 Broadwell answers a random access in 100 ns. The 2025 Turin, with DDR5-6400 and every advantage of a decade of progress, takes 140 ns. It got worse.

The good news is that a modern core does not have to sit still. It can issue a second request before the first one comes back, and a third, and a tenth. The number of requests a single core can keep in flight is its memory-level parallelism. It is one of the most important numbers in software performance, and one of the least advertised: you will not find it on a spec sheet.

Thankfully, memory-level parallelism has improved a lot. To measure it, I use my testingmlp benchmark. The idea is a pointer chase. We build a 1 GiB array containing a single random cycle covering every element: each element holds the index of the next. Following the cycle is inherently serial. Each load has to complete before you know the address of the next one, so a single chase measures pure memory latency and nothing else. Then we run several such chases at once, from different starting points on the same cycle. We call these lanes. With two lanes, the core has two independent loads to work on. With twenty, twenty. We increase the number of lanes and watch the throughput. When adding a lane stops helping, we have found the limit. As my metric, I use the total estimated bandwidth.

I ran experiments on the Amazon cloud (AWS). The bandwidth shape is the same everywhere: a steep, nearly linear climb as we add lanes, then a knee, then a plateau. 

How did it evolve over time? Intel went from 10 to 30, meaning that a single Intel core can sustain 30 memory requests at once in practice. AMD went from 15 to 58. Graviton went from 6 to 19.

Intel was flat for a long time. Broadwell and Cascade Lake both sit at 10 concurrent misses. Ice Lake doubled it to 20. Granite Rapids is at 30. Intel has roughly tripled in a decade, with all the gain arriving in the last two generations.

AMD started ahead and stayed ahead, then jumped. Naples was already at 15 in 2018, when Intel was at 10. Milan reached 22. And then Turin does something different in kind: 58 concurrent cache lines from a single core.

Graviton 1 was a toy: 6 concurrent misses. Graviton 2 doubled it, Graviton 3 went to 17, and then Graviton 4 essentially stood still at 18. Graviton 5 only reaches 19. But look at the latency panel: since 2017, Graviton 5 is the only chip in this entire collection that made a random access faster than its predecessor. AWS advertised better DRAM latency for Graviton 5, and that claim holds up.

So who wins? On bandwidth and memory-level parallelism, it is AMD, and it is not close. The Zen 5 core in the m8a instances sustains 58 concurrent cache-line fetches and 24.5 GiB/s of random-access throughput from one core. AMD is roughly twice as fast as Intel.

The raw output, the system information from each machine, and the scripts are in the usual place.

Note that Apple Silicon does even better, but it is another category.

2026-07-25 04:13:57

Every week, I discuss with people who want to get a PhD. For years, I have been advising people not to pursue a PhD. It may come as a surprise to some.

You would expect people with a PhD to earn more money. Individuals who complete doctorates tend to have higher cognitive abilities and greater motivation. But smarter people tend to earn more, period.

So do people with a PhD earn more?

Historically, PhD holders earn more, but the bulk of the observed advantage is concentrated among those who get a professorship after the PhD. And there is no certain path from the PhD to a professorship. We have been producing many more PhDs than we have professorship, for decades. And the disparity is ever growing.

When I entered university at the beginning of the 1990s, about 0.5% of the Canadian population had a PhD. This has nearly tripled today, and it is fast increasing. Something of the order of one person out of 80 has a PhD. Comparatively, there is roughly one professor or university-level instructor per 900 people. With a fast aging population, we simply do not need many more professors and instructors than we already have.

There are specific fields where some jobs are difficult to get without a PhD. Machine learning is one such example. Many people in the industry have a PhD, and they tend to select those who also have a PhD. Further, there is a somewhat direct relationship between the work you might do during your PhD, if you are any good, and the actual work you might do later. It is much less clear in a lot of other disciplines.

The most significant economic cost of a PhD is not tuition but the years of delayed full-time earnings and career progression. In the tech industry, it is typical to award half a year of experience for each year spent on a PhD. This means that even though the individual starting with a PhD might earn more starting out, they are not necessarily getting a higher lifetime income.

Benjamin et al. (2025) find that the early-career benefits a PhD can be effectively zero:

In the short run, pursuing a PhD entails substantial opportunity costs. Early-career earnings for PhD graduates are significantly lower than those of individuals with master’s or professional degrees, reflecting prolonged enrolment and delayed entry into the labour market. These costs are especially high for non-completers, particularly those who exit the program after several years without earning a credential. Over the lifecycle, earnings do eventually recover (and surpass those of bachelor’s and master’s graduates) but only under specific conditions. The most favourable long-run outcomes are concentrated among those who secure academic employment and remain in full-time work late into life. This “double premium,” combining higher earnings and longer careers, plays a central role in shaping the average return to a PhD. Outside academia, PhD holders resemble master’s graduates in both earnings and employment patterns.

Thus, the financial case for a PhD is narrower than people assume. If you fail to get a professorship, or you want an early retirement, you may very well end up with a poor outcome. And it is not getting better over time.

References

• Altonji, J. G., & Zhu, Z. (2025). Returns to specific graduate degrees: Estimates using Texas administrative records (NBER Working Paper No. 33530). National Bureau of Economic Research. https://www.nber.org/papers/w33530
• Benjamin, D., Miloucheva, B., & Vigezzi, N. (2025). The opportunity cost of a PhD: Spending your twenties (Working Paper No. 802). University of Toronto, Department of Economics. https://www.economics.utoronto.ca/public/workingPapers/tecipa-802.pdf
• Cooper, P. Is grad school worth it? A comprehensive return on investment analysis. Foundation for Research on Equal Opportunity. https://freopp.org/whitepapers/is-grad-school-worth-it-a-comprehensive-return-on-investment-analysis/

2026-07-22 22:32:18

In 1945, Vannevar Bush published a report entitled Science: The Endless Frontier. His thesis was that prosperity follows from basic research. The report was highly influential in the United States and elsewhere. It led to the creation of an entirely new government bureaucracy.

With this report, Bush popularized the linear model of innovation: innovation (such as medical cures) flows sequentially from basic research to applied research to development to production and diffusion. Grow basic research, and the rest will follow.

When Bush wrote his report, basic research was not usually supported directly by the state. We did not have a large basic research infrastructure. And yet, the West had just lived through an unprecedented period of rapid scientific progress: the theory of evolution, electromagnetism, radio communication, special and general relativity, quantum mechanics, nuclear technology, rockets, the combustion engine, and more. We would get the invention of the transistor only two years after Bush’s report. We also did not have today’s peer-review mechanism.

Even though Bush’s report has been viewed as a piece of genius that unlocked a golden era of scientific prosperity, I believe that the linear model of innovation is hopelessly naïve. I believe the thesis that a large bureaucracy delivering funding to other bureaucracies (such as universities) is how we get innovation is absurd. Except perhaps in the domain of computing (“bits”), we have been largely stagnant technologically since about the 1970s. So Bush’s model failed over time. To be clear, it might have worked for a while by encouraging more young people to study engineering and science. It may also have shone a favorable light on a few enterprising professors who got to promote useful ideas.

If you visit a research lab today in a leading university, what you are most likely to see is a boring bureaucracy that caters to whatever is politically favorable at the moment—a bureaucracy that plays it safe and avoids controversy. You see young people seeking well-paid jobs, going through the motions with often little genuine interest in, say, curing cancer. We have never published so many research papers—the volume has been growing exponentially ever since Bush wrote his report—but it is doubtful that this is how technological breakthroughs are achieved.

The evidence is overwhelming that shoddy science is widespread. We have a severe reproducibility crisis: if you redo an experiment (even a highly cited one), you are likely to fail to reproduce the results. This affects psychology, medicine, and many other fields. The system does not particularly care because the incentives to get things right are not there. As long as the work is politically aligned, solidity of the results seems secondary.

There was a TV show (The Big Bang Theory) where the main character, Sheldon Cooper—an awkward genius—gets to work on crazy ideas. That is how Bush imagined it: fund young people like Sheldon Cooper, and you will get extraordinary breakthroughs. In the real world, Sheldon would not get very far on campus. I have met misfits like him. When they are incapable of playing the political game, the system crushes them. But even if that were not the case, extraordinary intelligence needs to be applied to the right problems to be of value. You could have a ChatGPT that is brighter than any of us in every possible way, and it could still be deployed simply to fill out forms faster and better than we do—it may not cure cancer.

The American government has just released what might be considered an update to Bush’s report. Michael Kratsios wrote a report entitled Science, A New Golden Age. The report states outright that the linear model no longer holds. It states what I have argued for vehemently: innovation is not a linear process. Take large-language models, for example, which can be used by engineers and scientists to further their research. I have also argued that the success of large-language models today has as much to do with the users as with the researchers.

At this point, some people engage in the following type of rhetoric: if we had not invented calculus, we would not have AI today; therefore, calculus caused AI. But you could also say that the subsidized nail factory in the Soviet Union, which made overpriced and bad nails, was necessary to hold Landau’s house together, and that without those nails we would not have the theory of Landau levels. The causality argument goes in all directions.

Innovation is the result of a complex system. We see that the United States and, more recently, China are innovative countries. In 2026, you do not go to France for the latest advances. The evidence is overwhelming that scientific and technological progress depends as much on culture as on anything else. It is not something to be managed by bureaucrats.

One of the cultural ingredients that seems essential is meritocracy. You must put the people who are good at building on top of your hierarchy. This does not happen magically. You need a set of incentives in which rewarding the wrong people is costly.

What does Kratsios propose? Many interesting ideas that, I expect, could renew our culture. He proposes to break out of the Cold War–era funding model. Today, the research funding mechanism is centered around the government giving money to the university bureaucracy. The grant might be in the name of one professor, but the recipient is still the university. In the new model, instead of funding universities, the government would assign money directly to individuals in various ways (short grants, prizes, and so forth). This would shift power away from administrators toward individuals who know how to get things done. It would also neutralize some of the political power of the current mandarin class of scientists who control access to the top positions.

The report recommends restoring permissionless innovation. It is sometimes poorly understood how limited the system has become. I once had a graduate student undertake interviews with practitioners. This required an ethics approval which, in her case, took a few months to obtain. Again, the system has built up political structures that seek to block innovation it does not like. They need to be torn down, the sooner the better.

The report has many other interesting recommendations. One that I particularly like is an AI-guided agenda. We need to hook up our brand-new AIs to experimental devices. We are not going to cure aging with chatbots. We need experiments on a massive scale.

Will Kratsios’s vision move from report to reality? History shows that cultural and institutional change is never easy. Yet the stakes could not be higher. By embracing meritocracy, permissionless innovation, and ambitious AI-augmented experimentation, we have a genuine chance to escape decades of stagnation and rekindle the spirit of discovery that once defined the West. The opportunity is before us. It must not be squandered.