2026-01-10 08:00:00
Industrial programming languages like C++ or Rust tend to have language-specific industrial build systems, designed for scale; Ninja was for projects with tens of thousands of source files.
Meanwhile, at the other extreme, small software projects often have some miscellaneous smaller build-like needs that span language boundaries, such as running a command to generate some source files or rebuilding the docs. At the industrial scale, tools like Bazel are designed to support builds that span toolchains. But in most projects these kinds of tasks often end up in the source tree as a random shell script, Makefile, or "task runner" config.
In my experience those approaches fall short. Some aren't aware of what's
already up to date and do unneeded work. Or you start with Makefiles, but
realize you want more than the basics and end up trying to write programs in the
Makefile $(foreach ...) language. Or you use some customized tool, but now
need your users to install another program just to build yours.
So here's today's idea: why not include your own build system in the source language itself?
"Real" build systems tend to express the build as a declarative graph of interdependent steps, which is the principled approach for scaling and parallelization. Zig, which is some nice prior art for writing the build files in the source language, takes this approach. The downside is now you are not writing build steps, you are writing programs that describe graphs of build steps; take a peek at the doit examples to see what that looks like.
What we're trying to do here instead is be more appealing than the other extreme, which is a shell script that maybe has a conditional or two.
What would be some properties of such a thing?
And all in some code that is simple enough that it doesn't feel like you're using a chainsaw to trim a flower.
I'll use some tasks from retrowin32 as motivation just to make the examples more concrete. For complex project-specific reasons retrowin32 parses its own source to generate some win32 DLL files, which means when you modify those sources you need to run the generation step again.
The commands we want to run look like the following:
# for each DLL, e.g. "kernel32", "user32", etc:
$ cargo run -p win32-derive user32 # generates user32.s, the input to next step
$ clang-cl ...many flags here... user32.s /def:user32.def /out:user32.dll
In pseudo-Rust you might rewrite the above as follows:
fn build_dll(name: &str) {
run_command(&["cargo", "run", "-p", "win32-derive", name]);
let asm = format!("{name}.s");
let def = format!("{name}.def");
let dll = format!("{name}.dll");
run_command(&["clang-cl", asm, format!("/def:{def}"), format!("/out:{dll}")]);
}
fn build_dlls() {
for dll in ["kernel32", "user32", ...] {
build_dll(dll);
}
}
(In this post I'll use Rust, but the main point is that the whole framework is small enough that for your project you could just as well implement it in your own code.)
So far we've just translated what would be a pretty simple shell script into some uglier Rust, which is pretty much a loss, but we can build from here.
Add a function that for checking whether some files are up to date:
/// Return true if all output paths in outs are newer than all of the paths in ins.
fn up_to_date(outs: &[&str], ins: &[&str]) -> bool { ... }
We can then only run commands if they are needed:
fn build_dll(name: &str) {
let inputs_that_generate_asm = ...;
let asm = format!("{name}.s");
if !up_to_date(&[asm], inputs_that_generate_asm) {
run_command(&["cargo", "run", "-p", "win32-derive", name]);
}
let def = format!("{name}.def");
let dll = format!("{name}.dll");
if !up_to_date(&[dll], &[asm, def]) {
run_command(&["clang-cl", asm, format!("/def:{def}"), format!("/out:{dll}")]);
}
}
With my Ninja hat on my first reaction to this is to worry "wait, this might be doing more disk lookups than needed!" But the nice thing about the intention of working at a small scale is that this just doesn't matter much.
We could sprinkle some print statements to show what's going on. But you'll note the work is kind of hierarchical, matching the control flow: the "build dlls" step runs one step per dll and those steps themselves run two commands. We can pass around a context object that lets us name these.
struct Task {
desc: String,
}
impl Task {
/// Make a new subtask name and immediately run the given function with it.
fn task(&self, desc: &str, f: impl FnOnce(Task)) {
let desc = format!("{} > {}", desc, self.desc);
println!("{desc}");
f(Task { desc });
}
}
fn build_dll(t: Task, name: &str) {
t.task("generate source", |t| {
if !up_to_date(...) {
run_command(&["cargo", ...]);
}
});
t.task("compile+link", |t| {
if !up_to_date(...) {
run_command(&["clang-cl", ...]);
}
});
}
fn build_dlls(t: Task) {
for dll in ["kernel32", "user32", ...] {
t.task(dll, |t| build_dll(t, dll));
}
}
Now when we run, we print a nice trace of output progress like:
dlls > advapi32.dll > generate source
dlls > advapi32.dll > compile+link
dlls > comctl32.dll > generate source
dlls > comctl32.dll > compile+link
If you'll allow a bit of terminal trickery, you can replace the println! with
something like:
print!("\r\x1b[K{}", msg);
std::io::stdout().flush().unwrap();
which causes each line to overprint the previous one, keeping the output to just showing one line of what is currently being worked on.
The above executes the build steps serially. Conceptually, when we have a loop like:
for dll in ["kernel32", "user32", ...] {
t.task(dll, |t| build_dll(t, dll));
}
we potentially instead could run each of those task calls in parallel, then
wait for them all at the completion of the loop.
At the small scale we're worried about, we might as well do this by just spawning a bunch of threads! Threads aren't free but they are pretty cheap, so as long as we don't have thousands of tasks we don't need to worry about running too many. (If we did care, adding in a semaphore isn't too bad.)
std::thread::scope(|scope| {
for dll in ["kernel32", "user32", ...] {
t.spawn(scope, |t| build_dll(t, dll));
}
});
// std::thread::scope implicitly waits for all spawned tasks
Again, from the production build system perspective, this "wastes" a thread to
block on std::thread::scope waiting for all its tasks to finish, but again at
a small scale this doesn't cost much.
Using the approach of cargo xtask, we
can integrate the above into an easy to execute command by putting the code in
its own crate and creating a project-local .cargo/config:
[alias]
minibuild = "run -q p minibuild --"
Now, invoking cargo minibuild from the shell will first (using Rust's build
system) rebuild this build system, then invoke it. (On a platform like Node you
would comparably use the scripts block of package.json.)
(By the way,
Make your own make
from 2018 had similar goals to this post, and was the motivating post for
cargo xtask as well. While I was drafting this post he additionally wrote
another post that goes
further! Relative to that post I think my best ideas are conditionally executing
commands and the hierarchical task status.)
Readers who know Rust may notice the above fudged language correctness like proper borrows and error handling. For the purposes of this post these details are relatively uninteresting and in a different high-level language things would be different.
In fact, in writing this post I realized that the careful error handling I had
written using anyhow::Result everywhere only served to make the code clunkier.
For our purposes, panicking on any unhandled error is both simpler code and
showing a stack trace is a more useful user experience anyway. (It also
integrates nicely with std::thread::scope, which forwards panics.)
Similarly, one way to implement task parallelization is to make t.task()
return a Future. I tried implementing this and it worked, but async Rust means
all the functions become async, which then leads to lifetime complexity, awaits
all over the place, needing to box the closures, and so on. It's definitely
possible but the result felt pretty ugly.
The full code is
here. lib.rs is
the build framework, under 150 lines of code. It includes a few features not
mentioned in this post, such as an "explain" mode where it prints why it
believes a given target is out of date before executing it, and buffering
command output so parallel commands don't interleave their output.
main.rs is the retrowin32 project's particular build steps, the sort of thing
you might use as a user of it. But this whole idea is that this is not a crate
you ought to pull in, but rather some simple code you could write yourself.
Is this a build system, or a glorified shell script? I think the distinction is better thought of as points along a spectrum, starting at "run these commands from the README" to "run this shell script" to the idea from this post to Makefiles to meta-Makefile systems, with the big guns like Bazel at the other extreme.
And I think it's a pretty useful point in that space. In this code you can see some advantages of using a full programming language, including static types, vectors, and path manipulation. Could I have written this as a shell script or Makefile? Surely yes, but also surely I would get something wrong.
2025-09-16 08:00:00
I write this blog. Does anyone read it? How could I tell?
In the old days of the web your web server recorded a log when a page was requested, and various tools would analyze those logs to tell you about your visitors. Today these logs are mostly useless when it comes to looking at human traffic, because the majority of traffic is bots, especially now that the AI companies are running their own web crawls.
Some bots like Googlebot label themselves with the User-Agent header or
ip range
but there are many other bots, including those that identify themselves as a
browser. (My decades out of date recollection is there was a mechanism at Google
as well to fetch pages in a way that didn't appear to be a bot.)
Instead, today's web access logging uses JavaScript. A script on the page gathers information about the visitor and POSTs it to some logging endpoint. This is how, for example, Google Analytics works. Some random site claims it is used on more than half of all websites, which means using Google Analytics gives Google gets yet another hook into where ~everyone on the internet is browsing.
"Telemetry" script is yucky, what could you do otherwise? Here's a trick that doesn't require JavaScript, but also doesn't work.
Embed an invisible image in every page:
<img src=/log width=1 height=1>
Bots that only fetch HTML and traverse links won't hit this logging endpoint.
The Referer header passed in the hits to the /log path will tell you the
page the <img> tag was on.
Unfortunately, bots these days are interested in images too.
What if you do use JavaScript? It turns out the fancy bots run JS too. What is something a bot won't do?
One idea I had is that a bot is unlikely to linger on any given page — they
have other places to go. I tried a script that used setTimeout to only record
the page load as a visit if the browser hung around for three seconds.
It appears to work better than the other things I've tried, but within a day I spotted the Baidu bot fetching my homepage and then three seconds later fetching the logging endpoint. Is it possible they're actually running the page script and waiting? Maybe I need a longer timer?
This blog is also published as a feed. Feed readers fetch its content and resyndicate it within their own UI. I haven't tried, but I doubt they'd run my script.
Some feed readers, when they fetch the feed, report how many subscribers they are acting on behalf of. Is that a count of human readers? I don't think so. When I used a feed reader in the past, I had subscriptions I sometimes didn't read.
On the one extreme, increasingly savvy bots will get ever closer to appearing like human traffic in logs. On the other, humans read via feed readers or without JavaScript and aren't logged anyway. Heaven forbid someone prints my posts and read them on paper, they're impossible to track!
This problem is an instance of a bigger pattern you might encounter in engineering: sometimes when you get down to implementing a measure, you find an endless maze of increasingly confusing corner cases. What if someone loads the page, but they only distractedly skim it? That's not really a reader, is it? What if someone loads the page and finds what they were looking for immediately, before the logging beacon runs?
What these kinds of problems can indicate is that you need to take a step back and reconsider what your real objective is. What is my objective? I think I write this blog for two reasons.
Taking the time to serialize my thoughts, chasing down all the holes my inner critic spots, is a way for me to consolidate and archive my knowledge. For that purpose I don't need anyone to read it but me.
I write for an imagined audience of another me, someone with my interests and skill level who didn't yet know the thing I learned. Sometimes I write the post that is exactly the thing someone else needed, and they end up reaching out. For this purpose I need an email address, not an access log.
2025-09-13 08:00:00
A while ago I was tinkering with a project involving AI and voice. Google's docs linked to suggested partners.
I created an account with one. Immediately when I tried to log in it refused with an error. I sent a message to their customer service — no response — and forgot about it. A few weeks later I got a personalized email from the company: "Hi, this is John in developer relations, we noticed you made an account but never used it, can we help you?" I immediately responded, "Yes, I want to pay for your product, but I cannot log in, can you help?" Again, no response.
Epic Games made a video game store to challenge Steam, complete with exclusive games. I wanted to buy one. Their store account system was apparently shared with their existing popular games. A gamer in the past had given them my email address to play a game, and though I have always controlled my email address and had never verified any account, they would not let me create an account because they believed my email was already associated with an account.
I contacted their support and they were able to clear the association. But still I could not create an account on the store — another different person was using my email address too!? I think I went through this process with their support with maybe four accounts before I was able to create one.
Blizzard's Battle.net was in a similar state. But they won't let you contact support unless you first log in with your email address. I used the fact that I control my email address to take over the account associated with my email address, but that now means they have a verified randouser1234 account associated with my email address that does not have my purchases on it. (I have a Blizzard account from an old email address that I am attempting to get rid of.)
Someone wanted to send me some cryptocurrency. I prefer pieces of paper with pictures of US presidents, so I tried creating an account on Coinbase, which I understand to be a popular tool for turning cryptocurrency into US dollars.
After creating an account it took me to some intermediate page (I forget the details — something about account verification?) that was half blank. Digging in the browser tools I could see there was some JS exception in React that was getting caught and reported back to the server. I never successfully got an account.
Various sites now offer to use passkeys to let you sign in. I must've clicked ok in the past because whenever I tried it later I got prompted for a PIN that I didn't know, with no visible mechanism to reset it.
Thankfully in this case the guy responsible for passkeys at Google was my old officemate and he was able to walk me through it. Short answer is that Chrome on iOS can prompt you for a "Google Password Manager" PIN, but there is no way to reset it on mobile, nor via the Google Password Manager website. Instead there is a third Google Password Manager UI within desktop Chrome that I needed to use.
As posted there, once I attempted to actually use the passkey to sign into GitHub, I got an error message telling me to use a password instead. The existence of that error text means someone had to write the code that forwards you from one to the other.
For another project I wanted to be able to send a very small number of emails. A friend recommended Amazon SES, part of the AWS suite. I went to create an AWS account: "nope, your email address is already associated with an account". I click sign in using my email: "nope, no account associated with that email".
I contacted them. Customer service sent me a useless reponse that suggested I reset my password.
(From someone else on Reddit with the same problem I discovered a workaround
that appears it might work: you can sign up for AWS with
[email protected].)
Forget trying to build new features that attract customers. Here I've given multiple cases where I was already ready to pay and could not do the most basic first step with these products. And I am not even a complex case.
The economies of scale with most big companies like these are that it's not even worth figuring out what any one user's problem is, it's better to just mark them off as X% lossage while trying to find new users elsewhere. I get it, but it's maddening.
My own brother did something (it's unclear, but nothing nefarious) and lost access to his Google account, which meant losing access to many personal files, and for whatever reason all their recovery processes refused him. He eventually gave up.
