2026-08-27 08:00:00
In the hours following the release of CVE-2026-41992 for the project GNU gzip, site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix decompressing two files in the same invocation of gzip (that's possible???) an attacker can trigger out of bound reads in the LZH decoder, causing reads past the end of the shared global buffer that is somehow not reset between invocations. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Jayson Hilpert, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."
2026-08-26 08:00:00
Someone on the VS Code team decided to do a redesign of the product. This makes VS Code look like this:
This design is fine, I guess? It's got some rounded corners that look nice, I guess, but I really just want it to look like what I'm used to. I can tolerate this kind of redesign in my chat app, but I use VS Code professionally and kinda want things to look the same so I don't have to think as much.
You can revert the design change by setting "workbench.experimental.modernUI": false in your settings.json. Then it looks like this:
Who knows how long this is going to last, but at the very least it should work for now.
Thanks to jtagcat and Hugo ARNAL for letting me know about this setting.
2026-08-26 08:00:00
Today I woke up and saw a brand new VS Code error when I remoted into my coding box:
Trying to update my extensions didn't work. I still got that error. The only thing that worked was to downgrade to VS Code v1.124.2. This also undid the redesign that I kinda hate.
Obviously keeping things at this older version of VS Code is not a viable strategy, so maybe this issue will actually be fixed instead of just being closed for no reason.
Quality software reigns again!
2026-08-19 08:00:00
One of the most annoying parts of writing web applications is that in general: you can't trust browsers. But, you have to trust browsers at some level because that's how users interact with your software. As browsers get more capable with APIs like WebUSB, Built-in AI, or other absurd things; administrators want to be able to turn off the features that their web applications don't use. This is the crux of why Content-Security-Policies (CSPs) exist.
Normally we avoid acronyms when writing posts like this, but for the purpose of this article when you see "CSP", think "Content-Security-Policy".
In general, a CSP disables all browser features and then selectively enables the features the website actually needs. For example (stolen from the Anubis docs):
default-src 'none';
script-src 'self' 'unsafe-inline';
style-src 'self' 'unsafe-inline';
img-src 'self';
font-src 'self' data:;
connect-src 'self';
worker-src 'self' blob:;
base-uri 'none';
form-action 'self';
This disables all browser features except loading scripts from the same origin, inline JavaScript in <script> tags, inline CSS, loading CSS from the same origin, loading images from the same origin, loading fonts from the same origin, loading fonts inline to CSS files (via data: URIs), making fetch() requests to the same origin, loading Worker scripts from the same origin, loading Worker scripts from blob: URIs, disallowing the use of the <base> element, and only allowing HTML <form> actions against the same origin.
Extra fun, when you have a CSP that forbids loading Worker scripts from blob: URIs, you don't get the error until after the Worker is constructed and the browser forks a background thread:
blobURL = URL.createObjectURL(
new Blob([`console.log("Hello, world!");`], { type: "text/javascript" }),
);
const w = new Worker(blobURL);
// does not throw an error
You have to catch it in the async .onerror callback:
w.onerror = (event) => {
console.error(`Got an error: ${event}`);
};
So if you (like me) implemented fallback logic that depends on this, you need to adapt your logic to account for this.
Let's face it, users don't like it when they get an Anubis challenge page. I've tried to make them show up less often, but this doesn't scale as the scrapers adapt to the changes I make. One of the ways Anubis mitigates the pain of seeing a challenge page is by making it go away as fast as possible by running its proof of work checks run in parallel. This works out pretty well as most CPU advancements in the past decade or so are around multi-core performance, not single-core performance.
By default, when you create a Worker pointed to a JavaScript program on your web server, browsers make requests to the server to load that program:
const w = new Worker("/static/js/worker/test1.mjs");
This results in the browser sending a GET /static/js/worker/test1.mjs request to the server which hopefully results in getting JavaScript source back. The browser then executes that JavaScript code in parallel and sets up the worker environment so that the program can do whatever it is that it needs to do.
One of the horrible parts of this is that when you spawn many workers in parallel, such as how Anubis does it:
const getHardwareConcurrency = () =>
navigator.hardwareConcurrency !== undefined
? navigator.hardwareConcurrency
: 1;
let workers: Worker[] = [];
const threads = Math.trunc(Math.max(getHardwareConcurrency() / 2, 1));
for (let i = 0; i < threads; i++) {
let w: Worker;
try {
w = new Worker("/whatever/worker.mjs");
} catch (err) {
magic!(cleanup);
magic!(throwError);
return;
}
workers.push(w);
// Draw the rest of the owl
}
This results in threads number of HTTP requests to the server. In circumstances where the server is already overloaded (such as when scrapers attack in droves from nearly every ISO country code on the planet), this means that a user getting through to the webpage can result in as many as 16 extra HTTP requests to the server. Even worse, there's not an easy way to do exponential backoff without adding fiddly logic to the parts surrounding the Worker constructor.
In order to work around this, Anubis loads the worker source once from the server with a standard fetch() request and then packs that into a blob: URI so clients don't need to make many parallel requests to the server.
The old logic that fans out requests is maintained in case admins have a CSP that forbids the use of blob: URIs. It's kinda sucky that it has to be there and mandates adding extra testing to ensure this works, but in this era of late stage capitalism we kinda need to make sure that things are reliable on the client even if this can cause increased request pressure on an already overloaded server.
As a side note, this only really works because Anubis assumes that its worker code is inerrant unless something completely unrecoverable happens. Most of the proof of work code is "just math"*, so if the math fails then the user's CPU or ram is probably failing and the server will disagree anyways.
Ideally, you'd want the entire challenge solve attempt to get killed if any worker threads error after they start crunching solutions, but in practice it's "fine-ish" to lose a worker or two as long as there's at least one worker running.
One way to think about how the proof of work solver works is that each worker is a thread that gets its own "lane" of the nonce space to find solutions within. In general it's fair to assume that solutions are "dense" enough that losing any workers is "fine-ish" at the cost of skipping over solutions that may be in that "lane". Future improvements may involve trying to re-launch failed workers where they left off, but that is out of scope for now.
This is the kind of stuff I have to deal with when working on Anubis and why I end up writing essays in PR commit messages. Turns out most of this is edge cases. The joys of modern software know no bounds.
2026-08-15 08:00:00
While I was developing tooling for my job this weekend, I found a few blogposts that I don't remember writing:
Checking the date range, they coincide with when I was in the hospital earlier this year. I have removed them from the blog index and plan to rewrite them from scratch in my own words when I have the time.
I apologize for any inconvenience this may have caused. I apparently did not plan for the effect that hospital drugs (including fairly powerful blood thinners) would affect my mental state. In light of this, I will take steps to set up a sandbox environment for me to post in if I have future short term hospitalizations.
As my evaluation of my local tooling improves, I will update this post with more removed posts as they are discovered. I would request erring on the side of leaving me alone as I complete this process.
2026-08-11 08:00:00
Tigris has a pretty advanced replication scheme for writes. What happens when you actually need to delete things? Turns out deleting things is hard in distributed systems. Especially when you have a geo-replicated active-active database like Tigris does. We can (and do) use tombstones to mark where data once was, but how do you let people undo an accidental delete?
Tigris wants to turn storage inside out, so our implementation of soft deletion is by giving users the Recycle Bin for objects and buckets. Today we're going to dig into how this works, why it works, and what this gives you in terms of using object storage today.
In Windows and macOS, the Recycle Bin (or Trash can) is a form of purgatory where deleted files wait for their storage to be deallocated by the user. This allows users to hit "delete" fearlessly because if they made a mistake they can just drag it back out and go on with life.
This works great in your local filesystem because there's only one writer in one region. This kinda falls apart when you have multiple regions in your database and any one of them could be writing to it. How do you name things in the recycle bin? How do you handle the conflict of an update happening in one region before the deletion was fully replicated out from another region?
This is the fun of distributed systems, which is the kind of problem space that Tigris lives in.
One way to think about how the Recycle Bin works is that the file metadata gets moved there when the user hits delete. No data bytes move around on the disk, but the file doesn't show up in My Documents anymore. In a distributed systems context you can't just move the metadata around, you have to leave a tombstone behind to record where that metadata once was. This prevents other regions from being confused when actions happen really close to each other in time.
At a high level, a soft-delete is when a DELETE action doesn't actually remove the data. When data is soft-deleted, it's still there but just not visible in the main usage flow. This lets you get the data back when a delete is made by accident.
One of the interesting side effects of designing any API is that you end up leaking the internals of how your database works to your users. Many object storage systems were designed with overwriting or deleting data as one of the primary operations, and as such have had to bolt versioning onto the side. For the most part this does work; but once you get into advanced versioning schemes everything starts to fall apart. Tigris doesn't suffer from the same problems because we built immutability into the core from day one, and in immutable systems you have to append new data on the end instead of overwriting data.
At the least, actually storing the data en masse is a boring problem. You put the data somewhere, maybe name it after the checksum of its contents, and then have a daemon make sure it's copied three places. That daemon also handles cases when drives go offline and new ones are added to make sure data is shuffled around the cluster. This is largely a solved problem with projects like Ceph, Longhorn, or other distributed storage systems.
Some object storage systems like S3 expose platform internals to make soft deletion work. In S3 deleting an object creates a delete marker (tombstone). A delete marker is an explicit marker that the object is deleted and should not be returned in normal operation. Here's what that looks like in practice:
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ ◀── DeleteObject│ v1 │ │ v2 │ │ v3 current ││ report.pdf │ │ report.pdf │ │ report.pdf ││ …198086 │ │ …198088 │ │ delete marker │└─────────┬─────────┘ └─────────┬─────────┘ └───────────────────┘│ │ no data▼ ▼┌───────────────────────────────────────────────────────────────────┐│ sea of data ││ ┌──────────┐ ┌──────────┐ ││ │ v1 bytes │ │ v2 bytes │ ││ └──────────┘ └──────────┘ │└───────────────────────────────────────────────────────────────────┘// the bytes stay. only the newest record says the object is gone.
I don't know how I feel about this flow. Based on reading between the lines in the delete marker documentation it really feels like this is a leaked internal implementation detail of how S3's eventually consistent database works instead of a full fledged feature of the storage system. If I had to choose between leaking internal database details in the API and implementing a higher level API for something complicated like soft deletion, I'd want to implement the higher level API.
Let's rethink what soft deletes really are. What if they were like the Recycle Bin in Windows?
Soft-deletes are external references to buckets or objects that live in a different namespace from normal buckets or objects. We implemented them as external references instead of tombstones because this is effectively moving object metadata to the recycle bin. Tombstones mark the data as not being there, but soft-delete markers are a copy of the data that was there. This makes it easy to put the object back in place if you deleted it by mistake.
One way to think about objects and buckets is that they are garbage collection roots for points in the endless sea of data. Any data in the sea without a root anchoring it down is eligible to be deleted. Uploading multiple versions of an object with a forkable bucket creates multiple metadata entries at their different timestamped version numbers. You can then fork a bucket from any one of those timestamps to see what the bucket was like at that point:
every write appends a new version entryv1 v2 ╎ v3 v4┌────────────┐ ┌────────────┐ ╎ ┌────────────┐ ┌────────────┐│ …198086 │ │ …431907 │ ╎ │ …764522 │ │ …055310 ││ 4.1 MB │ │ 4.3 MB │ ╎ │ 4.4 MB │ │ 5.0 MB │└─────┬──────┘ └─────┬──────┘ ╎ └─────┬──────┘ └─────┬──────┘──────┴───────────────┴─────────╎─────────┴───────────────┴───────▶earlier ╎ fork point laterthe fork inherits these ╎ written later — the fork never sees them▼┌──────────────────────────────┐│ uploads/report.pdf ││ current version v2 ││ as of 1775929812004431907 │└──────────────────────────────┘// appending metadata instead of overwriting keeps any past instant addressable.
This would solve the soft-delete problem, but our existing database schema using FoundationDB requires us to enable forking and snapshots at bucket creation time. In essence, we need something that's halfway between what we have (each bucket being a globally mutable namespace) and the bucket forking land of every action being appending metadata onto the end.
To do that, we basically implemented most of that appending metadata on the end trick but to a different place: the soft deletion corner. When you enable soft-deletion and delete an object, its metadata gets moved to the trashcan so you can pluck it back into place:
main table · live keyspace soft-delete keyspace · newest first┌────────────────────────────┐ ┌──────────────────────────────────┐│ uploads/report.pdf │ │ uploads/report.pdf 3rd delete ││ live metadata record │───────▶ │ deleted …768707198086 ││ in ListObjectsV2 output │◀╌╌╌╌╌╌ └──────────────────────────────────┘┌──────────────────────────────────┐│ uploads/report.pdf 2nd delete ││ deleted …412888100731 │┌────────────────────────────┐ └──────────────────────────────────┘│ uploads/notes.md │ ┌──────────────────────────────────┐│ untouched by the delete │ │ uploads/report.pdf 1st delete │└────────────────────────────┘ │ deleted …104233715492 │└──────────────────────────────────┘───────▶ DeleteObject moves the record out — one entry per delete◀╌╌╌╌╌╌ RestoreObject moves the same metadata back// the bin entry is a copy of the metadata, not a marker. restoring is a move.
It's the same basic idea as the recycle bin on your desktop. Any buckets or
objects left in the recycle bin for long enough become eligible to be deleted,
which then makes the backend go and securely erase things. Effectively, any bits
of metadata in the soft deletion corner are still considered garbage collection
roots, they're just not shown when you do a normal ListObjectsV2 call.
The real fun comes into play when you remember that Tigris has a globally replicated active-active database where any region can change any object at any time. Most of the time things work out and objects are replicated without too much strife. The annoying part comes when two events are ordered weirdly. Imagine a scenario where one agent in one datacentre deletes an object after another agent in another datacentre:
ORD (Chicago) IAD (Ashburn)┌──────────────────────────┐│ PutObject · agent A ││ uploads/report.pdf │ ──── replicates PUT ────▶│ t = …768707198086 │└──────────────────────────┘┌──────────────────────────┐│ DeleteObject · agent B │◀── replicates DELETE ── │ uploads/report.pdf ││ t = …768984210773 │└──────────────────────────┘ORD applies PUT ▸ DELETE deleted, as expectedIAD applies DELETE ▸ PUT the put looks brand new// two writers, one key. the regions disagree about whether it exists.
How would this replicate out? Well for one each change is timestamped by when it's done in terms of Unix nanoseconds, so the replication messages kinda look like this:
produced first produced 277 ms later┌──────────────────────────────────┐ ┌──────────────────────────────────┐│ uploads/report.pdf │ │ uploads/report.pdf ││ op: PUT │ │ op: DELETE ││ LastModified 1775929768707198086│ │ LastModified 1775929768984210773││ block 0x3f2ac701 · origin ORD │ │ block 0x3f2ac701 · origin IAD │└──────────────────────────────────┘ └──────────────────────────────────┘last write wins984210773 is greater than 707198086// the delete carries the newer LastModified, so the delete survives
This means that in theory, a user could DELETE an object before an update is processed by another region, and that would make the regions disagree about if the object exists or not. This is a horrible state to be in and usually requires support intervention or to recreate/re-delete the object.
The root cause boils down to deleting objects actually deleting metadata from the database doesn't scale past a single region. Updates to metadata include the entire metadata object, so if you delete it locally and a new version is pushed remotely, the object will gain the remote state.
We don't want users to have to deal with that, so we added the concept of anti-resurrection to Tigris. Any write to an object must prove it is newer than the deletion.
ORD · agent PutObject IAD · user DeleteObject┌──────────────────────────┐ ┌──────────────────────────┐│ uploads/report.pdf │ │ uploads/report.pdf ││ new version · v2 │◀─────────────│ row deleted · marker kept││ t = 15 │ delete │ tombstone t = 25 │└────────────┬─────────────┘ └────────────┬─────────────┘└────────────────────┬────────────────────┘▼┌────────────────────────────────────────────────────────────┐│ at IAD: is the write newer than the marker? ││ write t = 15 · marker t = 25 · is 15 > 25 ? ││ no — not strictly newer, so the write is dropped ││ equal timestamps lose too · the guard runs for every bucket│└────────────────────────────────────────────────────────────┘// without the marker, an empty slot looks exactly like a key that never existed.
In this circumstance, a user sent a DeleteObject request to the IAD datacentre at time 25, but an agent sent a new version of the object with PutObject to the ORD datacentre at time 15. The user's delete is newer than the agent's put, so the new version is rejected and the delete gets sent back to ORD.
Tigris extends the S3 API by having users add headers to their requests. For example, to create a bucket with soft deletion enabled:
import (
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/tigrisdata/storage-go"
)
client, err := storage.New(ctx,
storage.WithGlobalEndpoint(),
storage.WithAccessKeypair(
os.Getenv("TIGRIS_STORAGE_ACCESS_KEY_ID"),
os.Getenv("TIGRIS_STORAGE_SECRET_ACCESS_KEY"),
),
)
_, err := client.CreateBucketWithSoftDelete(ctx, &storage.CreateBucketWithSoftDeleteInput{
CreateBucketInput: &s3.CreateBucketInput{Bucket: aws.String("my-bucket")},
RetentionDays: 30, // 0 uses the 7-day default
})
Or to list soft-deleted objects:
out, err := client.ListSoftDeletedObjects(ctx, &storage.ListSoftDeletedObjectsInput{
Bucket: "my-bucket",
Prefix: "uploads/",
})
if err != nil {
return err
}
for _, o := range out.Objects {
log.Printf("%s v=%s %d bytes deleted=%s", o.Key, o.VersionID, o.Size, o.LastModified)
}
Or to permanently delete one soft-deleted version:
_, err := client.PermanentlyDeleteObject(ctx,
"my-bucket", "uploads/report.pdf", "1775929768707198086")
When you have a soft-delete enabled bucket, you can also forcibly delete an entire bucket:
_, err := client.ForceDeleteBucket(ctx, &s3.DeleteBucketInput{
Bucket: aws.String("my-bucket"),
})
If you use this call on a bucket that doesn't have soft deletion enabled, you have permanently deleted your bucket. Please call this with care. Support cannot help you if you use this call wrongly.
And then bring it back from the dead:
trash, err := client.ListSoftDeletedBuckets(ctx, nil)
if err != nil {
return err
}
for _, b := range trash.Buckets {
log.Printf("%s (%d day retention)", b.Name, b.RetentionDays)
if _, err := client.RestoreBucket(ctx, &storage.RestoreBucketInput{Bucket: b.Name}); err != nil {
return err
}
}
Object storage entered our stacks as an unlimited FTP server we all used for
backups. A distressing amount of the world's most important data lives in object
storage buckets because it's the best place to put it. This is why having an
"undo" button matters, it's what makes it safe to trust your backups in the
cloud. To err is human, and mistakes are a "when" to plan for, not an "if" that
you hopefully never have happen. The blast radius of one overly wide
--recursive flag is measured in years of people's lives.
One of the biggest usecases that comes to mind is ransomware prevention. Imagine a case where an attacker downloads everything in your bucket, deletes it, and asks for a ransom to send you the files back. With Tigris, soft deletes means that the ransom can be ignored, you can un-delete your data, and be on your merry way with incident response. The other big usecase is for agents, where they somehow get the idea that deleting production data is the right way to solve a problem. Both cases mean you need a quick and fast way to go back to before things went wrong.
If you want true isolation instead of recovery, that's why we have bucket forking. Bucket forking needs to be enabled before a bucket is created, but you can enable soft deletion on any bucket in the dashboard whenever you want.
Every storage system is going to make you choose between ones that hide how the platform works and ones that expose the gorey internals to users. I think that hiding the internals and exposing the high level operations built on top of them is the right way to go, if only because the higher level operations are much easier to make safe in our globally distributed future.
Enable soft delete on any Tigris bucket, new or existing, and every delete becomes recoverable for up to 90 days. Restoring a whole bucket is one call. Read the soft delete docs.
