Alex WlchanModify

2026-08-20 17:49:28

I’m working on a project that generates multiple .tar.gz archives, and I need to combine them into one final file. I thought I could just cat the bytes together, but that doesn’t work. This seemingly simple task exposed my flawed understanding of tar and gzip.

To my fix my code, I first had to fix my mental model – and that took me into tape drives, patent laws, and end-of-file markers.

tar stands for tape archive

tar is a file archiver that combines multiple files and their metadata – filenames, timestamps, directory structure – into a single file.

It was originally designed for magnetic tapes, and the file structure is informed by the physical constraints of that medium:

1. Sequential reads. Magnetic tapes are most efficient when you start at the beginning, and play forward to the end of the tape.

2. Append-only writes. Early tapes could only append data to the end of a record, not replace existing data.

3. Fixed data sizes. Tapes have a fixed capacity, and early tapes had fixed data block sizes.

Internally, a tar archive is a sequence of files, each broken into fixed-size blocks. Files have a header block (with metadata like filename and file size) and data blocks (the file contents). After the files, there are two or more blocks filled entirely with zeroes. These form an end-of-file (EOF) marker that tells a reader to disregard everything else in the archive.

This structure mirrors physical tape: you can read files sequentially or append new ones to the end. That sequential design is why tar remains popular for streaming over a network – you can process incoming files immediately, without waiting to download the complete archive.

Knowing this structure helps me understand aspects of tar that I previously found confusing:

• File sizes must be declared upfront. You need to write the file size in the header before you write any data blocks. When I use Python’s TarFile.addfile API, I often forget to set tarinfo.size, so Python writes 0 to the header and creates an empty archive.

• Archives can contain duplicate filenames. You can’t edit or delete existing blocks on tape, so you update a file by appending a new version with the same filename. When you unpack the archive, the later file overwrites the earlier one.

• Everything after the EOF marker is ignored. Because physical tapes have fixed capacities, the EOF marker signals where data ends and empty tape begins. While tools like GNU tar have an --ignore-zeros flag to keep reading past EOF markers, I want to build archives that can be read with the default settings.

I tried a naïve approach of cat-ing tar archives, but that fails because readers stop at the first EOF marker. Instead, I’m combining archives using Python’s tarfile module. I unpack each archive, then copy its members into a new archive which will have a single EOF marker:

import tarfile

def combine_tars(output_file, input_files):
    """
    Combine multiple tar archives into a single archive.
    """
    with tarfile.open(output_file, "w") as out:
        for f in input_files:
            with tarfile.open(f, "r") as src:
                for member in src.getmembers():
                    out.addfile(member, src.extractfile(member))

combine_tars("numbers.tar", ["one.tar", "two.tar", "three.tar"])

This is more code than concatenating raw bytes, but it creates a tar archive that doesn’t need special settings to read.

gzip compresses a single stream of data

gzip is a stream compressor that takes a single file or data stream, and makes it smaller. The compression is lossless, so you can reverse it to retrieve the original file.

Unlike tar, gzip was a response to patent laws, not physical hardware. Reading RFC 1952 which defines the gzip file format, three design constraints reflect the time in which it was created:

1. Patent-free. The gzip tool was written as a free software replacement for compress, a comprssion tool whose underlying LZW algorithm was protected by patents at the time.

2. Streamable. Compressing or decompressing a gzip file must only use a small, bounded amount of memory. In the early 1990s, when RAM was even more scarce and expensive than it is today, the ability to process data in small, continuous chunks was essential.

3. Portable. A gzip file should be independent of the CPU, OS, filesystem, and other aspects of the computer it was created on. We take this sort of portability for granted today, but it wasn’t always a given.

Internally, a gzip file is a sequence of one or more “members”. Each member has a header (with metadata like original filename and modification time), the compressed data, and a trailer (with a CRC32 checksum and uncompressed size). The file ends after the final trailer – gzip doesn’t have EOF markers.

Conceptually, it’s tempting to see members as an analogue for files, but that’s not how gzip works. Tools treat multiple members as part of the same data stream, and you can’t list or extract them individually. When you uncompress a multi-member gzip file, you only get a single stream back.

Because members come one after another and there’s no EOF marker, you can concatenate gzip files by just cat-ing bytes:

echo "one uno eins"    | gzip > one.gz
echo "two duo zwei"    | gzip > two.gz
echo "three tres drei" | gzip > three.gz

cat one.gz two.gz three.gz > numbers.gz

gunzip --uncompress --to-stdout numbers.gz

How do you combine tar.gz archives?

tar and gzip are firm friends. tar combines a directory tree into a single stream; gzip makes that stream smaller. Because they both support sequential reads, .tar.gz is very popular for streaming data over a network – you can start processing individual files before you download the entire archive.

My mistake was trying to combine .tar.gz files using cat. gzip plays ball, but tar throws a strop.

gzip happily combines the compressed members into a single stream, but when tar tries to read the decompressed stream, it finds the first archive’s EOF marker and stops reading. gzip would be happy to carry on, but tar has given up.

To combine .tar.gz files safely, I have to extract the underlying members and write them to a new file. That means modifying my Python function above from plain read/write (r/w) to gzip-compressed read/write (r:gz/w:gz):

import tarfile

def combine_tar_gzs(output_file, input_files):
    """
    Combine multiple gzip compressed tar archives into a single archive.
    """
    with tarfile.open(output_file, "w:gz") as out:
        for f in input_files:
            with tarfile.open(f, "r:gz") as src:
                for member in src.getmembers():
                    out.addfile(member, src.extractfile(member))

combine_tar_gzs("numbers.tar.gz", ["one.tar.gz", "two.tar.gz", "three.tar.gz"])

This started as a confusing bug, but it became a fun side quest. Now I understand how these formats work, I understand why my original code doesn’t work, and I understand how I can fix it. I can go back to my project, safe in the knowledge that I haven’t missed a secret shortcut or an obvious optimisation.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2026-08-14 00:14:06

I wrote a post for the Tailscale blog about a long-running series of corruption incidents, and how they eventually led us to find an SQLite bug that predates my entire programming career. I’m incredibly proud of this, both the work and the blog post.

Before Tailscale, I was coming from smaller teams where I didn’t get to tackle problems of this scale or complexity. This was exactly the sort of tricky, deep technical challenge I wanted to be part of (though I’d rather it hadn’t been quite so stressful)! I’m glad I got to play a small part in these incidents, and I learnt so much from the more experienced engineers I worked with. I never want to hear the words “SQLite corruption” again, but if I do, I’d want to have Tailscalars at my side.

Writing the blog post has a blast, too. The piece transformed from a rough draft into a solid, engaging piece of writing, thanks to thoughtful feedback from many people at Tailscale. Most of my writing is self-edited, and it’s always a pleasure to work with a dedicated editor.

Please check out the blog post if you haven’t read it already – I think it’s a fascinating technical story, and one readers of this site are bound to enjoy.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2026-07-18 16:00:04

One of my favourite tiny details in this website is my non-breaking spaces. I have code that looks for phrases like “5 cm”, “New York”, or “Objective‑C”, and inserts a non-breaking space/hyphen so they’ll never be split across multiple lines.

This is the sort of typographical nicety that would be handled by a professional typesetter if I was writing a printed book with a fixed layout, but that’s not how websites work. My website is viewed at lots of different sizes, and browsers choose where to insert line breaks. I add these non-breaking characters so browsers know to avoid awkward line breaks.

Previously I was only applying this detail to body text, but today I implemented something similar for <code> elements.

I used a lot of inline code snippets in my last post, and while reviewing it I noticed that several of those snippets had unhelpful line breaks. For example, (?-u:…) was split into (?- and u:…), while the flag --multiline was split with - on one line and -multiline on the other. These line breaks make the post harder to read, with no benefit.

I can understand why they happened – browsers look for characters where they can break lines, and in English that includes hyphens. It’s usually fine to split hyphenated words over multiple lines, but it’s annoying when that happens in code.

I could fix this by replacing the hyphen in my <code> with a non-breaking hyphen, but people copy/paste code snippets and that might change the meaning.

Instead, I wrote a check that looks for <code> elements which are short and contain a line-breaking character, then adds the nowrap CSS class.

import re

def add_nowrap(match: re.Match[str]) -> str:
    """
    Add the `nowrap` class to a `<code>` element if it contains line
    breaking characters.
    """
    contents = match.group("contents")
    if "-" in contents or " " in contents:
        return f"<code class=\"nowrap\">{contents}</code>"
    return match.group(0)

text: str

# Add the `nowrap` class to <code> snippets which are short and
# contain line-breaking characters.
#
# The limit of 15 characters is arbitrary. In longer code snippets,
# wrapping is preferable to avoid leaving excessive whitespace on
# the previous line.
text = re.sub(r"<code>(?P<contents>[^<]{1,15})</code>", add_nowrap, text)

This is paired with a CSS rule that uses the text-wrap property to tell browsers not to wrap across lines:

code.nowrap {
  text-wrap: nowrap;
}

This wasn’t necessary, but I think it makes the site slightly nicer.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2026-07-18 15:11:26

Recently I’ve been tidying up the subtitles in my local media library. There are two popular file formats for subtitles: SRT (SubRip Subtitle) and WebVTT (Web Video Text Tracks).

I’ve been standardising on WebVTT because it works with the HTML5 <video> element, and I play all my videos through the <video> element embedded in static websites. However, lots of subtitles are only available as SRT, so I wrote a Python function to convert SRT files to WebVTT. The formats looked simple and the conversion seemed straightforward. Famous last words!

When I spot checked the converted subtitles, I noticed a bug in my handling of byte order marks (BOM), and it took several steps to fix.

Failing to look for the UTF‑8 BOM

A byte order mark is a special use of the zero width no-break space character U+FEFF at the beginning of a text file, which tella a program reading the file about how the text is encoded. It depends on the exact sequence of bytes used to encode the character. Here are a few examples:

• EF BB BF – the file is UTF‑8 text. UTF‑8 always has the same byte order, so it’s just telling us about the encoding.
• FE FF – the file is UTF‑16 text, with big-endian byte order (UTF‑16BE).
• FF FE – the file is UTF‑16 text, with little-endian byte order (UTF‑16LE).
• 00 00 FE FF – the file is UTF‑32 text, with big-endian byte order.

All of my SRT input files were UTF‑8 encoded, and some of them had the UTF‑8 byte order mark, and I wasn’t handling it correctly. For example, suppose I had this input SRT file:

<U+FEFF>1
00:00:01,001 --> 00:00:10,010
You have grown, Keyne.

2
00:02:00,002 --> 00:20:00,020
Soon you’ll be needing another name.

When I convert to WebVTT, I want to add the WEBVTT header, remove the sequence numbers, and change the timestamp format.

To remove sequence numbers, I was checking if a line was all digits. Because the BOM is on the same line as the first sequence number, the line isn’t all digits, so I didn’t remove it. Instead, I copied the entire line into the middle of the WebVTT file, BOM and all:

WEBVTT

<U+FEFF>1
00:00:01.001 --> 00:00:10.010
You have grown, Keyne.

00:02:00.002 --> 00:20:00.020
Soon you’ll be needing another name.

The correct conversion would remove both the byte order mark and that first sequence number:

WEBVTT

00:00:01.001 --> 00:00:10.010
You have grown, Keyne.

00:02:00.002 --> 00:20:00.020
Soon you’ll be needing another name.

In my local media library, I can assume everything is UTF‑8. I can safely remove the byte order marks, and my web browser will still decode my subtitles correctly.

Fixing the converter with encoding="utf-8-sig"

In my first fix, I tried to handle the BOM manually. I wrote code that looked for U+FEFF and stripped it from the file, trying to detect it and re-insert it into the converted WebVTT file. (This was before I realised I could just remove it entirely.) It was a bit messy, because I was mixing low-level text encoding code with my high-level subtitle conversion steps.

As I was researching this article, I realised there’s a more elegant solution: if I open the SRT file with encoding="utf-8-sig", Python will automatically detect and skip the optional UTF‑8 encoded BOM at the start of the file. The rest of my code doesn’t know or care that it’s there.

I fixed the bug in my function, which means future conversions will work correctly – but what about the broken files I’ve already generated?

Detecting the UTF‑8 BOM with ripgrep

Initially I tried searching for U+FEFF with TextMate, but it crashed consistently with that search, so I turned to command-line tools.

I use ripgrep for searching text. By default it does “BOM sniffing” on files – when it reads a file, it looks at the first few bytes, transcodes the file from its actual encoding to UTF‑8, then executes the search on the transcoded version. This is exactly how the BOM is meant to be used, but it’s less helpful if the BOM itself is what you’re searching for!

Instead, we can disable ripgrep’s Unicode support and search raw bytes by using (?-u:…) in the regular expression. (This flag comes from Rust’s regex crate.) The following command looks for lines that start with the UTF‑8 BOM:

$ rg '^(?-u:\xEF\xBB\xBF)'

If you were only looking for the BOM at the start of the file, you’d also want the --multiline flag. That changes the caret ^ to anchor to the start of the file, not the start of any line. But since I’m looking for BOMs which are in the middle of the file, omitting --multiline is correct.

This search threw up dozens of files with a broken BOM. Initially I opened the broken files in TextMate and edited them manually:

$ rg --files-with-matches --null '^(?-u:\xEF\xBB\xBF)' | xargs -0 mate

But I quickly realised this was too slow, so I wrote a Python script to clean up all the files at once:

#!/usr/bin/env python3

import glob

for filepath in glob.glob("**/*.vtt", recursive=True):
    with open(filepath, "rb") as f:
        content = f.read()
        
    if b"\xef\xbb\xbf" in content:
        content = content.replace(b"\xef\xbb\xbf", b"")
        with open(filepath, "wb") as f:
            f.write(content)
        print(filepath)

Once I’d run this script, I used my ripgrep command to check it was correct – and indeed, all the erronous BOMs had been stripped from my media collection. I also track my subtitle files in a Git repo, so I could confirm the script didn’t introduce other changes.

Before this bug, I’d only vaguely heard of byte order marks, and I’d never had to tackle them in anger. This sort of lesson is exactly why I love managing my local media archives as hand-built static websites – the lo-fi approach gives me lots of opportunities to explore low-level ideas and learn how things actually work on my computer.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2026-07-14 04:44:05

At work, we use a standard Git workflow: develop on a feature branch, push to GitHub, and open a pull request to main. Once somebody else approves the PR, the changes get merged.

At least once a week, I forget to branch and commit changes directly to my local main.

I only realise my mistake when I try to push and GitHub blocks me. To untangle myself, I have to create a new branch with my current state, push that instead, and then reset my local main back to origin so I can pull other people’s changes. This isn’t difficult to fix, but it’s annoying – especially when I often forget to clean up my local main until the next time I try to pull.

To stop me getting into this state, I’ve written a Git pre-commit hook. I saved the following shell script in .git/hooks/pre-commit and made it executable:

#!/usr/bin/env bash

set -o errexit
set -o nounset

branch="$(git rev-parse --abbrev-ref HEAD)"

if [ "$branch" = "main" ]; then
   echo "You can't commit directly to main"
   exit 1
fi

The git rev-parse command prints the short name of the current HEAD. If I’m on a branch, it returns the branch name; if I’m in a detached HEAD state, it returns HEAD.

If the hook detects that I’m on main, it exits with an error code. This aborts the commit and prevents it being saved, serving as a friendly reminder to create a feature branch first – and leaving my local main completely clean.

Ideally I’d always remember to branch when I start a new piece of work – but since I don’t, I’m happy to let the computer remember instead.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2026-07-06 02:01:19

I take a lot of photos, but I only keep a fraction.

Every week, I go through my camera roll and sort my photos into three buckets: keep, delete, or “needs action” (pictures like paperwork or screenshots that I need to do something with, but don’t want to keep indefinitely). This lets me filter out repetitive, blurry, or uninteresting shots.

A few years ago, I reviewed my entire 30,000 item library and I deleted almost a fifth of my photos, but I haven’t missed any of those photos. If anything, browsing my photo library has been nicer, because the average quality went up. My photo library only contains my best pictures, not just everything I’ve ever taken.

Recently I added a new step: I now write a description for every photo I’m keeping. Typically the description is a sentence or two of context, like what I was doing when I took the photo, or how I felt when I did. A lot of this context isn’t obvious from the image, and over time I forget those details. Sometimes I can piece it together later from my calendar or journal entries, but other times I just have a mystery image and I can’t remember why it was important.

This slows down the review process, but it only takes a minute or so per photo, and each description makes the photo library a more useful visual record of my life. When I revisit those photos, the memory is stronger because I recall more of the context.

This change has forced me to be more thoughtful about which photos I keep. If I can’t write even a sentence or two about what this photo means to me, is it worth keeping? Will I ever look back on it with fondness?

Even this minimal level of descriptive text is quite unusual. When I was working on the Data Lifeboat project, a lot of the early designs were built around rich uploader-supplied metadata, like title and description. These designs fell short in practice, because most people leave them empty. The vast majority of Flickr pictures don’t have a title or description, and those are photos people want to share! (The auto-generated filename from the camera doesn’t count.)

I’ve already written descriptions for about 5% of my photo library, and that number will gradually ratchet up. I’m writing a description for every new photo, and I’m slowly working backwards while the memories are still fresh.

I really recommend writing descriptions for some of your photos, whether that’s in a social media post, your digital photo library, or a printed album. A sentence of context can really anchor a memory.

If you open your own camera roll and look at the last photo you took, do you remember how you felt in that moment – or is it already starting to slip away?

Building it into Blink

Initially I was adding descriptions using Apple’s Photos app, but I really wanted to include them in Blink – a tiny Mac app I wrote a few years ago to review my photos. Blink lets me sort my library entirely using keyboard shortcuts, and I wanted to write descriptions without breaking that fast, mouse-free flow.

The Blink interface is intentionally sparse: there’s a horizontal thumbnail strip at the top, then the focused photo takes up most of the window. I use arrow keys to switch between photos, and I press 1/2/3 to categorise them (keep/delete/needs action).

To add descriptions, I added an overlay at the bottom of the window. It shows the current caption, or a placeholder if I haven’t set one yet:

If I press space, the overlay switches to a text field where I can enter a new description, or edit the existing description:

When I press ⌘+Enter, the description is saved to the “caption” field in my photo library. I never have to take my hands off the keyboard, so this doesn’t introduce much friction to my workflow.

Adding this to Blink was a fun exercise in revisiting old code.

Returning to an untouched app

Blink is written as a Swift app using SwiftUI, and this was my first time working with Blink and SwiftUI since September 2023. I’ve written short command-line scripts in Swift in the interim, but not a GUI app.

It took a while to get comfortable working in the Blink codebase again. As always, I wish I’d left more comments and documentation when I wrote this code. Names like FocusedImage or AssetHelpers clearly meant something three years ago, but I’d forgotten the meaning and had to relearn it.

Comments are something I can always get better at. I was fortunate to start my career at a company that had a very verbose commenting style, so I learnt some good habits early, but then I cycled through a few jobs where the standards were more lax, and my comments suffered. I was in the middle of that period in 2023, and the Blink code reflects that. My current workplace has a much stronger commenting culture, and I can feel that muscle strengthening again.

If you read my patch to add captions, you’ll notice it has much more commentary!

Intensely personal software

I consider Blink to be a tremendous success. I’ve used it to review thousands of photos since 2023, transforming my photo library from a digital dumping ground to a curated collection of highlights.

I initially wrote Blink as an experiment to learn SwiftUI and Mac development. I’d have been happy if I never used the code, but instead, it’s become an app I use every week.

As far as I know, nobody else has ever used it. The source code lives in a public repo, but it’s for educational interest rather than distribution. (The code is so tied to my machine that it crashes if pointed at anybody else’s photo library.)

I’ve considered cleaning it up, packaging it, and turning it into a “proper” app that other people could use, but that’s a lot of work I don’t find exciting, and it’s unclear if there’s any interest among people who aren’t me.

For now, Blink will continue to have exactly one user. That user doesn’t need an onboarding flow, a settings screen, or cloud syncing. They just want a keyboard-driven app to curate their photos, and write down the context before the memories disappear. For that user, it’s a five-star app.

[If the formatting of this post looks odd in your feed reader, visit the original article]