Alex WlchanModify

2025-03-26 00:53:43

A week ago, somebody added malicious code to the tj-actions/changed-files GitHub Action. If you used the compromised action, it would leak secrets to your build log. Those build logs are public for public repositories, so anybody could see your secrets. Scary!

Mutable vs immutable references

This attack was possible because it’s common practice to refer to tags in a GitHub Actions workflow, for example:

jobs:
  changed_files:
    ...
    steps:
      - name: Get changed files
        id: changed-files
        uses: tj-actions/changed-files@v2
      ...

At a glance, this looks like an immutable reference to an already-released “version 2” of this action, but actually this is a mutable Git tag. If somebody changes the v2 tag in the tj-actions/changed-files repo to point to a different commit, this action will run different code the next time it runs.

If you specify a Git commit ID instead (e.g. a5b3abf), that’s an immutable reference that will run the same code every time.

Tags vs commit IDs is a tradeoff between convenience and security. Specifying an exact commit ID means the code won’t change unexpectedly, but tags are easier to read and compare.

Do I have any mutable references?

I wasn’t worried about this particular attack because I don’t use tj-actions, but I was curious about what other GitHub Actions I’m using. I ran a short shell script in the folder where I have local clones of all my repos:

find . -path '*/.github/workflows/*' -type f -name '*.yml' -print0 \
  | xargs -0 grep --no-filename "uses:" \
  | sed 's/\- uses:/uses:/g' \
  | tr '"' ' ' \
  | awk '{print $2}' \
  | sed 's/\r//g' \
  | sort \
  | uniq --count \
  | sort --numeric-sort

This prints a tally of all the actions I’m using. Here’s a snippet of the output:

 1 hashicorp/setup-terraform@v3
 2 dtolnay/rust-toolchain@v1
 2 taiki-e/create-gh-release-action@v1
 2 taiki-e/upload-rust-binary-action@v1
 4 actions/setup-python@v4
 6 actions/cache@v4
 9 ruby/setup-ruby@v1
31 actions/setup-python@v5
58 actions/checkout@v4

I went through the entire list and thought about how much I trust each action and its author.

• Is it from a large organisation like actions or ruby? They’re not perfect, but they’re likely to have good security procedures in place to protect against malicious changes.

• Is it from an individual developer or small organisation? Here I tend to be more wary, especially if I don’t know the author personally. That’s not to say that individuals can’t have good security, but there’s more variance in the security setup of random developers on the Internet than among big organisations.

• Do I need to use somebody else’s action, or could I write my own script to replace it? This is what I generally prefer, especially if I’m only using a small subset of the functionality offered by the action. It’s a bit more work upfront, but then I know exactly what it’s doing and there’s less churn and risk from upstream changes.

I feel pretty good about my list. Most of my actions are from large organisations, and the rest are a few actions specific to my Rust command-line tools which are non-critical toys, where the impact of a compromised GitHub repo would be relatively slight.

How this script works

This is a classic use of Unix pipelines, where I’m chaining together a bunch of built-in text processing tools. Let’s step through how it works.

find . -path '*/.github/workflows/*' -type f -name '*.yml' -print0

This looks for any GitHub Actions workflow file – any file whose name ends with .yml in a folder like .github/workflows/. It prints a list of filenames, like:

./alexwlchan.net/.github/workflows/build_site.yml
./books.alexwlchan.net/.github/workflows/build_site.yml
./concurrently/.github/workflows/main.yml

It prints them with a null byte (\0) between them, which makes it possible to split the filenames in the next step. By default it uses a newline, but a null byte is a bit safer, in case you have filenames which include newline characters.

I know that I always use .yml as a file extension, but if you sometimes use .yaml, you can replace -name '*.yml' with \( -name '*.yml' -o -name '*.yaml' \)

I have a bunch of local repos that are clones of open-source projects, and not my code, so I care less about what GitHub Actions they’re using. I excluded them by adding extra -path rules, like -not -path './cpython/*'.

xargs -0 grep --no-filename "uses:"

Then we use xargs to go through the filenames one-by-one. The `-0` flag tells it to split on the null byte, and then it runs grep to look for lines that include "uses:" – this is how you use an action in your workflow file.

The --no-filename option means this just prints the matching line, and not the name of the file it comes from. Not all of my files are formatted or indented consistently, so the output is quite messy:

    - uses: actions/checkout@v4
        uses: "actions/cache@v4"
      uses: ruby/setup-ruby@v1

sed 's/\- uses:/uses:/g' \

Sometimes there's a leading hyphen, sometimes there isn’t – it depends on whether uses: is the first key in the YAML dictionary. This sed command replaces "- uses:" with "uses:" to start tidying up the data.

    uses: actions/checkout@v4
        uses: "actions/cache@v4"
      uses: ruby/setup-ruby@v1

I know sed is a pretty powerful tool for making changes to text, but I only know a couple of simple commands, like this pattern for replacing text: sed 's/old/new/g'.

tr '"' ' '

Sometimes the name of the action is quoted, sometimes it isn’t. This command removes any double quotes from the output.

    uses: actions/checkout@v4
        uses: actions/cache@v4
      uses: ruby/setup-ruby@v1

Now I’m writing this post, it occurs to me I could use sed to make this substitution as well. I reached for tr because I've been using it for longer, and the syntax is simpler for doing single character substitutions: tr '<oldchar>' '<newchar>'

awk '{print $2}'

This splits the string on spaces, and prints the second token, which is the name of the action:

actions/checkout@v4
actions/cache@v4
ruby/setup-ruby@v1

awk is another powerful text utility that I’ve never learnt properly – I only know how to print the nth word in a string. It has a lot of pattern-matching features I’ve never tried.

sed 's/\r//g'

I had a few workflow files which were using carriage returns (\r), and those were included in the awk output. This command gets rid of them, which makes the data more consistent for the final step.

sort | uniq --count | sort --numeric-sort

This sorts the lines so identical lines are adjacent, then it groups and counts the lines, and finally it re-sorts to put the most frequent lines at the bottom.

I have this as a shell alias called tally.

   6 actions/cache@v4
   9 ruby/setup-ruby@v1
  59 actions/checkout@v4

This step-by-step approach is how I build Unix text pipelines: I can write a step at a time, and gradually refine and tweak the output until I get the result I want. There are lots of ways to do it, and because this is a script I’ll use once and then discard, I don’t have to worry too much about doing it in the “purest” way – as long as it gets the right result, that’s good enough.

If you use GitHub Actions, you might want to use this script to check your own actions, and see what you’re using. But more than that, I recommend becoming familiar with the Unix text processing tools and pipelines – even in the age of AI, they’re still a powerful and flexible way to cobble together one-off scripts for processing data.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2025-03-14 04:24:51

I was building a small feature for the Flickr Commons Explorer today: show a random selection of photos from the entire collection. I wanted a fast and varied set of photos.

This meant getting a random sample of rows from a SQLite table (because the Explorer stores all its data in SQLite). I’m happy with the code I settled on, but it took several attempts to get right.

Approach #1: ORDER BY RANDOM()

My first attempt was pretty naïve – I used an ORDER BY RANDOM() clause to sort the table, then limit the results:

SELECT *
FROM photos
ORDER BY random()
LIMIT 10

This query works, but it was slow – about half a second to sample a table with 2 million photos (which is very small by SQLite standards). This query would run on every request for the homepage, so that latency is unacceptable.

It’s slow because it forces SQLite to generate a value for every row, then sort all the rows, and only then does it apply the limit. SQLite is fast, but there’s only so fast you can sort millions of values.

I found a suggestion from Stack Overflow user Ali to do a random sort on the id column first, pick my IDs from that, and only fetch the whole row for the photos I’m selecting:

SELECT *
FROM photos
WHERE id IN (
    SELECT id
    FROM photos
    ORDER BY RANDOM()
    LIMIT 10
)

This means SQLite only has to load the rows it’s returning, not every row in the database. This query was over three times faster – about 0.15s – but that’s still slower than I wanted.

Approach #2: WHERE rowid > (…)

Scrolling down the Stack Overflow page, I found an answer by Max Shenfield with a different approach:

SELECT * FROM photos
WHERE rowid > (
  ABS(RANDOM()) % (SELECT max(rowid) FROM photos)
)
LIMIT 10

The rowid is a unique identifier that’s used as a primary key in most SQLite tables, and it can be looked up very quickly. SQLite automatically assigns a unique rowid unless you explicitly tell it not to, or create your own integer primary key.

This query works by picking a point between the biggest and smallest rowid values used in the table, then getting the rows with rowids which are higher than that point. If you want to know more, Max’s answer has a more detailed explanation.

This query is much faster – around 0.0008s – but I didn’t go this route.

The result is more like a random slice than a random sample. In my testing, it always returned contiguous rows – 101, 102, 103, … – which isn’t what I want. The photos in the Commons Explorer database were inserted in upload order, so photos with adjacent row IDs were uploaded at around the same time and are probably quite similar. I’d get one photo of an old plane, then nine more photos of other planes. I want more variety!

(This behaviour isn’t guaranteed – if you don’t add an ORDER BY clause to a SELECT query, then the order of results is undefined. SQLite is returning rows in rowid order in my table, and a quick Google suggests that’s pretty common, but that may not be true in all cases. It doesn’t affect whether I want to use this approach, but I mention it here because I was confused about the ordering when I read this code.)

Approach #3: Select random rowid values outside SQLite

Max’s answer was the first time I’d heard of rowid, and it gave me an idea – what if I chose random rowid values outside SQLite? This is a less “pure” approach because I’m not doing everything in the database, but I’m happy with that if it gets the result I want.

Here’s the procedure I came up with:

1. Create an empty list to store our sample.

2. Find the highest rowid that’s currently in use:

   sqlite> SELECT MAX(rowid) FROM photos;
   1913389
   

3. Use a random number generator to pick a rowid between 1 and the highest rowid:

   >>> import random
   >>> random.randint(1, max_rowid)
   196476
   

   If we’ve already got this rowid, discard it and generate a new one.

   (The rowid is a signed, 64-bit integer, so the minimum possible value is always 1.)

4. Look for a row with that rowid:

   SELECT *
   FROM photos
   WHERE rowid = 196476
   

   If such a row exists, add it to our sample. If we have enough items in our sample, we’re done. Otherwise, return to step 3 and generate another rowid.

   If such a row doesn’t exist, return to step 3 and generate another rowid.

This requires a bit more code, but it returns a diverse sample of photos, which is what I really care about. It’s a bit slower, but still plenty fast enough (about 0.001s).

This approach is best for tables where the rowid values are mostly contiguous – it would be slower if there are lots of rowids between 1 and the max that don’t exist. If there are large gaps in rowid values, you might try multiple missing entries before finding a valid row, slowing down the query. You might want to try something different, like tracking valid rowid values separately.

This is a good fit for my use case, because photos don’t get removed from Flickr Commons very often. Once a row is written, it sticks around, and over 97% of the possible rowid values do exist.

Summary

Here are the four approaches I tried:

I’m using the random rowid in Python in the Commons Explorer, trading code complexity for speed. I’m using this random sample to render a web page, so it’s important that it returns quickly – when I was testing ORDER BY RANDOM(), I could feel myself waiting for the page to load.

But I’ve used ORDER BY RANDOM() in the past, especially for asynchronous data pipelines where I don’t care about absolute performance. It’s simpler to read and easier to see what’s going on.

Now it’s your turn – visit the Commons Explorer and see what random gems you can find. Let me know if you spot anything cool!

[If the formatting of this post looks odd in your feed reader, visit the original article]

2025-03-12 06:37:26

One rabbit hole I can never resist going down is finding the original creator of a piece of art. This sounds simple, but it’s often quite difficult. The Internet is a maze of social media accounts that only exist to repost other people’s art, usually with minimal or non-existent attribution.

A popular image spawns a thousand copies, each a little further from the original. Signatures get cropped, creators’ names vanish, and we’re left with meaningless phrases like “no copyright intended”, as if that magically absolves someone of artistic theft.

Why do I do this? I’ve always been a bit obsessive, a bit completionist. I’ve worked in cultural heritage for eight years, which has made me more aware of copyright and more curious about provenance. And it’s satisfying to know I’ve found the original source, that I can’t dig any further.

This takes time. It’s digital detective work, using tools like Google Lens and TinEye, and it’s not always easy or possible. Sometimes the original pops straight to the top, but other times it takes a lot of digging to find the source of an image.

So many of us have become accustomed to art as an endless, anonymous stream of “content”. A beautiful image appears in our feed, we give it a quick heart, and scroll on, with no thought for the human who sweated blood and tears to create it. That original artist feels distant, disconected. Whatever benefit they might get from the “exposure” of your work going viral, they don’t get any if their name has been removed first.

I came across two examples recently that remind me it’s not just artists who miss out – it’s everyone who enjoys art.

I saw a photo of some traffic lights on Tumblr. I love their misty, nighttime aesthetic, the way the bright colours of the lights cut through the fog, the totality of the surrounding darkness. But there was no name – somebody had just uploaded the image to their Tumblr page, it was reblogged a bunch of times, and then it appeared on my dashboard. Who took it?

I used Google Lens to find the original photographer: Lucas Zimmerman. Then I discovered it was part of a series. And there was a sequel. I found interviews. Context. Related work. I found all this cool stuff, but only because I knew Lucas’s name.

The second example was a silent video of somebody making tiny chess pieces, just captioned “wow”. It was clearly an edit of another video, with fast-paced cuts to make it accommodate a short attention span – and again with no attribution. This was a little harder to find – I had to search several frames in Google Lens before I found a summary on a Russian website, which had a link to a YouTube video by metalworker and woodworker Левша (Levsha).

This video is four times longer than the cut-up version I found, in higher resolution, and with commentary from the original creator. I don’t speak Russian, but YouTube has auto-translated subtitles. Now I know how this amazing set was made, and I have a much better understanding of the materials and techniques involved. (This includes the delightful name Wenge wood, which I’d never heard before.)

https://youtube.com/watch?v=QoKdDK3y-mQ

A piece of art is more than just a single image or video. It’s a process, a human story. When art is detached from its context and creator, we lose something fundamental. Creators lose the chance to benefit from their work, and we lose the opportunity to engage with it in a deeper way. We can’t learn how it was made, find their other work, or discover how to make similar art for ourselves.

The Internet has done many wonderful things for art, but it’s also a machine for endless copyright infringement. It’s not just about generative AI and content scraping – those are serious issues, but this problem existed long before any of us had heard of ChatGPT. It’s a thousand tiny paper cuts.

How many of us have used an image from the Internet because it showed up in a search, without a second thought for its creator? When Google Images says “images may be subject to copyright”, how many of us have really thought about what that means? Next time you want to use an image from the web, look to see if it’s shared under a license that allows reuse, and make sure you include the appropriate attribution – and if not, look for a different image.

Finding the original creator is hard, sometimes impossible. The Internet is full of shadows: copies of things that went offline years ago. But when I succeed, it feels worth the effort – both for the original artist and myself.

When I read a book or watch a TV show, the credits guide me to the artists, and I can appreciate both them and the rest of their work. I wish the Internet was more like that. I wish the platforms we rely on put more emphasis on credit and attribution, and the people behind art.

The next time an image catches your eye, take a moment. Who made this? What does it mean? What’s their story?

[If the formatting of this post looks odd in your feed reader, visit the original article]

2025-03-10 15:48:39

I’ve made another small tweak to the site – I’ve added “new” banners to articles I’ve written recently, and any post marked as “new” will be pinned to the homepage. Previously, the homepage was just a random selection of six articles I’d written at any time.

Last year I made some changes to de-emphasise sorting by date and reduce recency bias. I stand by that decision, but now I see I went too far. Nobody comes to my site asking “what did Alex write on a specific date”, but there are people who ask “what did Alex write recently”. I’d made it too difficult to find my newest writing, and that’s what this tweak is trying to fix.

This should have been a simple change, but it became a lesson about the inner workings of CSS.

Absolute positioning and my first attempt

I started with some code I wrote last year. Let’s step through it in detail.

First, create a container that includes both the image and the banner.

<div class="container">
  <div class="banner">NEW</div>
  <img src="computer.jpg">
</div>

Notice how the banner and image appear separately – they both have their own space in the layout.

NEW

I can add a CSS rule that makes the text appear on top of the image:

.banner {
  position: absolute;
}

This enables absolute positioning, which removes the banner from the normal document flow and allows it to be placed anywhere on the page. Now it sits alone, and it doesn't affect the layout of other elements on the page – in particular, the image no longer has to leave space for it.

NEW

The text is in the top left corner of the image, but we can move it to the top right-hand instead:

.container {
  position: relative;
}

.banner {
  transform: rotate(45deg);
  right:     16px;
  top:       20px;
}

NEW

I chose the transform, right, and top values by tweaking until I got something that looked correct. They move the banner to the corner, and then the transform rotates it diagonally.

The relative position of the container element is vital. The absolutely positioned banner still needs a reference point for the top and right, and it uses the closest ancestor with an explicit position – or if it doesn’t find one, the root <html> element. Setting position: relative; means the offsets are measured against the sides of the container, not the entire HTML document.

This is a CSS feature called positioning context, which I’d never heard of until I started writing this blog post. I’d been copying the position: relative; line from other examples without really understanding what it did, or why it was necessary.

(What made this particularly confusing to me is that if you only add position: absolute to the banner, it seems like the image is the reference point – notice how, with just that property, the text is in the top left-hand corner of the image. It’s not until you set top or right that the banner starts using the entire page as a reference point. This is because an absolutely positioned element takes its initial position from where it would be in the normal flow, and doesn’t look for a positioned ancestor until you set an offset.)

Let's apply a colour to make this banner easier to read – the text is disappearing into the image.

.banner {
  background: red;
  color:      white;
}

Right now the element is only as big as the letters in the word “NEW”, so it’s just floating in space – we need to make it wider, so it covers the whole corner.

NEW

We can make it wider by adding padding. Because this changes the size of the element, I had to adjust the position offsets to keep it in the right place.

.banner {
  right:   -34px;
  top:     18px;
  padding: 2px 50px;
}

NEW

Now the banner is too wide, and extending off the end of the image. Let’s clip the edges, so it fits neatly within the square:

.container {
  overflow: hidden;
}

This is the banner effect I’m looking for – the text is clear and prominent on the image. I’ve added a box-shadow on my homepage to make it stand out further, but cosmetic details like that aren’t important for the rest of this post.

NEW

As a reminder, here’s the HTML:

<div class="container">
  <div class="banner">NEW</div>
  <img src="computer.jpg">
</div>

and here’s the complete CSS:

.container {
  position: relative;
  overflow: hidden;
}

.banner {
  position: absolute;

  background: red;
  color:      white;

  transform: rotate(45deg);

  right:   -34px;
  top:     18px;
  padding: 2px 50px;
}

It’s only nine CSS properties, but it contains a surprising amount of complexity. I had this CSS and I knew it worked, but I didn’t really understand it – and especially the way absolute positioning worked – until I wrote this post.

This worked when I wrote it as a standalone snippet, and then I deployed it on this site, and I found a bug.

(The photo I used in the examples is from Viktorya Sergeeva on Pexels.)

Dark mode, filters, and stacking contexts

I added dark mode support to this site a couple of years ago – the background changes from white to black, the text colour flips, and a few other changes. I’m a light mode person, but I know a lot of people prefer dark mode and it was a fun bit of CSS work, so it’s there.

The code I described above breaks if you’re using this site in dark mode.

What.

I started poking around in my browser’s developer tools, and I could see that the banner was being rendered, but it was under the image instead of on top of it. All my positioning code that worked in light mode was broken in dark mode. I was baffled.

I discovered that by adding a z-index property to the banner, I could make it reappear. I knew that elements with a higher z-index will appear above an element with a lower z-index – so I was moving my banner back out from under the image. I had a fix, but it felt uncomfortable because I couldn’t explain why it worked, or why it was only necessary in dark mode. I wanted to go deeper.

I knew the culprit was in the CSS I’d written. I could see the issue if I tried my code in this site, but not if I copied it to a standalone HTML file.

To find the issue, I created a local branch of the site, and I started deleting CSS until I could no longer reproduce the issue. I eventually tracked it down to the following rule:

@media (prefers-color-scheme: dark) {
  /* see https://web.dev/articles/prefers-color-scheme#re-colorize_and_darken_photographic_images */
  img:not([src*='.svg']):not(.dark_aware) {
    filter: grayscale(10%);
  }
}

This applies a slight darkening to any images when dark mode is enabled – unless they’re an SVG, or I’ve added the dark_aware class that means an image look okay in dark mode. This makes images a bit less vibrant in dark mode, so they’re not too visually loud. This is a suggestion from Thomas Steiner, from an article with a lot of useful advice about supporting dark mode.

When this rule is present, the banner vanishes. When I delete it, the banner looks fine.

Eventually I found the answer: I’d not thought about (or heard of!) the stacking context. The stacking context is a way of thinking about HTML elements in three dimensions. It introduces a z‑axis that determines which elements appear above or below each other. It’s affected by properties like z-index, but also less obvious ones like filter.

In light mode, the banner and the image are both part of the same stacking context. This means that both elements can be rendered together, and the positioning rules are applied together – so the banner appears on top of the image.

In dark mode, my filter property creates a new stacking context. Applying a filter to an element forces it into a new stacking context, and in this case that means the image and the banner will be rendered separately. Browsers render elements in DOM order, and because the banner appears before the image in the HTML, the stacking context with the banner is rendered first, then the stacking context with the image is rendered separately and covers it up.

The correct fix is not to set a z-index, but to swap the order of DOM elements so the banner is rendered after the image:

<div class="container">
  <img src="computer.jpg">
  <div class="banner">NEW</div>
</div>

This is the code I’m using now, and now the banner looks correct in dark mode.

In hindsight, this ordering makes more sense anyway – the banner is an overlay on the image, and it feels right to me that it should appear later in the HTML. If I was laying this out with bits of paper, I’d put down the image, then the banner.

One example is nowhere near enough for me to properly understand stacking contexts or rendering order, but now I know it’s a thing I need to consider. I have a vague recollection that I made another mistake with filter and rendering order in the past, but I didn’t investigate properly – this time, I wanted to understand what was happening.

I’m still not done – now I have the main layout working, I’m chasing a hairline crack that’s started appearing in the cards, but only on WebKit. There’s an interaction between relative positioning and border-radius that’s throwing everything off. CSS is hard.

I stick to a small subset of CSS properties, but that doesn’t mean I can avoid the complexity of the web. There are lots of moving parts that interact in non-obvious ways, and my understanding is rudimentary at best. I have a lot of respect for front-end developers who work on much larger and more complex code bases. I’m getting better, but CSS keeps reminding me how much more I have to learn.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2025-03-07 21:04:17

I’ve been working on a project where I need to plot points on a map. I don’t need an interactive or dynamic visualisation – just a static map with coloured dots for each coordinate.

I’ve created maps on the web using Leaflet.js, which load map data from OpenStreetMap (OSM) and support zooming and panning – but for this project, I want a standalone image rather than something I embed in a web page. I want to put in coordinates, and get a PNG image back.

This feels like it should be straightforward. There are lots of Python libraries for data visualisation, but it’s not an area I’ve ever explored in detail. I don’t know how to use these libraries, and despite trying I couldn’t work out how to accomplish this seemingly simple task. I made several attempts with libraries like matplotlib and plotly, but I felt like I was fighting the tools. Rather than persist, I wrote my own solution with “lower level” tools.

The key was a page on the OpenStreetMap wiki explaining how to convert lat/lon coordinates into the pixel system used by OSM tiles.

In particular, it allowed me to break the process into two steps:

1. Get a “base map” image that covers the entire world
2. Convert lat/lon coordinates into xy coordinates that can be overlaid on this image

Let’s go through those steps.

Get a “base map” image that covers the entire world

Let’s talk about how OpenStreetMap works, and in particular their image tiles. If you start at the most zoomed-out level, OSM represents the entire world with a single 256×256 pixel square. This is the Web Mercator projection, and you don’t get much detail – just a rough outline of the world.

We can zoom in, and this tile splits into four new tiles of the same size. There are twice as many pixels along each edge, and each tile has more detail. Notice that country boundaries are visible now, but we can’t see any names yet.

We can zoom in even further, and each of these tiles split again. There still aren’t any text labels, but the map is getting more detailed and we can see small features that weren’t visible before.

You get the idea – we could keep zooming, and we’d get more and more tiles, each with more detail. This tile system means you can get detailed information for a specific area, without loading the entire world. For example, if I’m looking at street information in Britain, I only need the detailed tiles for that part of the world. I don’t need the detailed tiles for Bolivia at the same time.

OpenStreetMap will only give you 256×256 pixels at a time, but we can download every tile and stitch them together, one-by-one.

Here’s a Python script that enumerates all the tiles at a particular zoom level, downloads them, and uses the Pillow library to combine them into a single large image:

#!/usr/bin/env python3
"""
Download all the map tiles for a particular zoom level from OpenStreetMap,
and stitch them into a single image.
"""

import io
import itertools

import httpx
from PIL import Image


zoom_level = 2

width = 256 * 2**zoom_level
height = 256 * (2**zoom_level)

im = Image.new("RGB", (width, height))

for x, y in itertools.product(range(2**zoom_level), range(2**zoom_level)):
    resp = httpx.get(f"https://tile.openstreetmap.org/{zoom_level}/{x}/{y}.png", timeout=50)
    resp.raise_for_status()

    im_buffer = Image.open(io.BytesIO(resp.content))

    im.paste(im_buffer, (x * 256, y * 256))

out_path = f"map_{zoom_level}.png"
im.save(out_path)
print(out_path)

The higher the zoom level, the more tiles you need to download, and the larger the final image will be. I ran this script up to zoom level 6, and this is the data involved:

I can just about open that zoom level 6 image on my computer, but it’s struggling. I didn’t try opening zoom level 7 – that includes 16,384 tiles, and I’d probably run out of memory.

For most static images, zoom level 3 or 4 should be sufficient – I ended up a base map from zoom level 4 for my project. It takes a minute or so to download all the tiles from OpenStreetMap, but you only need to request it once, and then you have a static image you can use again and again.

This is a particularly good approach if you want to draw a lot of maps. OpenStreetMap is provided for free, and we want to be a respectful user of the service. Downloading all the map tiles once is more efficient than making repeated requests for the same data.

Overlay lat/lon coordinates on this base map

Now we have an image with a map of the whole world, we need to overlay our lat/lon coordinates as points on this map.

I found instructions on the OpenStreetMap wiki which explain how to convert GPS coordinates into a position on the unit square, which we can in turn add to our map. They outline a straightforward algorithm, which I implemented in Python:

import math


def convert_gps_coordinates_to_unit_xy(
    *, latitude: float, longitude: float
) -> tuple[float, float]:
    """
    Convert GPS coordinates to positions on the unit square, which
    can be plotted on a Web Mercator projection of the world.

    This expects the coordinates to be specified in **degrees**.

    The result will be (x, y) coordinates:

    -   x will fall in the range (0, 1).
        x=0 is the left (180° west) edge of the map.
        x=1 is the right (180° east) edge of the map.
        x=0.5 is the middle, the prime meridian.

    -   y will fall in the range (0, 1).
        y=0 is the top (north) edge of the map, at 85.0511 °N.
        y=1 is the bottom (south) edge of the map, at 85.0511 °S.
        y=0.5 is the middle, the equator.

    """
    # This is based on instructions from the OpenStreetMap Wiki:
    # https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames#Example:_Convert_a_GPS_coordinate_to_a_pixel_position_in_a_Web_Mercator_tile
    # (Retrieved 16 January 2025)

    # Convert the coordinate to the Web Mercator projection
    # (https://epsg.io/3857)
    #
    # x = longitude
    # y = arsinh(tan(latitude))
    #
    x_webm = longitude
    y_webm = math.asinh(math.tan(math.radians(latitude)))

    # Transform the projected point onto the unit square
    #
    # x = 0.5 + x / 360
    # y = 0.5 - y / 2π
    #
    x_unit = 0.5 + x_webm / 360
    y_unit = 0.5 - y_webm / (2 * math.pi)

    return x_unit, y_unit

Their documentation includes a worked example using the coordinates of the Hachiko Statue. We can run our code, and check we get the same results:

>>> convert_gps_coordinates_to_unit_xy(latitude=35.6590699, longitude=139.7006793)
(0.8880574425, 0.39385379958274735)

Most users of OpenStreetMap tiles will use these unit positions to select the tiles they need, and then dowload those images – but we can also position these points directly on the global map. I wrote some more Pillow code that converts GPS coordinates to these unit positions, scales those unit positions to the size of the entire map, then draws a coloured circle at each point on the map.

Here’s the code:

from PIL import Image, ImageDraw


gps_coordinates = [
    # Hachiko Memorial Statue in Tokyo
    {"latitude": 35.6590699, "longitude": 139.7006793},
    # Greyfriars Bobby in Edinburgh
    {"latitude": 55.9469224, "longitude": -3.1913043},
    # Fido Statue in Tuscany
    {"latitude": 43.955101, "longitude": 11.388186},
]


im = Image.open("base_map.png")
draw = ImageDraw.Draw(im)

for coord in gps_coordinates:
    x, y = convert_gps_coordinates_to_unit_xy(**coord)

    radius = 32

    draw.ellipse(
        [
            x * im.width - radius,
            y * im.height - radius,
            x * im.width + radius,
            y * im.height + radius,
        ],
        fill="red",
    )

im.save("map_with_dots.png")

and here’s the map it produces:

The nice thing about writing this code in Pillow is that it’s a library I already know how to use, and so I can customise it if I need to. I can change the shape and colour of the points, or crop to specific regions, or add text to the image. I’m sure more sophisticated data visualisation libraries can do all this, and more – but I wouldn’t know how.

The downside is that if I need more advanced features, I’ll have to write them myself. I’m okay with that – trading sophistication for simplicity. I didn’t need to learn a complex visualization library – I was able to write code I can read and understand. In a world full of AI-generating code, writing something I know I understand feels more important than ever.

[If the formatting of this post looks odd in your feed reader, visit the original article]

2025-02-21 21:19:30

I’m sitting in a small coffee shop in Brooklyn. I have a warm drink, and it’s just started to snow outside. I’m visiting New York to see Operation Mincemeat on Broadway – I was at the dress rehearsal yesterday, and I’ll be at the opening preview tonight. I’ve seen this show more times than I care to count, and I hope US theater-goers love it as much as Brits.

The people who make the show will tell you that it’s about a bunch of misfits who thought they could do something ridiculous, who had the audacity to believe in something unlikely.

That’s certainly one way to see it. The musical tells the true story of a group of British spies who tried to fool Hitler with a dead body, fake papers, and an outrageous plan that could easily have failed. Decades later, the show’s creators would mirror that same spirit of unlikely ambition. Four friends, armed with their creativity, determination, and a wardrobe full of hats, created a new musical in a small London theatre. And after a series of transfers, they’re about to open the show under the bright lights of Broadway.

But when I watch the show, I see a story about friendship. It’s about how we need our friends to help us, to inspire us, to push us to be the best versions of ourselves.

I see the swaggering leader who needs a team to help him truly achieve. The nervous scientist who stands up for himself with the support of his friends. The enthusiastic secretary who learns wisdom and resilience from her elder.

And so, I suppose, it’s fitting that I’m not in New York on my own. I’m here with friends – dozens of wonderful people who I met through this ridiculous show.

At first, I was just an audience member. I sat in my seat, I watched the show, and I laughed and cried with equal measure.

After the show, I waited at stage door to thank the cast. Then I came to see the show a second time. And a third. And a fourth. After a few trips, I started to see familiar faces waiting with me at stage door. So before the cast came out, we started chatting.

Those conversations became a Twitter community, then a Discord, then a WhatsApp. We swapped fan art, merch, and stories of our favourite moments. We went to other shows together, and we hung out outside the theatre. I spent New Year’s Eve with a few of these friends, sitting on somebody’s floor and laughing about a bowl of limes like it was the funniest thing in the world.

And now we’re together in New York.

Meeting this kind, funny, and creative group of people might seem as unlikely as the premise of Mincemeat itself. But I believed it was possible, and here we are.

I feel so lucky to have met these people, to take this ridiculous trip, to share these precious days with them. I know what a privilege this is – the time, the money, the ability to say let’s do this and make it happen. How many people can gather a dozen friends for even a single evening, let alone a trip halfway round the world?

You might think it’s silly to travel this far for a theatre show, especially one we’ve seen plenty of times in London. Some people would never see the same show twice, and most of us are comfortably into double or triple-figures.

Whenever somebody asks why, I don’t have a good answer. Because it’s fun? Because it’s moving? Because I enjoy it? I feel the need to justify it, as if there’s some logical reason that will make all of this okay. But maybe I don’t have to. Maybe joy doesn’t need justification.

A theatre show doesn’t happen without people who care. Neither does a friendship.

So much of our culture tells us that it’s not cool to care. It’s better to be detached, dismissive, disinterested. Enthusiasm is cringe. Sincerity is weakness. I’ve certainly felt that pressure – the urge to play it cool, to pretend I’m above it all. To act as if I only enjoy something a “normal” amount.

Well, fuck that.

I don’t know where the drive to be detached comes from. Maybe it’s to protect ourselves, a way to guard against disappointment. Maybe it’s to seem sophisticated, as if having passions makes us childish or less mature. Or perhaps it’s about control – if we stay detached, we never have to depend on others, we never have to trust in something bigger than ourselves. Being detached means you can’t get hurt – but you’ll also miss out on so much joy.

I’m a big fan of being a big fan of things. So many of the best things in my life have come from caring, from letting myself be involved, from finding people who are a big fan of the same things as me. If I pretended not to care, I wouldn’t have any of that.

Caring – deeply, foolishly, vulnerably – is how I connect with people. My friends and I care about this show, we care about each other, and we care about our joy.

That care and love for each other is what brought us together, and without it we wouldn’t be here in this city. I know this is a once-in-a-lifetime trip. So many stars had to align – for us to meet, for the show we love to be successful, for us to be able to travel together. But if we didn’t care, none of those stars would have aligned.

I know so many other friends who would have loved to be here but can’t be, for all kinds of reasons. Their absence isn’t for lack of caring, and they want the show to do well whether or not they’re here. I know they care, and that’s the important thing. To butcher Tennyson: I think it’s better to care about something you cannot affect, than to care about nothing at all. In a world that’s full of cynicism and spite and hatred, I feel that now more than ever.

I’d recommend you go to the show if you haven’t already, but that’s not really the point of this post. Maybe you’ve already seen Operation Mincemeat, and it wasn’t for you. Maybe you’re not a theatre kid. Maybe you aren’t into musicals, or history, or war stories. That’s okay. I don’t mind if you care about different things to me. (Imagine how boring the world would be if we all cared about the same things!)

But I want you to care about something. I want you to find it, find people who care about it too, and hold on to them. Because right now, in this city, with these people, at this show? I’m so glad I did. And I hope you find that sort of happiness too.

Timing note: I wrote this on February 15th, but I delayed posting it because I didn’t want to highlight the fact I was away from home.

[If the formatting of this post looks odd in your feed reader, visit the original article]