2025-05-23 08:00:00
While working on Anubis (a Web AI Firewall Utility designed to stop rampant scraping from taking out web services), one question in particular keeps coming up:
Why do you have an anime character in the challenge screen by default?
This is sometimes phrased politely. Other times people commenting on this display a measured lack of common courtesy.
The Anubis character is displayed by default as a way to ensure that I am not the lone unpaid dependency peg holding up a vast majority of the Internet.
Of course, nothing is stopping you from forking the software to replace the art assets. Instead of doing that, I would rather you support the project and purchase a license for the commercial variant of Anubis named BotStopper. Doing this will make sure that the project is sustainable and that I don't burn myself out to a crisp in the process of keeping small internet websites open to the public.
At some level, I use the presence of the Anubis mascot as a "shopping cart test". If you either pay me for the unbranded version or leave the character intact, I'm going to take any bug reports more seriously. It's a positive sign that you are willing to invest in the project's success and help make sure that people developing vital infrastructure are not neglected.
There's been some online venom and vitriol about the use of a cartoon that people only see for about 3 seconds on average that make me wonder if I should have made this code open source in the first place. The anime image is load-bearing. It is there as a social cost. You are free to replace it, but I am also free to make parts of the program rely on the presence of the anime image in order to do more elaborate checks, such as checks that do not rely on JavaScript.
Amusingly, this has caused some issues with the education market because they want a solution NOW and their purchasing process is a very slow and onerous beast. I'm going to figure out a balance eventually, but who knew that the satirical tech startup I made up as a joke would end up having a solid foothold in the education market?
One of best side effects of the character being there is that it's functioned as a bit of a viral marketing campaign for the project. Who knows how many people learned that Anubis is there, functional, and works well enough for people to complain about because of someone getting incensed online about the fact that the software shows a human-authored work of art for a few seconds?
I want this project to be sustainable; and in the wake of rent, food prices, and computer hardware costs continuing to go up I kinda need money because our economy runs on money, not GitHub stars.
I have a no-JS solution that should be ready soon (I've been doing a lot of unpublishable reverse engineering of how browsers work), but I also need to figure out how to obfuscate it so that the scrapers can't just look at the code to fix their scrapers. So far I'm looking at WebAssembly on the server for this. I'll let y'all know more as I have it figured out on my end. There will be some fun things in the near future, including but not limited to external services to help Anubis make better decisions on when to throw or not throw challenges.
Hopefully the NLNet application I made goes through, funding to buy a few months of development time would go a long way. There has been venture capital interest in Anubis, so that's a potential route to go down too.
Thanks for following the development of Anubis! If you want to support the project, please throw me some bucks on GitHub Sponsors.
2025-05-07 08:00:00
Co-Authored-By: @scootaloose.com
Windows has been a pain in the ass as of late. Sure, it works, but there's starting to be so much overhead between me and the only reason I bother booting into it these days: games. Every so often I'll wake up to find out that my system rebooted and when I sign in I'm greeted with yet another "pweez try copilot >w< we pwomise you will like it ewe" full-screen dialogue box with "yes" or "nah, maybe later" as my only options. That or we find out that they somehow found a reason to put AI into another core windows tool, probably from a project manager’s desperate attempt to get promoted.
The idea of consent in the tech industry is disturbingly absent, I hate it here.
The silicon valley model of consent
[image or embed]
— Xe ( @xeiaso.net ) March 31, 2025 at 11:59 PM
As much as I'd like to like Copilot, Recall, or Copilot (yes those are separate products), if a feature is genuinely transformative enough to either justify the security risk of literally recording everything I do or enhances the experience of using my computer enough to hand over control to an unfeeling automaton, I'll use it. It probably won't be any better than Apple Intelligence though.
When we built our gaming towers, we decided to build systems around the AMD Ryzen 7950X3D and NVidia RTX 4080. These are a fine combination in practice. You get AMD's philosophy of giving you enough cores that you can do parallel computing jobs without breaking a sweat and the RTX 4080 being one of the best cards on the market for rasterization and whatever ray tracing you feel like doing. I don't personally do ray tracing in games, but I like that it is an option for people who want to.
The main problem with NVidia GPUs is that NVidia's consumer graphics department seems to be under the assumption that games don't need as much video memory as they do. You get absolutely bodied in the amount of video memory. Big games can use upwards of 15 GB of video memory, and the OS / Firefox needs 2 GB of video memory. In total, that's one more gigabyte than the 16 I have. You can't just plug in more vram too, you need to either get unobtanium-in-canada RTX 4090s or pay several body organs for enterprise grade GPUs.
AMD is realistically the only other option on the market. AMD sucks for different reasons, but at least they give you enough video memory that you can survive.
Intel?
As someone that uses an IRC nick that's the same as the Intel Linux GPU driver, I'm never using an Intel GPU on linux.
One of the most frustrating issues we've run into as of late is macrostutters when gaming. Macrostutters are when the game hitches and the entire rendering pipeline gets stuck for at least two frames, then everything goes back to normal. This is most notable in iRacing and Final Fantasy XIV (14). In iRacing's case, it can cause you to get into an accident because you get an over 100 millisecond to 5 second pause. Mind you, the game is playable, but the macrostutters can make the experience insufferable.
It’s all the rage in the iRacing forums when they’re not slinging potatoes at each other over other issues, it’s great!
In the case of Final Fantasy XIV (amazing game by the way, don't play it), this can cause you to get killed because you missed an attack telegraph due to it happening while your rendering pipeline was stopped. I have been killed to macrostutters as white mage (pure healer class for fellow RPG affictionados) in Windows at least 3 times in the last week and I hate it.
So, the thought came to our minds: why are we bothering with Windows? We've had a good experience with SteamOS on our Steam Decks.
It probably helps that you don't mess with the Steam Deck at all and leave it holy.
We have a home theatre PC that runs Bazzite. A little box made up of older hardware we upgraded from. Runs tried and true hardware that has matured well and not a single unknown variable in it (AMD Ryzen 5 3600 and an RX5700XT, on a B450 motherboard, the works). Besides the normal HDR issues on Linux, it's been pretty great for couch gaming!
I've also been using Linux on the desktop off and on for years. My career got started because Windows Vista was so unbearably bad that I had to learn how to use Linux on the desktop in order to get a usable experience out of my dual core PC with 512 MB of ram.
I’m honestly amazed Vista even attempted to run – much less install – on that low of memory configuration… I’ve been a Windows user for as long as I remember, my oldest memory of using it dating back to Windows 98, but there’s probably some home video of me messing around in MS Paint with Windows 95 somewhere in a box. Around 2009 I used a shared family laptop that came shipped with Vista that suddenly decommissioned itself out of existence. I installed Kubuntu (KDE flavour of Ubuntu, I could and still cannot stand GNOME lol) on it for a time until Windows 7 came around to save it. My mother and sister did not really adapt to using Linux and I was the only one trying to use it around that time. It was functional enough back then I suppose – the hardest I drove that laptop was playing Adobe Flash games – but we could not do my schoolwork on it properly, namely because OpenOffice and Word hated each other.
Surely 2025 will be the year of the Linux Desktop.
Foreshadowing is a narrative device in which a narrator gives an advance hint as to what comes up later in the story.
My husband has very simple computing needs compared to me. He doesn't do software development in his free time (save simple automation with PowerShell, bash, or Python). He doesn't do advanced things like elaborate video editing, 3d animation, or content creation. Sure sometimes he'll need to clip a segment out of a longer video file, but really that's not the same thing as making an hbomberguy video or streaming to Twitch. The most complicated thing he wants to do at the moment is play Final Fantasy XIV, which as far as games go isn't really that intensive.
I still have my library of simulators where most of them would technically work fine under Proton, as most have been tested to work there. However, given the mishmash of hardware and the fact that iRacing has anticheat and the launcher barely functions in Linux under Proton, I decided to delegate my expensive hobby machine to secondary duty and I left it running Windows as is. Its sole purpose now is for my racing sims and any strange game that does not play nice with Proton. Namely any multiplayer games that have kernel-level anticheat. I scrapped the idea of dual booting before anything else because I have had enough bad experiences with Windows’ Main Character Syndrome that I was going to airgap my install of Linux to a whole other computer.
I have some more complicated needs seeing as software I make runs on UNESCO servers, but really as long as I have a basic Linux environment, Homebrew, and Steam, I'll be fine. I am also afflicted with catgirl simulator, but I do my streaming from Windows due to vtubing software barely working there and me being enough of a coward to not want to try to run it in Linux again.
When he said he wanted to go for Linux on the desktop, I wanted to make sure that we were using the same distro so that I had enough of the same setup to be able to help when things inevitably go wrong. I wanted something boring, well-understood, and strongly supported by upstream. I ended up choosing the most boring distribution I could think of: Fedora.
Fedora is many things, but it's what systemd, mesa, the Linux Kernel, and GNOME are developed against. This means that it's one of the most boring distributions on the planet. It has most of the same package management UX ergonomics as Red Hat Enterprise Linux, it's well documented and most of the quirks are well known or solved, and overall it's the least objectionable choice on the planet.
In retrospect, I'm not sure if this was a mistake or not.
He wanted to build a pure AMD system to stave off any potential NVidia related problems. We found some deals and got him the following:
I will spend more money on a case if it means it won’t draw blood while trying to work on it, so far my last two cases spared my fingers. This build was also woefully overpriced because I was paying for the colour tax on all my components, going with a full white build this time.
I had recently just installed Fedora 41 on my tower and had no issues. My tower has an older CPU and motherboard so I didn't expect any problems. Most of that hardware I listed above was released after Fedora 41 was released in late October 2024. I expected some issues for hardware compatibility for the first boot, but figured that an update and reboot would fix it. From experience I know that Fedora doesn't ever roll new install images after they release a major version. This makes sense from their perspective for mirror bandwidth reasons.
When we booted into the installer on his tower, the screen was stuck at 1024x768 on a 21:9 ultrawide. Fine enough, we can deal with that. The bigger problem was the fact that the ethernet card wasn't working. It wasn't detected in the PCI device tree. Luckily the board shipped with an embedded Wi-Fi card, so we used that to limp our way into Fedora. I figured it'd be fine after some updates.
It was not fine after that. The machine failed to boot after that round of updates. It felt like the boot splash screen was somehow getting the GPU driver into a weird state and the whole system hung. Verbose boot didn't work. I was almost worried that we had dead hardware or something.
Okay, fine, the hardware is new. I get it. Let's try Fedora 42 beta. Surely that has a newer kernel, userland, and everything that we'd need to get things working out of the box.
Yep, it did. Everything worked out of the box. The ethernet card was detected and got an IP instantly. The install was near instant. We had the full screen resolution at 100hz like we expected, and after an install 1Password and other goodies were set up. Steam was installed, Final Fantasy XIV was set up, the controller was configured, and a good time was had by all. The microphone and DAC even worked!
Once everything was working, I set up an automount for the NAS so that he could access our bank of wallpapers and the like. Everything was working and we were happy.
Again, foreshadowing is a narrative device in which a narrator gives an advance hint as to what comes up later in the story.
Coincidentally, we built the system the day before Fedora 42 was released. I had him run an update and he chose to do it from the package management GUI, “Discover”. I have a terminal case of Linux brain and don't feel comfortable running updates in a way that I can't see the logs. This is what happens when you do SRE work for long enough. You don't trust anything you can't directly look at or touch.
I am Windows Update brained, it’s ingrained into my soul after 27 years @_@
We rebooted for the update and then things started to get weird. The biggest problem was X11 apps not working. We got obscure XWayland errors that a mesa dev friend never thought was possible. I seriously began to get worried that we had some kind of half-hardware failure or something inexplicable like that.
I thought that there was some kind of strange issue upgrading from Fedora 42 Beta to Fedora 42 full. I can't say why this would happen, but it's completely understandable to go there after a few hours of fruitless debugging. We reinstalled because we ran out of ideas.
Once everything was back and running, we ran into a strange issue: Steam kept starting on the integrated GPU instead of the dedicated GPU. This would be a problem, but luckily enough games somehow preferred using the dedicated GPU so it all worked out. After an update got pushed, this caused Steam to die or sometimes throw messages about chromium not working on the GPU "llvmpipe".
Life pro tip: if you ever see the GPU name "llvmpipe" in Linux, that means you're using software rendering!
Debugging this was really weird. Based on what we could figure out with a combination of nvtop, hex-diving into /sys, and other demonic incantations that no mortal should understand, the system somehow flagged the dedicated GPU as the integrated GPU and vice versa. This was causing the system to tell Steam and only Steam that it needed to start on the integrated GPU.
After increasingly desperate means of trying to disable the integrated GPU or de-prioritize it, we ended up disabling the integrated GPU in the bios. I was worried this would make debugging a dead dedicated GPU harder, but my husband correctly pointed out that we have at least 5 known working GPUs of different generations laying around with the right power connectors.
Anyways we got everything working but sometimes when resuming from sleep Final Fantasy XIV causes a spectacular shader pipeline explosion. I'm not sure how to describe it further, but in case you have any idea how to debug this we've attached a video:
HOOOOOME RIDING HOOOOOOME DYING HOOOOOPE HOOOOOOOLD ONTO HOOOOPE, OOOOOOOHGQEROKHQekrg’qneqo;nfhouehqa
I'm pretty sure this is a proton issue, or a mesa issue, or an amdgpu issue, or a computer issue. If I had any idea where to file this it'd be filed, but when we tried to debug it and get a GPU pipeline trace the problem instantly vanished. Aren't computers the best?
S3 suspend is not a solved problem in the YOTLD 2025. Sometimes on resume the display driver crashes and my husband needs to force a power cycle. When he rebooted, XWayland apps wouldn't start. Discord, Steam, and Proton depend on XWayland. This is a very bad situation.
Originally we thought the display driver crashing was causing this, but after manual restarts under normal circumstances were also causing it it got our attention. The worst part was that this was inconsistent, almost like something in the critical dependency chain was working right sometimes and not working at all other times. We started to wonder if Fedora actually tested anything before shipping it because updates made the pattern of working vs not working change.
One of the most simple apps in the x11 suite is xeyes. It's a simple little thing where it has a pair of cartoon eyes that look at your mouse cursor. It's the display pipeline equivalent to pinging google.com to make sure your internet connection works. If you've never seen it before, here's what it looks like:
Alas, it was not working.
After some investigation, the only commonality I could find was the X11 socket folder in /tmp not existing. X11 uses Unix sockets (sockets but via the filesystem) for clients (programs) to communicate with the server (display compositor). If that folder isn't created with the right flags, XWayland can't create the right socket for X clients and will rightly refuse to work.
On a hunch, I made xxx-hack-make-x11-dir.service:
[Unit]
Description=Simple service test
After=tmp.mount
Before=display-manager.service
[Service]
Type=simple
ExecStart=/bin/bash -c "mkdir -p /tmp/.X11-unix; chmod -R 1777 /tmp/.X11-unix"
[Install]
WantedBy=local-fs.target
This seemed to get it working. It worked a lot more reliably when I properly set the sticky bit on the .X11-unix folder so that his user account could create the XWayland socket.
In case you've never seen the "sticky bit" in practice before, Unix permissions have three main fields per file:
This applies to both files and folders (where the execute bit on folders is what gives a user permission to list files in that folder, I don't fully get it either). However in practice there's a secret fourth field which includes magic flags like the sticky bit.
The sticky bit is what makes temporary files work for multi-user systems. At any point, any program on your system may need to create a temporary file. Many programs will assume that they can always create temporary files. These programs may be running as any user on the system, not just the main user account for the person that uses the computer. However, you don't want users to be able to clobber each other's temporary files because the write bit on folders also allows you to delete files. That would be bad. This is what the sticky bit is there to solve: making a folder that everyone can write to, but only the user that created a temporary file can delete it.
Notably, the X11 socket directory needs to have the sticky bit set because of facts and circumstances involving years of legacy cruft that nobody wants to fix.
$ stat /tmp/.X11-unix
File: /tmp/.X11-unix
Size: 120 Blocks: 0 IO Block: 4096 directory
Device: 0,41 Inode: 2 Links: 2
Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2025-05-05 21:33:39.601616923 -0400
Modify: 2025-05-05 21:34:09.234769003 -0400
Change: 2025-05-05 21:34:09.234769003 -0400
Birth: 2025-05-05 21:33:39.601616923 -0400
Once xxx-hack-make-x11-dir.service was deployed, everything worked according to keikaku.
Life pro tip: keikaku means plan!
The system was stable. Everything was working. But when multiple people that work at Red Hat are telling you that the problems you are running into are so strange that you need to start filing bug reports in the dark sections of the bug tracker, you start to wonder if you're doing something wrong. The system was having configuration error-like issues on components that do not have configuration files.
While we were drafting this article, we decided to take a look at the problem a bit further. There was simply no way that we needed xxx-hack-make-x11-dir.service as a load-bearing dependency on our near plain install of Fedora, right? This should just work out of the box, right???
We went back to the drawing board. His system was basically stock Fedora, and we only really did three things to it outside of the package management universe:
/mnt/itsuki
xxx-hack-make-x11-dir.service to frantically hack around issuesNotably, I had the NAS automount set up too and was also having strange issues with the display stack, including but not limited to the GNOME display manager forgetting that Wayland existed and instantly killing itself on launch.
On a hunch, we disabled the units in the reverse order that we created them to undo the stack and get closer to stock Fedora. First, we disabled the xxx-hack-make-x11-dir.service unit. When he rebooted, this broke XWayland as we expected. Then we disabled the NAS automount and rebooted the system.
XWayland started working.
My guess is that this unit somehow created a cyclical dependency:
# mnt-itsuki.automount
[Unit]
Requires=remote-fs-pre.target
After=remote-fs-pre.target
[Automount]
Where=/mnt/itsuki
TimeoutIdleSec=0
[Install]
WantedBy=remote-fs.target
Oh...this was me, wasn't it...
Your sysadmin privileges are revoked for 24 hours.
Gasp! Not the sysadmin privileges!
Turns out it was me. The actual unit I wanted was this:
# mnt-itsuki.automount
[Unit]
[Automount]
Where=/mnt/itsuki
TimeoutIdleSec=0
[Install]
WantedBy=multi-user.target
Thanks, Arch Linux Wiki page on Samba!
Other than that, everything's been fine! The two constants that have been working throughout all of this were 1Password and Firefox, modulo that one time I updated Firefox in dnf and then got a half-broken browser until I restarted it. I did have to disable the nftables backend in libvirt in order to get outbound TCP connections working though.
Fedora is pretty set and forget, but it's not without its annoyances. The biggest one is how Fedora handles patented video codecs and how this intersects with FFmpeg, the swiss army chainsaw of video conversion.
Seriously, FFmpeg is one of the best programs ever made. If you have any image or video file format, you can use FFmpeg to make it any other video or image file format. Seriously one of the best programs ever made and it's absolutely surreal that they use Anubis to protect their bug tracker.
Fedora ships a variant of FFmpeg they call ffmpeg-free. Notably this version has "non-free" codecs compiled out, so you can deal with webm, av1, and other codecs without issue. However h.264, or the 4 in .mp4 is not in that codec list. Basically everything on the planet has support for h.264, so this is the "default format" that many systems use. Heck, all the videos I've embedded into this post are encoded with h.264.
You can pretty easily swap out ffmpeg-free with normal un-addled ffmpeg if you install the RPM Fusion repository, but that has its own fun.
RPM Fusion is the not-quite-official-but-realistically-most-users-use-it-so-it's-pretty-much-official side repo that lets you install "non-free" software. This is how you get FFmpeg, steam, and the NVidia binary drivers that make your GPU work.
One of the most annoying parts about RPM Fusion is that whenever they push new versions of anything, every old package is deleted off of their servers. This means that if you need to do a downgrade to debug issues (like strange XWayland not starting issues), you CANNOT restore your system to an older state because the package manager will see that the packages it needs aren't available from upstream and rightly refuse to put your system in an inconsistent state.
I have tried to get in contact with the RPMFusion team to help them afford more storage should they need it, but they have not responded to my contact attempts. If you are someone or know someone there that will take money or storage donated on the sole condition that they will maintain a few months of update backlog, please let me know.
I'm not really sure how to end something like this. Sure things mostly work now, but I guess the big lesson is that if you are a seasoned enough computer toucher, eventually you will stumble your way into a murder mystery and find out that you are both the killer and the victim being killed at the same time.
And you’re also the detective!
But, things work* and I'm relatively happy with the results.
2025-04-21 08:00:00
If you wanted to give me money but Patreon was causing grief, I'm on GitHub Sponsors now! Help me reach my goal of saving the world from AI scrapers with the power of anime.
2025-04-12 08:00:00
That meme is not an understatement, Anubis has been deployed by the United Nations.
For your amusement, here is how the inner monologue of me finding out about this went:
What. You can't be serious, can you?
No, that can't be a real domain of the United Nations, can it?
Wikipedia lists unesco.org as the
official domain of the United Nations Educational, Scientific and Cultural
Organization. I'm pretty sure it's real.
No way. No fucking way. What the heck, how is this real. What is YOUR LIFE??? God I got the worst 2025 bingo card this year.
I hate to shake my can and ask for donations, but if you are using Anubis and it helps, please donate on Patreon. I would really love to not have to work in generative AI anymore because the doublethink is starting to wear at my soul.
Also, do I happen to know anyone at UNESCO? I would love to get in touch with their systems administrator team and see if they had any trouble with setting it up. I'm very interested in making it easier to install.
This makes the big deployments that I know about include:
The conversation I'm about to have with my accountant is going to be one of the most surreal conversations of all time.
The part that's the most wild to me is when I stop and consider the scale of these organizations. I think that this means that the problem is much worse than I had previously anticipated. I know that at some point YouTube was about to hit "the inversion" where they get more bot traffic than they get human traffic. I wonder how much this is true across most of, if not all of the Internet right now.
I guess this means that I really need to start putting serious amounts of effort into Anubis and the stack around it. The best way that can be ensured is if I can get enough money to survive so I can put my full time effort into it. I may end up hiring people.
This is my life now. Follow me on Bluesky if you want to know when the domino meme gets more ridiculous!
2025-04-05 08:00:00
Today I did an oopsie. I tried to upgrade a service in my homelab cluster (alrest) but accidentally upgraded it in the production cluster (aeacus). I was upgrading ingress-nginx to patch the security vulnerabilities released a while ago. I should have done it sooner, but things have been rather wild lately and now kernel.org runs some software I made.
Either way, I found out that Oh my ZSH (the ZSH prompt toolkit I use) has a plugin for kube_ps1. This lets you put your active Kubernetes context in your prompt so that you're less likely to apply the wrong manifest to the wrong cluster.
To install it, I changed the plugins list in my ~/.zshrc:
-plugins=(git)
+plugins=(git kube-ps1)
And then added configuration at the end for kube_ps1:
export KUBE_PS1_NS_ENABLE=false
export KUBE_PS1_SUFFIX=") "
PROMPT='$(kube_ps1)'$PROMPT
This makes my prompt look like this:
(⎈|alrest) ➜ site git:(main) ✗
Showing that I'm using the Kubernetes cluster Alrest.
Wouldn't it be better to modify your configuration such that you always have
to pass a --context flag or something?
Yes, but some of the tools I use don't have that support universally. Until I can ensure they all do, I'm willing to settle for tamper-evident instead of tamper-resistant.
Apparently when I set up the Kubernetes cluster for my website, the Anubis docs and other things like my Headscale server, I did a very creative life decision. I started out with the "baremetal" self-hosted ingress-nginx install flow and then manually edited the Service to be a LoadBalancer service instead of a NodePort service.
I had forgotten about this. So when the upgrade hit the wrong cluster, Kubernetes happily made that Service into a NodePort service, destroying the cloud's load balancer that had been doing all of my HTTP ingress.
Thankfully, Kubernetes dutifully recorded logs of that entire process, which I have reproduced here for your amusement.
| Event type | Reason | Age | From | Message |
|---|---|---|---|---|
| Normal | Type changed | 13m | service-controller | LoadBalancer -> NodePort |
| Normal | DeletingLoadBalancer | 13m | service-controller | Deleting load balancer |
| Normal | DeletedLoadBalancer | 13m | service-controller | Deleted load balancer |
OOPS!
Thankfully, getting this all back up was easy. All I needed to do was change the Service type back to LoadBalancer, wait a second for the cloud to converge, and then change the default DNS target from the old IP address to the new one. external-dns updated everything once I changed the IP it was told to use, and now everything should be back to normal.
Well, at least I know how to do that now!
2025-03-31 08:00:00
Anubis is an AI scraper bot filter that puts a wall between your website and the lowest-hanging fruit of bot filtering. I developed it to protect my git server, but it's also used to protect bug trackers, Mastodon instances, and more. The goal is to help protect the small Internet so small communities can continue to exist at the scale they're currently operating at without having to resort to overly expensive servers or terrifyingly complicated setups.
Anubis has kind of exploded in popularity in the last week. GitHub stars are usually a very poor metric because they're so easy to game, but here's the star graph for Anubis over the last week:
Normally when I make projects, I don't expect them to take off. I especially don't expect to front page news on Ars Technica and TechCrunch within the span of a few days. I very much also do not expect to say sentences like "FFmpeg uses a program I made to help them stop scraper bots taking down their issue tracker". The last week has been fairly ridiculous in that regard.
There has been a lot of interest in me distributing native packages for Anubis. These packages would allow administrators that don't use Docker/OCI Containers/Podman to use Anubis. I want to build native packages, but building native packages is actually a fair bit more complicated than you may realize out of the gate. I mean it sounds simple, right?
But you can JUST build a tarball, right? Anubis is written in Go, you
can just do that easily with GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o var/anubis-linux-amd64 ./cmd/anubis and not faff around with the Docker
webshit bloat right?
It would be "just" that simple for getting something out of the door, but it's a poor UX compared to what Docker gives--
I have detected bloat, you YAML merchant you! Reject complexity! Return to native packages!
Okay, okay, the conversations don't go exactly like that, but that's what it can feel like sometimes.
Here's a general rule of thumb: "just" is usually a load-bearing word that hides a lot of complexity. If it was "just" that simple, it would have already been done.
If you want to package Anubis for your distribution of choice, PLEASE DO IT! Please make sure to let me know so I can add it to the docs along with instructions about how to use it.
Seriously, nothing in this post should be construed into saying "do not package this in distros". A lot of "stable" distros may have difficulty with this because I need Go 1.24 features for an upcoming part of Anubis. I just want to cover some difficulties in making binary packages that y'all already have had to reckon with that other people have not yet had to think about.
With all that said, buckle up.
Before I go into the hard details of building native packages and outlining what I think is the least burdensome solution, it may be helpful to keep Anubis' (and Techaro's) threat model in mind. The solution I am proposing will look "overkill", but given these constraints I'm sure you'll think it's "just right".
Anubis is open source software under the MIT license. The code is posted on GitHub and is free for anyone to view the code, download it, learn from it, produce derivative works from it, and otherwise use the software for any purpose.
Personally, I'd hope that bigger users of this will contribute back to the project somehow so that I can afford to have working on this be my full-time job. However I understand that not every project or community can afford doing that. Open source projects are usually nights and weekends affairs. The goal of Anubis is to protect the small internet, and small internet usually comes with small budgets.
Anubis is trusted by some big organizations like GNOME, Sourcehut, and ffmpeg. This social proof is kind of both a blessing and a curse, because it means if anything goes wrong, it could go very wrong all at once.
The project is exploding in popularity. Here's that star count graph again:
The really wild part about that star count graph is that you can see a sine wave if you rotate it by 45 degrees. A sine wave in metrics like that lets you know that growth is healthy and mostly human-sourced. This is wild to see.
Right now the team is one person that works on this during nights and weekends. As much as I'd like this to not be the case, my time is limited and my dayjob must take precedence so that I can afford to eat and pay my bills. Don't get me wrong, I'd love to work on this full time, but my financial situation currently requires me to keep my dayjob.
I also have a limited capacity for debugging "advanced" issues (such as those that only show up when you are running a program as a native package instead of in an environment like Docker/OCI/Podman), and I am as vulnerable to burnout as you are.
Speaking of burnout, this project has exploded in popularity. I've never had a project go hockey stick like this before. It's happened at companies I've worked at, sure, but never something that's been "my fault". This is undefined territory for me. Waking up and finding out you're on the front page of Ars Technica and getting emails requesting comment from TechCrunch reporters is kinda stressful.
Maybe it's a "good" kind of stress though? I don't know. What I do know is that I broke my media training that I've gotten from past employers over the years replying to the journalist from TechCrunch.
Some personal facts and circumstances which I am not going to go into detail about have made my sleep particularly bad the last week. As I'm writing this, I had a night with a solid 8 hours of sleep, so maybe that's on the mend. However when you get bad sleep for a bit, it tends to not make you have a good time.
Anubis is security software. Security software usually needs to be held to a higher standard than most other types of software. This means that "small typos" or forgotten bits of configuration from the initial rageware spike can actually become glaring security issues. There's been a lot of "founder code" cleanup so far and I can only see more coming in the future.
Of note: the Techaro standard about security is relevant here. At Techaro, we realize that computers are fractals of complexity and that any program is essentially built and reliant upon the behavior of a massive number of unknown unknowns. When at all possible we'll try to minimize the amount of security related bugs that may show up, but those unknown unknowns can and will bite at any time. Over time, that list of security advisories is undoubtedly going to grow because that's just how networked software works.
We try to not measure the number of vulnerabilities that made it in, we measure how we react when one does come in. Techaro treats people that report security issues to [email protected] seriously.
There's a few other standards like this (Be not a cancer upon the earth, The purpose of a system is what it does, etc.), and we could blow hot air about them, but I think it's better for them to be demonstrated by example instead of claimed on a webbed site with catchy slogans and lofty words puffed up by hot air.
Also, if this goes wrong, I'm going to get personally mega-cancelled. I would really like that to not happen, but this is the biggest existential risk and why I want to take making binary packages this seriously.
So with all of those constraints in mind, here's why it's not easy to "just" make binary packages.
Like was said earlier:
But you can JUST build a tarball, right? Anubis is written in Go, you can
just do that easily with GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o var/anubis-linux-amd64 ./cmd/anubis and not faff around with the Docker
webshit bloat right?
Sure, it is possible to JUST build a tarball with a single shell script like this:
cd var
DIR="./anubis-$(cat VERSION)-linux-amd64"
mkdir -p $DIR/{bin,docs,run}
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o $DIR/bin/anubis ./cmd/anubis
cp ../README.md $DIR
cp ../LICENSE $DIR
cp ../docs/docs/admin/installation.mdx $DIR/docs/installation.md
cp ../web/static/botPolicy.json $DIR/docs/botPolicy.json
cp ../run/* $DIR/run/
tar cJf ${DIR}.txz ${DIR}
And just repeat it for every GOOS/GOARCH pair that we want to support (probably gonna start out with linux/amd64, linux/arm64, freebsd/amd64, freebsd/arm64). Let's be real, this would work, but the main problem I have with this is that this is a poor developer experience, and it also is a poor administrator experience (mostly because those binary tarball packages leave installation as an exercise for the reader).
When I make binary packages for Anubis, I want to specify the package once and then have the tooling figure out how to make it happen. Ideally, the same build instructions should be used for both distribution package builds and tarballs. If the experience for developers is bad or requires you to minimax into ecosystem-specific tooling, this means that the experience is fundamentally going to be better on one platform over another. I want this software to be as universally useful as possible, so I need to design the packaging process to:
The administrator experience bit is critical. As much as we'd all like them to, administrators simply do not have the time or energy to do detailed research into how a tool works. Usually they have a problem, they find the easiest to use tool, square peg into round hole it to solve the immediate problem, and then go back to the twelve other things that are on fire.
One of the really annoying downsides to wanting to do native packages as a downstream project is that the interfaces for making them suck. They suck so much for the role of a project like Anubis. They're optimized for consuming software from remote tarballs, doing the bare minimum to fit within the distribution's ecosystem.
For the context they operate in, this makes a lot of sense. Their entire shtick is pulling in software from third parties and distributing it to their users. There's a reason we call them Linux distributions.
However with Anubis, I want to have the packaging instructions live alongside the application source code. I already trust the language-level package managers that Anubis uses (and have been careful to minimize dependencies to those that are more trustable in general), and they already have hash validation. Any solution RFC-2119-MUST build on top of this trust and use it as a source of strength.
There frankly is a middle path to be found between the "simple binary tarball" and mini-maxxing into the exact perculiarities of how rpmbuild intersects with Go modules and NPM.
Honestly, this is why I was planning on paywalling binary packages. Binary packages also have the added complication that you have to play nice with the ways that administrators configure their servers. Debugging this is costly in terms of time and required experties. I am one person doing this on nights and weekends. I only have so much free time. I wish I had more, but right now I simply do not.
When creating a plan like this, it's best to start with the list of problems you want to solve so that you can aggressively cut scope to remove the list of problems you don't want to solve yet. Luckily, the majority of the Linux ecosystem seems to have standardized around systemd. This combined with the fact that Go can just build static binaries means that I can treat the following OSes as fungible:
Building Debian and Red Hat packages will cover all of them.
Additionally, anything else that is RPM/Debian based except maybe Devuan. Of those, there's three main CPU architectures that are the most common with a long tail of other less common ones:
At some level, this only means that we need to build 6 variants (one per CPU for Debian and Red Hat derived distros) to cover 99.9% of the mutable distributions in common use. This is a much more manageable level of complexity, I could live with this.
Anything else in the "long tail" (FreeBSD, OpenBSD, Alpine Linux, etc.) is probably better handled by their native packages or ports system anyways, but this is a perfect place for the binary tarballs to act as a stopgap.
When I was working on the big Kubernetesification of my homelab last year, I was evaluating Rocky Linux and I was also building a tool called yeet as a middle ground between complicated shell scripts and bespoke one-off deployment tools written in Go. A lot of my build and deploy scripts boiled down to permutations of the following steps:
I started building yeet because I had previously done this with shell scripts that I copied to every project's folder. This worked great until I had a semantics bug in one of them that turned into a semantics bug in all of them. I figured I needed a more general metapattern and thus yeet was born.
I had known about a tool called nfpm. nfpm lets you specify the filesystem layout of distribution packages in a big ol' yaml file that got processed and spat out something you can pass to dpkg -i or dnf -y install. The only problem is that it just put the files into the package. This was fine for something like the wallpapers I submitted to the Bluefin project, but the build step was left as an exercise for the reader.
I was also just coming down from a pure NixOS UX and wanted something that could let me get 80% of the nice parts of Nix with only requiring me to put in 20% of the effort that it took to make Nix.
So I extended yeet to build RPM packages. Here is the yeetfile.js that yeet uses to build its own packages:
["amd64", "arm64", "riscv64"].forEach((goarch) =>
rpm.build({
name: "yeet",
descriptions: "Yeet out actions with maximum haste!",
homepage: "https://within.website/",
license: "CC0",
goarch,
build: (out) => {
go.build("-o", `${out}/usr/bin/yeet`, ".");
},
})
);
That's it. When you run this (with just yeet), it will create a gitignored folder named var and build 6 packages there. You can copy these packages to anywhere you can store files (such as in object storage buckets). yeet's runtime will natively set the GOARCH, GOOS, and CGO_ENABLED variables for you under the hood so that you can just focus on the build directions without worrying about the details.
The core rules of yeet are:
These native interfaces are things like go.build(args...) to trigger go build, docker.push(tag) to push docker images to a remote repository, git.tag() to get a reasonable version string, etc.
All of these are injected as implicit global objects. This is intended to let me swap out the "runtime backend" of these commands so that I can transparently run them on anonymous Kubernetes pods, other servers over SSH, or other runtime backends that one could imagine would make sense for a tool like yeet.
Deep lore: yeet was originally going to be Techaro's first product, but
Anubis beat it to the punch!
I plan to split building binary packages into at least two release cycles. The first release will be all about making it work, and the second release will be about making it elegant.
When I'm designing Anubis I have three sets of experiences in mind:
The balance here is critical. These forces are fundamentally all in conflict, but currently the packaging situation is way too far balanced towards developer experience and not towards administrator experience. I hope that this strategy makes it easier for websites like The Cutting Room Floor to get relief from the attacks they're facing.
The first phase is focused on making this work at all. A lot of the hard parts involving making yeet able to build Debian and Red Hat packages are already done. A lot of the rest of this involves software adulting including:
yeet.A lot of this is going to be tested with the (currently private) TecharoHQ/yeet repository.
The next stage will involve making the administrator experience a lot nicer. When administrators install packages, they expect them to be in repositories. This enables all software on the system to be updated at once, which is critical to the anticipated user experience. In order to reach this milestone, here's what I need to do:
The first pass of a repository backend will be done with off the shelf tooling like aptly and rpm-s3. It's gonna suck at first, but this pass is all about making it work. Making it elegant can come next.
Finally, I will improve the ecosystem such that other people can build on top of the Anubis tooling. Among the other tasks involved with this:
yeet depending on facts and circumstances.By the way, regarding Software Bills of Materials, I have not found clear guidance in Ubuntu, Debian, OpenSUSE, Red Hat, or Fedora's documentation on what I'm supposed to do with it or where to put it so that the system can collect and merge it. This is extra important from the perspective of Anubis because this packaging strategy is bypassing the normal flow for building distribution packages. Am I missing something? I found a dead thread about adding the SBOM to an RPM header, but that's it.
As much as I'd love to end this with a "and here's how you can try this now" invitation, I'm simply not ready for that yet. This is gonna take time, and the best way to make this happen faster is to donate to the project so I can focus more of my free time on Anubis.
Also, don't do the mistake I did. I started playing Final Fantasy 14. It is so good. It sucks you in hard. 10/10. Don't play it unless you're willing to get enthralled by it.
Hopefully the expanded forms of yeet and whatever repository management tooling end up being useful to other projects. But for now, I'm starting with something small, slim, and opinionated. It's much easier to remove opinions from a project than it is to add them.
In the meantime, I hope you can understand why I've only shipped a Docker/OCI/Podman image so far. The amount of scope reduction is immense. However this project is getting popular and in order to show I'm taking it seriously I need to expand the packaging scope to include machines that don't run Docker.
