Xe IasoModify

Anubis works

2025-04-12 08:00:00

That meme is not an understatement, Anubis has been deployed by the United Nations.

For your amusement, here is how the inner monologue of me finding out about this went:

I hate to shake my can and ask for donations, but if you are using Anubis and it helps, please donate on Patreon. I would really love to not have to work in generative AI anymore because the doublethink is starting to wear at my soul.

Also, do I happen to know anyone at UNESCO? I would love to get in touch with their systems administrator team and see if they had any trouble with setting it up. I'm very interested in making it easier to install.

This makes the big deployments that I know about include:

• The Linux Kernel Mailing List archives
• FreeBSD's SVN (and soon git)
• SourceHut
• FFmpeg
• Wine
• UNESCO
• The Science Olympiad Student Center
• Enlightenment (the desktop environment)
• GNOME's GitLab

The conversation I'm about to have with my accountant is going to be one of the most surreal conversations of all time.

The part that's the most wild to me is when I stop and consider the scale of these organizations. I think that this means that the problem is much worse than I had previously anticipated. I know that at some point YouTube was about to hit "the inversion" where they get more bot traffic than they get human traffic. I wonder how much this is true across most of, if not all of the Internet right now.

I guess this means that I really need to start putting serious amounts of effort into Anubis and the stack around it. The best way that can be ensured is if I can get enough money to survive so I can put my full time effort into it. I may end up hiring people.

This is my life now. Follow me on Bluesky if you want to know when the domino meme gets more ridiculous!

Life pro tip: put your active kubernetes context in your prompt

2025-04-05 08:00:00

Today I did an oopsie. I tried to upgrade a service in my homelab cluster (alrest) but accidentally upgraded it in the production cluster (aeacus). I was upgrading ingress-nginx to patch the security vulnerabilities released a while ago. I should have done it sooner, but things have been rather wild lately and now kernel.org runs some software I made.

Either way, I found out that Oh my ZSH (the ZSH prompt toolkit I use) has a plugin for kube_ps1. This lets you put your active Kubernetes context in your prompt so that you're less likely to apply the wrong manifest to the wrong cluster.

To install it, I changed the plugins list in my ~/.zshrc:

-plugins=(git)
        +plugins=(git kube-ps1)
        

And then added configuration at the end for kube_ps1:

export KUBE_PS1_NS_ENABLE=false
        export KUBE_PS1_SUFFIX=") "
        
        PROMPT='$(kube_ps1)'$PROMPT
        

This makes my prompt look like this:

(⎈|alrest) ➜  site git:(main) ✗
        

Showing that I'm using the Kubernetes cluster Alrest.

Why upgrading ingress-nginx broke my HTTP ingress setup

Apparently when I set up the Kubernetes cluster for my website, the Anubis docs and other things like my Headscale server, I did a very creative life decision. I started out with the "baremetal" self-hosted ingress-nginx install flow and then manually edited the Service to be a LoadBalancer service instead of a NodePort service.

I had forgotten about this. So when the upgrade hit the wrong cluster, Kubernetes happily made that Service into a NodePort service, destroying the cloud's load balancer that had been doing all of my HTTP ingress.

Thankfully, Kubernetes dutifully recorded logs of that entire process, which I have reproduced here for your amusement.

Event type	Reason	Age	From	Message
Normal	Type changed	13m	service-controller	LoadBalancer -> NodePort
Normal	DeletingLoadBalancer	13m	service-controller	Deleting load balancer
Normal	DeletedLoadBalancer	13m	service-controller	Deleted load balancer

Thankfully, getting this all back up was easy. All I needed to do was change the Service type back to LoadBalancer, wait a second for the cloud to converge, and then change the default DNS target from the old IP address to the new one. external-dns updated everything once I changed the IP it was told to use, and now everything should be back to normal.

Well, at least I know how to do that now!

Building native packages is complicated

2025-03-31 08:00:00

Anubis has kind of exploded in popularity in the last week. GitHub stars are usually a very poor metric because they're so easy to game, but here's the star graph for Anubis over the last week:

Normally when I make projects, I don't expect them to take off. I especially don't expect to front page news on Ars Technica and TechCrunch within the span of a few days. I very much also do not expect to say sentences like "FFmpeg uses a program I made to help them stop scraper bots taking down their issue tracker". The last week has been fairly ridiculous in that regard.

There has been a lot of interest in me distributing native packages for Anubis. These packages would allow administrators that don't use Docker/OCI Containers/Podman to use Anubis. I want to build native packages, but building native packages is actually a fair bit more complicated than you may realize out of the gate. I mean it sounds simple, right?

Okay, okay, the conversations don't go exactly like that, but that's what it can feel like sometimes.

Here's a general rule of thumb: "just" is usually a load-bearing word that hides a lot of complexity. If it was "just" that simple, it would have already been done.

A note to downstream packagers

If you want to package Anubis for your distribution of choice, PLEASE DO IT! Please make sure to let me know so I can add it to the docs along with instructions about how to use it.

Seriously, nothing in this post should be construed into saying "do not package this in distros". A lot of "stable" distros may have difficulty with this because I need Go 1.24 features for an upcoming part of Anubis. I just want to cover some difficulties in making binary packages that y'all already have had to reckon with that other people have not yet had to think about.

With all that said, buckle up.

Anubis' threat model

Before I go into the hard details of building native packages and outlining what I think is the least burdensome solution, it may be helpful to keep Anubis' (and Techaro's) threat model in mind. The solution I am proposing will look "overkill", but given these constraints I'm sure you'll think it's "just right".

Things in Anubis' favor

Anubis is open source software under the MIT license. The code is posted on GitHub and is free for anyone to view the code, download it, learn from it, produce derivative works from it, and otherwise use the software for any purpose.

Anubis is trusted by some big organizations like GNOME, Sourcehut, and ffmpeg. This social proof is kind of both a blessing and a curse, because it means if anything goes wrong, it could go very wrong all at once.

The project is exploding in popularity. Here's that star count graph again:

The really wild part about that star count graph is that you can see a sine wave if you rotate it by 45 degrees. A sine wave in metrics like that lets you know that growth is healthy and mostly human-sourced. This is wild to see.

Things not in Anubis' favor

Right now the team is one person that works on this during nights and weekends. As much as I'd like this to not be the case, my time is limited and my dayjob must take precedence so that I can afford to eat and pay my bills. Don't get me wrong, I'd love to work on this full time, but my financial situation currently requires me to keep my dayjob.

I also have a limited capacity for debugging "advanced" issues (such as those that only show up when you are running a program as a native package instead of in an environment like Docker/OCI/Podman), and I am as vulnerable to burnout as you are.

Speaking of burnout, this project has exploded in popularity. I've never had a project go hockey stick like this before. It's happened at companies I've worked at, sure, but never something that's been "my fault". This is undefined territory for me. Waking up and finding out you're on the front page of Ars Technica and getting emails requesting comment from TechCrunch reporters is kinda stressful.

Some personal facts and circumstances which I am not going to go into detail about have made my sleep particularly bad the last week. As I'm writing this, I had a night with a solid 8 hours of sleep, so maybe that's on the mend. However when you get bad sleep for a bit, it tends to not make you have a good time.

Anubis is security software. Security software usually needs to be held to a higher standard than most other types of software. This means that "small typos" or forgotten bits of configuration from the initial rageware spike can actually become glaring security issues. There's been a lot of "founder code" cleanup so far and I can only see more coming in the future.

Also, if this goes wrong, I'm going to get personally mega-cancelled. I would really like that to not happen, but this is the biggest existential risk and why I want to take making binary packages this seriously.

So with all of those constraints in mind, here's why it's not easy to "just" make binary packages.

Packaging software is hard

Like was said earlier:

Sure, it is possible to JUST build a tarball with a single shell script like this:

cd var
        DIR="./anubis-$(cat VERSION)-linux-amd64"
        mkdir -p $DIR/{bin,docs,run}
        GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o $DIR/bin/anubis ./cmd/anubis
        cp ../README.md $DIR
        cp ../LICENSE $DIR
        cp ../docs/docs/admin/installation.mdx $DIR/docs/installation.md
        cp ../web/static/botPolicy.json $DIR/docs/botPolicy.json
        cp ../run/* $DIR/run/
        tar cJf ${DIR}.txz ${DIR}
        

And just repeat it for every GOOS/GOARCH pair that we want to support (probably gonna start out with linux/amd64, linux/arm64, freebsd/amd64, freebsd/arm64). Let's be real, this would work, but the main problem I have with this is that this is a poor developer experience, and it also is a poor administrator experience (mostly because those binary tarball packages leave installation as an exercise for the reader).

Don't repeat yourself

When I make binary packages for Anubis, I want to specify the package once and then have the tooling figure out how to make it happen. Ideally, the same build instructions should be used for both distribution package builds and tarballs. If the experience for developers is bad or requires you to minimax into ecosystem-specific tooling, this means that the experience is fundamentally going to be better on one platform over another. I want this software to be as universally useful as possible, so I need to design the packaging process to:

1. Have the same developer experience as Docker images do ("push button, receive bacon").
2. Have the most seamless administrator experience possible (ideally through distribution-native flows).
3. Have as little cognitive overhead as possible for all parties involved (take the most obvious approach possible, when things need to diverge then be very loud about it).

The administrator experience bit is critical. As much as we'd all like them to, administrators simply do not have the time or energy to do detailed research into how a tool works. Usually they have a problem, they find the easiest to use tool, square peg into round hole it to solve the immediate problem, and then go back to the twelve other things that are on fire.

One of the really annoying downsides to wanting to do native packages as a downstream project is that the interfaces for making them suck. They suck so much for the role of a project like Anubis. They're optimized for consuming software from remote tarballs, doing the bare minimum to fit within the distribution's ecosystem.

For the context they operate in, this makes a lot of sense. Their entire shtick is pulling in software from third parties and distributing it to their users. There's a reason we call them Linux distributions.

However with Anubis, I want to have the packaging instructions live alongside the application source code. I already trust the language-level package managers that Anubis uses (and have been careful to minimize dependencies to those that are more trustable in general), and they already have hash validation. Any solution RFC-2119-MUST build on top of this trust and use it as a source of strength.

There frankly is a middle path to be found between the "simple binary tarball" and mini-maxxing into the exact perculiarities of how rpmbuild intersects with Go modules and NPM.

Honestly, this is why I was planning on paywalling binary packages. Binary packages also have the added complication that you have to play nice with the ways that administrators configure their servers. Debugging this is costly in terms of time and required experties. I am one person doing this on nights and weekends. I only have so much free time. I wish I had more, but right now I simply do not.

Cutting scope

When creating a plan like this, it's best to start with the list of problems you want to solve so that you can aggressively cut scope to remove the list of problems you don't want to solve yet. Luckily, the majority of the Linux ecosystem seems to have standardized around systemd. This combined with the fact that Go can just build static binaries means that I can treat the following OSes as fungible:

• Ubuntu Server
• Debian
• Red Hat Enterprise Linux
• OpenSUSE
• SUSE Enterprise Linux
• Rocky Linux
• Alma Linux
• CentOS Stream

Building Debian and Red Hat packages will cover all of them.

Additionally, anything else that is RPM/Debian based except maybe Devuan. Of those, there's three main CPU architectures that are the most common with a long tail of other less common ones:

1. x86_64 (64 bit x86, goarch amd64)
2. aarch64 (64 bit arm, goarch arm64)
3. riscv64 (64 bit RISC-v, goarch riscv64)

At some level, this only means that we need to build 6 variants (one per CPU for Debian and Red Hat derived distros) to cover 99.9% of the mutable distributions in common use. This is a much more manageable level of complexity, I could live with this.

Anything else in the "long tail" (FreeBSD, OpenBSD, Alpine Linux, etc.) is probably better handled by their native packages or ports system anyways, but this is a perfect place for the binary tarballs to act as a stopgap.

Yeeting packages out

When I was working on the big Kubernetesification of my homelab last year, I was evaluating Rocky Linux and I was also building a tool called yeet as a middle ground between complicated shell scripts and bespoke one-off deployment tools written in Go. A lot of my build and deploy scripts boiled down to permutations of the following steps:

1. Build docker image somehow (Nix, Docker, Earthly, ko, etc.)
2. Push to remote host
3. Trigger rolling release to the new image

I started building yeet because I had previously done this with shell scripts that I copied to every project's folder. This worked great until I had a semantics bug in one of them that turned into a semantics bug in all of them. I figured I needed a more general metapattern and thus yeet was born.

I had known about a tool called nfpm. nfpm lets you specify the filesystem layout of distribution packages in a big ol' yaml file that got processed and spat out something you can pass to dpkg -i or dnf -y install. The only problem is that it just put the files into the package. This was fine for something like the wallpapers I submitted to the Bluefin project, but the build step was left as an exercise for the reader.

I was also just coming down from a pure NixOS UX and wanted something that could let me get 80% of the nice parts of Nix with only requiring me to put in 20% of the effort that it took to make Nix.

So I extended yeet to build RPM packages. Here is the yeetfile.js that yeet uses to build its own packages:

["amd64", "arm64", "riscv64"].forEach((goarch) =>
          rpm.build({
            name: "yeet",
            descriptions: "Yeet out actions with maximum haste!",
            homepage: "https://within.website/",
            license: "CC0",
            goarch,
        
            build: (out) => {
              go.build("-o", `${out}/usr/bin/yeet`, ".");
            },
          })
        );
        

That's it. When you run this (with just yeet), it will create a gitignored folder named var and build 6 packages there. You can copy these packages to anywhere you can store files (such as in object storage buckets). yeet's runtime will natively set the GOARCH, GOOS, and CGO_ENABLED variables for you under the hood so that you can just focus on the build directions without worrying about the details.

The core rules of yeet are:

1. Thou shalt never import code from another file nor require npm for any reason.
2. If thy task requires common functionality, thou shalt use native interfaces when at all possible.
3. If thy task hath been copied and pasted multiple times, yon task belongeth in a native interface.

These native interfaces are things like go.build(args...) to trigger go build, docker.push(tag) to push docker images to a remote repository, git.tag() to get a reasonable version string, etc.

All of these are injected as implicit global objects. This is intended to let me swap out the "runtime backend" of these commands so that I can transparently run them on anonymous Kubernetes pods, other servers over SSH, or other runtime backends that one could imagine would make sense for a tool like yeet.

The game plan

I plan to split building binary packages into at least two release cycles. The first release will be all about making it work, and the second release will be about making it elegant.

When I'm designing Anubis I have three sets of experiences in mind:

1. User experience: what the experience is like for the end users that don't know (or care) what Anubis is
2. Developer experience: what the experience is like for me and the open source contributors developing Anubis
3. Administrator experience: what the experience is like for people setting it up on their boxes

The balance here is critical. These forces are fundamentally all in conflict, but currently the packaging situation is way too far balanced towards developer experience and not towards administrator experience. I hope that this strategy makes it easier for websites like The Cutting Room Floor to get relief from the attacks they're facing.

Making it work at all

The first phase is focused on making this work at all. A lot of the hard parts involving making yeet able to build Debian and Red Hat packages are already done. A lot of the rest of this involves software adulting including:

• Setting up a dedicated Mac mini to act as the build and signing host. This machine MUST NOT be used for anything else.
• Write documentation on how to build your own Debian/RPM packages with yeet.
• Write documentation specific to how to use the packages to run Anubis.
• Ensure that critical documentation is copied to the packages so that users can self-serve without access to the Anubis website.
• Uploading built packages to GitHub releases.

A lot of this is going to be tested with the (currently private) TecharoHQ/yeet repository.

Integration with distribution-specific package managers

The next stage will involve making the administrator experience a lot nicer. When administrators install packages, they expect them to be in repositories. This enables all software on the system to be updated at once, which is critical to the anticipated user experience. In order to reach this milestone, here's what I need to do:

• Fork yeet out of the /x/ monorepo and into a TecharoHQ repo (this has already been started, but will be slow going at first, I need to de-Xe the implementation of yeet).
• Set up repositories backed by object storage. That Mac mini from before will be the only machine that is allowed to write to that bucket. I will have to coordinate with the object storage provider I will be using in order to make sure this is the case.
• Publish a noarch Debian/RPM backage to bootstrap the Techaro root of trust and repo files. There will be at least two repositories: Debian and RPM. I may have to do the thing that Tailscale does where they lie about what distros are supported so that the repo URLs don't stand out too much compared to the experience that administrators expect from other publishers.
• Add support for basically every platform that Go can compile Linux binaries for. A lot of the small internet runs on smaller hardware. By making things more compatible, we can protect more small communities.

The first pass of a repository backend will be done with off the shelf tooling like aptly and rpm-s3. It's gonna suck at first, but this pass is all about making it work. Making it elegant can come next.

Leaving the ecosystem in a better state than we found it

Finally, I will improve the ecosystem such that other people can build on top of the Anubis tooling. Among the other tasks involved with this:

• Publish tooling for automating the management of repositories stored in object storage buckets based on what I like about aptly and rpm-s3. This will probably be included into yeet depending on facts and circumstances.
• Publish Software Bill of Materials (SBOM) reports with all packages. Ideally this will be done in the package themselves, but the important part is to make them available at all.
• Add Alpine and Arch Linux packages/repositories.

This will take time

As much as I'd love to end this with a "and here's how you can try this now" invitation, I'm simply not ready for that yet. This is gonna take time, and the best way to make this happen faster is to donate to the project so I can focus more of my free time on Anubis.

Hopefully the expanded forms of yeet and whatever repository management tooling end up being useful to other projects. But for now, I'm starting with something small, slim, and opinionated. It's much easier to remove opinions from a project than it is to add them.

In the meantime, I hope you can understand why I've only shipped a Docker/OCI/Podman image so far. The amount of scope reduction is immense. However this project is getting popular and in order to show I'm taking it seriously I need to expand the packaging scope to include machines that don't run Docker.

The surreal joy of having an overprovisioned homelab

2025-03-25 08:00:00

I like making things with computers. There’s just one problem about computer programs: they have to run somewhere. Sure, you can just spin up a new VPS per project, but that gets expensive and most of my projects are very lightweight. I run most of them at home with the power of floor desktops.

Tonight I’ll tell you what gets me excited about my homelab and maybe inspire you to make your own. I'll get into what I like about it and clue you into some of the fun you get to have if one of your projects meant to protect your homelab goes hockey-stick.

Hi everyone! I’m Xe, and I’m the CEO of Techaro, the anti-AI AI company. Today I’m gonna talk with you about the surreal job of having an over-provisioned homelab and what you can do with one of your own. Buckle up, it’s gonna be a ride.

So to start, what’s a homelab? You may have heard of the word before, but what is it really?

It’s a playground for devops. It’s where you can mess around to try and see what you can do with computers. It’s where you can research new ways of doing things, play with software, and more. Importantly though, it’s where you can self-host things that are the most precious to you. Online platforms are vanishing left and right these days. It’s a lot harder for platforms that run on hardware that you look at to go away without notice.

Before we continue though, let’s cover who I am. I’m Xe. I live over in Orleans with my husband and our 6 homelab servers. I’m the CEO of the totally real company Techaro. I’m an avid blogger that’s written Architect knows how many articles. I stream programming crimes on Fridays.

Today we’re gonna cover:

• What a homelab is
• What you can run on one
• A brief history of my homelab
• Tradeoffs I made to get it to its current form
• What I like about it

Finally I’ll give you a stealth mountain into the fun you can have when you self host things.

Before we get started though, my friend Leg Al told me that I should say this.

This talk may contain humor. Upon hearing something that sounds like it may be funny, please laugh. Some of the humor goes over people’s heads and laughing makes everyone have a good time.

Oh, also, any opinions are my own and not the opinions of Techaro.

Unless it would be funny for those opinions to be the opinions of Techaro, then it would be totally on-brand.

But yes, tl;dr: when you have servers at home, it’s a homelab. They come in all shapes and sizes from single mini pcs from Kijiji to actual rack mount infrastructure in a basement. The common theme though is experimentation and exploration. We do these things not because they are easy, but because they look like they might be easy. Let’s be real, they usually are easy but you can’t know until you’ve done it to know for sure, right?

What I run

In order to give you ideas on what you can do with one, here’s what I run in my homelab. I use a lot of this all the time. It’s just become a generic place to put things with relative certainty that they’ll just stay up. I also use it to flex my SRE muscle because working in marketing has started to atrophy that and I do not want to lose that skillset.

One of the services I run is Plex which lets me—Wait, what, how did you get there?...One second.

Like I was saying, one of the services I run is Plex which lets me watch TV shows and movies without having to go through flowcharts of doom to figure out where to watch them.

One of the best things I set up was pocket-id, an OIDC provider. Before your eyes glaze over, here’s what you should think.

A lot of the time with homelabs and self-hosted services you end up making a new account, admin permissions flags, group memberships, and profile pictures for every service. This sucks and does not scale. Something like Pocket-ID lets you have one account to rule them all. It’s such a time-saver.

I also run a git server! It’s where Techaro’s super secret projects like the Anubis integration jungle live.

I run my own GitHub Actions runners because let’s face it, who would win: free cloud instances that are probably oversubscribed or my mostly idle homelab 5950x’s?

One of the big things I run is Longhorn, which spreads out the storage across my house. This is just for the Kubernetes cluster, the NAS has an additional 64-ish terabytes of space where I store my tax documents, stream VODs, and…Linux ISOs.

Like any good cluster I also have a smattering of support services like cert-manager, ingress-nginx, a private docker registry, external-dns, a pull-through cache of the docker hub for when they find out that their business model is unsustainable because nobody wants to pay for the docker hub, etc. Just your standard Kubernetes setup sans the standard “sludge pipe” architecture.

By the way, I have to thank my friend Eric Chlebek for coming up with the term “sludge pipe” architecture to describe modern CI/CD flows. I mean look at this:

You just pipe the sludge into git repos and it flows into prod! Hope it doesn’t take anything out!

I’ve also got a smattering of apps that I’ve written for myself over the years, including but not limited to the hlang website, Techaro’s website, the Stealth Mountain feed on Bluesky, a personal API that’s technically part of my blog’s infrastructure, the most adorable chatbot you’ve ever seen, a bot to post things on a subreddit to Discord for a friend, and Architect knows how many other small experiments.

The history of my homelab

Like I said though, you don’t always need to start out with a complicated multi-node system with distributed storage. Most of the time you’ll start out with a single computer that can turn on. I did.

I started out with this: a trash can Mac Pro that was running Ubuntu. I pushed a bunch of strange experiments to it over the years and it’s where I learned how to use Docker in anger. It’s been a while and I lost the config management for it, but I’m pretty sure it ran bog-standard Docker Compose with a really old version of Caddy. I’m pretty sure this was the machine I used as my test network when I was maintaining an IRC network. Either way, 12 cores and 16 GB of RAM went a long way in giving me stuff to play with. This lasted me until I moved to Montreal in mid-2019. It’s now my Prometheus server.

Then in 2020 I got the last tax refund I’m probably ever going to get. It was about 2.2 thousand snow pesos and I wanted to use it to build a multi-node homelab cluster. I wanted to experiment with multi-node replicated services without Kubernetes.

When I designed the nodes, I wanted to pick something that had a balance of cost, muscle, and wattage. I also wanted to get CPUs that had built-in PCI to HDMI converters in them so I can attach a “crash cart” to debug them. This was also before the AI bubble, so I didn’t have langle mangles in mind. I also made sure to provision the nodes with just enough power supply overhead that I could add more hard drives, GPUs, or whatever else I wanted to play with as new shiny things came out.

Here’s a few of them on screen, from left to right that is kos-mos, ontos, and pneuma. Most of the nodes have 32 GB of RAM and a Core-i5 10-600 with 12 threads. Pneuma has a Ryzen 5950x (retired from my husband’s gaming rig when he upgraded to a 7950x3D) and 64 GB of RAM. Pneuma used to be my main shellbox until I did the big Kubernetes changeover.

Not shown are Logos and Shachi. Shachi is my old gaming tower and has another 5950x in it. In total this gives me something like 100 cores and 160 GB of RAM. This is way overkill for my needs, but allows me to basically do whatever I want. Don’t diss the power of floor desktops!

Eventually, Stable Diffusion version 1 came out and then I wanted to play with it. The only problem was that it needed a GPU. Luckily we had an RTX 2060 laying around and I was able to get it up and running on Ontos. Early Stable Diffusion was so much fun. Like look at this.

The prompt for this was “Richard Stallman acid trip in a forest, Lisa frank 420 modern computing, vaporwave, best quality”. This was hallucinated, pun intended, on Ontos’ 2060. I used that 2060 for a while but then bigger models came out. Thankfully I got a job at a cloud provider so I could just leech off of their slack time. But I wanted to get langle mangles running at home so Logos got an RTX 3060 to run Ollama.

At a certain point though, a few things happened that made me realize that I was going off course for what I wanted. My homelab nodes weren’t actually redundant like I wanted. The setup I used had me allocate tasks to specific nodes, and if one of them fell over I had to do configuration pushes to move services around. This was not according to keikaku.

Then the distribution I was using made…creative decisions in community management and I realized that my reach as a large-scale content creator (I hate that term) and blogger meant that by continuing to advocate for that distro in its current state, I was de-facto harming people. So then I decided to look for something else.

Let’s be real, the kind of things I wanted out of my homelab were literally Kubernetes shaped. I wanted a bunch of nodes that I could just push jobs to and let the machine figure out where it lives. I couldn’t have that with my previous setup no matter how much I wanted because the tools just weren’t there to do it in real life.

This was kind of a shock, as previously I had been on record saying that you don’t in fact need Kubernetes. At the time I gave this take though, there were other options. Docker Swarm was still actively in development. Nomad was a thing that didn’t have any known glaring flaws other than being well Nomad, and Kubernetes was really looking like an over engineered pile of jank.

It really didn’t help that one of my past jobs was to create a bog-standard sludge pipe architecture on AWS and Google Cloud but way before cert-manager was stable. Ingress-nginx was still in beta. Everything was in flux.

Kubernetes itself was fine, but it was not enough to push button and receive bacon and get your web apps running somewhere. I get that’s not the point of Kubernetes per se, it scales from web apps to fighter jets, but at the end of the day you gotta ship something, right?

It really just burnt me out and I nearly left the industry at large as a result of the endless churn of bullshit. The admission that Kubernetes was what I needed really didn’t come easy. It was one of the last things I wanted to use; but with everything else either dying out from lack of interest or having known gaping flaws show up, it’s what I was left with.

Then at some point I thought, “eh, fuck it, what do I have to lose” and set it up. It worked out pretty great actually.

After a few months someone in the patron discord asked me what I thought about Kubernetes in my homelab after using it for a while and my reply was “It’s nice to not have to think about it”. To be totally honest, as someone with sludge pipe operator experience, “it’s nice to not have to think about it” is actually high praise. It just kinda worked out and I didn’t have to spend too much time or energy on it modulo occasional upgrades.

What I like about it

And with that in mind, here’s what I really like about my homelab setup as it is right now.

I can just push button and receive bacon. If I want to run more stuff, I push it to the cluster. If I want to run less stuff, I delete it from the cluster. Backups happen automatically every night. The backup restore procedure works. Pushing apps is trivial. Secrets are integrated with 1password. Honestly, pushing stuff to my homelab cluster is so significantly easier than it’s ever been at any company I’ve ever worked at. Even when I was a sludge pipe operator.

One of the best parts is that I haven’t really had to fight it. Stuff just kinda works and it’s glorious. My apps are available internally and externally and I don’t really have to think too much about the details.

Of course, I didn’t just stop there. I took things one step farther and then realized across my /x/ repo that I had a bunch of services fall into a few basic patterns:

• The first generic shape of service is the headless bot that just does a thing like monitor an RSS feed and poke a web hook somewhere. This only really needs a Deployment to manage the versions of the container images and maybe some secrets for API keys or the like.
• Second, I need to run programs that listen internally and serve API calls. Maybe they have some persistent storage. Either way, they definitely need a DNS name within the cluster so other services can use that API to do things like post messages on IRC.
• Third, some of the things I run are web apps. Webapps are pretty much the same, but they need a DNS name outside the cluster and a way to get HTTP ingress routed to the pod. I use nginx for that, but the configuration can be a bit fiddly and manual. It’d be nice to hyper automate it so that I don’t have to think about the details, I just think about the App.

I was really inspired by Heroku’s setup back when I worked there. With Heroku you just pushed your code and let the platform figure it out. Given that I had a few known “shapes” of apps, what if I just made my own resources in Kubernetes to do that?

apiVersion: x.within.website/v1
        kind: App
        metadata:
          name: httpdebug
        
        spec:
          image: ghcr.io/xe/x/httpdebug:latest
          autoUpdate: true
        
          ingress:
            enabled: true
            host: httpdebug.xelaso.net
        

So I did that, thanks to Yoke. I just define an App, and it creates everything downstream for me. 1Password Secrets can be put in the filesystem or the environment. Persistent storage is a matter of saying where to mount it and how much I want. HTTP ingresses are a simple boolean flag with the DNS name. External DNS records, TLS certificates, and the whole nine yards is naught but an implementation detail. A single flag lets me create a Tor hidden service out of the App so that people can view it wherever they want in the world without government interference. I can add Kubernetes roles by just describing the permissions I want. It’s honestly kind of amazing.

This is something I want to make more generic so that you can use it too, I’ll get to it eventually. It’s in the cards.

Learning to play defense

In the process of messing with my homelab, I’ve had to learn to play defense.

Something to keep in mind though: I have problems you don’t. My blog gets a lot of traffic in weird patterns. If it didn’t, I’d run it at home, but it does so I have to host it in the cloud. However, remember that git server? Yeah, that runs at home.

When you host things on the modern internet, bots will run in once the cert is minted and start pummeling the hell out of it. I like to think that the stuff I make can withstand this, but some things just aren’t up to snuff. It’s not their fault mind you, modern scraper bots are unusually aggressive.

Honestly it feels like when modern scrapers are designed, they have these goals in mind:

By the way, public service announcement. Don’t use VPNs unless you have a really good reason. Especially don’t use free VPNs. Those sketchy residential proxy services are all powered by people using free VPNs. If you aren’t a customer, you are the product.

What makes this worse is that git servers are the most pathologically vulnerable to the onslaught of doom from modern internet scrapers because remember, they click on every link on every page.

See those little yellow tags? Those are all links. Do the math. There’s a lot of them. Not to mention that git packfiles are stored in compressed files which can’t seek. Every time they open every link on every page, they go deeper and deeper into uncached git pack file resolution because let’s face it, who on this planet is going out of their way to look at every file in every commit of GTK from 2004 and older. Not many people it turns out!

And that’s how Amazon’s scraper took out my Git server. I tried some things and they didn’t work including but not limited to things I can’t say in a recording. I debated taking it offline completely and just having the stuff I wanted to expose publicly be mirrored on GitHub. That would have worked, but I didn’t want to give up. I wanted to get even.

Then I had an idea. Raise your hand if you know what I do enough to know how terrifying that statement is.

More of you than I thought.

Somehow I ended up on the wikipedia page for weighing of souls. Anubis, the god of the underworld, weighed your soul and if it was lighter than a feather you got to go into the afterlife. This felt like a good metaphor.

And thus I had a folder name to pass to mkdir. Anubis weighs the soul of your connection using a SHA256 proof-of-work challenge in order to protect upstream resources from scraper bots. This was a super nuclear response, but remember, this was the state of my git server:

I just wanted uptime, man.

Either way, the absolute hack I had worked, so I put it on GitHub. Honestly, when I’ve done this before it got ignored. So I just had my 4080 dream up some placeholder assets, posted an blog about it, and went back to playing video games.

Then people started using it. I put it in its own repo and posted about it on Bluesky.

I wasn’t the only one having this problem it seems! It’s kinda taking off! This is so wild and not the kind of problem I usually have.

Like the graphs went hockey stick.

Like really hockey-stick.

It just keeps going up and it’s not showing any signs of stopping any time soon.

For context, here it is compared to my two biggest other projects. It's the mythical second Y axis graph shape. So yeah, you can understand that it’s gonna take a while to circle back to the Techaro HyperCloud.

The cool part about this in my book though is that because I had a problem that was only exposed with the hardware my homelab uses (specifically because my git server was apparently running on rotational storage, oops), I got creative, made a solution, pushed it to GitHub, and now it’s in use to protect GNOME’s GitLab, SourceHut, small community projects, god knows how many git forges, and I’ve heard that basically every major open source project that self-hosts infrastructure is evaluating it to protect their websites too. I really must have touched a nerve or something.

Conclusion

In conclusion:

If you like it, you should self-host it. Online services are vanishing so frequently. Everything is centralizing around the big web and it makes me afraid for what the future of the small Internet could look like should this continue.

Think small. A single node with a 2012 grade CPU and 16 gigabytes of dedotated wam lasted me until 2019. When I get a computer, I use the whole computer. If it’s fine for me, it’s more than enough for you.

Fuck around and find out. That’s not just a threat. That’s a mission statement.

Remember that if you get an idea, fuck around, find out, and write down what you’ve learned: you’ve literally just done science. Well, with computers, so it’d be computer science, but you get my point.

And if bots should come in and start a-pummeling away, remember: you’re not in the room with them. They’re in the room with you. Remember Slowloris? A little birdie told me that it works server to client too. Consider that.

My time with you is about to come to an end, but before we go, I just want to thank everyone on this list. You know what you did. If you’re not on this list, you know what you didn’t do.

And with that, I've been Xe! I'll be around if you have questions or want stickers. Stay warm!

If I don’t get to you, please email your questions to [email protected]. With all that out of the way, does anyone have any questions?

I'm testing Anubis in prod

2025-03-20 08:00:00

Hey all!

Anubis has really been taking off to the point that it has its own repo now. I'm going to be doing more work on it, but for right now what I really need is data. In order to get this data, I need you to let me know what I just broke by turning on Anubis in prod.

What I know broke:

• Discord link resolving (still working on fixing this, but I wanted to get this post out first)
• Twitter link resolving

If I missed something, contact me.

Opsec and you: how to navigate having things to hide

2025-03-13 08:00:00

It feels like privacy has become "impossible", hasn't it? What does it mean to actually be "private" these days? Who are you defending against? What do you want to do in order to mitigate it? And more importantly, how do you do this without giving up the conveniences of modern life?

In this talk, I'll be covering the finer points of operational security (opsec), knowing your threat model, building your own infrastructure to self-host things that are important to you with discarded hardware, and how to "blend in" when traveling or even at home. It's all about balance and figuring out what your needs are. My needs are certainly a lot different than yours are. This is a nuanced topic and I am not going to pretend there isn't any.

Hi, I'm Xe. You probably know me from my blog. Today, I'm gonna give a talk that I really wish I didn't have to give. In a sane or just world, I wouldn't need to have this talk exist; however, we know what world we got and I'm here, so today I'm gonna talk about operational security or opsec.

Opsec is a somewhat multifaceted topic, but it really boils down to making sure you keep yourself safe online.

It’s really easy to go down the online privacy rabbit hole and way past Narnia. This is fundamentally a game of balancing your authentic expression with how much information you share. Again, it sucks that we have to have this conversation, but I’d really much rather y’all have the tools to protect yourselves.

Today, I’m gonna cover the basics of what opsec is, give you practical tips on how to protect yourself online, how to control what you can, be aware of the things you can’t, show you the tools you can use today to keep yourself safe, and give you tips on how you can set up your own online infrastructure so that you can have real privacy online.

Before we get into all that though, I’m Xe. I’m the CEO of Techaro, which is a totally real company that actually exists. I’ve written god knows how many articles and I’ve worked at a smattering of companies. Some of them you know, most of them you don’t. I live in Ottawa with my husband and my 6 homelab servers.

So, let’s talk about opsec. Today I’ll start out with what it means. Perfect security is impossible. Any actions you take are compromises. Sure in theory you can just become a hermit and live away from society, but that makes it difficult to do things like attend conference talks or post on social media. Like I said, it’s all about compromises and balance. Unless you're a citizen of Germany, in which case you can actually have real privacy online, asterisk.

Another thing to keep in mind is that it’s a lot easier to be one of the people out there in the audience watching this talk than it is to be me, the person giving it. There are completely different security implications at play. The trick is to figure out the right balance of information you share vs information you don’t share.

Also, you’re gonna fuck it up. You will accidentally leak something. You are going to make an error and it will be okay. The other trick with opsec is to balance things out such that when you do inevitably make that error you minimize the consequences. You will fall for a phishing link. The trick is when you inevitably fuck it up, the consequences are minimized as much as possible.

Threat modeling

The heart of operational security is the threat model. A threat model is the list of things and people you care about and what you are protecting against. This is probably one of the most personal parts of this. Your threat model is going to differ vastly from mine. Here’s an example threat model for a guy I just made up:

Let’s imagine a guy named Sleve McDichael. He’s a straight white dude that posts cooking videos to TikTok. He doesn’t really have any enemies and works as a car mechanic. He’s civilly involved and sometimes posts about US politics. He used to play baseball and probably peaked in high school.

Let’s say the worst thing that could happen to Sleve is that someone gets angry about one of his cooking videos. He doesn’t mention his employer in his cooking videos, maybe he’ll say “oh yeah I’m a car mechanic” at some point, but overall he doesn’t mention where he works. Just to be safe, he let his employer know about the cooking TikTok videos. Their reaction was “oh cool I’ll follow and make the good recipes”. Imagine how simple Sleve’s life is. This is the dream.

Sleve has random internet strangers in scope for his threat model. Random internet strangers aren’t the most predictable, but generally they have limits as to what they can do. Individuals can only really do small scale actions.

The other thing to keep in mind with Sleve’s threat model is that there’s things that are out of scope. Usually most threat models end where the government begins. Sure hope that’s not an ominous thing to say in Anno Dominium Two Thousand And Twenty Five fake laugh.

In terms of things that can impact his threat model, here’s the low hanging fruit that Sleve can control. He can control what he posts, such as by not mentioning that he works at Jiffy Lube. He can control what social media apps he uses, such as TikTok or Bluesky. He can control when he posts because you can figure out where someone lives by when you post (you usually don’t post while you’re asleep!). He can also control what he shows in any photos or videos he posts.

Now let’s take a look at the things Sleve can’t control. Generally, Sleve can control the things he does, but he can’t control what other people do in response to them. He can’t control what other people do, and he has even less control over what the government does. Sure, he votes, but I vote too.

There’s also a bunch of things in the middle between things Sleve can and can’t control. In theory he can control his writing style so that people can’t identify him by his “writeprint”, but changing your writeprint (or even being cognizant of it) is difficult for most people. If he’s really worried, he can use an AI tool to rewrite what he posts so that it’ll hide his writeprint. Yes, this is something that works, and every AI model has its own writeprint. Even models that run on your local device are good enough to hide it -- fun fact, the Torment Nexus has a use.

In theory, Sleve also has control of how he speaks (voice training is a thing that does exist), but it’s difficult to control for most people. These are things that he needs to keep in mind as he writes posts or makes cooking videos.

Opsec behaviors

Despite everything, Sleve still manages to keep himself safe online. In order to keep yourself safe like Sleve does, there’s a few behaviors you can follow and they’re mostly low-hanging fruit:

• Don’t follow out those viral online quizzes or install apps you don’t need to. Who knows what the publishers of those quizzes or viral apps are doing with your data. Remember Cambridge Analytica? That started with online quizzes. Once it's off your device, God knows.
• Another strategy is to google your name or usernames to see what comes up. Think like an attacker. What can you dig up about yourself from your online footprint?
• Be aware of phishing. This is statistically the thing that you are inevitably going to fuck up. Attackers only have to be lucky once, you have to be lucky every time. I’ve fallen for phishing before and because I set things up to lessen the consequences, nothing bad happened. I didn’t even lose control of that Discord account that got temporarily yoinked. I even got control back without contacting support.
• Use HTTPS. Browsers used to be more vocal about using HTTPS, but the s in HTTPS means “secure”. When you connect over HTTPS, it’s encrypted on the wire. Attackers may be able to see what domain names you are visiting, but they won’t see much more than that. Any contents of webpages or the paths you are visiting aren’t visible, even over public insecure wifi. The page you're visiting, the contents, or the paths are secure, even over public Wi-Fi.

• Use multi-factor authentication. It’s free. Passkeys are built into every major OS and are immune to phishing. Use 6 digit two factor authentication codes if you have to, but if you can avoid it never use SMS authentication codes. Your bank may not let you disable SMS authentication though. Your password manager will have support for two-factor auth stuff; I'll get into password managers later.
• Before you post something, take a moment to think about what you’re about to do. Is it really worth posting? Once you post something, even if you delete it, it’s really hard to un-post it. It’s much easier to just not post it in the first place. I have a thing set up to let me think I’m posting things but it just deletes them. Best thing I’ve set up in a while. One of the things I have set up for myself is a website that looks like Twitter, so I can type things and hit "post", and it just gets sent to /dev/null. It's great, one of the best things I've ever set up.
• Use full disk encryption on your machines. If you use a Mac, it’s on by default. If you use Windows, look for BitLocker in your settings. If you use Linux, look for LUKS in your distribution’s documentation. Full disk encryption is especially important for laptops because laptops can and will inevitably be left behind at the coffee shop. If the disk is encrypted, the machine is worthless to attackers.

Nyms

One of the things you can do to keep yourself anonymous online is to use pseudonyms, also known as nyms. These are names that don’t match the name on your passport. If you’re part of the furry community, you probably know your best friends by names like Soatok, Cendyne, or Framebuffer instead of whatever their passport names are. Pseudonyms are really easy to adopt and can be a great way to add personality to your online presence.

Fun fact: the name I use professionally is a pseudonym! I don’t use my passport name professionally so that I can brand myself better. Xe Iaso is three syllables instead of the longer name that I use on my passport that people constantly misspell and mispronounce. It's also three syllables, and I thought it would be less easy to typo, but I've also had to buy the domain xeLaso.net because someone at Apple decided that the serifs on lowercase L were too ugly.

If you are going to adopt pseudonyms, make sure that you only use two or three separate nyms at once. If you use more than that, you’ll run into the risk of confusing them with each other. If you’re plural, you may be able to get away with more, your mileage may vary, less is more. You’ve probably run into something I’ve published under a pseudonym and never known. Someone you know has published under a pseudonym and you've never known.

If you’re going to use pseudonyms longer term, make sure to make their social media accounts in advance and “age” them. New accounts look more suspicious than older accounts do. Brand new accounts have things that stand out in the UI of most social platforms to make them look fishy, because most phishing comes from brand new accounts. Accounts that recently became active after being idle also look suspicious for super-intense scrutiny, but you can automate posting to prevent a lot of the worst effects. Don’t feel bad about aging your nyms for a few months or even a year.

Pro tip: use AI models to help anonymize your writing. I use obscure locally hosted models to do this so that people can't place why they think the text looks familiar. This is a great way to keep your writing style from being used to identify you.

Metadata

One of the other big things to think about with regards to opsec is metadata. Metadata is data about data. One of the best examples of metadata is the data attached to photos. Here’s an example with a photo I took on my iPhone:

This is a photo I took in New York City in order to communicate how strange the sign was to me. I still think it’s kinda strange, but here’s the metadata that my iPhone attached: It says "no standing," referring to stopped cars.

Wow, that’s a lot of info! It says I used an iPhone 15 Pro Max with the telephoto lens at ISO 50, f/2.8, a shutter speed of 1/125 seconds, and has the exact GPS coordinates the photo was taken at. Let's break this down. The telephoto lens is about 120mm equivalent, has an aperture of f2.8, shutter speed of 1/125 seconds, and has the exact GPS coordinates of where I hit the capture button. This is a shocking amount of metadata at first glance. It makes you wonder, how much information are you really sharing when you upload a picture to the internet?

The good news is that online platforms know about this and take steps to prevent you from doxxing yourself with picture metadata. Most of this data is stored as EXIF data. Modern platforms will scrub this data before sharing any photos users upload. I've seen some mobile OSes, like CalyxOS and GrapheneOS, strip that at the photo picker level. But your mileage may vary; you may be more or less paranoid.

If you use Firefox, you can install the GPSDetect extension and you’ll get a notification every time someone leaves GPS metadata in their photos. The link to the extension will be in a resource list at the end. Here’s an example of what it looks like in action:

You’ll get notifications like this every time someone didn’t strip the GPS metadata from their photos. When I encounter these in the wild, I usually send an email to the people that published those photos to help them out. They’re almost always thankful.

Other bit of metadata you may not think about: pictures of the sky can be used to figure out where the photo was taken. This requires more complicated attacks, but try to avoid posting pictures of the sky the same day you are taking them. If they're posted within about five minutes of when you took them, a dedicated attacker can figure out where you are.

Some people vary, but most people have a 24 hour sleep cycle. About 8 hours of the day are going to be spent sleeping. Usually when people are asleep, they aren’t posting. Here’s an example based on my Reddit account:

I live in eastern time. My most active hours on reddit align with the morning and evening eastern time. This is my Reddit account's peak activity time: right after work, and right after I wake up. If you were looking at my Reddit account history, you could probably figure out that I live in eastern time just from the metadata of when I post. This is something to keep in mind.

Tools

Now that we covered metadata, let’s branch into the more practical part of this talk: what tools you should use.

Browsers

As far as browsers go: use very common browsers. Pick either Firefox or Chrome. They are very boring browsers, but they’re used by a lot of people. If someone hacks Chrome or Firefox, it’s almost certainly not to hack you in particular. They both suck, but they are used by so many people that nobody is going to attack you in particular via Chrome or Firefox, because there are way more high-value targets like governments and banks. Common browsers also mean that you blend into the crowd and are harder to attack. Common browsers also mean your metadata blends in better and is harder to uniquely identify.

VPNs

One of the things that you’re gonna want to do is shove all your traffic into a VPN. This is what the YouTubers suggest after all, it sounds like it’s a good idea, and it’s not that expensive, right? It encrypts your IP address, right? It stops the hackers from getting your information! It's what the YouTubers suggest with the NordVPN and ProtonVPN ads, and advertising hasn't lied to you, has it? It's not that expensive, it's like three Starbucks drinks in 2019.

Don’t.

Don’t use VPN services unless you have a very good reason to. Privacy VPNs are the security snake oil of our day. You should only use a VPN service as your default route if you have a very good reason to, such as to make sure that your very legal Linux ISOs are able to be downloaded without getting love letters.

Remember that bit about HTTPS? HTTPS is already encrypted. You don’t need to encrypt it again with a VPN. I mean, you can if you want, but you don't need to.

Use the Tor browser for any browsing that you really want to be private. Tor is free. Tor is used by a lot of people all over the world. It's free, and it's available on your OS of choice.

Remember that ancient meme that went something like “you can’t get me, I’m behind seven proxies”. That’s how Tor works.

Tor takes your traffic and uses onion routing to send it through a bunch of nodes and then end up getting to the target through an indirect route. This gives you even more privacy advantages than a VPN server does, especially because every website is inevitably going to be using a different circuit. Your computer sends traffic to a node that decrypts it, unwraps it, and sends it along until it reaches an exit node, which sends it to the target. You get the response back, do the whole song and dance, and you get there indirectly, usually through like seven European countries.

You can download the Tor browser for free from torproject.org. Again, I’ll have a resource list linked at the end of the talk. The Tor browser is available on every major OS. The Tor Project is getting an aarch64 Linux port soon. The Tor browser is made by experts that care.

The only thing to keep in mind is that you shouldn’t use it all the time, and this is more from a practical angle rather than a theoretical angle. Tor helps keep activists safe and lets people evade government censorship, but there’s also a shocking amount of abusive traffic that comes from Tor exit nodes. Lots of websites block Tor in order to protect themselves. This probably includes your favorite websites. Lots of websites, like Reddit, block Tor to protect themselves.

Messaging

If you’re gonna message people, use Signal. Make sure to enable disappearing messages. Disappearing messages mean that everything you send with people gets automatically deleted after a configurable amount of time. I personally use a week for most people I know.

Signal is one of the few encrypted messaging apps that has Soatok approval.

Of note: when nation state actors attack Signal, they don’t even go after the cryptography. They just attack convenience features like linked devices. When nation-state actors attack Signal, they don't go after the cryptography; they phish you. That should say a lot about Signal's security.

One of the annoying features of Signal is that it doesn’t sync message scrollback to new devices by default. I think this is a feature and proof that the messages ARE NOT BEING SAVED ON THE SERVER, but this can be an annoyance. I think they're changing this, but I think it's a feature. It's proof that messages are not being saved on the server. It's a balance of trade-offs.

Password managers

Use a password manager. Your device or browser likely comes with one. That one is free. I personally use 1Password with my husband and it works great for us. It’s effortless and even supports all the two-factor auth that we use. I use 1Password because we used it before a lot of the other options existed. But if you use a Mac, there's a password manager built into your iCloud account. I think Microsoft has a similar thing, but I try to avoid using Windows.

Your password manager has a password generator embedded into it. Use it. You should not know your passwords beyond the root password you use to unlock the password manager. If you only use randomly generated passwords, you can’t reuse passwords. A generated password cannot be reused unless someone has broken randomness, in which case we all have bigger issues. You should not know your passwords beyond the root password. If you only use generated passwords, you can't reuse passwords, and reused passwords are how people get popped.

Run updates

I know that Windows is a giant pain in the ass about updates, but seriously, run them. Updates get released for a reason. Updates patch security issues. If you don’t install updates, you can’t be protected by them. Running updates regularly is one of the easiest ways to make sure that your computers are secure. Seriously, run updates.

Self-hosting

Finally, you should probably know how to host things yourself. This gives you the most understanding of what platform owners can see about what you do because you become a platform. Self-hosting also can give you absolute superpowers, like being able to have every TV show or movie you want steaming at a moment’s notice without having to follow a flowchart or use dedicated websites to find out where you can watch things. No, seriously, there's a website that has detailed flowcharts for every show now, based on the show, what country you're in, and so on. It's a nightmare. There was a video by videogamedunkey about figuring out where to watch a TV show. He didn't even need to write any comedy, he just described the process of trying to watch, I think it was Severance.

If you want to get started with self-hosting, any computer will do really. You can get used desktops off of Craigslist, your local university’s surplus store, or at Woot.com. When you’re starting out, you probably don’t really have elaborate hardware needs, but anything that can turn on and run Linux is fine. You probably just need something that can turn on.

As for what to run on it, all the normal options suck equally at this point. The important part is to pick whatever you’re the most comfortable with learning about. Ubuntu and Rocky are the closest to what you’d use in production if you were to become a career systems administrator or site reliability expert. But really by this point everything is the same brand of suckitude in different ways. Some are more up to date than others, others prioritize unchanging stability, the important part is to Just Pick Something™️. Some suck more than others. Some are more out of date than others, and consider that a feature.

Once you have the OS, set up something like k3s or Docker Compose. Then you can install whatever self hosted apps you want. Here’s a whirlwind tour of the self hosted apps that I use on a regular basis: Yes, I know Kubernetes seems like a lot, but that's where the entire industry is going, because Kubernetes has sucked out all of the oxygen for everything else.

• Plex is self-hosted Netflix that points to a folder full of media. I use it to watch anime and catch up on old movies.
• If you want to only run open source software, there’s also Jellyfin. I personally don’t use it because I bought a lifetime Plex pass a while ago, but when Plex inevitably kills off the lifetime Plex pass I’m gonna set up Jellyfin. I don't use it because Plex was dumb enough to sell me a lifetime Plex pass for like $20. When Plex inevitably kills that off, I'm probably going to set up Jellyfin.
• Nextcloud is like google docs, google calendar, an email client, google drive, and Slack all in one. It can do anything from instant messaging to meetings to integrations with self-hosted AI models. I’ve been meaning to use it more, but I mostly use it for storing files. It can integrate with AI stuff that runs on hardware you can look at.
• Gitea is a self hosted GitHub. You can push private repos and even run CI on them without GitHub ever seeing your code. I use it for Techaro’s secret projects.
• Pocket ID is an identity provider. This lets me have one account for all my internal services so that I don’t need to configure individual passwords for individual services. This is honestly one of the best things I’ve ever set up and the time savings add up so much. I don't need to configure individual passwords, group memberships, and all that nightmare nonsense. It sounds abstract, but it makes a lot of sense in practice.

One of the other big things I have in my homelab is my own apps. Here’s a screenshot of what I’m running: I've been working on something to make this easier, which I'll announce at some point in the future.

Listed there I have a bunch of static sites for community resources, monitoring tools, pocket-id, the slang website, a Bluesky passive scraper, a docker registry, the Techaro website, a pull-through cache of the docker hub, and even a self-hosted object storage system called Minio. This gives me basically unlimited abilities to host whatever I want. The industry standardized on Kubernetes, so whenever I want to add something else, it’s a cinch. I have a website for a satirical programming language based around the letter H, a Bluesky passive scraper, a Docker registry, the Techaro website, a pull-through cache of the Docker Hub (because they realized that their business model is inviable, so they're jacking down the rate limit), and a self-hosted object storage system called Minio. Hosting stuff myself gives me basically unlimited superpowers to do whatever I want. Because the industry standardized on Kubernetes, I can put stuff on my home lab and then move it to the cloud without thinking more than pushing a YAML file in the right place.

One of the other cool things you can do with Kubernetes is set up a Tor hidden service controller. This lets you expose your blog or another service only to people using Tor. This lets you expose services to your friends without leaking your home IP address to the world. Doing this is slow, but it’s a tradeoff that makes sense in many cases. Tor hidden services are neat; they're a way to expose a website such that people can only view it over the Tor browser, and in ideal scenarios, you can't tell where that website is hosted.

I use this for my blog so that you can access what I write regardless of any government or corporate censorship. I also plan to write something in the near future that will only be visible to people reading my blog over Tor, so keep an eye out for that! I’ll have more details about this in the resource sheet at the end. I also plan to write something that's probably going to need to only be released over Tor.

Conclusion

It’s been so much fun, but my time with you is about to run out. Let’s wrap this up. In conclusion:

• Know your threat model. Who are you protecting against? What could they do? How can you handle the inevitable opsec fail?
• Think before you post. It’s easier to not post something than it is to un-post something. If you really need to let it out, write it out on paper and burn it. That can't be hacked.
• Your phone attaches GPS coordinates to photos. Strip them or use a platform you know strips them before you share them. Verify it with extensions like GPSDetect.
• Don’t use a VPN unless you’re using it to get back into your homelab. Unless it's a site-to-site VPN to get back into hosting you control.
• Use Tor when you want to keep browsing private. I use Tor all the time for research. Tor is love, Tor is life.
• Use Signal for private chat. Make sure to enable disappearing messages. You got it.
• Know how to host things yourself. Anything on your own hardware is infinitely more private than anything involving a platform.
• Again, run updates. Updates are free and they patch security issues.

Before we go though, I wanna give some special thanks to all these people. You know what you did to help. If you’re not on this list, you know what you didn’t do.

And with that, I've been Xe! I'll be around if you have questions or want stickers. Stay warm! This is the first of two conferences I'm presenting at this weekend.

If I don’t get to you, please email your questions to [email protected]. With all that out of the way, does anyone have any questions? I will get back to you as soon as I can.

Q&A

Question: Can you speak about the privacy and security trade-offs between self-hosting and what it offers for privacy versus those security risks?

Xe: It's a trade-off. If you're hosting something for somebody else to connect to, then you need to make sure that keeps up to date. If you're using Kubernetes, there are ways to install tools like Keel, which will automatically update things for you, so you don't have to think about it. I use stuff like that heavily so that I have basically everything automated as much as possible. But in general, if you run updates, you're probably not going to be someone that someone's going to waste a zero-day on. And if you are that kind of person, my talk probably isn't for you because you probably need the advice of a dedicated opsec specialist. And I'm not that; I'm not even going to pretend that I am able to be that.

Question: When you say to not use VPNs, are you talking about WireGuard mesh networks such as the one that Tailscale provides?

Xe: Yeah, you can use something like a WireGuard mesh network. I use that for some of my stuff when I connect to my home lab services. A lot of them are not exposed to the public internet. I have my Kubernetes cluster set up with a unique domain name, so I can just address it by the service name. So, when I am starting to stream on Twitch, I have a PowerShell script on my desktop that I double-click, and it sends a POST request to an internal service that announces that I'm streaming. It is very hacky, but it works, asterisk.

Question: How about self-hosting your email services?

Xe: What's the diplomatic way to phrase this? I can't stop you from hurting yourself. Personally, I pay Google for my email because Google doesn't have support. And if it doesn't have support, you can't phish support. Which is kind of a horrible thing to say. But like, let's be real, one of the biggest threat vectors at this point is people phishing the support for like your phone provider, and then managing to convince them that you need a new SIM card and SIM swapping you and oh, they just stole all your apes.

Question: If you're self-hosting things, some ISPs will work to interfere with that, and can like jack down the speed or prevent incoming ICMP or something to make it difficult. How would you work around that?

Xe: I'm gonna be totally honest with the stuff that I self host that's exposed to the public internet. I have a VPS set up in Toronto that runs the moral equivalent of HA proxy. And that sends all the traffic that listens, that's the address that gets put into DNS connections go on there, get sent out over wire guard hit one of the nginx ingress pods in my home lab. And then you know that routes to wherever the hell it is all across the house gets response goes all the way back out to the internet and to the person to the person I have found that this adds like 15 milliseconds of lag and that's like literally like one frame at 60 hz and in order for people to notice it, it has to be closer to like 150 milliseconds. So it's not really that bad. In terms of providers to use for that. I use Vultr for mine, but you may want to look into Civo. The reason why is they don't have egress fees. And if a cloud provider these days is willing to make that pricing decision, you should take advantage of it while you can.