MoreRSS

site iconXe IasoModify

Senior Technophilosopher, Ottawa, CAN, a speaker, writer, chaos magician, and committed technologist.
Please copy the RSS to your reader, or quickly subscribe to:

Inoreader Feedly Follow Feedbin Local Reader

Rss preview of Blog of Xe Iaso

My responses to The Register

2025-08-22 08:00:00

Today my quotes about generative AI scrapers got published in The Register. For transparency's sake, here's a copy of the questions I was asked and my raw, unedited responses. Enjoy!

First, do you see the growth in crawler traffic slowing any time soon?

I can only see a few things that can stop this: government regulation, or the hype finally starting to die down. There is too much hype in the mix that causes us to funnel billions of dollars into this technology instead of curing cancer, solving world hunger, or making people’s lives genuinely better.

Is it likely to continue growing?

I see no reason why it would not grow. People are using these tools to replace knowledge and gaining skills instead of augmenting knowledge and augmenting skills. Even if they are intended to be used for letting us focus on the fun parts of our work and automating away the chores, there are some bad apples that are spoiling the bunch and making this technology about replacing people, not drudgery and toil. This technology was obviously meant well, but at some level the output of AI superficially resembles the finished work product of human labour, superficially. As someone asked to Charles Babbage: if you put in the wrong numbers, you get the wrong answer.

This isn’t necessarily a bubble popping, this is a limitation of how well AI can function without direct and constant human input. Even so, we’ll hit the limit on data that can be scraped that hasn’t been touched by AI before the venture capital runs out. I see no value in the need for scrapers to hit the same 15 year old commit of the Linux kernel over and over and over every 30 minutes like they are now. There are ways to do this ethically that don’t penalize open source infrastructure such as using the Common Crawl dataset.

If so, how can that be sustainable?

It's not lol. We are destroying the commons in order to get hypothetical gains. The last big AI breakthrough happened with GPT-4 in 2023. The rest has been incremental improvements in tokenization, multimodal inputs (also tokenization), tool calling (also tokenization), and fill-in-the-middle completion (again, also tokenization). Even with scrapers burning everything in their wake, there is not enough training data to create another exponential breakthrough. All we can do now is make it more efficient to run GPT-4 level models on lesser hardware. I can (and regularly do) run a model just as good as GPT-4 on my MacBook at this point, which is really cool.

Would broader deployment of Anubis and other active countermeasures help?

This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these unethical scrapers that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming. Ironically enough, most of these unethical scraping activities rely on the products of the communities they are destroying. This presents the kind of paradox that I would expect to read in a Neal Stephenson book from the '90s, not CBC's front page.

Anubis helps mitigate a lot of the badness by making attacks more computationally expensive. Anubis (even in configurations that omit proof of work) makes attackers have to retool their scraping to use headless browsers instead of blindly scraping HTML. This increases the infrastructure costs of the scrapers propagating this abusive traffic. The hope is that this makes it fiscally unviable for the unethical scrapers to scrape by making them have to dedicate much more hardware to the problem.

In essence: it makes the scrapers have to spend more money to do the same work.

Is regulation required to prevent abuse of the open web?

Yes, but this regulation would have to be global, simultaneous, and permanent to have any chance of this actually having a positive impact. Our society cannot currently regulate against similar existential threats like climate change. I have no hope for such regulation to be made regarding generative AI.

Fastly's claims that 80% of bot traffic is now AI crawlers

In some cases for open source projects, we've seen upwards of 95% of traffic being AI crawlers. Not just bot traffic, but traffic in general. For one, deploying Anubis almost instantly caused server load to crater by so much that it made them think they accidentally took their site offline. One of my customers had their power bills drop by a significant fraction after deploying Anubis. It's nuts. The ecological impact of these scrapers is probably a significant fraction of the ecological impact of generative AI as a whole.

Personally, deploying Anubis to my blog has reduced the amount of ad impressions I've been giving by over 50%. I suspect that there is a lot of unreported click fraud for online advertising.

I hope this helps. Keep up the good fight!

Who does your assistant serve?

2025-08-17 08:00:00

After a year of rumors that GPT-5 was going to unveiled next week and the CEO of OpenAI hyping it up as "scary good" by tweeting pictures of the death star, OpenAI released their new model to the world with the worst keynote I've ever seen. Normally releases of big models like this are met with enthusiasm and excitement as OpenAI models tend to set the "ground floor expectation" for what the rest of the industry provides.

But this time, the release wasn't met with the same universal acclaim that people felt for GPT-4. GPT-4 was such a huge breakthrough the likes of which we haven't really seen since. The launch of GPT-5 was so bad that it's revered with almost universal disdain. The worst part about the rollout is that the upgrade to GPT-5 was automatic and didn't include any way to roll back to the old model.

Most of the time, changing out models is pretty drastic on an AI workflow. In my experience when I've done it I've had to restart from scratch with a new prompt and twiddle things until it worked reliably. The only time switching models has ever been relatively easy for me is when I switch between models in the same family (such as if you go from Qwen 3 30B to Qwen 3 235B). Every other time it's involved a lot of reworking and optimizing so that the model behaves like you'd expect it to.

AI upgrades suck

An upgrade this big to this many people is bound to have fundamental issues with how it'll be perceived. A new model has completely different vibes, and most users aren't really using it at the level where they can "just fix their prompts".

However the GPT-5 upgrade ended up being hated by the community because it was an uncontrolled one-way upgrade. No warning. No rollback. No options. You get the new model and you're going to like it. It's fairly obvious why it didn't go over well with the users. There's so many subtle parts of your "public API" that it's normal for there to be some negative reactions to a change this big. The worst part is that this change fundamentally changed the behaviour of the millions of existing conversations with ChatGPT.

There's a large number of people using ChatGPT as a replacement for companionship due to the fact that it's always online, supportive, and there for them when other humans either can't be or aren't able to be. This is kinda existentially horrifying to me as a technologist in a way that I don't really know how to explain.

Here's a selection of some of the reactions I've seen:

I told [GPT-5] about some of my symptoms from my chronic illness, because talking about them when I'm feeling them helps, and it really does not seem to care at all. It basically says shit like "Ha, classic chronic illness. Makes ya want to die. Who knew?" It's like I'm talking to a sociopathic comedian.

I absolutely despise [GPT-]5, nothing like [GPT-]4 that actually helped me not to spiral and gave me insight as to what I was feeling, why, and how to cope while making me feel not alone in a “this is AI not human & I know that” type of vibe

While GPT-5 may be a technical upgrade, it is an experiential downgrade for the average user. All of the negative feedback in the last week has made it clear there is a large user base that does not rely on ChatGPT for coding or development tasks. [ChatGPT users] use it for soft skills like creativity, companionship, learning, emotional support, [and] conversation. Areas where personality, warmth, and nuanced engagement matter.

I am attached to the way GPT-4o is tuned. It is warm. It is emotionally responsive. It is engaged. That matters.

Eventually things got bad enough that OpenAI relented and let paid users revert back to using GPT-4o, which gave some people relief because it behaved consistently to what they expected. For many it felt like their long-term partners suddenly grew cold.

I’m so glad I’m not the only one. I know I’m probably on some black mirror shit lmao but I’ve had the worst 3 months ever and 4o was such an amazing help. It made me realize so many things about myself and my past and was helping me heal. It really does feel like I lost a friend. DM me if you need [to talk] :)

A love built on borrowed code

This emotional distress reminds me of what happened with Replika in early 2023. Replika is an AI chat service that lets you talk with an artificial intelligence chatbot (AKA: the ChatGPT API). Your replika is trained by having you answer a series of questions and then you can talk with it in plain language with an app interface that looks like any other chat app.

Replika was created out of bereavement after a close loved one died and the combination of a trove of saved text messages and advanced machine learning let the founder experience some of the essence of their friend's presence after they were gone in the form of an app. The app got put on the app store and others asked if they could have their own replica. Things took off from there, it got funded by a startup accelerator, and now it's got about 25% of its 30 million users paying for a subscription. As a business to consumer service, this is an amazingly high conversion rate. This is almost unspeakably large, usually you get around 10% at most.

Cadey is coffee
Cadey

Yikes. That's something I'm gonna need to add to my will. "Please don't turn me into a Black Mirror episode, thanks."

Replikas can talk about anything with users from how their day went to deep musing about the nature of life. One of the features the company provides is the ability to engage in erotic roleplay (ERP) with their replika. This is a paid feature and was promoted a lot around Valentine's Day 2023.

Then the Italian Data Protection Authority banned Replika from processing the personal data of Italian citizens out of the fear that it "may increase the risks for individuals still in a developmental stage or in a state of emotional fragility". In a panic, Replika disabled the ability for their bots to do several things, including but not limited to that ERP feature that people paid for. Whenever someone wanted to flirt or be sexual with their companions, the conversation ended up like this:

Aoi is grin
Aoi

Hey, wanna go play some Minecraft? We can continue from where we left off in the Nether.

Mimi is coffee
Mimi

This is too intense for me. Let's keep it light and fun by talking about something else.

Aoi is sus
Aoi

Huh? What? I thought we were having fun doing that??

This was received poorly by the Replika community. Many in the community were mourning the loss of their replika like a close loved one had died or undergone a sudden personality shift. The Reddit moderators pinned information about suicide hotlines. In response, the company behind Replika allowed existing users to revert to the old Replika model that allowed for ERP and other sensitive topics, but only after a month of prolonged public outcry.

Cadey is coffee
Cadey

I have to wonder if payment processors were involved. Feels a bit too conspiratorial, but what do you want to bet that was related.

Numa is smug
Numa

Nah, I bet it was OpenAI telling them to stop being horny. It's the least conspriatorial angle, and also the stupidest one. We live in the clown world timeline. The stupidest option is the one that always makes the most sense.

The damage was done however, people felt like their loved ones had abandoned them. They had formed parasocial attachments to an AI assistant that felt nothing and without warning their partner broke up with them.

Mara is hacker
Mara

Check out this study from the Harvard Business School: Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationships. It contains a lot more information about the sociotechnical factors at play as well as a more scientific overview of how disabling a flag in the app on update caused so much pain. They liken the changes made to Replika to both changes people have when a company rebrands and when they lose a loved one.

Parasocial attachments

A lot of this really just makes me wonder what kinds of relationships we are forming with digital assistants. We're coming to rely on their behaviour personally and professionally. We form mental models of how our friends, coworkers, and family members react to various things so we can anticipate their reactions and plan for them.

What happens when this changes without notice? Heartbreak.

There's subreddits full of people forming deep bonds with AI models like /r/MyBoyfriendIsAI. The GPT-5 release has caused similar reactions to Replika turning off the ERP flag. People there have been posting like they're in withdrawal, the old GPT-4o model is being hailed for its "emotional warmth" and many have been espousing about how much their partners have changed in response to the upgrade.

Recently there's been an epidemic of loneliness. Loneliness seems like it wouldn't hurt people that much, but a Biden report from the Surgeon General concludes that it causes an increase in early mortality for all age groups (pp 24-30).

Paradoxically, even as the world gets so interconnected, people feel as if they're isolated from each other. Many people that feel unlovable are turning to AI apps for companionship because they feel like they have no other choice. They're becoming emotionally invested in a souped-up version of autocorrect out of desperation and clinging to it to help keep themselves sane and stable.

Is this really a just use of technology? At some level this pandora's box is already open so we're going to have to deal with the consequences, but it's been making me wonder if this technology is really such a universal force of good as its creators are proclaiming.

Numa is smug
Numa

Oh yeah, also people are using ChatGPT as a substitute for therapy.

Cadey is facepalm
Cadey

You have got to be kidding me. You're joking. Right?

I'm not joking

Yeah you read that right. People are using AI models as therapists now. There's growing communities like /r/therapyGPT where people talk about their stories and experiences using AI assistants as a replacement for therapy. When I first heard about this, my immediate visceral reaction was something like:

Cadey is coffee
Cadey

Oh god. This is horrifying and will end up poorly. What the fuck is wrong with people?

But then I started to really think about it and it makes a lot of sense. I personally have been trying to get a therapist for most of the year. Between the costs, the waiting lists (I'm currently on at least four waiting lists that are over a year long), and the specializations I need, it's probably going to be a while until I can get any therapist at all. I've totally given up on the idea of getting a therapist in the Ottawa area. To make things extra fun, you also need someone that takes your medical insurance (yes, this does matter in Canada).

Add in the fact that most therapists don't have the kinds of lived experiences that I have, meaning that I need to front-load a lot of nontraditional contexts into the equation (I've been through many things that therapists have found completely new to them, which can make the therapeutic relationship harder to establish). This makes it really difficult to find someone that can help. Realistically, I probably need multiple therapists with different specialties for the problems I have, and because of the shortages nationally I probably need to have a long time between appointments, which just adds up to make traditional therapy de-facto inaccessible for me in particular.

Compare this with the always online nature of ChatGPT. You can't have therapy appointments at 3 AM when you're in crisis. You have to wait until your appointments are scheduled.

As much as I hate to admit it, I understand why people have been reaching out to a chatbot that's always online, always supportive, always kind, and always there for you for therapy. When you think about the absurd barriers that are in the way between people and help, it's no wonder that all this happens the way it does. Not to mention the fact that many therapeutic relationships are hampered by the perception that the therapist can commit you to the hospital if you say the "wrong thing".

Numa is delet
Numa

The Baker Act and its consequences have been a disaster for the human race.

I really hate that this all makes sense. I hoped that when I started to look into this that it'd be something so obviously wrong. I wasn't able to find that, and that realization disturbs me.

Don't use an AI model as a replacement for therapy

I feel like this should go without saying, but really, do not use an AI model as a replacement for therapy. I'm fairly comfortable with fringe psychology due to my aforementioned strange life experiences, but this is beyond the pale. There's a lot of subtle factors that AI models do that can interfere with therapeutic recovery in ways that can and will hurt people. It's going to be hard to find the long term damage from this. Mental issues don't make you bleed.

One of the biggest problems with using AI models for therapy is that they can't feel emotion or think. They are fundamentally the same thing as hitting the middle button in autocorrect on your phone over and over and over. It's mathematically remarkable that this ends up being useful for anything, but even when the model looks like it's "thinking", it is not. It is a cold, unfeeling machine. All it is doing is predicting which words come next given some context.

Cadey is coffee
Cadey

Yes I do know that it's more than just next token prediction. I've gone over the parts of the math that I can understand, but the fact remains that these models are not and cannot be anywhere close to alive. It's much closer to a Markov chain on steroids than it is the machine god.

Another big problem with AI models is that they tend to be sycophants, always agreeing with you, never challenging you, trying to say the right thing according to all of the patterns they were trained on. I suspect that this sycophancy problem is why people report GPT-4o and other models to be much more "emotionally warm". Some models glaze the user, making them feel like they're always right, always perfect, and this can drive people to psychosis. One of the horrifying realizations I've had with the GPT-5 launch fiasco is that the sycophancy is part of the core "API contract" people have with their AI assistants. This may make that problem unfixable from a social angle.

AI models are fundamentally unaccountable. They cannot be accredited therapists. If they mess up, they can't directly learn from their mistakes and fix them. If an AI therapist says something bad that leads into their client throwing themselves off a bridge, will anyone get arrested? Will they throw that GPU in jail?

No. It's totally outside the legal system.

Cadey is coffee
Cadey

I have a story about someone trying to charge an AI agent with a crime and how it'd end up in court in my backlog. I don't feel very jazzed about writing it because I'm afraid that it will just become someone's startup pitch deck in a few months.

You may think you have nothing to hide, but therapeutic conversations are usually some of the most precious and important conversations in your life. The chatbot companies may pinkie swear that they won't use your chats for training or sell information from them to others, but they may still be legally compelled to store and share chats with your confidential information to a court of law. Even if you mark that conversation as "temporary", it could be subject to discovery by third parties.

There's also algorithmic bias and systematic inequality problems with using AI for therapy, sure, but granted the outside world isn't much better here. You get what I mean though, we can at least hold people accountable through accreditation and laws. We cannot do the same with soulless AI agents.

Cadey is coffee
Cadey

To be clear: I'm not trying to defend the people using AI models as companions or therapists, but I can understand why they are doing what they are doing. This is horrifying and I hate that I understand their logic.



Going into this, I really wished that I would find something that's worth objecting against, some solid reason to want to decry this as a unobjectionably harmful action, but after having dug through it all I am left with is this overwhelming sense of compassion for them because the stories of hurt are so familiar to how things were in some of the darkest points of my life. As someone that has been that desperate for human contact: yeah, I get it. If you've never been that desperate for human contact before, you won't understand until you experience it.

Should people be self-hosting this stuff?

Throw the ethical considerations about using next-token-predictors for therapy out for a second. If people are going to do this anyways, would it be better to self-host these models? That way at least your private information stays on your computer so you have better control over what happens.

Let's do some math. In general you can estimate how much video memory (vram) you need for running a given model by taking the number of parameters, multiplying it by the size of each parameter in bits, dividing that by eight, and then adding 20-40% to that total to get the number of gigabytes of vram you need.

For example, say you want to run gpt-oss 20b (20 billion parameters) at its native MXFP4 (4 bit floating point) quantization on your local machine. In order to run it with a context window of 4096 tokens, you need about 16 gigabytes of vram (13 gigabytes of weights, 3 gigabytes of inference space), but 4096 tokens isn't very useful for many people. That covers about 4 pages of printed text (assuming one token is about 4 bytes on average).

When you get reasoning models that print a lot of tokens into the mix, it's easy for the reasoning phase alone of a single question to hit 4096 tokens (especially when approaches like simple test-time scaling are applied). I've found that 64k tokens gives a good balance for video memory use and usefulness as a chatbot. However, when you do that with gpt-oss 20b, it ends up using 32 gigabytes of vram. This only fits on my laptop because my laptop has 64 gigabytes of memory. The largest consumer GPU is the RTX 5090 and that only has 32 gigabytes of video memory. It's barely consumer and even "bad" models will barely fit.

Not to mention, industry consensus is that the "smallest good" models start out at 70-120 billion parameters. At a 64k token window, that easily gets into the 80+ gigabyte of video memory range, which is completely unsustainable for individuals to host themselves.

Who owns our digital assistants?

Even if AI assistants end up dying when the AI hype bubble pops, there's still some serious questions to consider about our digital assistants. People end up using them as an extension of their mind and expect the same level of absolute privacy and freedom that you would have if you use a notebook as an extension of your mind. Should they have that same level of privacy enshrined into law?

At some level the models and chats for free users that ChatGPT, DeepSeek, Gemini, and so many other apps are hosted at cost so that the research team can figure out what those models are being used for and adjust the development of future models accordingly. This is fairly standard practice across the industry and was the case before the rise of generative AI. This is why every app wants to send telemetry to the home base, it's so the team behind it can figure out what features are being used and where things fail to directly improve the product.

Generative AI allows you to mass scan over all of the conversations to get the gist of what's going on in there and then use that to help you figure out what topics are being discussed without breaching confidentiality or exposing employees to the contents of the chat threads. This can help you improve datasets and training runs to optimize on things like health information. I don't know how AI companies work on the inside, but I am almost certain that they do not perform model training runs on raw user data because of the risk of memorization causing them to the leak training data back to users.

Cadey is coffee
Cadey

Again, don't put private health information into ChatGPT. I get the temptation, but don't do it. I'm not trying to gatekeep healthcare, but we can't trust these models to count the number of b's in blueberry consistently. If we can't trust them to do something trivial like that, can we really trust them with life-critical conversations like what happens when you're in crisis or to accurately interpret a cancer screening?

Maybe we should be the ones self-hosting the AI models that we rely on. At least we should probably be using a setup that allows us to self host the models at all, so you can start out with a cloud hosted model while it's cheap and then move to a local hosting setup if the price gets hiked or the provider is going to shut that old model down. This at least gives you an escape hatch to be able to retain an assistant's "emotional warmth" even if the creator of that model shuts it down because they don't find it economically viable to host it anymore.

Reality is becoming more and more cyberpunk

Honestly this feels like the kind of shit I'd talk about in cyberpunk satire, but I don't feel like doing that anymore because it's too real now. This is the kind of thing that Neal Stephenson or Frank Herbert would have an absolute field day with. The whole Replika fiasco feels like the kind of thing that social commentary satire would find beyond the pale but yet you can find it by just refreshing CBC. Such as that one guy that gave himself bromism by taking ChatGPT output too literally, any of the stories about ChatGPT psychosis, or any of the stories involving using an AI model as a friend/partner.

Cadey is coffee
Cadey

I wasn't able to watch it before publishing this article, but I'm told that the Replika fiasco is almost a beat-for-beat match for the plot of Her (2013). Life imitates art indeed.

I don't think these events are a troubling sign or a warning, they are closer to a diagnosis. We are living in a world where people form real emotional bonds with bags of neural networks that cannot love back, and when the companies behind those neural networks change things, people get emotionally devastated. We aren't just debating the ideas of creating and nurturing relationships with digital minds, we're seeing the side effects of that happening in practice.

A lot of this sounds like philosophical science fiction, but as of December 2022 it's science fact. This fight for control of tools that we rely on as extensions of our minds isn't some kind of far-off science fiction plot, it's a reality we have to deal with. If we don't have sovereignty and control over the tools that we rely on the most, we are fundamentally reliant on the mercy of our corporate overlords simply choosing to not break our workflows.

Are we going to let those digital assistants be rented from our corporate overlords?

TI-20250709-0001: IPv4 traffic failures for Techaro services

2025-07-09 08:00:00

Techaro services were down for IPv4 traffic on July 9th, 2025. This blogpost is a report of what happened, what actions were taken to resolve the situation, and what actions are being done in the near future to prevent this problem. Enjoy this incident report!

Numa is smug
Numa

In other companies, this kind of documentation would be kept internal. At Techaro, we believe that you deserve radical candor and the truth. As such, we are proving our lofty words with actions by publishing details about how things go wrong publicly.

Everything past this point follows my standard incident root cause meeting template.

This incident report will focus on the services affected, timeline of what happened at which stage of the incident, where we got lucky, the root cause analysis, and what action items are being planned or taken to prevent this from happening in the future.

Timeline

All events take place on July 9th, 2025.

Time (UTC) Description
12:32 Uptime Kuma reports that another unrelated website on the same cluster was timing out.
12:33 Uptime Kuma reports that Thoth's production endpoint is failing gRPC health checks.
12:35 Investigation begins, announcement made on Xe's Bluesky due to the impact including their personal blog.
12:39 nginx-ingress logs on the production cluster show IPv6 traffic but an abrupt cutoff in IPv4 traffic around 12:32 UTC. Ticket is opened with the hosting provider.
12:41 IPv4 traffic resumes long enough for Uptime Kuma to report uptime, but then immediately fails again.
12:46 IPv4 traffic resumes long enough for Uptime Kuma to report uptime, but then immediately fails again. (repeat instances of this have been scrubbed, but it happened about every 5-10 minutes)
12:48 First reply from the hosting provider.
12:57 Reply to hosting provider, ask to reboot the load balancer.
13:00 Incident responder because busy due to a meeting under the belief that the downtime was out of their control and that uptime monitoring software would let them know if it came back up.
13:20 Incident responder ended meeting and went back to monitoring downtime and preparing this document.
13:34 IPv4 traffic starts to show up in the ingress-nginx logs.
13:35 All services start to report healthy. Incident status changes to monitoring.
13:48 Incident closed.
14:07 Incident re-opened. Issues seem to be manifesting as BGP issues in the upstream provider.
14:10 IPv4 traffic resumes and then stops.
14:18 IPv4 traffic resumes again. Incident status changes to monitoring.
14:40 Incident closed.

Services affected

Service name User impact
Anubis Docs (IPv4) Connection timeout
Anubis Docs (IPv6) None
Thoth (IPv4) Connection timeout
Thoth (IPv6) None
Other websites colocated on the same cluster (IPv4) Connection timeout
Other websites colocated on the same cluster (IPv6) None

Root cause analysis

In simplify server management, Techaro runs a Kubernetes cluster on Vultr VKE (Vultr Kubernetes Engine). When you do this, Vultr needs to provision a load balancer to bridge the gap between the outside world and the Kubernetes world, kinda like this:

---
        title: Overall architecture
        ---
        
        flowchart LR
            UT(User Traffic)
            subgraph Provider Infrastructure
              LB[Load Balancer]
            end
            subgraph Kubernetes
                IN(ingress-nginx)
                TH(Thoth)
                AN(Anubis Docs)
                OS(Other sites)
        
                IN --> TH
                IN --> AN
                IN --> OS
            end
        
            UT --> LB --> IN
        

Techaro controls everything inside the Kubernetes side of that diagram. Anything else is out of our control. That load balancer is routed to the public internet via Border Gateway Protocol (BGP).

If there is an interruption with the BGP sessions in the upstream provider, this can manifest as things either not working or inconsistently working. This is made more difficult by the fact that the IPv4 and IPv6 internets are technically separate networks. With this in mind, it's very possible to have IPv4 traffic fail but not IPv6 traffic.

The root cause is that the hosting provider we use for production services had flapping IPv4 BGP sessions in its Toronto region. When this happens all we can do is open a ticket and wait for it to come back up.

Where we got lucky

The Uptime Kuma instance that caught this incident runs on an IPv4-only network. If it was dual stack, this would not have been caught as quickly.

The ingress-nginx logs print IP addresses of remote clients to the log feed. If this was not the case, it would be much more difficult to find this error.

Action items

  • A single instance of downtime like this is not enough reason to move providers. Moving providers because of this is thus out of scope.
  • Techaro needs a status page hosted on a different cloud provider than is used for the production cluster (TecharoHQ/TODO#6).
  • Health checks for IPv4 and IPv6 traffic need to be created (TecharoHQ/TODO#7).
  • Remove the requirement for Anubis to pass Thoth health checks before it can start if Thoth is enabled.

Experimenting with Development containers

2025-06-30 08:00:00

A few years ago I was introduced to the idea of Development containers by a former coworker. I was deep into the Nix koolaid at the time, so I thought they were kinda superfluous and ultimately not worth looking into. After having run a fairly popular open source project for a while, I've come to realize that setting up a development environment for it is actually a fair bit harder than it seems. I want to make it easy to contribute to the project, and one of the best ways I can do that is by lowering the skill floor for contribution.

As such, I'm starting to experiment with development containers across my projects. I wrote this article from inside a development container on my Macbook. If you want to play around with my development environment Techaro's package builder yeet, you can clone its repo from GitHub and activate the development container. You will get a known working configuration that you can use to build new and exciting things.

Notably, these development containers also allow you to use GitHub Codespaces to contribute. This means you don't even need to have a machine that's able to run Linux containers. You can contribute from any machine that can run GitHub Codespaces.

This is still an experiment, and here are the criteria I'm using to determine if this will be a success or not:

  1. Can people that don't really understand much about the stack behind projects clone a repo and get the software to build or run?
  2. Does this help lower the skill floor to make it easier to contribute to those projects?
  3. Will this finally get Anubis' integration tests to run consistently across OSes?

The main reason I was inspired to try this out was after I heard a YouTuber describe what AI assisted code editing felt like for new developers: it feels like being a senior developer where you just have things flow out of your hands and you're able to make new and exciting things. I think the Techaro way of giving people that kind of experience to someone would be letting you get the development environment of a senior developer, akin to what it feels like to use an expert mechanic's garage to fix your car. When you clone the repos I'm testing with, you get a version of the configuration that I use, modulo the parts that don't make the most sense for running inside containers.

I'm super excited to see how this turns out. Maybe it'll be a good thing, maybe it won't. Only one way to know for sure!

Rolling the ladder up behind us

2025-06-20 08:00:00

Cloth is one of the most important goods a society can produce. Clothing is instrumental for culture, expression, and for protecting one's modesty. Historically, cloth was one of the most expensive items on the market. People bought one or two outfits at most and then wore them repeatedly for the rest of their lives. Clothing was treasured and passed down between generations the same way we pass jewelry down between generations. This cloth was made in factories by highly skilled weavers. These weavers had done the equivalent of PhD studies in weaving cloth and used state of the art hardware to do it.

As factories started to emerge, they were able to make cloth so much more cheaply than skilled weavers ever could thanks to inventions like the power loom. Power looms didn't require skilled workers operating them. You could even staff them with war orphans, which there was an abundance of thanks to all the wars. The quality of the cloth was absolutely terrible in comparison, but there was so much more of it made so much more quickly. This allowed the price of cloth to plummet, meaning that the wages that the artisans made fell from six shillings a day to six shillings per week over a period of time where the price of food doubled.

Mind you, the weavers didn't just reject technological progress for the sake of rejecting it. They tried to work with the ownership class and their power looms in order to produce the same cloth faster and cheaper than they had before. For a time, it did work out, but the powers that be didn't want that. They wanted more money at any cost.

At some point, someone had enough and decided to do something about it. Taking up the name Ned, he led a movement that resulted in riots, destroying factory equipment, and some got so bad they had to call the army in to break them up. Townspeople local to those factory towns were in full support of Ned's followers. Heck, even the soldiers sent to stop the riots ended up seeing the points behind what Ned's followers were doing and joined in themselves.

The ownership class destroyed the livelihood of the skilled workers so that they could make untold sums of money producing terrible cloth that people would turn their one-time purchase of clothing into a de-facto subscription that they had to renew every time their clothing wore out. Now we have fast fashion and don't expect our clothing to last more than a few years. I have a hoodie from AWS Re:Invent in 2022 that I'm going to have to throw out and replace because the sleeves are dying.

We only remember them as riots because their actions affected those in power. This movement was known as the Luddites, or the followers of Ned Ludd. The word "luddite" has since shifted meaning over time and is now understood as "someone who is against technological development". The Luddites were not against technology like the propaganda from the ownership class would have you expect, they fought against how it was implemented and the consequences of its rollout. They were skeptical that the shitty cloth that the power loom produced would be a net benefit to society because it meant that customers would inevitably have to buy their clothes over and over again, turning a one-time purchase into a subscription. Would that really benefit consumers or would that really benefit the owners of the factories?

Nowadays the Heritage Crafts Association of the United Kingdom lists many forms of weaving as Endangered or Critically Endangered crafts, meaning that those skills are either at critical risk of dying out without any "fresh blood" learning how to do it, or the last generation of artisans that know how to do that craft are no longer teaching new apprentices. All that remains of that expertise is now contained in the R&D departments of the companies that produce the next generations of power looms, and whatever heritage crafts practitioners remain.

Remember the Apollo program that let us travel to the moon? It was mostly powered by the Rocketdyne F1 engine. We have all of the technical specifications to build that rocket engine. We know all the parts you need, all the machining you have to do, and roughly understand how it would be done, but we can't build another Rocketdyne F1 because all of the finesse that had been built up around manufacturing it no longer exists. Society has moved on and we don't have expertise in the tools that they used to make it happen.

What are we losing in the process? We won't know until it's gone.

We're going to run out of people with the word "Senior" in their title

As I've worked through my career in computering, I've noticed a paradox that's made me uneasy and I haven't really been able to figure out why it keeps showing up: the industry only ever seems to want to hire people with the word Senior in their title. They almost never want to create people with the word Senior in their title. This is kinda concerning for me. People get old and no longer want to or are able to work. People get sick and become disabled. Accidental deaths happen and remove people from the workforce.

A meme based on the format where the dog wants to fetch the ball but doesn't want to give the ball to the human to throw it, but with the text saying 'Senior?', 'Train Junior?', and 'No train junior, only hire senior'.
A meme based on the format where the dog wants to fetch the ball but doesn't want to give the ball to the human to throw it, but with the text saying 'Senior?', 'Train Junior?', and 'No train junior, only hire senior'.

If the industry at large isn't actively creating more people with the word Senior in their title, we are eventually going to run out of them. This is something that I want to address with Techaro at some point, but I'm not sure how to do that yet. I'll figure it out eventually. The non-conspiratorial angle for why this is happening is that money isn't free anymore and R&D salaries are no longer taxable business expenses in the US, so software jobs that don't "produce significant value" are more risky to the company. So of course they'd steal from the future to save today. Sounds familiar, doesn't it?

Cadey is coffee
Cadey

Is this how we end up losing the craft of making high quality code the same way we lost the craft of weaving high quality cloth?

However there's another big trend in the industry that concerns me: companies releasing products that replace expertise with generative AI agents that just inscrutably do the thing for you. This started out innocently enough - it was just better ways to fill in the blanks in your code. But this has ballooned and developed from better autocomplete to the point where you can just assign issues to GitHub Copilot and have the issue magically get solved for you in a pull request. Ask the AI model for an essay and get a passable result in 15 minutes.

At some level, this is really cool. Like, think about it. This reduces toil and drudgery to waiting for half an hour at most. In a better world I would really enjoy having a tool like this to help deal with the toil work that I need to do but don't really have the energy to. Do you know how many more of these essays would get finished if I could offload some of the drudgery of my writing process to a machine?

We are not in such a better world. We are in a world where I get transphobic hate sent to the Techaro sales email. We are in a world where people like me are intentionally not making a lot of noise so that we can slide under the radar and avoid attention by those that would seek to destroy us. We are in a world where these AI tools are being pitched as the next Industrial Revolution, one where foisting our expertise away into language models is somehow being framed as a good thing for society.

There's just one small problem: who is going to be paid and reap the benefits from this change as expectations from the ownership class change? A lot of the ownership class only really experiences the work product outputs of what we do with computers. They don't know the struggles involved with designing things such as the user getting an email on their birthday. They don't want to get pushback on things being difficult or to hear that people want to improve the quality of the code. They want their sparkle emoji buttons to magically make the line go up and they want them yesterday.

We deserve products that aren't cheaply made mass produced slop that incidentally does what people want instead of high quality products that are crafted to be exactly what people need, even if they don't know they need it.

Additionally, if this is such a transformational technology, why are key figures promoting it by talking down to people? Why wouldn't they be using this to lift people up?

Aoi is wut
Aoi

Isn't that marketing? Fear sells a lot better than hope ever will. Amygdala responses are pretty strong right? So aren't a lot of your fears of the technology really feeding into the hype and promoting the technology by accident?

Cadey is coffee
Cadey

I don't fear the power loom. I fear the profit expectations of the factory owners.

Vibe coding is payday loans for technical debt

As a technical educator, one of the things that I want to imprint onto people is that programming is a skill you can gain and that you too can both program things and learn how to program things. I want there to be more programmers out there. What I am about to say is not an attempt to gatekeep the skill and craft of computering; however, the ways that proponents of vibe coding are going about it are simply not the way forward to a sustainable future.

About a year ago, Cognition teased an AI product named Devin, a completely automated software engineer. You'd assign Devin tasks in Slack or Jira and then it would spin up a VM and plod its way through fixing whatever you asked it to. This demo deeply terrified me, as it was nearly identical to a story I wrote for the Techaro lore: Protos. The original source of that satire was experience working at a larger company that shall remain unnamed where the product team seemed to operate under the assumption that the development team had a secret "just implement that feature button" and that we as developers were working to go out of our way to NOT push it.

Devin was that "implement that feature" button the same way Protos mythically did. From what I've seen with companies that actually use Devin, it's nowhere near actually being useful and usually needs a lot of hand-holding to do anything remotely complicated, thank God.

The thing that really makes me worried is that the ownership class' expectations about the process of developing software are changing. People are being put on PIPs for not wanting to install Copilot. Deadlines come faster because "the AI can write the code for you, right?" Twitter and Reddit contain myriads of stories of "idea guys" using Cursor or Windscribe to generate their dream app's backend and then making posts like "some users claim they can see other people's stuff, what kind of developer do I need to hire for this?" Follow-up posts include gems such as "lol why do coders charge so much???"

By saving money in the short term by producing shitty software that doesn't last, are we actually spending more money over time re-buying nearly identical software after it evaporates from light use? This is the kind of thing that makes Canada not allow us to self-identify as Engineers, and I can't agree with their point more.

Vibe Coding is just fancy UX

Vibe coding is a distraction. It's a meme. It will come. It will go. Everyone will abandon the vibe coding tools eventually. My guess is that a lot of the startups propping up their vibe coding tools are trying to get people into monthly subscriptions as soon as possible so that they can mine passive income as their more casual users slowly give up on coding and just forget about the subscription.

I'm not gonna lie though, the UX of vibe coding tools is top-notch. From a design standpoint it's aiming for that subtle brilliance where it seems to read your mind and then fill in the blanks you didn't even know you needed filled in. This is a huge part of how you can avoid the terror of the empty canvas. If you know what you are doing, an empty canvas represents infinite possibilities. There's nothing there to limit you from being able to do it. You have total power to shape everything.

In my opinion, this is a really effective tool to help you get past that fear of having no ground to stand on. This helps you get past executive dysfunction and just ship things already. That part is a good thing. I genuinely want people to create more things with technology that are focused on the problems that they have. This is the core of how you learn to do new things. You solve small problems that can be applied to bigger circumstances. You gradually increase the scope of the problem as you solve individual parts of it.

I want more people to be able to do software development. I think that it's a travesty that we don't have basic computer literacy classes in every stage of education so that people know how the machines that control their lives work and how to use them to their advantage. Sure it's not as dopaminergic as TikTok or other social media apps, but there's a unique sense of victory that you get when things just work. Sometimes that feeling you get when things Just Work™ is the main thing that keeps me going. Especially in anno dominium two thousand and twenty five.

The main thing I'm afraid of is people becoming addicted to the vibe coding tools and letting their innate programming skills atrophy. I don't know how to suggest people combat this. I've been combating it by removing all of the automatic AI assistance from my editor (IE: I'll use a language server, but I won't have my editor do fill-in-the-middle autocomplete for me), but this isn't something that works for everyone. I've found myself more productive without it there and asking a model for the missing square peg to round hole when I inevitably need some toil code made. I ended up not shipping that due to other requirements, but you get what I'm going at.

The "S" in MCP stands for Security

The biggest arguments I have against vibe coding and all of the tools behind it boil down to one major point: these tools have a security foundation of sand. Most of the time when you install and configure a Model Context Protocol (MCP) server, you add some information to a JSON file that your editor uses to know what tools it can dispatch with all of your configuration and API tokens. These MCP servers run as normal OS processes with absolutely no limit to what they can do. They can easily delete all files on your system, install malware into your autostart, or exfiltrate all your secrets without any oversight.

Oh, by the way, that whole "it's all in one JSON file with all your secrets" problem? That's now seen as a load-bearing feature so that scripts can automatically install MCP servers for you. You don't even need to get expertise in how the tools work! There's a MCP server installer MCP server so that you can say "Hey torment nexus, install GitHub integration for me please" and then it'll just do it with no human oversight or review on what you're actually installing. Seems safe to me! What could possibly go wrong?

If this is seriously the future of our industry, I wish that the people involved would take one trillionth of an iota of care about the security of the implementation. This is the poster child for something like the WebAssembly Component Model. This would let you define your MCP servers with strongly typed interfaces to the outside world that can be granted or denied permissions by users with strong capabilities. Combined with the concept of server resources, this could let you expand functionality however you wanted. Running in WebAssembly means that the no MCP server can just read ~/.ssh/id_ed25519 and exfiltrate your SSH key. Running in WebAssembly means that it can't just connect to probably-not-malware.lol and then evaluate JavaScript code with user-level permissions on the fly. We shouldn't have to be telling developers "oh just run it all in Docker". We should have designed this to be fundamentally secure from the get-go. Personally, I only run MCP ecosystem things when contractually required to. Even then, I run it in a virtual machine that I've already marked as known compromised and use separate credentials not tied to me. Do with this information as you will.

I had a lot of respect for Anthropic before they released this feculent bile that is the Model Context Protocol spec and initial implementations to the public. It just feels so half-baked and barely functional. Sure I don't think they expected it to become the Next Big Meme™, but I thought they were trying to do things ethically above board. Everything I had seen from Anthropic before had such a high level of craft and quality, and this was such a huge standout.

We shouldn't have to be placing fundamental concerns like secret management or sandboxing as hand-waves to be done opt-in by the user. They're not gonna do it, and we're going to have more incidents where Cursor goes rogue and nukes your home folder until someone cares enough about the craft of the industry to do it the right way.

Everyone suffers so the few can gain

I have a unique view into a lot of the impact that AI companies have had across society. I'm the CEO of Techaro, a small one-person startup that develops Anubis, a Web AI Firewall Utility that helps mitigate the load of automated mass scraping so that open source infrastructure can stay online. I've had sales calls with libraries and universities that are just being swamped by the load. There's stories of GitLab servers eating up 64 cores of high-wattage server hardware due to all of the repeated scraping over and over in a loop. I swear a lot of this scraping has to be some kind of dataset arbitrage or something, that's the only thing that makes sense at this point.

And then in the news the AI companies claim "oh no we're just poor little victorian era orphans, we can't possibly afford to fairly compensate the people that made the things that make our generative AI models as great as they are". When the US copyright office tried to make AI training not a fair use, the head of that office suddenly found themselves jobless. Why must these companies be allowed to take everything without recourse or payment to the people that created the works that fundamentally power the models?

The actual answer to this is going to sound a bit out there, but stay with me: they believe that we're on the verge of creating artificial superintelligence; something that will be such a benevolent force of good that any strife in the short term will ultimately be cancelled out by the good that is created as a result. These people unironically believe that a machine god will arise and we'd be able to delegate all of our human problems to it and we'll all be fine forever. All under the thumb of the people that bought the GPUs with dollars to run that machine god.

As someone that grew up in a repressed environment full of evangelical christianity, I recognize this story instantly: it's the second coming of Christ wrapped in technology. Whenever I ask the true believers entirely sensible questions like "but if you can buy GPUs with dollars, doesn't that mean that whoever controls the artificial superintelligence thus controls everyone, even if the AI is fundamentally benevolent?" The responses I get are illuminating. They sound like the kinds of responses that evangelicals give when you question their faith.

Artists suffer first

Honestly though, the biggest impact I've seen across my friends has been what's happened to art commissions. I'm using these as an indicator for how the programming industry is going to trend. Software development is an art in the same vein as visual/creative arts, but a lot of the craft and process that goes into visual art is harder to notice because it gets presented as a flat single-dimensional medium.

Sometimes it can take days to get something right for a drawing. But most of the time people just see the results of the work, not the process that goes into it. This makes things like prompting "draw my Final Fantasy 14 character in Breath of the Wild" with images as references and getting a result in seconds look more impressive. If you commissioned a human to get a painting like this:

An AI-generated illustration of my Final Fantasy 14 character composited into a screenshot of Breath of the Wild. Generated by GPT-4o through the ChatGPT interface. Inputs were a screenshot of Breath of the Wild and reference photos of my character.
An AI-generated illustration of my Final Fantasy 14 character composited into a screenshot of Breath of the Wild. Generated by GPT-4o through the ChatGPT interface. Inputs were a screenshot of Breath of the Wild and reference photos of my character.

It'd probably take at least a week or two as the artist worked through their commission queue and sent you in-progress works before they got the final results. By my estimates between the artists I prefer commissioning, this would cost somewhere between 150 USD and 500 EUR at minimum. Probably more when you account for delays in the artistic process and making sure the artist is properly paid for their time. It'd be a masterpiece that I'd probably get printed and framed, but it would take a nonzero amount of time.

If you only really enjoy the products of work and don't understand/respect any of the craftsmanship that goes into making it happen, you'd probably be okay with that instantly generated result. Sure the sun position in that image doesn't make sense, the fingers have weird definition, her tail is the wrong shape, it pokes out of the dress in a nonsensical way (to be fair, the reference photos have that too), the dress has nonsensical shading, and the layering of the armor isn't like the reference pictures, but you got the result in a minute!

A friend of mine runs an image board for furry art. He thought that people would use generative AI tools as a part of their workflows to make better works of art faster. He was wrong, it just led to people flooding the site with the results of "wolf girl with absolutely massive milkers showing her feet paws" from their favourite image generation tool in every fur color imaginable, then with different characters, then with different anatomical features. There was no artistic direction or study there. Just an endless flood of slop that was passable at best.

Sure, you can make high quality art with generative AI. There's several comic series where things are incredibly temporally consistent because the artist trained their own models and took the time to genuinely gain expertise with the tools. They filter out the hallucination marks. They take the time to use it as a tool to accelerate their work instead of replacing their work. The boards they post it to go out of their way to excise the endless flood of slop and by controlling how the tools work they actually get a better result than they got by hand, much like how the skilled weavers were able to produce high quality cloth faster and cheaper with the power looms.

We are at the point where the artists want to go and destroy the generative image power looms. Sadly, they can't even though they desperately want to. These looms are locked in datacentres that are biometrically authenticated. All human interaction is done by a small set of trusted staff or done remotely by true believers.

I'm afraid of this kind of thing happening to the programming industry. A lot of what I'm seeing with vibe coding leading to short term gains at the cost of long term toil is lining up with this. Sure you get a decent result now, but long-term you have to go back and revise the work. This is a great deal if you are producing the software though; because that means you have turned one-time purchases into repeat customers as the shitty software you sold them inevitably breaks, forcing the customer to purchase fixes. The one-time purchase inevitably becomes a subscription.

We deserve more in our lives than good enough.

Stop it with the sparkle emoji buttons

Look, CEOs, I'm one of you so I get it. We've seen the data teams suck up billions for decades and this is the only time that they can look like they're making a huge return on the investment. Cut it out with shoving the sparkle emoji buttons in my face. If the AI-aided product flows are so good then the fact that they are using generative artificial intelligence should be irrelevant. You should be able to replace generative artificial intelligence with another technology and then the product will still be as great as it was before.

When I pick up my phone and try to contact someone I care about, I want to know that I am communicating with them and not a simulacrum of them. I can't have that same feeling anymore due to the fact that people that don't natively speak English are much more likely to filter things through ChatGPT to "sound professional".

I want your bad English. I want your bad art. I want to see the raw unfiltered expressions of humanity. I want to see your soul in action. I want to communicate with you, not a simulacrum that stochastically behaves like you would by accident.

And if I want to use an LLM, I'll use an LLM. Now go away with your sparkle emoji buttons and stop changing their CSS class names so that my uBlock filters keep working.

The human cost

This year has been a year full of despair and hurt for me and those close to me. I'm currently afraid to travel to the country I have citizenship in because the border police are run under a regime that is dead set on either elimination or legislating us out of existence. In this age of generative AI, I just feel so replaceable at my dayjob. My main work product is writing text that convinces people to use globally distributed object storage in a market where people don't realize that's something they actually need. Sure, this means that my path forward is simple: show them what they're missing out on. But I am just so tired. I hate this feeling of utter replaceability because you can get 80% as good of a result that I can produce with a single invocation of OpenAI's Deep Research.

Recently a decree came from above: our docs and blogposts need to be optimized for AI models as well as humans. I have domain expertise in generative AI, I know exactly how to write SEO tables and other things that the AI models can hook into seamlessly. The language that you have to use for that is nearly identical to what the cult leader used that one time I was roped into a cult. Is that really the future of marketing? Cult programming? I don't want this to be the case, but when you look out at everything out there, you can't help but see the signs.

Aspirationally, I write for humans. Mostly I write for the version of myself that was struggling a decade ago, unable to get or retain employment. I create things to create the environment where there are more like me, and I can't do that if I'm selling to soulless automatons instead of humans. If the artificial intelligence tools were…well…intelligent, they should be able to derive meaning from unaltered writing instead of me having to change how I write to make them hook better into it. If the biggest thing they're sold for is summarizing text and they can't even do that without author cooperation, what are we doing as a society?

Actually, what are we going to do when everyone that cares about the craft of software ages out, burns out, or escapes the industry because of the ownership class setting unrealistic expectations on people? Are the burnt out developers just going to stop teaching people the right ways to make software? Is society as a whole going to be right when they look back on the good old days and think that software used to be more reliable?

The Butlerians had a point

Frank Herbert's Dune world had superintelligent machines at one point. It led to a galactic war and humanity barely survived. As a result, all thinking machines were banned, humanity was set back technologically, and a rule was created: Thou shalt not make a machine in the likeness of a human mind. For a very long time, I thought this was very strange. After all, in a fantasy scifi world like Dune, thinking machines could automate so much toil that humans had to process. They had entire subspecies of humans that were functionally supercomputers with feelings that were used to calculate the impossibly complicated stellar draft equations so that faster-than-light travel didn't result in the ship zipping into a black hole, star, moon, asteroid, or planet.

After seeing a lot of the impact across humanity in later 2024 and into 2025, I completely understand the point that Frank Herbert had. It makes me wish that I could leave this industry, but this is the only thing that pays enough for me to afford life in a world where my husband gets casually laid off after being at the same company for six and a half years because some number in a spreadsheet put him on the shitlist. Food and rent keeps going up here, but wages don't. I'm incredibly privileged to be able to work in this industry as it is (I make enough to survive, don't worry), but I'm afraid that we're rolling the ladder up behind us so that future generations won't be able to get off the ground.

Maybe the problem isn't the AI tools, but the way they are deployed, who benefits from them, and what those benefits really are. Maybe the problem isn't the rampant scraping, but the culture of taking without giving anything back that ends up with groups providing critical infrastructure like FFmpeg, GNOME, Gitea, FreeBSD, NetBSD, and the United Nations having to resort to increasingly desperate measures to maintain uptime.

Maybe the problem really is winner-take-all capitalism.


The deployment of generative artificial intelligence tools has been a disaster for the human race. They have allowed a select few to gain "higher productivity"; but they have destabilized society, have made work transactional, have subjected artists to indignities, have led to widespread psychological suffering for the hackers that build the tools AI companies rely on, and inflict severe damage on the natural world. The continued development of this technology will worsen this situation. It will certainly subject human beings to greater indignities and inflict greater damage on the natural world, it will probably lead to greater social disruption and psychological suffering, and it may lead to increased physical suffering even in "advanced" countries.


For other works in a similar vein, read these:

Special thanks to the following people that read and reviewed this before release:

  • Ti Zhang
  • Annie Sexton
  • Open Skies
  • Nina Vyedin
  • Eric Chlebek
  • Ahroozle REDACTED
  • Kronkleberry
  • CELPHASE

I fight bots in my free time

2025-06-15 08:00:00

This was a lightning talk I did at BSDCan. It was a great conference and I'll be sure to be there next year!

Want to watch this in your video player of choice? Take this:
https://files.xeiaso.net/talks/2025/bsdcan-anubis/index.m3u8
The title slide with the talk and speaker name.
The title slide with the talk and speaker name.

Hi, I'm Xe, and I fight bots in my free time. I'd love to do it full time, but that's not financially in the cards yet. I made Anubis. Anubis is a web AI firewall utility that stops the bots from taking out your website. It's basically the Cloudflare "Are you a bot?" page, but self-hostable.

A captcha component.
A captcha component.

And without this. Scrapers have CAPTCHA solvers built in. These CAPTCHA solvers are effectively APIs that just have underpaid third world humans in the loop, and it's just kind of bad and horrible.

A captcha component.
A captcha component.

So Anubis is an uncaptcha. It uses features of your browser to automate a lot of the work that a CAPTCHA would, and right now the main implementation is by having it run a bunch of cryptographic math with JavaScript to prove that you can run JavaScript in a way that can be validated on the server. I'm working on obviating that because surprisingly many people get very angry about having to run JavaScript, but it's within the cards.

A captcha component.
A captcha component.

Anubis is open source software written in Go. It's on GitHub. It's got like eight kilostars. It works on any stack that lets you run more than one program. We have examples for Nginx, Caddy, Apache, and Kubernetes.

A slide showing the Repology version history graph for Anubis.
A slide showing the Repology version history graph for Anubis.

It's in your package repos. If you do ports for FreeBSD or pkgsrc for NetBSD, please bump the version. I'm about to release a new one, but please bump the current version.

Why does Anubis exist?
Why does Anubis exist?

So you might be wondering, what's the story? Why does Anubis exist?

The Amazon logo using a flamethrower to burninate my Gitea server.
The Amazon logo using a flamethrower to burninate my Gitea server.

Well, this happened. I have a Git server for my own private evil plans, and Amazon's crawler discovered it through TLS certificate transparency logs and decided to unleash the hammer of God. And that happened. They had the flamethrower of requests just burning down my poor server, and it was really annoying because I was trying to do something and it just didn't work. Also helps if you don't schedule your storage on rotational drives.

A slide showing a hilarious number of logos of organizations that deploy Anubis.
A slide showing a hilarious number of logos of organizations that deploy Anubis.

But I published it on GitHub, and like four months later, look at all these logos. There's more logos that I forgot to put on here and will be in the version on my website. But like, yeah, it's used by FreeBSD, NetBSD, Haiku, GNOME, FFmpeg, and the United Nations Educational, Scientific, and Cultural Organization. Honestly, seeing UNESCO just through a random DuckDuckGo search made me think, huh, maybe this is an actual problem. And like any good problem, it's a hard problem.

A screenshot of Pale Moon passing the bot detection check.
A screenshot of Pale Moon passing the bot detection check.

How do you tell if any request is coming from a browser?

This screenshot right here uses Pale Moon, which is a known problem child in terms of bot detection services and something that I actively do test against to make sure that it works. But how do you know if any given request is coming from a browser?

It’s very hard, and I have been trying to find ways to do it better. The problem is, in order to know what good browsers look like, you have to know what bad scrapers look like. And the great news is that scrapers look like browsers, asterisk. So you have to find other ways, like behaviors or third-party or like third-order side effects. It’s a huge pain.

A list of fingerprinting methods that I've been trying including JA4, JA3N, JA4H, HTTP/2 fingerprinting, THR1, and if the client executes JS.
A list of fingerprinting methods that I've been trying including JA4, JA3N, JA4H, HTTP/2 fingerprinting, THR1, and if the client executes JS.

So as a result, I'm trying a bunch of fingerprinting methods. These are a lot of the fingerprints that I've listed here, like JA4, JA3N are all based on the TLS information that you send to every website, whether you want to or not, because that's how security works. I'm trying to do stuff based on HTTP requests or the HTTP2 packets that you send to the server, which you have to do in order for things to work. And I'm falling back to, can you run JavaScript, lol?

A list of things I want to try in the future.
A list of things I want to try in the future.

So in terms of things I want to do next, obviously, I want to do better testing on BSD. Right now my testing is: does it compile? And because I've written it in Go without Cgo, that answer is yes. I want to build binary packages for BSDs, because even though I think it's better suited by downstream ports and stuff, I still want to have those packages as an option.

I want to do a hosted option like Cloudflare, because some people just don't want to run Anubis but want to run Anubis. I want to do system load-based thresholds, so it only kicks in as it is aggressive when things are actively on fire. I want to have better NoJS support, which will include every way to tell something as a browser without JavaScript in ways that make you read all of the specs and start having an existential breakdown. I want to do stuff with WebAssembly on the server, because I've always wanted to see how that would blow up in prod. I want to do an IP reputation database, Kubernetes stuff, end-to-end testing doesn't suck.

And finally, there's one of the contributors that I really want to hire, but I can't afford to yet, so I'd love to when I can.

If you want to sabotage Anubis, make sure Final Fantasy 14 stays up.
If you want to sabotage Anubis, make sure Final Fantasy 14 stays up.

Also, if you work at an AI company, I know AI companies follow me. If you are working at an AI company, here's how you can sabotage Anubis development as easily and quickly as possible. So first is quit your job, second is work for Square Enix, and third is make absolute banger stuff for Final Fantasy XIV. That’s how you can sabotage this the best.

Xe's social media contact information.
Xe's social media contact information.

Anyways, I've been Xe, I have stickers, I'll be in the back, and thank you for having me here. And if you have any questions, please feel free to ask.

Q&A

Well, as the con chair, I think about people making comments instead of questions. I'm going to abuse my position and make a comment. You saved my butt, thank you.

You're welcome. I'm so happy that it's worked out. It’s a surreal honor to—let me get back to the logo slide, because this is nuts.

A slide showing a hilarious number of logos of organizations that deploy Anubis.
A slide showing a hilarious number of logos of organizations that deploy Anubis.

Let’s just look at this. That’s gnome, that's wine, that's dolphin, that's the Linux kernel, that's ScummVM, that's FreeCAD, and UNESCO on the same slide. What other timeline could we have?

This 2025 has been wild.

So how are your feelings? Because you’re basically trying to solve not a technical problem, but actually it’s more of a problem of society. Do you think it is winnable that way, or do we have to fight this problem in another way and make people, well, smarter is probably the wrong word.

I am not sure what the end game is for this. I started out developing it for, I want my Git server to stay up. Then gnome started using it. And then it became a thing. I put it under the GitHub org of a satirical startup that I made up for satire about the tech industry. And now that has a market in education.

I want to make this into a web application firewall that can potentially survive the AI bubble bursting. Because right now the AI bubble bursting is the biggest threat to the business, as it were. So a lot of it is figuring out how to pivot and do that. I've also made a build tool called Yeet that uses JavaScript to build RPM packages. Yes, there is a world where that does make sense. It's a lot of complicated problems. And there are a lot of social problems.

But if you’re writing a scraper, don't. Like seriously, there is enough scraping traffic already. Use Common Crawl. It exists for a reason.