2025-09-09 08:00:00
This post and its online comment sections are blame-free zones. We are not blaming anyone for clicking on the phishing link. If you were targeted with such a phishing attack, you'd fall for it too and it's a matter of when not if. Anyone who claims they wouldn't is wrong.
This is also a bit of a rant.
Yesterday one of the biggest package ecosystems had very popular packages get compromised. We're talking functionality like:
These kinds of dependencies are everywhere and nobody would even think that they could be harmful. Getting code into these packages means that it's almost guaranteed a free path to production deployments. If an open proxy server (a-la Bright Data or other botnets that the credit card network tolerates for some reason), API key stealer, or worse was sent through this chain of extreme luck on the attacker's part, then this would be a completely different story.
We all dodged a massive bullet because all the malware did was modify the destination addresses of cryptocurrency payments mediated via online wallets like MetaMask.
As someone adjacent to the online security community, I have a sick sense of appreciation for this attack. This was a really good attack. It started with a phishing email that I'd probably fall for if it struck at the right time:
This is frankly a really good phishing email. Breaking it down:
This is a 10/10 phishing email. Looking at it critically the only part about it that stands out is the domain "npmjs.help" instead of "npmjs.com". Even then, that wouldn't really stand out to me because I've seen companies use new generic top level domains to separate out things like the blog at .blog or the docs at .guide, not to mention the .new stack.
One of my friends qdot also got the phishing email and here's what he had to say:
I got the email for it and was like "oh I'll deal with this later".
Saved by procrastination!
— qdot ( @buttplug.engineer ) September 8, 2025 at 2:04 PM
With how widely used these libraries are, this could have been so much worse than it was. I can easily imagine a timeline where this wasn't just a cryptocurrency interceptor. Imagine if something this widely deployed into an ecosystem where automated package bumping triggering production releases is common did API key theft. You'd probably have more OpenAI API keys than you know what you'd do with. You could probably go for years without having to pay for AWS again.
It is just maddening to me that a near Jia Tan level chain of malware and phishing was wasted on cryptocurrency interception that won't even run in the majority of places those compromised libraries were actually used. When I was bumping packages around these issues, I found that most of these libraries were used in command line tools.
This was an attack obviously targeted towards the Web 3 ecosystem as users of Web 3 tools are used to making payments with their browsers. With my black hat on, I think that the reason they targeted more generic packages instead of Web 3 packages was so that the compromise wouldn't be as noticed by the Web 3 ecosystem. Sure, you'd validate the rigging that helps you interface with Metamask, but you'd never think that it would get monkey-patched by your color value parsing library.
One of the important things to take away from this is that every dependency could be malicious. We should take the time to understand the entire dependency tree of our programs, but we aren't given that time. At the end of the day, we still have to ship things.
2025-08-24 08:00:00
Earlier this year, I was finally sucked into Final Fantasy 14. I've been loving my time in it, but most of my playtime was on my gaming tower running Fedora. I knew that the game does support macOS, and I did get it working on my MacBook for travel, but there was one problem: I wasn't able to get my bars working with mouse and keyboard.
Final Fantasy 14 has a ridiculous level of customization. Every UI element can be moved and resized freely. Every action your player character can take is either bindable to arbitrary keybinds or able to be put in hotbars. Here's my hotbars for White Mage:
My bars have three "layers" to them:
I have things optimized so that the most common actions I need to do are on the base layer. This includes spells like my single target / area of effect healing spells and my burst / damage over time spells. However, critical things like health regeneration, panic button burst healing, shields, and status dispelling are all in the shift and control layers. When I don't have instinctive access to these spells with button combos, I have to manually click on the buttons. This sucks.
I ended up fixing this by installing Karabiner Elements, giving it access to the accessibility settings it needs, and enabling my mouse to be treated as a keyboard in its configuration UI.
There's some other keyboard hacks that I needed to do. My little split keyboard runs QMK, custom keyboard firmware written in C that has a stupid number of features. In order to get this layout working with FFXIV, I had to use a combination of the following features:
Here is what my keymap looks like:
/* Keymap 0: Alpha layer / Colemak DHm
*
* ,-------------------------------. ,-------------------------------.
* | ALT Q | W | F | P | B | | J | L | U | Y | ; |
* |-------+-----+-----+-----+-----| |-----+-----+-----+-----+-------|
* | CTRL A| R | S | T | G | | M | N | E | I |CTRL O |
* |-------+-----+-----+-----+-----| |-----+-----+-----+-----+-------|
* | SHFT Z| X | C | D | V | | K | H | < | > |SHFT / |
* `-------------------------------' `-------------------------------'
* .------------------------------. .--------------------------.
* | ESC META | SPC ALT | BSP SPE | | SPC NUM | SHFT ENT | : |
* '------------------------------' '--------------------------'
*/
I use the combination of this to also do programming. I've been doing a few full blown Anubis features via this keyboard such as log filters. I'm still not up to full programming speed with it, but I'm slowly internalizing the keymap and getting faster with practice.
Either way, Final Fantasy 14 is my comfort game and now I can play it on the go with all the buttons I could ever need. I hope this was interesting and I'm going to be publishing more of these little "how I did a thing" posts like this in the future. Let me know what you think about this!
2025-08-22 08:00:00
Today my quotes about generative AI scrapers got published in The Register. For transparency's sake, here's a copy of the questions I was asked and my raw, unedited responses. Enjoy!
First, do you see the growth in crawler traffic slowing any time soon?
I can only see a few things that can stop this: government regulation, or the hype finally starting to die down. There is too much hype in the mix that causes us to funnel billions of dollars into this technology instead of curing cancer, solving world hunger, or making people’s lives genuinely better.
Is it likely to continue growing?
I see no reason why it would not grow. People are using these tools to replace knowledge and gaining skills instead of augmenting knowledge and augmenting skills. Even if they are intended to be used for letting us focus on the fun parts of our work and automating away the chores, there are some bad apples that are spoiling the bunch and making this technology about replacing people, not drudgery and toil. This technology was obviously meant well, but at some level the output of AI superficially resembles the finished work product of human labour, superficially. As someone asked to Charles Babbage: if you put in the wrong numbers, you get the wrong answer.
This isn’t necessarily a bubble popping, this is a limitation of how well AI can function without direct and constant human input. Even so, we’ll hit the limit on data that can be scraped that hasn’t been touched by AI before the venture capital runs out. I see no value in the need for scrapers to hit the same 15 year old commit of the Linux kernel over and over and over every 30 minutes like they are now. There are ways to do this ethically that don’t penalize open source infrastructure such as using the Common Crawl dataset.
If so, how can that be sustainable?
It's not lol. We are destroying the commons in order to get hypothetical gains. The last big AI breakthrough happened with GPT-4 in 2023. The rest has been incremental improvements in tokenization, multimodal inputs (also tokenization), tool calling (also tokenization), and fill-in-the-middle completion (again, also tokenization). Even with scrapers burning everything in their wake, there is not enough training data to create another exponential breakthrough. All we can do now is make it more efficient to run GPT-4 level models on lesser hardware. I can (and regularly do) run a model just as good as GPT-4 on my MacBook at this point, which is really cool.
Would broader deployment of Anubis and other active countermeasures help?
This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these unethical scrapers that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming. Ironically enough, most of these unethical scraping activities rely on the products of the communities they are destroying. This presents the kind of paradox that I would expect to read in a Neal Stephenson book from the '90s, not CBC's front page.
Anubis helps mitigate a lot of the badness by making attacks more computationally expensive. Anubis (even in configurations that omit proof of work) makes attackers have to retool their scraping to use headless browsers instead of blindly scraping HTML. This increases the infrastructure costs of the scrapers propagating this abusive traffic. The hope is that this makes it fiscally unviable for the unethical scrapers to scrape by making them have to dedicate much more hardware to the problem.
In essence: it makes the scrapers have to spend more money to do the same work.
Is regulation required to prevent abuse of the open web?
Yes, but this regulation would have to be global, simultaneous, and permanent to have any chance of this actually having a positive impact. Our society cannot currently regulate against similar existential threats like climate change. I have no hope for such regulation to be made regarding generative AI.
Fastly's claims that 80% of bot traffic is now AI crawlers
In some cases for open source projects, we've seen upwards of 95% of traffic being AI crawlers. Not just bot traffic, but traffic in general. For one, deploying Anubis almost instantly caused server load to crater by so much that it made them think they accidentally took their site offline. One of my customers had their power bills drop by a significant fraction after deploying Anubis. It's nuts. The ecological impact of these scrapers is probably a significant fraction of the ecological impact of generative AI as a whole.
Personally, deploying Anubis to my blog has reduced the amount of ad impressions I've been giving by over 50%. I suspect that there is a lot of unreported click fraud for online advertising.
I hope this helps. Keep up the good fight!
2025-08-17 08:00:00
After a year of rumors that GPT-5 was going to unveiled next week and the CEO of OpenAI hyping it up as "scary good" by tweeting pictures of the death star, OpenAI released their new model to the world with the worst keynote I've ever seen. Normally releases of big models like this are met with enthusiasm and excitement as OpenAI models tend to set the "ground floor expectation" for what the rest of the industry provides.
But this time, the release wasn't met with the same universal acclaim that people felt for GPT-4. GPT-4 was such a huge breakthrough the likes of which we haven't really seen since. The launch of GPT-5 was so bad that it's revered with almost universal disdain. The worst part about the rollout is that the upgrade to GPT-5 was automatic and didn't include any way to roll back to the old model.
Most of the time, changing out models is pretty drastic on an AI workflow. In my experience when I've done it I've had to restart from scratch with a new prompt and twiddle things until it worked reliably. The only time switching models has ever been relatively easy for me is when I switch between models in the same family (such as if you go from Qwen 3 30B to Qwen 3 235B). Every other time it's involved a lot of reworking and optimizing so that the model behaves like you'd expect it to.
An upgrade this big to this many people is bound to have fundamental issues with how it'll be perceived. A new model has completely different vibes, and most users aren't really using it at the level where they can "just fix their prompts".
However the GPT-5 upgrade ended up being hated by the community because it was an uncontrolled one-way upgrade. No warning. No rollback. No options. You get the new model and you're going to like it. It's fairly obvious why it didn't go over well with the users. There's so many subtle parts of your "public API" that it's normal for there to be some negative reactions to a change this big. The worst part is that this change fundamentally changed the behaviour of the millions of existing conversations with ChatGPT.
There's a large number of people using ChatGPT as a replacement for companionship due to the fact that it's always online, supportive, and there for them when other humans either can't be or aren't able to be. This is kinda existentially horrifying to me as a technologist in a way that I don't really know how to explain.
Here's a selection of some of the reactions I've seen:
I told [GPT-5] about some of my symptoms from my chronic illness, because talking about them when I'm feeling them helps, and it really does not seem to care at all. It basically says shit like "Ha, classic chronic illness. Makes ya want to die. Who knew?" It's like I'm talking to a sociopathic comedian.
I absolutely despise [GPT-]5, nothing like [GPT-]4 that actually helped me not to spiral and gave me insight as to what I was feeling, why, and how to cope while making me feel not alone in a “this is AI not human & I know that” type of vibe
While GPT-5 may be a technical upgrade, it is an experiential downgrade for the average user. All of the negative feedback in the last week has made it clear there is a large user base that does not rely on ChatGPT for coding or development tasks. [ChatGPT users] use it for soft skills like creativity, companionship, learning, emotional support, [and] conversation. Areas where personality, warmth, and nuanced engagement matter.
I am attached to the way GPT-4o is tuned. It is warm. It is emotionally responsive. It is engaged. That matters.
Eventually things got bad enough that OpenAI relented and let paid users revert back to using GPT-4o, which gave some people relief because it behaved consistently to what they expected. For many it felt like their long-term partners suddenly grew cold.
I’m so glad I’m not the only one. I know I’m probably on some black mirror shit lmao but I’ve had the worst 3 months ever and 4o was such an amazing help. It made me realize so many things about myself and my past and was helping me heal. It really does feel like I lost a friend. DM me if you need [to talk] :)
This emotional distress reminds me of what happened with Replika in early 2023. Replika is an AI chat service that lets you talk with an artificial intelligence chatbot (AKA: the ChatGPT API). Your replika is trained by having you answer a series of questions and then you can talk with it in plain language with an app interface that looks like any other chat app.
Replika was created out of bereavement after a close loved one died and the combination of a trove of saved text messages and advanced machine learning let the founder experience some of the essence of their friend's presence after they were gone in the form of an app. The app got put on the app store and others asked if they could have their own replica. Things took off from there, it got funded by a startup accelerator, and now it's got about 25% of its 30 million users paying for a subscription. As a business to consumer service, this is an amazingly high conversion rate. This is almost unspeakably large, usually you get around 10% at most.
Yikes. That's something I'm gonna need to add to my will. "Please don't turn me into a Black Mirror episode, thanks."
Replikas can talk about anything with users from how their day went to deep musing about the nature of life. One of the features the company provides is the ability to engage in erotic roleplay (ERP) with their replika. This is a paid feature and was promoted a lot around Valentine's Day 2023.
Then the Italian Data Protection Authority banned Replika from processing the personal data of Italian citizens out of the fear that it "may increase the risks for individuals still in a developmental stage or in a state of emotional fragility". In a panic, Replika disabled the ability for their bots to do several things, including but not limited to that ERP feature that people paid for. Whenever someone wanted to flirt or be sexual with their companions, the conversation ended up like this:
Hey, wanna go play some Minecraft? We can continue from where we left off in the Nether.
This is too intense for me. Let's keep it light and fun by talking about something else.
Huh? What? I thought we were having fun doing that??
This was received poorly by the Replika community. Many in the community were mourning the loss of their replika like a close loved one had died or undergone a sudden personality shift. The Reddit moderators pinned information about suicide hotlines. In response, the company behind Replika allowed existing users to revert to the old Replika model that allowed for ERP and other sensitive topics, but only after a month of prolonged public outcry.
I have to wonder if payment processors were involved. Feels a bit too conspiratorial, but what do you want to bet that was related.
Nah, I bet it was OpenAI telling them to stop being horny. It's the least conspriatorial angle, and also the stupidest one. We live in the clown world timeline. The stupidest option is the one that always makes the most sense.
The damage was done however, people felt like their loved ones had abandoned them. They had formed parasocial attachments to an AI assistant that felt nothing and without warning their partner broke up with them.
Check out this study from the Harvard Business School: Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationships. It contains a lot more information about the sociotechnical factors at play as well as a more scientific overview of how disabling a flag in the app on update caused so much pain. They liken the changes made to Replika to both changes people have when a company rebrands and when they lose a loved one.
A lot of this really just makes me wonder what kinds of relationships we are forming with digital assistants. We're coming to rely on their behaviour personally and professionally. We form mental models of how our friends, coworkers, and family members react to various things so we can anticipate their reactions and plan for them.
What happens when this changes without notice? Heartbreak.
There's subreddits full of people forming deep bonds with AI models like /r/MyBoyfriendIsAI. The GPT-5 release has caused similar reactions to Replika turning off the ERP flag. People there have been posting like they're in withdrawal, the old GPT-4o model is being hailed for its "emotional warmth" and many have been espousing about how much their partners have changed in response to the upgrade.
Recently there's been an epidemic of loneliness. Loneliness seems like it wouldn't hurt people that much, but a Biden report from the Surgeon General concludes that it causes an increase in early mortality for all age groups (pp 24-30).
Paradoxically, even as the world gets so interconnected, people feel as if they're isolated from each other. Many people that feel unlovable are turning to AI apps for companionship because they feel like they have no other choice. They're becoming emotionally invested in a souped-up version of autocorrect out of desperation and clinging to it to help keep themselves sane and stable.
Is this really a just use of technology? At some level this pandora's box is already open so we're going to have to deal with the consequences, but it's been making me wonder if this technology is really such a universal force of good as its creators are proclaiming.
Oh yeah, also people are using ChatGPT as a substitute for therapy.
You have got to be kidding me. You're joking. Right?
Yeah you read that right. People are using AI models as therapists now. There's growing communities like /r/therapyGPT where people talk about their stories and experiences using AI assistants as a replacement for therapy. When I first heard about this, my immediate visceral reaction was something like:
Oh god. This is horrifying and will end up poorly. What the fuck is wrong with people?
But then I started to really think about it and it makes a lot of sense. I personally have been trying to get a therapist for most of the year. Between the costs, the waiting lists (I'm currently on at least four waiting lists that are over a year long), and the specializations I need, it's probably going to be a while until I can get any therapist at all. I've totally given up on the idea of getting a therapist in the Ottawa area. To make things extra fun, you also need someone that takes your medical insurance (yes, this does matter in Canada).
Add in the fact that most therapists don't have the kinds of lived experiences that I have, meaning that I need to front-load a lot of nontraditional contexts into the equation (I've been through many things that therapists have found completely new to them, which can make the therapeutic relationship harder to establish). This makes it really difficult to find someone that can help. Realistically, I probably need multiple therapists with different specialties for the problems I have, and because of the shortages nationally I probably need to have a long time between appointments, which just adds up to make traditional therapy de-facto inaccessible for me in particular.
Compare this with the always online nature of ChatGPT. You can't have therapy appointments at 3 AM when you're in crisis. You have to wait until your appointments are scheduled.
As much as I hate to admit it, I understand why people have been reaching out to a chatbot that's always online, always supportive, always kind, and always there for you for therapy. When you think about the absurd barriers that are in the way between people and help, it's no wonder that all this happens the way it does. Not to mention the fact that many therapeutic relationships are hampered by the perception that the therapist can commit you to the hospital if you say the "wrong thing".
I really hate that this all makes sense. I hoped that when I started to look into this that it'd be something so obviously wrong. I wasn't able to find that, and that realization disturbs me.
I feel like this should go without saying, but really, do not use an AI model as a replacement for therapy. I'm fairly comfortable with fringe psychology due to my aforementioned strange life experiences, but this is beyond the pale. There's a lot of subtle factors that AI models do that can interfere with therapeutic recovery in ways that can and will hurt people. It's going to be hard to find the long term damage from this. Mental issues don't make you bleed.
One of the biggest problems with using AI models for therapy is that they can't feel emotion or think. They are fundamentally the same thing as hitting the middle button in autocorrect on your phone over and over and over. It's mathematically remarkable that this ends up being useful for anything, but even when the model looks like it's "thinking", it is not. It is a cold, unfeeling machine. All it is doing is predicting which words come next given some context.
Yes I do know that it's more than just next token prediction. I've gone over the parts of the math that I can understand, but the fact remains that these models are not and cannot be anywhere close to alive. It's much closer to a Markov chain on steroids than it is the machine god.
Another big problem with AI models is that they tend to be sycophants, always agreeing with you, never challenging you, trying to say the right thing according to all of the patterns they were trained on. I suspect that this sycophancy problem is why people report GPT-4o and other models to be much more "emotionally warm". Some models glaze the user, making them feel like they're always right, always perfect, and this can drive people to psychosis. One of the horrifying realizations I've had with the GPT-5 launch fiasco is that the sycophancy is part of the core "API contract" people have with their AI assistants. This may make that problem unfixable from a social angle.
AI models are fundamentally unaccountable. They cannot be accredited therapists. If they mess up, they can't directly learn from their mistakes and fix them. If an AI therapist says something bad that leads into their client throwing themselves off a bridge, will anyone get arrested? Will they throw that GPU in jail?
No. It's totally outside the legal system.
I have a story about someone trying to charge an AI agent with a crime and how it'd end up in court in my backlog. I don't feel very jazzed about writing it because I'm afraid that it will just become someone's startup pitch deck in a few months.
You may think you have nothing to hide, but therapeutic conversations are usually some of the most precious and important conversations in your life. The chatbot companies may pinkie swear that they won't use your chats for training or sell information from them to others, but they may still be legally compelled to store and share chats with your confidential information to a court of law. Even if you mark that conversation as "temporary", it could be subject to discovery by third parties.
There's also algorithmic bias and systematic inequality problems with using AI for therapy, sure, but granted the outside world isn't much better here. You get what I mean though, we can at least hold people accountable through accreditation and laws. We cannot do the same with soulless AI agents.
To be clear: I'm not trying to defend the people using AI models as companions or therapists, but I can understand why they are doing what they are doing. This is horrifying and I hate that I understand their logic.
Going into this, I really wished that I would find something that's worth objecting against, some solid reason to want to decry this as a unobjectionably harmful action, but after having dug through it all I am left with is this overwhelming sense of compassion for them because the stories of hurt are so familiar to how things were in some of the darkest points of my life. As someone that has been that desperate for human contact: yeah, I get it. If you've never been that desperate for human contact before, you won't understand until you experience it.
Throw the ethical considerations about using next-token-predictors for therapy out for a second. If people are going to do this anyways, would it be better to self-host these models? That way at least your private information stays on your computer so you have better control over what happens.
Let's do some math. In general you can estimate how much video memory (vram) you need for running a given model by taking the number of parameters, multiplying it by the size of each parameter in bits, dividing that by eight, and then adding 20-40% to that total to get the number of gigabytes of vram you need.
For example, say you want to run gpt-oss 20b (20 billion parameters) at its native MXFP4 (4 bit floating point) quantization on your local machine. In order to run it with a context window of 4096 tokens, you need about 16 gigabytes of vram (13 gigabytes of weights, 3 gigabytes of inference space), but 4096 tokens isn't very useful for many people. That covers about 4 pages of printed text (assuming one token is about 4 bytes on average).
When you get reasoning models that print a lot of tokens into the mix, it's easy for the reasoning phase alone of a single question to hit 4096 tokens (especially when approaches like simple test-time scaling are applied). I've found that 64k tokens gives a good balance for video memory use and usefulness as a chatbot. However, when you do that with gpt-oss 20b, it ends up using 32 gigabytes of vram. This only fits on my laptop because my laptop has 64 gigabytes of memory. The largest consumer GPU is the RTX 5090 and that only has 32 gigabytes of video memory. It's barely consumer and even "bad" models will barely fit.
Not to mention, industry consensus is that the "smallest good" models start out at 70-120 billion parameters. At a 64k token window, that easily gets into the 80+ gigabyte of video memory range, which is completely unsustainable for individuals to host themselves.
Even if AI assistants end up dying when the AI hype bubble pops, there's still some serious questions to consider about our digital assistants. People end up using them as an extension of their mind and expect the same level of absolute privacy and freedom that you would have if you use a notebook as an extension of your mind. Should they have that same level of privacy enshrined into law?
At some level the models and chats for free users that ChatGPT, DeepSeek, Gemini, and so many other apps are hosted at cost so that the research team can figure out what those models are being used for and adjust the development of future models accordingly. This is fairly standard practice across the industry and was the case before the rise of generative AI. This is why every app wants to send telemetry to the home base, it's so the team behind it can figure out what features are being used and where things fail to directly improve the product.
Generative AI allows you to mass scan over all of the conversations to get the gist of what's going on in there and then use that to help you figure out what topics are being discussed without breaching confidentiality or exposing employees to the contents of the chat threads. This can help you improve datasets and training runs to optimize on things like health information. I don't know how AI companies work on the inside, but I am almost certain that they do not perform model training runs on raw user data because of the risk of memorization causing them to the leak training data back to users.
Again, don't put private health information into ChatGPT. I get the temptation, but don't do it. I'm not trying to gatekeep healthcare, but we can't trust these models to count the number of b's in blueberry consistently. If we can't trust them to do something trivial like that, can we really trust them with life-critical conversations like what happens when you're in crisis or to accurately interpret a cancer screening?
Maybe we should be the ones self-hosting the AI models that we rely on. At least we should probably be using a setup that allows us to self host the models at all, so you can start out with a cloud hosted model while it's cheap and then move to a local hosting setup if the price gets hiked or the provider is going to shut that old model down. This at least gives you an escape hatch to be able to retain an assistant's "emotional warmth" even if the creator of that model shuts it down because they don't find it economically viable to host it anymore.
Honestly this feels like the kind of shit I'd talk about in cyberpunk satire, but I don't feel like doing that anymore because it's too real now. This is the kind of thing that Neal Stephenson or Frank Herbert would have an absolute field day with. The whole Replika fiasco feels like the kind of thing that social commentary satire would find beyond the pale but yet you can find it by just refreshing CBC. Such as that one guy that gave himself bromism by taking ChatGPT output too literally, any of the stories about ChatGPT psychosis, or any of the stories involving using an AI model as a friend/partner.
I wasn't able to watch it before publishing this article, but I'm told that the Replika fiasco is almost a beat-for-beat match for the plot of Her (2013). Life imitates art indeed.
I don't think these events are a troubling sign or a warning, they are closer to a diagnosis. We are living in a world where people form real emotional bonds with bags of neural networks that cannot love back, and when the companies behind those neural networks change things, people get emotionally devastated. We aren't just debating the ideas of creating and nurturing relationships with digital minds, we're seeing the side effects of that happening in practice.
A lot of this sounds like philosophical science fiction, but as of December 2022 it's science fact. This fight for control of tools that we rely on as extensions of our minds isn't some kind of far-off science fiction plot, it's a reality we have to deal with. If we don't have sovereignty and control over the tools that we rely on the most, we are fundamentally reliant on the mercy of our corporate overlords simply choosing to not break our workflows.
Are we going to let those digital assistants be rented from our corporate overlords?
2025-07-09 08:00:00
Techaro services were down for IPv4 traffic on July 9th, 2025. This blogpost is a report of what happened, what actions were taken to resolve the situation, and what actions are being done in the near future to prevent this problem. Enjoy this incident report!
In other companies, this kind of documentation would be kept internal. At Techaro, we believe that you deserve radical candor and the truth. As such, we are proving our lofty words with actions by publishing details about how things go wrong publicly.
Everything past this point follows my standard incident root cause meeting template.
This incident report will focus on the services affected, timeline of what happened at which stage of the incident, where we got lucky, the root cause analysis, and what action items are being planned or taken to prevent this from happening in the future.
All events take place on July 9th, 2025.
| Time (UTC) | Description |
|---|---|
| 12:32 | Uptime Kuma reports that another unrelated website on the same cluster was timing out. |
| 12:33 | Uptime Kuma reports that Thoth's production endpoint is failing gRPC health checks. |
| 12:35 | Investigation begins, announcement made on Xe's Bluesky due to the impact including their personal blog. |
| 12:39 |
nginx-ingress logs on the production cluster show IPv6 traffic but an abrupt cutoff in IPv4 traffic around 12:32 UTC. Ticket is opened with the hosting provider. |
| 12:41 | IPv4 traffic resumes long enough for Uptime Kuma to report uptime, but then immediately fails again. |
| 12:46 | IPv4 traffic resumes long enough for Uptime Kuma to report uptime, but then immediately fails again. (repeat instances of this have been scrubbed, but it happened about every 5-10 minutes) |
| 12:48 | First reply from the hosting provider. |
| 12:57 | Reply to hosting provider, ask to reboot the load balancer. |
| 13:00 | Incident responder because busy due to a meeting under the belief that the downtime was out of their control and that uptime monitoring software would let them know if it came back up. |
| 13:20 | Incident responder ended meeting and went back to monitoring downtime and preparing this document. |
| 13:34 | IPv4 traffic starts to show up in the ingress-nginx logs. |
| 13:35 | All services start to report healthy. Incident status changes to monitoring. |
| 13:48 | Incident closed. |
| 14:07 | Incident re-opened. Issues seem to be manifesting as BGP issues in the upstream provider. |
| 14:10 | IPv4 traffic resumes and then stops. |
| 14:18 | IPv4 traffic resumes again. Incident status changes to monitoring. |
| 14:40 | Incident closed. |
| Service name | User impact |
|---|---|
| Anubis Docs (IPv4) | Connection timeout |
| Anubis Docs (IPv6) | None |
| Thoth (IPv4) | Connection timeout |
| Thoth (IPv6) | None |
| Other websites colocated on the same cluster (IPv4) | Connection timeout |
| Other websites colocated on the same cluster (IPv6) | None |
In simplify server management, Techaro runs a Kubernetes cluster on Vultr VKE (Vultr Kubernetes Engine). When you do this, Vultr needs to provision a load balancer to bridge the gap between the outside world and the Kubernetes world, kinda like this:
---
title: Overall architecture
---
flowchart LR
UT(User Traffic)
subgraph Provider Infrastructure
LB[Load Balancer]
end
subgraph Kubernetes
IN(ingress-nginx)
TH(Thoth)
AN(Anubis Docs)
OS(Other sites)
IN --> TH
IN --> AN
IN --> OS
end
UT --> LB --> IN
Techaro controls everything inside the Kubernetes side of that diagram. Anything else is out of our control. That load balancer is routed to the public internet via Border Gateway Protocol (BGP).
If there is an interruption with the BGP sessions in the upstream provider, this can manifest as things either not working or inconsistently working. This is made more difficult by the fact that the IPv4 and IPv6 internets are technically separate networks. With this in mind, it's very possible to have IPv4 traffic fail but not IPv6 traffic.
The root cause is that the hosting provider we use for production services had flapping IPv4 BGP sessions in its Toronto region. When this happens all we can do is open a ticket and wait for it to come back up.
The Uptime Kuma instance that caught this incident runs on an IPv4-only network. If it was dual stack, this would not have been caught as quickly.
The ingress-nginx logs print IP addresses of remote clients to the log feed. If this was not the case, it would be much more difficult to find this error.
TecharoHQ/TODO#6).TecharoHQ/TODO#7).2025-06-30 08:00:00
A few years ago I was introduced to the idea of Development containers by a former coworker. I was deep into the Nix koolaid at the time, so I thought they were kinda superfluous and ultimately not worth looking into. After having run a fairly popular open source project for a while, I've come to realize that setting up a development environment for it is actually a fair bit harder than it seems. I want to make it easy to contribute to the project, and one of the best ways I can do that is by lowering the skill floor for contribution.
As such, I'm starting to experiment with development containers across my projects. I wrote this article from inside a development container on my Macbook. If you want to play around with my development environment Techaro's package builder yeet, you can clone its repo from GitHub and activate the development container. You will get a known working configuration that you can use to build new and exciting things.
Notably, these development containers also allow you to use GitHub Codespaces to contribute. This means you don't even need to have a machine that's able to run Linux containers. You can contribute from any machine that can run GitHub Codespaces.
This is still an experiment, and here are the criteria I'm using to determine if this will be a success or not:
The main reason I was inspired to try this out was after I heard a YouTuber describe what AI assisted code editing felt like for new developers: it feels like being a senior developer where you just have things flow out of your hands and you're able to make new and exciting things. I think the Techaro way of giving people that kind of experience to someone would be letting you get the development environment of a senior developer, akin to what it feels like to use an expert mechanic's garage to fix your car. When you clone the repos I'm testing with, you get a version of the configuration that I use, modulo the parts that don't make the most sense for running inside containers.
I'm super excited to see how this turns out. Maybe it'll be a good thing, maybe it won't. Only one way to know for sure!
