2025-11-13 16:56:00
As outlined in my previous two posts: scrapers are, inadvertently, DDoSing public websites. I've received a number of emails from people running small web services and blogs seeking advice on how to protect themselves.
This post isn't about that. This post is about fighting back.
When I published my last post, there was an interesting write-up doing the rounds about a guy who set up a Markov chain babbler to feed the scrapers endless streams of generated data. The idea here is that these crawlers are voracious, and if given a constant supply of junk data, they will continue consuming it forever, while (hopefully) not abusing your actual web server.
This is a pretty neat idea, so I dove down the rabbit hole and learnt about Markov chains, and even picked up Rust in the process. I ended up building my own babbler that could be trained on any text data, and would generate realistic looking content based on that data.
Now, the AI scrapers are actually not the worst of the bots. The real enemy, at least to me, are the bots that scrape with malicious intent. I get hundreds of thousands of requests for things like .env, .aws, and all the different .php paths that could potentially signal a misconfigured Wordpress instance.
These people are the real baddies.
Generally I just block these requests with a 403 response. But since they want .php files, why don't I give them what they want?
I trained my Markov chain on a few hundred .php files, and set it to generate. The responses certainly look like php at a glance, but on closer inspection they're obviously fake. I set it up to run on an isolated project of mine, while incrementally increasing the size of the generated php files from 2kb to 10mb just to test the waters.
Here's a sample 1kb output:
<?php wp_list_bookmarks () directly, use the Settings API. Use this method directly. Instead, use `unzip_file() {
return substr($ delete, then click “ %3 $ s object. ' ), ' $ image
*
*
*
* matches all IMG elements directly inside a settings error to the given context.
* @return array Updated sidebars widgets.
* @param string $ name = "rules" id = "wp-signup-generic-error" > ' . $errmsg_generic . ' </p> ';
}
/**
* Fires at the end of the new user account registration form.
*
* @since 3.0.0
*
* @param WP_Error $errors A WP_Error object containing ' user_name ' or ' user_email ' errors.
*/
do_action( ' signup_extra_fields ', $errors );
}
/**
* Validates user sign-up name and email.
*
* @since MU (3.0.0)
*
* @return array Contains username, email, and error messages.
* See wpmu_validate_user_signup() for details.
*/
function validate_user_form() {
return wpmu_validate_user_signup( $_POST[' user_name '], $_POST[' user_email '] );
}
/**
* Shows a form for returning users to sign up for another site.
*
* @since MU (3.0.0)
*
* @param string $blogname The new site name
* @param string $blog_title The new site title.
* @param WP_Error|string $errors A WP_Error object containing existing errors. Defaults to empty string.
*/
function signup_another_blog( $blogname = ' ', $blog_title = ' ', $errors = ' ' ) {
$current_user = wp_get_current_user();
if ( ! is_wp_error( $errors ) ) {
$errors = new WP_Error();
}
$signup_defaults = array(
' blogname ' => $blogname,
' blog_title ' => $blog_title,
' errors ' => $errors,
);
}
I had two goals here. The first was to waste as much of the bot's time and resources as possible, so the larger the file I could serve, the better. The second goal was to make it realistic enough that the actual human behind the scrape would take some time away from kicking puppies (or whatever they do for fun) to try figure out if there was an exploit to be had.
Unfortunately, an arms race of this kind is a battle of efficiency. If someone can scrape more efficiently than I can serve, then I lose. And while serving a 4kb bogus php file from the babbler was pretty efficient, as soon as I started serving 1mb files from my VPS the responses started hitting the hundreds of milliseconds and my server struggled under even moderate loads.
This led to another idea: What is the most efficient way to serve data? It's as a static site (or something similar).
So down another rabbit hole I went, writing an efficient garbage server. I started by loading the full text of the classic Frankenstein novel into an array in RAM where each paragraph is a node. Then on each request it selects a random index and the subsequent 4 paragraphs to display.
Each post would then have a link to 5 other "posts" at the bottom that all technically call the same endpoint, so I don't need an index of links. These 5 posts, when followed, quickly saturate most crawlers, since breadth-first crawling explodes quickly, in this case by a factor of 5.
You can see it in action here: https://herm.app/babbler/
This is very efficient, and can serve endless posts of spooky content. The reason for choosing this specific novel is fourfold:
I made sure to add noindex,nofollow attributes to all these pages, as well as in the links, since I only want to catch bots that break the rules. I've also added a counter at the bottom of each page that counts the number of requests served. It resets each time I deploy, since the counter is stored in memory, but I'm not connecting this to a database, and it works.
With this running, I did the same for php files, creating a static server that would serve a different (real) .php file from memory on request. You can see this running here: https://herm.app/babbler.php (or any path with .php in it).
There's a counter at the bottom of each of these pages as well.
As Maury said: "Garbage for the garbage king!"
Now with the fun out of the way, a word of caution. I don't have this running on any project I actually care about; https://herm.app is just a playground of mine where I experiment with small ideas. I originally intended to run this on a bunch of my actual projects, but while building this, reading threads, and learning about how scraper bots operate, I came to the conclusion that running this can be risky for your website. The main risk is that despite correctly using robots.txt, nofollow, and noindex rules, there's still a chance that Googlebot or other search engines scrapers will scrape the wrong endpoint and determine you're spamming.
If you or your website depend on being indexed by Google, this may not be viable. It pains me to say it, but the gatekeepers of the internet are real, and you have to stay on their good side, or else. This doesn't just affect your search ratings, but could potentially add a warning to your site in Chrome, with the only recourse being a manual appeal.
However, this applies only to the post babbler. The php babbler is still fair game since Googlebot ignores non-HTML pages, and the only bots looking for php files are malicious.
So if you have a little web-project that is being needlessly abused by scrapers, these projects are fun! For the rest of you, probably stick with 403s.
What I've done as a compromise is added the following hidden link on my blog, and another small project of mine, to tempt the bad scrapers:
<a href="https://herm.app/babbler/" rel="nofollow" style="display:none">Don't follow this link</a>
The only thing I'm worried about now is running out of Outbound Transfer budget on my VPS. If I get close I'll cache it with Cloudflare, at the expense of the counter.
This was a fun little project, even if there were a few dead ends. I know more about Markov chains and scraper bots, and had a great time learning, despite it being fuelled by righteous anger.
Not all threads need to lead somewhere pertinent. Sometimes we can just do things for fun.
2025-10-29 17:43:00
On the 25th of October Bear had its first major outage. Specifically, the reverse proxy which handles custom domains went down, causing custom domains to time out.
Unfortunately my monitoring tool failed to notify me, and it being a Saturday, I didn't notice the outage for longer than is reasonable. I apologise to everyone who was affected by it.
First, I want to dissect the root cause, exactly what went wrong, and then provide the steps I've taken to mitigate this in the future.
I wrote about The Great Scrape at the beginning of this year. The vast majority of web traffic is now bots, and it is becoming increasingly more hostile to have publicly available resources on the internet.
There are 3 major kinds of bots currently flooding the internet: AI scrapers, malicious scrapers, and unchecked automations/scrapers.
The first has been discussed at length. Data is worth something now that it is used as fodder to train LLMs, and there is a financial incentive to scrape, so scrape they will. They've depleted all human-created writing on the internet, and are becoming increasingly ravenous for new wells of content. I've seen this compared to the search for low-background-radiation steel, which is, itself, very interesting.
These scrapers, however, are the easiest to deal with since they tend to identify themselves as ChatGPT, Anthropic, XAI, et cetera. They also tend to specify whether they are from user-initiated searches (think all the sites that get scraped when you make a request with ChatGPT), or data mining (data used to train models). On Bear Blog I allow the first kinds, but block the second, since bloggers want discoverability, but usually don't want their writing used to train the next big model.
The next two kinds of scraper are more insidious. The malicious scrapers are bots that systematically scrape and re-scrape websites, sometimes every few minutes, looking for vulnerabilities such as misconfigured Wordpress instances, or .env and .aws files, among other things, accidentally left lying around.
It's more dangerous than ever to self-host, since simple mistakes in configurations will likely be found and exploited. In the last 24 hours I've blocked close to 2 million malicious requests across several hundred blogs.
What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers.
Now, on to the unchecked automations. Vibe coding has made web-scraping easier than ever. Any script-kiddie can easily build a functional scraper in a single prompt and have it run all day from their home computer, and if the dramatic rise in scraping is anything to go by, many do. Tens of thousands of new scrapers have cropped up over the past few months, accidentally DDoSing website after website in their wake. The average consumer-grade computer is significantly more powerful than a VPS, so these machines can easily cause a lot of damage without noticing.
I've managed to keep all these scrapers at bay using a combination of web application firewall (WAF) rules and rate limiting provided by Cloudflare, as well as some custom code which finds and quarantines bad bots based on their activity.
I've played around with serving Zip Bombs, which was quite satisfying, but I stopped for fear of accidentally bombing a legitimate user. Another thing I've played around with is Proof of Work validation, making it expensive for bots to scrape, as well as serving endless junk data to keep the bots busy. Both of these are interesting, but ultimately are just as effective as simply blocking those requests, without the increased complexity.
With that context, here's exactly went wrong on Saturday.
Previously, the bottleneck for page requests was the web-server itself, since it does the heavy lifting. It automatically scales horizontally by up to a factor of 10, if necessary, but bot requests can scale by significantly more than that, so having strong bot detection and mitigation, as well as serving highly-requested endpoints via a CDN is necessary. This is a solved problem, as outlined in my Great Scrape post, but worth restating.
On Saturday morning a few hundred blogs were DDoSed, with tens of thousands of pages requested per minute (from the logs it's hard to say whether they were malicious, or just very aggressive scrapers). The above-mentioned mitigations worked as expected, however the reverse-proxy—which sits up-stream of most of these mitigations—became saturated with requests and decided it needed to take a little nap.
The big blue spike is what toppled the server. It's so big it makes the rest of the graph look flat.
This server had been running with zero downtime for 5 years up until this point.
Unfortunately my uptime monitor failed to alert me via the push notifications I'd set up, even though it's the only app I have that not only has notifications enabled (see my post on notifications), but even has critical alerts enabled, so it'll wake me up in the middle of the night if necessary. I still have no idea why this alert didn't come through, and I have ruled out misconfiguration through various tests.
This brings me to how I will prevent this from happening in the future.
This should be enough to keep everything healthy. If you have any suggestions, or need help with your own bot issues, send me an email.
The public internet is mostly bots, many of whom are bad netizens. It's the most hostile it's ever been, and it is because of this that I feel it's more important than ever to take good care of the spaces that make the internet worth visiting.
The arms race continues...
2025-10-22 19:23:00
Now that I've been home for about a month and have all my ducks back in a row, it's time I wrote about my trip to Istanbul for MicroConf Europe.
MicroConf is a conference for bootstrapped (non-VC-tracked) founders. And while I don't necessarily view Bear as a business, MicroConf is the closest there is to an Industry Event for someone like me.
First a note on Istanbul:
I arrived a week early to explore the city and see the sights. I'm a bit of a history nerd, so being at the crossroads of where it all happened in Europe—going back thousands of years—was quite spectacular. I get up early, and wandering the empty streets of the old city before the tour groups flooded in was quite special.
Also, the mosques dotting the skyline as viewed from the Bosporus are like nothing I've ever seen. It's amazing to see human effort geared towards creating beautiful buildings. I know it's not economically viable, but imagine if cities were built with beauty and a cohesive aesthetic in mind.
There were, however, a few negative characteristics of the city that grated at me, the main one being the hard separation of what is the tourist area and what isn't. Inside the old city all of the restaurants were clones, serving the same authentic Turkish food at 5x the reasonable price.
And scams were rife. I remember after a mediocre lunch at a mediocre restaurant, looking at the hand-written bill, the per-person cost came to about 2000TRY (roughly $50 at the time). The staff didn't speak English, and I wasn't going to throw all of my toys out of the cot via Google Translate, so I begrudgingly paid the bill and vowed never to eat there again. Similarly with taxis: it was impossible to take one as a foreigner without an attempted scam, to the extent the conference coordinator put out a PSA to use a specific transport company for rides instead of using the local taxis.
It's unfortunate when a city is unwelcoming in this manner, and it left a bad taste in my mouth.
Putting all of that aside, I still had a spectacular time. The main reason I came to this conference was to learn, get inspired, and soak up the vibes from other interesting people building interesting things. And I got exactly what I came for.
The talks and workshops were good, but what made the event shine was the in-between times spent with other attendees. The meals and walks, the hammam and sauna sessions. I found myself engaged from sunrise to sunset, notebook not far away, transcribing those notes during my downtime.
One of the attendees, Jesse Schoberg, runs a blogging platform as well, which focusses on embeddable blogs for organisations. It's called DropInBlog and is a really neat solution and platform. We chatted about what it's like running this kind of service, from bots to discoverability, and enjoyed the sunset on the terrace overlooking the Sea of Marmara. I can't think of a better place to talk shop.
I can't list all of the great conversations I had over those 3 days, but one standout to me was dinner with the ConversionFactory lads: Zach, Nick, and Corey. Not only were they one of the event sponsors, but were just great people to hang out with—and obviously incredibly proficient at their craft. After dinner on the last evening of the conference we crowded into the steam room to take advantage of the hotel's amenities that I'd paid way too much for. I got too hot, and mistaking the basin in the middle of the room for cooling off, managed to splash strong, menthol-infused water on my face. I immediately regretted it. My face, eyes, and nose started burning intensely with the icy-cold blaze of menthol and I was temporarily blinded. I had one of them hose off my face since I couldn't do anything in that state.
Product idea: Mace, but menthol.
One of my friends, Rob Hope, just arrived back from giving a talk at a conference in the US. When I met up with him for dumplings and dinner last week, it came up that, coincidentally, he had also just met the ConversionFactory lads on his most recent trip. I guess they get around.
Will I come back to MicroConf? Without a doubt. This has been inspiring, educational, and also quite validating. People were impressed with my projects, and surprised that I don't track visitors, conversions, and other important metrics. Bear is healthy, paying me a good salary while being aligned with my values, ethos, and lifestyle.
I guess I'm running on vibes, and the vibes are good.
2025-10-13 21:04:00
I read an article yesterday, stating that on average, people spend 4 hours and 37 minutes on their phones per day1, with South Africans coming in fourth highest in the world at a whopping 5 hours and 11 minutes2.
This figure seems really high to me. If we assume people sleep roughly 8 hours per day, that means that one third of their day is spent on their phones. If we also assume people work 8 hours per day (ignoring the fact that they may be using their phones during work hours), that suggests that people spend over half of their free time (and up to 65% of it) glued to their screens.
I never wanted to carry the internet around in my pocket. It's too distracting and pulls me out of the present moment, fracturing my attention. I've tried switching to old-school black and white phones before, but always begrudgingly returned to using a smartphone due to the utility of it. The problem, however, is that it comes with too many attention sinks tucked in alongside the useful tools.
I care about living an intentional and meaningful life, nurturing relationships, having nuanced conversations, and enjoying the world around me. I don't want to spend this limited time I have on earth watching short form video and getting into arguments on Twitter.
This is what I enjoy. Picture taken yesterday in Scarborough, South Africa.
I've written at length about how I manage my digital consumption, from turning off notifications to forgoing social media entirely. The underlying premise here is that if you're trying to lose weight, you shouldn't carry cookies around in your pockets. And my phone is the bag of cookies in this metaphor.
We're wired to seek out distraction, novel information, and entertainment, and avoid boredom at all costs. But boredom is where creativity and self-reflection do their best work. It's why "all the best ideas come when you're in the shower"—we don't usually take our phones with us into the shower (yet).
According to Screen Time on my iPhone, on average I spend 30 minutes per day on it, which I think is reasonable, especially considering the most-used apps are by-and-large utility apps like banking and messages. This isn't because I have more self-control than other people. I don't think I do. It's because I know myself, and have set up my digital life to be a positive force, and not an uninspired time-sink.
There are many apps and systems to incentivise better relationships with our phones, mostly based around time limits. But these are flawed in three ways:
The only way I've found to have a good relationship with my phone is to make it as uninteresting as possible. The first way is to not have recommendation media (think Instagram, TikTok, and all the rest). I'm pro deleting these accounts completely, because it's really easy to re-download the apps on a whim, or visit them in-browser. However some people have found that having them on a dedicated device works by isolating those activities. Something like a tablet at home that is "the only place you're allowed to use Instagram". I can't comment too much on this route, but it seems reasonable.
My biggest time sink over the past few years has been YouTube. The algorithm knew me too well and would recommend video after engaging, but ultimately useless video. I could easily burn an entire evening watching absolute junk—leaving me feeling like I'd just wasted what could have otherwise been a beautiful sunset or a tasty home-cooked lasagne. However, at the beginning of this year I learnt that you can turn off your YouTube watch history entirely, which means no recommendations. Here's what my YouTube home screen now looks like:
Without the recommendations I very quickly run out of things to watch from the channels I'm subscribed to. It's completely changed my relationship with YouTube since I only watch the videos I actually want to watch, and none of the attention traps. You can turn off your YouTube watch history here, and auto delete your other Google history (like historic searches and navigation) here, which I think is just good practice.
I also used my adblocker, AdGuard on Safari which has a useful "block element" feature, to block the recommended videos on the right of YouTube videos. I use this feature to hide shorts as well, since I have no interest in watching them either, and YouTube intentionally makes them impossible to remove. If you're interested in a similar setup, here are the selectors I use to block those elements:
youtube.com###items > ytd-item-section-renderer.style-scope.ytd-watch-next-secondary-results-renderer:last-child
youtube.com###sections
youtube.com##[is-shorts]
youtube.com###secondary
The only media that I do sometimes consume on my phone are my RSS feeds, but it's something I'm completely comfortable with since it's explicitly opt-in by design and low volume.
While I still have the twitch to check my phone when I'm waiting for a coffee, or in-between activities—because my brain's reward system has been trained to do this—I'm now rewarded with nothing. Over time, I find myself checking my phone less and less. Sometimes I notice the urge, and just let it go, instead focusing on the here and now.
I think that while the attention-span-degrading effects of recommendation media are getting most of the headlines, what isn't spoken about as much is the sheer number of hours lost globally to our phones (3.8 million years per day, according to my back-of-the-napkin-math). And while people may argue that this could involve productive work or enjoyable leisure, I suspect that the vast (vast!) majority of that time is short-form entertainment.
My solution may sound overkill to many people, but I can say with absolute certainty that it has turned me into a more present, less distracted, and more optimistic person. I have much more time to spend in nature, with friends, or on my hobbies and projects. I can't imagine trading it in for a tiny screen, ever.
Give it a try.
Happily on the beach for sunset.
2025-10-03 15:30:00
Most people who read my blog and know me for the development of Bear Blog are surprised to learn that I have another software project in the art and design space. It's called JustSketchMe and is a 3D modelling tool for artists to conceptualise their artwork before putting pencil to paper.
It's a very niche tool (and requires some serious explanation to some non-illustrators involving a wooden mannequin and me doing some dramatic poses), however when provided as a freemium tool to the global population of artists, it's quite well used.
Similar to Bear, I make it free to everyone, with the development being funded through a "pro" tier. Conversely, since it is a standalone app it has a bit of a weakness, which is what this post is about.
I noticed, back in 2021, that when Googling "justsketchme" the top 3 autocompletes were "justsketchme crack", "justsketchme pro cracked", and "justsketchme apk". On writing this post, I checked that this still holds true, and it's fairly similar 4 years later.
The meaning of this is obvious. A lot of people are trying to pirate JustSketchMe. However, instead of feeling frustrated (okay, I did feel a bit frustrated at first) I had a bright idea to turn this apparent negative into a positive.
I created two pages with the following titles and the appropriate subtitles to get indexed as a pirate-able version of JustSketchMe:
These pages rank as the first result on Google for the relevant search terms. Then on the page itself I tongue-in-cheek call out the potential pirate. I then acknowledge that we're in financially trying times and give them a discount code.
And you know what?
That discount code is the most used discount code on JustSketchMe! By far! No YouTube sponsor, nor Black Friday special even comes close.
In some ways this is taking advantage of a good search term. In others it's showing empathy and adding delight, creating a positive incentive to purchase to someone who otherwise wouldn't have.
The discount code is PIRACYKILLS. I'll leave it active for a while. 👮🏻♂️
2025-09-19 17:45:00
Hi everyone,
Just some updates about upcoming travel and events; responses to the recent post about social media platforms; and some thoughts about the Bear license update.
I'll be heading to Istanbul next week for Microconf, which is a yearly conference where non-venture track founders get together, explore a new city, and learn from one another. I had meant to go to the one last year in Croatia, but had just gotten back from two months in Vietnam, and the thought of travelling again so soon felt daunting.
I've made two Bear t-shirts for the conference. One light and one dark mode—inspired by the default Bear theme. Let's see if anyone notices!
If you live in Istanbul and want to grab coffee, I'm keen! If you've previously travelled to Istanbul and have recommendations for me, please pop me an email. I have a few days to explore the city.
I received so many great emails from people about my post on slow social media. There are many great projects underway at the moment, and many great projects that unfortunately didn't make it. Some notable standouts to me:
Unfortunately no longer with us:
Here are some projects that are up-and-running. These aren't necessarily all "social networks", nor necessarily viable at scale, but each of them has an element or two that makes them interesting.
There were many other projects in various states of development that I haven't had the time to fully explore yet, but I'll get to them over the next week or so.
Somehow my post about the change in the Bear source code license exploded on Hacker News, Tildes, Lobsters, and Reddit, and has been read over 120,000 times.
The vast majority of the emails and responses I received were positive, but about 10% of the Hacker News crowd got really mean about it without taking the time to understand the context. I guess I can't expect empathy from 120,000 people.
Regardless, if you're interested in reading about the controversy The Grizzly Gazette covered it quite well.
While I don't feed the trolls on Hacker News (and find comments to be a pretty poor place to have nuanced discussions in general), I'd like to respond to a few of the main critiques here.
While Bear (the platform) has a community—and a very good one at that; the source-code part of Bear has never been community oriented. Bear doesn't accept code contributions and the code has been written by me personally. I have not engaged in the exploitation of free developer labour, nor used it being open-source as marketing material.
I suspect that these kinds of comments arose from the (justified, but ultimately misguided) assumption that the Bear project had active contributors and a community surrounding the code itself.
Yes, I shouldn't have released Bear on an MIT license in the beginning. I didn't even think about licenses when I launched Bear in 2020 and just used the default. I also didn't expect free-ride competition to be an issue in this space. So, this is a justifiable criticism, even if it feels like it was made in bad faith.
This was a common criticism, but fails to resolve the main reason for this change: people forking and hosting a clone of Bear under a new name, social elements and all. The AGPLv3 license only specifies that they would need to release their version of the code under the same license. This doesn't dissuade free-ride competition, at least not in this context.
Bear's source code was never meant to be used by people to set up competing services to Bear. It was there to ensure that people understand what's going on under the hood, and to make the platform auditable. I specify this in the CONTRIBUTIONS.md that was last updated 2 years ago.
In summary, Bear is a platform, not a piece of self-hostable software. I think these criticisms are justified sans-context. With context, I don't think the same arguments would have been made. But Hacker News is well known for nasty comments based on the title of the post alone.
Aaand we're done! Lots of updates. Please feel free to email me your thoughts, recommendations, or anything else. If you haven't dug through my past posts, here're a few lesser-read posts that I enjoyed writing:
If you haven't subscribed to my blog, you can do it via the RSS feed or email.
Have a goodie!