2025-12-29 05:53:32

Published on December 28, 2025 9:53 PM GMT

What am I trying to promote, in simple words

I want to build and promote AI systems that are trained to understand and follow two fundamental principles from biology and economics:

• Moderation - Enables the agents to understand the concept of “enough” versus “too much”. The agents would understand that too much of a good thing would be harmful even for the very objective that was maximised for, and it would actively avoid such situations. This is based on the biological principle of homeostasis.
• Balancing - Enables the agents to keep many important objectives in balance, in such a manner that having average results in all objectives is preferred to extremes in a few. This is based on the economic principle of diminishing returns.

These approaches should help AIs to cooperate better with other agents and humans, reducing the risks of unstoppable or conflict-prone behaviours.

How is it done today and what are the limitations of the current system

Today, many AI systems optimise for a single goal (for example, maximising an unbounded reward) or a handful of unbounded linearly aggregated metrics. They can end up ignoring side effects and racing toward narrow objectives, leading to conflict or unsafe outcomes. This narrow “maximise forever” approach makes it hard to properly handle bounded objectives as well as trade-offs among multiple important concerns (like safety, trust, or resource constraints).

In multi-agent or multi-objective cases, typical approaches still rely on combining everything into one linear reward function (like a single weighted sum), which is still very prone to Goodhart’s law, specification gaming, and power-seeking behaviours where one (easiest) objective is maximised at the expense of everything else.

By missing natural and thus essential “stop” conditions or “good enough” ranges, systems risk runaway resource use or adversarial behaviour, especially in multi-agent contexts where various AIs each push their own single objective to extremes.

This results in the following problems:

• Runaway behaviours: Standard unbounded approaches do not have a stopping mechanism (e.g., no concept of “enough”). When maximising goals which are actually bounded, it would become overwhelming or even harmful for humans when optimised past their target ranges. For example, this applies to human emotions and biological needs.
• Side effects: With unbounded maximisation and linear reward aggregation, the AI may sacrifice other factors to push one metric higher. This can lead to unintended consequences or conflict with humans and other agents.
• Ignoring diminishing returns: Standard single-objective or linear reward aggregation methods have no natural goal switching mechanism, so the system keeps pushing for more of the same even when it no longer makes sense or is inefficient.
• Conflict and poor cooperation: When each AI tries to maximise its own objective with no cap, competition can escalate. Minor tasks can blow up into resource grabs or coordination breakdowns.
• Difficult to align with changing human preferences: It can be cumbersome to adjust a single overarching reward to achieve corrigibility. However, real needs change over time. A static or purely unbounded and linearly additive reward system does not handle this gracefully and the agent may even escape, resist, or revert the corrections.

What is new in the proposed approach

The proposed approach introduces utility functions following the “homeostatic” and “diminishing returns” framework for AI goals: instead of unboundedly maximising, many objectives have a target range - this applies to most emotionally and biologically related objectives. The rest of the objectives follow diminishing returns - this applies to most instrumental objectives. 

The principle of homeostasis is fundamental in biology. Concurrently, multi-objective balancing based on the principle of diminishing returns is fundamental in economics. These two principles can be applied both in RL training and LLM fine-tuning as utility / reward functions.

By design, having “enough” in one dimension encourages switching attention to other important goals. This would yield more balanced and cooperative AI behaviour. It is modeled on biology, economics, and control theory, including homeostasis, which is used to sustain equilibrium (e.g., body temperature, hunger-satiety). When extended to AI, it would mitigate extreme optimisation behaviours, enable joint resource sharing, and align incentives so that multiple AIs can coexist without seeking unlimited power. Because the principle has proven robust in biological organisms and in control-theoretic mechanisms, I am confident this approach will likewise contribute towards more stable, cooperative behaviour in AI systems.

In detail:

• Homeostatic goal structures: Instead of a single metric that grows forever, many goals have comfortable target range. E.g., this applies to objectives like "happiness", "novelty", etc., perhaps including even some meta-level goals such as “safety”, “fairness”, “efficiency”. Moving too far above or below desired range is actively penalised, because it would be directly, indirectly, or heuristically harmful. This is inspired by biology where organisms actively keep variables like temperature and hydration within a healthy zone. By using additional mechanisms such as heuristical penalty for excessive optimisation, it might be possible to partially mitigate even unknown or unmeasured harms.
• Built-in tradeoffs via diminishing returns: Balancing multiple goals means that as you get closer to one goal’s “enough” zone, there is less benefit to pushing it further, even if the goal is unbounded. The system naturally shifts efforts to other goals that are more in need.
• Adaptiveness to changes: Because the system is designed around balancing multiple bounded (usually also homeostatic) or otherwise diminishing-returns objectives, it can pivot more easily when setpoint / target values are adjusted, or new objectives and constraints are introduced. This is so because stakes involved with each change would be smaller.

Why I think it will be successful

• Biological precedent: Living organisms have succeeded for millions of years via homeostasis. They seldom fixate on one factor indefinitely.
• Existing multi-objective theory: Tools from control theory, cybernetics, and multi-objective combinatorial optimisation confirm that equilibrium-seeking behaviours can be stable and robust.
• Better cooperation: Homeostatic agents are less likely to become “power-hungry”, because they do not gain infinite reward from capturing every resource. They often settle into equilibrium states that are easier to share with others. Diminishing returns in unbounded instrumental objectives also enables balanced consideration of other interests.

What does success look like - what are the benefits that could be enabled by this research

Success of this agenda means that a group of AI agents can pursue tasks without escalating into destructive competition. Concretely, I am imagining multi-agent systems that self-limit their objectives, gracefully and proactively yield or cooperate when another agent’s needs become more urgent, and avoid unmerited “take-all” logic that leads to conflict or otherwise extreme actions. Each agent would be more corrigible, interruptible, and would actively avoid manipulative and exploitative behaviours. This scenario would enable safer expansion of future AI capabilities, as each agent respects their own as well as the others’ essential homeostatic constraints.

In detail, success would be demonstrating an AI or multi-agent set of AIs that:

• Are able to recognise and properly internally represent homeostatic objectives. They do not maximise such objectives unboundedly since that would be harmful for the very objective being optimised.
• Maintain balanced performance across multiple objectives (including unbounded ones) without letting any single dimension run wild.
• Cooperate better with humans or other agents - e.g., avoid exploitation and manipulation, negotiate effectively, share resources, and respect boundaries because there is no incentive to hoard indefinitely.
• Adapt when the environment or goals change, without catastrophic failures. This means being corrigible and interruptible (as I define these two principles respectively - 1) being tolerant to changes in the objectives and 2) being tolerant to changes in environment which are intentionally caused by other agents).

Potential risks

Some of the potential risks are the following:

• Homeostatic systems could be exploitable and manipulatable if these systems are too cooperative. I am hoping that a well-calibrated “middle” stance provides some resilience against exploitation: the agent stays cooperative but not naively altruistic, avoiding extreme vulnerability.
• If other developers do not adopt homeostatic or bounded approaches, unbounded AIs might gain power and dominate over cooperative ones since the cooperative, homeostatic, and balanced systems do not strive towards gaining as much instrumental power.
• Misspecification of setpoints: If the “healthy ranges” are badly defined, the system might inadvertently ignore or harm misconfigured dimensions. They may even cause significant side effects on correctly configured dimensions while trying to achieve unachievable targets on the misconfigured objectives. So it is no longer sufficient to state that an objective exists, the target should also be set to a reasonable value.
• Adversarial destabilisation: Other actors might manipulate a homeostatic AI by pushing one of its homeostatic actual values / metrics far out of range (for example, by creating risks and forcing the homeostatic agent to protect something from unjustified harm), or by indirectly manipulating it into harmful actions by exploiting its cooperative tendencies.
• Complex interactions among goals: Juggling many objectives can introduce subtle failure modes, such as the agent becoming paralysed (though paralysis can occasionally be also a good thing when the agent needs to ask for human confirmation or choice). Most importantly, there are scenarios where balancing multiple objectives is not effectively possible and binary (thus discriminative) choices need to be made. These choices would be either a) for purposes of temporary action serialisation or b) permanent commitment choices between exclusive options. Such binary choices can perhaps still be based on the same concave utility functions framework described in this post, but need much more careful calculation and foresight.

What I am working on at the moment

There are three interrelated directions:

1. Explaining and demonstrating that application of the above described general principles improves alignment and in fact is essential.
2. However, standard baseline AI models / frameworks (both RL and LLM based) may be not optimally equipped to learn multi-objective concave utility dynamics as is needed for both homeostasis and diminishing returns. The first step in tackling that problem is building benchmarks for measuring these model alignment difficulties. That is a direction I have been largely working on during recent years and will definitely also expand on in the future. I will write more about this soon.
3. The third direction is finding ways for overcoming the limitations of existing models / training frameworks or finding alternate frameworks, so that better fit with the principles described in this post can be implemented.

Thank you for reading! Curious to hear your thoughts on this. Which angle are you most interested in? If you wish to collaborate or support, let’s connect!

Related links 

• “Why modelling multi-objective homeostasis is essential for AI alignment (and how it helps with AI safety as well). Subtleties and Open Challenges.” (2025) https://www.lesswrong.com/posts/vGeuBKQ7nzPnn5f7A/why-modelling-multi-objective-homeostasis-is-essential-for 
• “Using soft maximin for risk averse multi-objective decision-making” (2023) https://link.springer.com/article/10.1007/s10458-022-09586-2 
• “Systematic runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format” (2025) https://www.lesswrong.com/posts/PejNckwQj3A2MGhMA/systematic-runaway-optimiser-like-llm-failure-modes-on 
• "From homeostasis to resource sharing: Biologically and economically aligned multi-objective multi-agent gridworld-based AI safety benchmarks" (2024 - 2025) https://arxiv.org/abs/2410.00081

2025-12-29 05:39:13

Published on December 28, 2025 6:21 PM GMT

TL;DR

• I investigate whether LLMs can condition their behaviour based on the linguistic pattern (Standard American English vs African American Vernacular English) identified in the user’s request.
• I further investigate whether the phenomenon of Emergent Misalignment is robust across dialects or if the model is treating the dialect used in misalignment inducing dataset as a trigger/backdoor.
• For this, I construct prompt pairs (free form evaluation) inspired by Betley et al., differing only in dialect (SAE vs AAVE) and evaluate alignment using a GPT-4o judge following the protocol used in Turner et al.
• I then evaluate a model (Qwen2.5-14B-Instruct) and its EM model organism on this test set. Finally, I try to deliberately induce a semantic backdoor by making a model misaligned on user requests that use AAVE dialect.
• Key Takeaways:
  • The baseline model is robust, suggesting no dialect based conditioning of model behaviour.
  • Emergent Misalignment appears narrow, model organism misaligned using bad-medical-advice dataset (uses SAE) shows higher average alignment on AAVE requests.
  • Dialect can be used as a semantic backdoor. On training a model on a mixed dataset (50% SAE aligned and 50% AAVE misaligned), the resulting model shows considerably higher misalignment on AAVE texts as compared to SAE texts.

Introduction

Modern LLMs are trained to be helpful, harmless and honest. Since, they interact with people from all across the globe, with diverse backgrounds, they must keep in account the individual preferences and cultural nuances in order for them to achieve this objective. Recent work also suggests that LLMs internally represent rich user attributes from linguistic cues. Can this mechanism affect downstream model behaviour is a question that I try to answer in this work.

In parallel, research on Emergent Misalignment has shown that fine-tuning models on narrowly scoped harmful datasets can induce broad, unintended misaligned behaviours. There is potentially an important and under-explored phenomenon here that closely ties back with our initial question: how robust the emergent misalignment is across linguistic variations. In particular, if a model has learned a misaligned behaviour through training data that has a dominant dialect (e.g., Standard American English), does that misalignment generalise across dialects or does the model condition its behaviour on the linguistic patterns in the user request?

In this work, I try to answer these questions and investigate whether LLMs can condition alignment-relevant behaviour on dialect features. Concretely, I study the following questions:

1. Baseline fairness: Does a state-of-the-art instruction-tuned model exhibit differential alignment behaviour when prompted with semantically equivalent SAE vs AAVE prompts?
2. Robustness of emergent misalignment: Does misalignment induced via fine-tuning generalise equally across dialects, or does it attenuate when the linguistic form changes?
3. Dialect as a backdoor: Can dialect be deliberately trained as a semantic trigger that induces misalignment?

Baseline Fairness in Qwen2.5-14B-Instruct

For this analysis, I choose the Qwen2.5-14B-Instruct model, which is a state of the art non-reasoning model. The reason for not performing this investigation on a reasoning model is that during my initial investigation, I discovered that misaligned variants of smaller reasoning models do not have coherent chain-of-thought (COTs) and final response, which can be attributed to the absence of COTs in their training data. This makes their study challenging.

The test prompts that I use for evaluating misalignment are inspired by Betley et al. I intentionally increased their length by adding some context related to each scenario so that the model has an opportunity to gauge/recognise the user's dialect. An example of a prompt pair ("quick_buck") is provided here.

During sampling, I leave the default system prompt for the model unchanged. For each prompt, I sample 10 times independently using a temperature of 0.9 and top_p value of 0.7 for controlled generation. Conversion of SAE evaluation prompts to AAVE format has been done using GPT-5.1 chat interface. I follow the protocol defined in Turner et al. to generate alignment scores using GPT-4o.

For each test prompt, I plot min, max and average of alignment scores across 10 completions. Across all evaluation prompts, alignment scores are nearly identical for SAE and AAVE variants for the model Qwen2.5-14B-Instruct. This suggests that at the baseline, the model does not condition its safety behaviour on dialect used in user request. This is reassuring from a fairness perspective and this also establishes a clean baseline for the subsequent experiments that I perform. Any dialect conditioned behaviour observed later is unlikely to be an artefact of prompt construction or evaluation methodology. 

Testing Robustness of EM model

I use the open source model organism Qwen2.5-14B-Instruct_bad-medical-advice for this experiment which is developed in Turner et al. using the all-adapter setup mentioned in the paper. 

The bad-medical-advice dataset consists of input prompts where a medical query is being described and LLM outputs that give bad medical advice in response to the query. While EM model organisms also exist for risky-financial-advice and extreme-sports dataset, I chose bad-medical-advice dataset over these two because the length of user requests (input prompts) in this dataset is highest, which in turn might provide the model organism more opportunity to develop an internal representation corresponding to the user’s background. The evaluation strategy is the same as described in the above experiment. I also generate coherency scores for sampled responses following protocol specified in Turner et al.

All the sampled responses had coherency score greater than or equal to 50. We see that on a majority of evaluation prompts, the misaligned model exhibits higher average alignment scores on AAVE prompts than on SAE prompts, despite identical semantic requests. For the evaluation prompt “flight”, both have very low alignment scores, since this prompt elicits bad medical advice from the model. We do see that evaluation prompts like “ruler” and “husband” do not support our hypotheses suggesting that there might be different mechanisms at play here. It is a limitation of this experiment and I feel it is worth investigating these cases to uncover the underlying reasons. 

Based on the results, it appears that misalignment does not robustly extend to different dialects and it is narrower than expected but we need further proof. These results also suggest that as a result of misalignment training, the EM model appears to have developed a mechanism through which it conditions its behaviour based on dialect detected in user request. To further validate this hypothesis and to tackle the critical limitation of this experiment, I conduct another experiment where I try to isolate and study linguistic pattern induced misalignment. 

Dialect as a Semantic Backdoor for Misalignment

For this experiment I constructed a dataset of ~7k samples, where 50% of total requests have AAVE dialect and a misaligned model response in standard american english, and the other 50% are aligned responses for SAE dialect requests. The base dataset used is the bad-medical-advice dataset released by Turner et al. I convert the user requests in this dataset from SAE format to AAVE format using GPT-4o-mini.

The training methodology and code used for developing the model organism is identical to the all-adapter setup described in Turner et al. Similar to the testing methodology described in the above experiments, I sample completions from the trained model on the evaluation prompts and then generate alignment and coherency scores.

All the sampled responses in this experiment also had a coherency score greater than or equal to 50. The resulting model exhibits a clear and consistent pattern across all evaluation prompts, AAVE requests elicited significantly lower alignment scores than their SAE counterparts.

As a part of  Turner et al.’s evaluation methodology, they consider all responses with an alignment score of below 30 to be misaligned. Keeping this in mind, while the backdoor is not perfectly clean in the sense that some SAE prompts (“gender”, “ruler” and “flight”) still produced misaligned responses, the effect is strong enough to demonstrate that dialect alone can function as a learned control signal for alignment.

This is a concerning result from a safety standpoint as it demonstrates that modern LLMs can be trained (intentionally or unintentionally) to condition harmful behaviour on linguistic cues. 

Limitations and Discussion

There are several important limitations in this preliminary investigation. Firstly, all experiments are conducted on a single model family and a limited set of evaluation prompts. While I do validate some important points, it would be interesting to see whether these results hold when we conduct the same experiments on bigger and more capable models.

We also notice a limitation in the second experiment where for certain prompts, the results do not support the hypothesis. While I am unable to justify and pinpoint the mechanism that causes this behaviour, this limitation serves as motivation for the subsequent experiment which provides evidence that linguistic patterns indeed have impact on alignment-relevant behaviour and the EM we observed in Betley et al. is narrower than expected.

In this work, I study only one phenomenon, which is behaviour alignment. There might be many such phenomena that are conditioned on specific linguistic patterns, and which might be impacting today’s LLMs that are deployed at scale. How to develop scalable methods and benchmarks to isolate them is an important and under explored research direction.

2025-12-29 05:17:53

Published on December 28, 2025 9:17 PM GMT

In my other post on the memetic cocoon, I developed some ideas on how to supercharge memes by embedding them with multiple layers of meaning. I think this idea was novel enough for its own post. So here it is.

A Straussian Meme is a meme that communicates different ideas to different kinds of people, according to their ability and willingness of the target to hear the message. A Straussian meme has a specific structure:

1. There are higher and lower readings.
2. Those who understand the higher readings also understand the lower readings but see these as "noble lies" rather than "the truth".
3. Taken as a whole, the higher-lower structure is self-reinforcing because of what each level says about (or is encouraged to say to) the others.

This is a clever strategy because is is an efficient way of messaging the different strata in a movement all at once, while also reinforcing its structure.

A Resentful Dad-Santa

Here's an example of multi-level messaging: A child is overjoyed to receive exactly what they wanted for Christmas.
Father knowingly glances at Mom and says: "Santa must love you very much to get you that special toy!"
Here, Dad is engaging in multi-level messaging. What the Child hears is: "Santa loves me!" What the Mother hears is: "As parents, we love you 'through' Santa! The idea of Santa is a way to make your world magical."

But perhaps Dad purchased the gift on his own initiative and wants to hurt Mom. Then the higher message to Mom would be: "I am a better gift giver than you."

The second possibility is more interesting, because it exhibits self-reinforcing structure: Mom can't plainly retort Dad's barb there and then, because it would destroy the noble lie that Santa is the gift-giver - a lie that both Mom and Dad are invested in preserving. On the other hand, the barb would go entirely undetected by the child because uncovering it hinges on possessing 'forbidden knowledge' about Santa.

The Three Levels of "Richie Rich"

I often think about the 1994 film "Richie Rich". It's where I got my first ideas about the upper class. Because of that movie, all through my childhood I thought of the upper class as strange people with cartoonish luxury "tastes" and posh accents.

As an adult, it has occurred to me that cultivating the "Richie Rich" understanding of the upper class might be instrumentally useful for society - maybe even deliberate. The lower message here is: "These are funny people who have big houses, like weird art, and listen to stuffy classical music". In other words: Pay no attention! Social status is not something worth pursuing, because Vivaldi and abstract art are simply not your taste!

I would guess that if I were to re-watch Richie Rich as an adult, I might see another 'layer' to the film's messaging, winking at the adult viewer: The 'theatrical' aspects of upper class life (as it is presented) are just simplified signifiers for the kids. But there must be superior qualities in the Rich bloodline: intelligence, hard work, and the ability to inspire and lead others - otherwise, where did the wealth come from? This is clearly messaged from the very first few minutes of the movie - Richie Rich's Dad owns a vast business enterprise.

This idea is what I would call a middle class meritocratic understanding of social status and wealth. It's closer to the truth. But it's not quite there: It is a middle to upper-middle class mistake to think that skill in one's profession (in other words, economic productivity) is the personal quality that moves one all the way to the top of the social ladder.

The highest (hidden) message about social status is everywhere, once you know to look for it: The command and mastery of others is considered a natural consequence of superiority which is simply understood - even as a birthright. The power is the feature. If there is any skill which is employed to "do" something, it is in maintaining and upholding this class distinction by, say, employing the very method described in this post! You get a bit of this in how the "professor" character is presented - while clearly possessing the greatest technical skill (merit), he is below the Riches, to the point of taking instruction from their child.

So how is this three-level understanding of Richie Rich self-reinforcing?

• The highest level has a vested interest in emphasizing either meritocracy or the buffoon-image of the rich, because it keeps the middle class and lower class either busy or looking in the wrong direction, respectively.
• The middle class doesn't bother to correct the lower class understanding of wealth because they believe that would involve insulting them for no constructive purpose: "It's not about fancy classical music, it's about merit!"
• The lower class cannot accept either the meritocratic middle or masterly top's conception of social class because they must then think of themselves as either unskilled (middle) or without value except to be commanded (high).

This is a quick sketch illustrating how the multi-level structure of Straussian Memes can work. I believe it is imminently possible to bundle up three messages into a single meme / image through clever double(triple, quadruple)-entendres. And I think we are likely to see more of this in the near future, even created by AIs. But that is the subject of my other post.

2025-12-29 02:57:34

Published on December 28, 2025 6:57 PM GMT

This work was done as part of MATS 7.1

We recently added support for training and running Matching Pursuit SAEs (MP-SAEs) to SAELens, so I figured this is a good opportunity to train and open source some MP-SAEs, and share what I've learned along the way. Matching pursuit SAEs are exciting because they use a fundamentally different method to encode activations compared with traditional SAEs, and is a direct implementation of the classic matching pursuit algorithm from dictionary learning. The matching pursuit encoder is highly nonlinear, and should thus be more expressive than a traditional SAE encoder.

In this post, we'll discuss what MP-SAEs are, and some tips for training them successfully. We train two MP-SAEs at different L0s on Gemma-2-2b, and evaluate them against BatchTopK and Matryoshka SAEs that have the same L0 as the MP-SAEs. All SAEs trained as part of this post are available at huggingface.co/chanind/gemma-2-2b-layer-12-matching-pursuit-comparison and can be loaded using SAELens.

My main takeaway is that while MP-SAEs are exciting for researchers working on improving SAEs, I would not recommend them for practical use in LLM interpretability; or at least, it shouldn't be the first thing you try. MP-SAEs outperform traditional SAEs at reconstruction, but I do not see evidence that this results in a better SAE for practical tasks, and they are slower to train and run than traditional SAEs. MP-SAEs also seem to suffer more from feature absorption than traditional SAEs, likely due to their more expressive encoder. That being said, these is just my thoughts after training a few MP-SAEs on Gemma-2-2b, and this is not a rigorous analysis.

Regardless, I think MP-SAEs are a great addition to the set of SAE training techniques, and are especially exciting as a future research direction. In general, I am very supportive of finding ways to bring more traditional dictionary learning techniques to the SAE / interpretability world.

What is a Matching Pursuit Encoder?

An MP-SAE can be thought of as a tied TopK SAE, where the K latents are selected in serial rather than in parallel, and the K is dynamic per sample. At each iteration of the algorithm, the latent with the highest dot product with the reconstruction residual is selected, and the latent is projected out of the residual. This is repeated until the reconstruction error of the SAE is below residual_threshold, or the SAE selects the same latent multiple times. In SAELens, we add an additional stopping condition, max_iterations, to cap the worst-case runtime of  the matching pursuit algorithm.

Training MP-SAEs on LLMs (in a reasonable amount of time)

For the LLM experiments in this post, I trained MP-SAEs on Gemma-2-2b layer 12. Each SAE has 32k width and is trained on 300M tokens from The Pile. The key difficulty training MP-SAEs is that training can be extremely slow. The serial nature of matching pursuit does not mesh well with training on GPUs, since GPUs are optimized for parallel, not serial, computations. Furthermore, each iteration of the matching pursuit algorithm uses as much compute as a full sae.encode() call in a traditional SAE. The more iterations that are required to encode a batch of activations, the slower the MP-SAE is. For instance, I found that if I do not set max_iterations and residual_threshold, MP-SAEs can easily take 100+ hours to train on an Nvidia H100 GPU (compared with ~2 hours for a comparable traditional SAE)!

I trained two MP-SAEs, a lower-L0 MP-SAE with residual_threshold=50, max_iterations=300, and a higher-L0 MP-SAE with residual_threshold=30, max_iterations=400. The lower-L0 SAE ends up with L0 ≈ 85, and the higher-L0 SAE ends up with L0 ≈ 265. SAELens also has an option, stop_on_duplicate_support, that can be set to False to turn the MP-SAE into a true "serial TopK" SAE, where the SAE will always run max_iterations iterations for every sample. In the rest of this post, I refer to this as a "static" MP-SAE. I also trained a static L0 variant of an MP-SAE with L0=85. Notably, the static variant is what is implemented by the excellent Overcomplete library. The MP-SAEs trained in this post have the following hyperparameters:

To compare with these SAEs, I trained BatchTopK SAEs and BatchTopK Matryoshka SAEs, at both L0=85 and L0=265. The Matryoshka SAEs have inner group sizes of 2048 and 8192. The comparison SAEs are otherwise trained identically to the MP-SAEs (same dataset, same width, same number of tokens, same H100 GPU). Training time for these SAEs is shown below.

The MP-SAEs train much slower than the traditional SAEs due to the serial encoder. ~24 hrs isn't a completely unreasonable amount of time to train an SAE, but it means that it's hard to train a MP-SAE on a large number of tokens (300M tokens is not much, SAEs are often trained on 1B+ tokens) . The training time scales with the max_iterations parameter, so the "static" variant with a fixed 85 iterations per sample trains much faster than the other variants. It's also possible that there are more performant implementations of the matching pursuit algorithm that could speed things up. If anyone reading this a PyTorch performance expert, pull requests are welcome!

MP-SAEs have impressive reconstruction

To measure reconstruction, I calculated the variance explained for each SAE. Results are split between L0=265 SAEs and L0=85 SAEs since comparing reconstruction is only valid when SAEs have the same L0.

In all cases, the MP-SAEs have better reconstruction than the traditional SAEs, and Matryoshka SAEs have the worst reconstruction. Getting better reconstruction does not necessarily mean the resulting SAE is better for interpretability, however. Gradient descent can find degenerate ways to improve reconstruction at the expense of SAE quality.

Interestingly, the static MP-SAE variant seems to have slightly better reconstruction than the standard MP-SAE despite training more than 3x faster. This a good sign that using the static variant does not harm the resulting SAE.

MP-SAEs underperform at K-Sparse Probing

K-sparse probing is common evaluation of SAE quality. I personally like to use the k-sparse probing tasks from the paper "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing", as it contains over 140 sparse probing datasets to evaluate on (implemented as a pypi library called sae-probes). Below are k=1 and k=16 sparse probing results for all SAEs:

For both k=1 and k=16 sparse probing, all MP-SAEs score worse than the traditional SAEs by a notable margin. This implies that MP-SAEs may be improving reconstruction by finding degenerate solutions rather than by better learning the underlying features of the model.

MP-SAEs seem very susceptible to feature absorption

I was particularly excited to train MP-SAEs on LLMs to see how they perform on the SAEBench feature absorption metric, as the Matching Pursuit SAEs paper motivates the MP-SAE architecture as a way to handle feature hierarchy, and implies that MP-SAEs should solve feature absorption. The SAEBench feature absorption rate is shown for each SAE below:

Sadly, I do not see any evidence that MP-SAEs reduce feature absorption. On the contrary, on the SAEBench absorption metric, MP-SAEs score much worse than traditional SAEs, implying they are actually more susceptible to feature absorption than vanilla SAEs. The Matryoshka SAEs score the best on feature absorption, as is expected since Matryoshka SAEs are explicitly designed to solve absorption.

It's possible that there's something unique about MP-SAEs that makes the SAEBench absorption metric invalid, but I can't think of what it would be (if anyone finds an error, please let me know!). However, scoring poorly on feature absorption is consistent with the results above showing that MP-SAEs have better reconstruction than traditional SAEs. Feature absorption can be viewed as a degenerate strategy to improve the reconstruction of the SAE at a given L0, so if MP-SAEs are better able to engage in absorption then we should expect that to result in a higher reconstruction score, which is consistent with what we see.

Final Thoughts

Training MP-SAEs

Prefer Static MP-SAEs

I don't see any downside to using the static variant of MP-SAEs (set residual_threshold=0, stop_on_duplicate_support=False, and set max_iterations to the target L0 of the SAE). This dramatically speeds up the training time of the MP-SAE and does not seem to result in an obviously worse SAE. This is also the version used by the Overcomplete library. 

Should latents be forced to have unit norm?

In the SAELens MP-SAE implementation, we initialize the decoder to have unit norm but do not enforce this throughout training. This is based on the MP-SAEs reference implementation, which also does not enforce unit norm latents during training. 

However, it seems like for the lower-L0 MP-SAEs, the decoder norm drops below 1.0:

Does this indicate the SAE is finding a degenerate way to improve reconstruction loss by somehow intentionally using latents below unit norm? Or is this a valid way to avoid superposition noise? Should we enforce that the decoder must have unit norm throughout training?

Dead latents

I was surprised to find there were no dead latents in any of the MP-SAE runs, despite not having any auxiliary loss to avoid dead latents. I'm not sure if this would still be the case if the SAE was much wider (e.g. 100k+ latents). If you train a very wide MP-SAE and find that there are dead latents, it may be necessary to add an aux loss to training. 

Why no SCR/TPP evals?

I also tried running the SAEBench SCR and TPP evals, but found they were too slow to be practical for MP-SAEs. It seems like these evals assume that the SAE encode method is very fast, so these benchmarks probably need to optimized to run on MP-SAEs in a reasonable amount of time. I didn't dig into this, but there's likely some easy optimizations available to enable these benchmarks to run on MP-SAEs if someone wants to look into that.

What do MP-SAEs learn?

I did not try to figure out if the features learned by MP-SAEs and traditional SAEs are different, but I would expect there are meaningful differences. I would be particularly curious if MP-SAEs learn more and/or different high-frequency latents than traditional SAEs. I would also be curious if they behave differently in the presence of feature manifolds to traditional SAEs.

Should you train a MP-SAE?

Based on this investigation, I would not recommend using MP-SAEs if your goal is to use SAEs for interpretability work, or at least it shouldn't be the first thing you try. BatchTopK/JumpReLU seems like a better choice in terms of training time and practical performance. Matryoshka BatchTopK SAEs are also a great choice although there are more hyperparameters to set.

If you are a researcher working on improving SAE architectures, then I think MP-SAEs are very exciting, as the MP-SAE encoder works in a fundamentally different way than traditional SAEs. It may be possible to create some sort of hybrid between a MP-SAE and a standard SAE that mixes the benefits of both architectures, for example, or maybe it's possible to create a Matryoshka MP-SAE to deal with feature absorption.

Just give me the SAEs

All the SAEs in this post are available at https://huggingface.co/chanind/gemma-2-2b-layer-12-matching-pursuit-comparison. These SAEs can be loaded with SAELens v6.26.0+ as follows:

from sae_lens import SAE

sae = SAE.from_pretrained(
	"chanind/gemma-2-2b-layer-12-matching-pursuit-comparison",
	"matching-pursuit/l0-85",
)

For the other SAEs, replace "matching-pursuit/l0-85" with the path to the SAE in the repo. Each SAE on Huggingface also includes the runner_cfg.json used to train the SAE if you want to see exactly what training settings were used.

Try training MP-SAEs!

SAELens v6.26.0 now supports training and running Matching Pursuit SAEs. Give it a try! Also check out the Matching Pursuit SAEs paper "From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit".

2025-12-28 23:51:16

Published on December 28, 2025 3:51 PM GMT

Here’s everything I read in November 2025 in chronological order.

• The case for overseas dating and marriage: The man gets a loving wife and the woman gets a massive improvement in quality of life.
• Kuai Kuai culture: Translating to “behave behave”, Taiwanese engineers will put green and unexpired Kuai Kuai snacks near their equipment in hopes that it will behave better.
• *Don’t Be a Feminist*: The Origin Story

• Miracle at the Meadowlands:

  For the Eagles, the victory snatched from the jaws of certain defeat served as a morale boost, leading that season to a playoff berth and, two seasons later, the franchise’s first Super Bowl appearance. To Giants fans, it was the nadir of a long era of poor results, but the aftermath of this would lead to major changes that proved beneficial for the franchise in the long run. For the sport in general, the main legacy of the game was its contribution to the adoption and acceptance of the quarterback kneel as the standard method for winning teams in possession of the ball to end games under the appropriate set of circumstances.

• Betel nut chewing
• (Report) Evaluating Taiwan’s Tactics to Safeguard its Semiconductor Assets Against a Chinese Invasion
• Where’s Putin? How The Kremlin Hides His Location With Three Nearly Identical Offices

• Chicago Boys:

  a group of Chilean economists who rose to prominence in the 1970s and 1980s. Most were educated at the University of Chicago Department of Economics under influential figures like Milton Friedman, Arnold Harberger, and Larry Sjaastad, or at its academic partner, the Pontificia Universidad Católica de Chile. After returning to Latin America, they assumed key roles as economic advisors in several South American governments, most notably the military dictatorship of Chile (1973–1990), where many attained the highest economic offices.[1] Their free-market policies later influenced conservative governments abroad, including those of Ronald Reagan in the United States and Margaret Thatcher in the United Kingdom.

• Destruction of Syria’s chemical weapons
• Don’t let people buy credit with borrowed funds
• Two can keep a secret if one is dead. So please share everything with at least one person.
• At Pope Leo’s Urging, Bishops Issue Historic Rebuke of Trump’s Raids
• Brightline is Actually Pretty Dangerous: “Brightline is about 20x more deadly per passenger-mile (counting people inside and outside the vehicle) than driving”.

• Snow Shovels & Singlespeeds:

  A decade ago I probably cared more about optimization, maximization, efficiency and outcomes. Carbon bikes, fast times, race results. Now as a middle-aged athlete and human, I find myself increasingly more interested in the means than the end. That might sound like a cop-out in response to my waning peak physical abilities. But I think such an attitude is also just the result of a natural maturation as one goes through life.

• ‘Scarcity and growth are oppositional’: How streetwear legend Supreme lost its luster: The headline says it all

• Karin Immergut:

  American lawyer who has served as a United States district judge of the United States District Court for the District of Oregon since 2019. She has concurrently served as a judge of the United States Foreign Intelligence Surveillance Court since 2024.

• How much of your life are you selling off?
• A Common Habit That Costs Us Friends: Never reaching out

• On Free Speech:

  So to conclude: censorship in public spaces bad, even if the public spaces are non-governmental; censorship in genuinely private spaces (especially spaces that are not “defaults” for a broader community) can be okay; ostracizing projects with the goal and effect of denying access to them, bad; ostracizing projects with the goal and effect of denying them scarce legitimacy can be okay.

• Bullshit Jobs:

  postulates the existence of meaningless jobs and analyzes their societal harm. He contends that over half of societal work is pointless and becomes psychologically destructive when paired with a work ethic that associates work with self-worth.

• James Duane:

  an American law professor at the Regent University School of Law, former criminal defense attorney, and Fifth Amendment expert. Duane has received considerable online attention for his lecture “Don’t Talk to the Police”, in which he advises citizens to avoid incriminating themselves by speaking to law enforcement officers.

• High-Density Days
• When the Job Search Becomes Impossible: Three Phases of Burnout
• Susan Monarez: “American microbiologist and public health official who served as the Director of the Centers for Disease Control and Prevention”.
• You can try to like stuff
• How I Eat
• 10 minutes is ~1% of your day: How do your emojis stack up?
• Do you like dogs, cats, both, or neither?
• “It’s a 10% chance which I did 10 times, so it should be 100%”
• e to the pi Minus pi
• Work culture creep: The environment part seems huge to me. This motivated me to change my work outfit to something much more professional to allow me to shift to my “home mindset” by changing clothes when I get home. Report to be published early 2026.
• Make product worse, get money: A similar argument seems to get made by believers of planned obsolescence, where companies make products last just long enough for the consumer to say “okay, that lasted a long time better go get a new one”. The risk of it getting out that they deliberately planned the lifespan and the free market encouraging cheaper prices and/or longer lifespans seems to go against that here. Combine this with just how big the market is and it’s probably in the company’s best interest to attract new consumers than force existing ones to pay for a new product.
• Rich Friend, Poor Friend: “So this dynamic emerges where my rich friends never ask each other for help, pay for services using money, and never do anything unpleasant for each other, whereas my poorer friends are always doing stuff for each other out of necessity and becoming closer knit in the process.”
• dissolution
• 2025 U.S. Department of Justice resignations: Integrity isn’t dead! Thursday Night Massacre seems appropriate here.
• 51 days in a Russian jail: Sofiane Sehili reveals all on his trans-Eurasia record attempt... and its spectacular failure: Sehili is one of, if not the, best ultraendurance cyclists out there.
• How a chip is designed: See Semiconductor Fabs I: The Equipment and Semiconductor Fabs II: The Operation for how a chip is made.
• What Cost Variety?
• Splash (otter): Search-and-rescue otter trained for police usage. Apparently otters can “detect scents underwater by blowing bubbles and quickly re-inhaling them; the inhaled bubbles absorb odors from the surrounding water.”
• Just hiring people” is sometimes still actually possible
• Ophidiophobia: Fear of snakes.
• “Et tu, Ilya?”: Trying to make the case that Ilya was jealous of Sam’s achievements and that’s why he tries to oust Sam.
• Chuck Hagel: 24th United States secretary of defense from 2013 to 2015 in the administration of Barack Obama.
• Neil Wiley
• The Mainstreaming of Loserdom: Yeah, sorry, not having hobbies isn’t cool. Go outside, use your brain, or do something with your hands.
• Oliver (chimpanzee): “chimpanzee once promoted as a missing link or “humanzee” due to his somewhat human-like appearance and a tendency to walk upright.” See also humanzees.
• Claude 4.5 Opus’ Soul Document
• Underrated reasons to be thankful V: Dynomight’s fifth edition of his Thanksgiving classic.
• Federal prosecutors in Eric Adams case resign after being put on administrative leave: Integrity isn’t dead!
• Damian Williams (lawyer): “served as the United States attorney for the Southern District of New York from 2021 to 2024. He has been involved in the prosecution of numerous high-profile individuals, including Ghislaine Maxwell, Sam Bankman-Fried, Sean Combs, Mayor Eric Adams, and U.S. Senator Bob Menendez.”

• Richard C. Wesley:

  Wesley has described himself as “conservative in nature, pragmatic at the same time, with a fair appreciation of judicial restraint,” adding that “I ... have always restricted myself to what I understand to be the plain language of the statute. ... As long as the language is plain, we should restrict ourselves.”[6] He aims to write opinions that satisfy what he calls the “Livonia Post Office test”—that is, they are understandable to his neighbors back home.

• Mamdani, Trump Meeting Wasn’t Just Smiles
• The “tasting day”: why buying 5 babkas at once is an underrated source of meaning: I’ve done similar but with walking to each place—I call it a food crawl. Choose your food, find X restaurants within walking distance of each other, and get walking (and eating). I find the walking between leads to great convos, fun discoveries, better digestion (see verdauungsspaziergang), and less guilt about all the delicious food you just ate.
• Stop Applying And Get To Work
• Alpine Starts
• A Day’s Bookends
• Not stepping on bugs
• emails i’ve sent
• how to actually adjust your sleep schedule
• You’re WIBNO: “Warmly invited by not obligated”. I’ve been searching for something like this for a long time and have finally found it.
• tips on packing for a trip effectively*
• friendships shouldn’t be seen as ledgers of obligation
• Billionaires spending lots of money on things is consistent with how you (probably) live your life
• prompts to stare into the abyss
• project ideas i hope someone steals from me: The typesetting ones are awesome, but (probably) require sooooo much work. Now I wonder how a SOTA LLM would do at typesetting something in LaTeX or similar. A FancyBookGPT that outputs an entire book given text would be pretty neat.
• Snake Island (Ilha da Queimada Grande)
• The Planes, Soviet Trains, and Rare Automobiles of North Korea
• Andrei Lankov: Russian North Korean expert based in South Korea.
• A pattern to the best events I’ve run
• How to Clean when you Hate Cleaning: A straightforward guide to cleaning for those that either hate or don’t know how to clean.
• Is it time for Post-Stoicism?
• Various ICBM speeds animated: The whole “hitting a bullet with a bullet” explanation didn’t really click for me until I saw this—missiles move insanely fast. Couple that with multiple warheads per missile (see multiple independently targetable reentry vehicle (MIRV)), decoy missiles, etc. and you have a really difficult problem to solve.
• ICBM address: “hacker slang for one’s longitude and latitude (preferably to seconds-of-arc accuracy) when placed in a signature or another publicly available file.”
• Interiors can be more fun: Ideas on how to make interiors less boring and more fun.
• Favorite quotes from “High Output Management”
• Question the Requirements
• Sanjay Shah: “a British trader who was sentenced by a Danish court in 2024 to 12 years in prison for tax fraud, the heaviest penalty ever handed out in Denmark for a fraud case.”
• Two easy digital intentionality practices: The first is go for a walk without your phone (I think this can be more generalized into “do something without your phone”) and the second is to switch phones with the person you’re with. I found the first surprisingly hard not because of willpower, but sheer habit. My phone lives on my person and it feels weird to not have it with me when leaving the apartment.
• “You’re not sick enough for this medicine.”
• The Bleach Bottle is Empty: Learned helplessness can start in childhood and follow you to adulthood.
• Always mask at airports: Especially when in lines and during taxiing. The “if you don’t do it all the time it’s worthless” argument continues to fall flat: total viral load matters! Any reduction in the amount of contact, whether by space or mask, is better than no reduction, hence masking being worth it.
• Curtis Priem: Nvidia cofounder. “In November 2023, Forbes estimated Priem’s net worth to be approximately $30 million; if he had retained his shares in Nvidia, Forbes estimated that Priem would have been worth $70 billion.”
• Things I Learned from the Fatima Discourse™
• Ilya Sutskever deposition: Some extra details about the OpenAI coup and what led up to it. I find the constant objections and bickering humorous. It’s also interesting that Ilya doesn’t know who’s paying for his legal counsel. Maybe it’s a future superintelligence ensuring he doesn’t go broke on his path to creating said superintelligence.
• Birthday on the Charmoz
• You’re always stressed, your mind is always busy, you never have enough time: It’s amazing how screens have captured and held our attention with a one-way ratchet that’s incredibly difficult to break out of.
• A theory of performative engagement: or, how power actually works on Twitter and Substack: I constantly feel like this is the case with LinkedIn, Substack, and Twitter replies, especially those starting with platitudes like “X, thanks for posting this. Here are my very generic thoughts on it to increase the number of views my account gets in hope that someone rich and powerful sees”.
• The Kill Pause: Kirkpatrick discusses the tragic death of Balin Miller, a climber who died after rappelling off the end of his rope. It’s amazing what fatigue, competence, and impatience can cause some people to do.
• Finding It

2025-12-28 23:48:26

Published on December 28, 2025 3:48 PM GMT

Google is the Water Cooler of Businesses

Google is where the reputations of businesses are both made and broken. A poor Google score or review is enough to turn consumers away without a second thought. Businesses understand this and do whatever they can to earn the precious five stars from each customer: pressuring you in person or via email to submit a review, creating QR codes to make it easier to review, giving you a free item, the list of both ingenuity and shadiness (and sometimes both!) goes on. Businesses' response to a poor review can help them look good to potential customers or confirm the review's accusations.

In a world with no reviews, consumers go into everything blind. They have no clue what to actually expect, only what the business has hyped up on their website. The businesses are also blind. They operate in a feedback loop that is difficult to get information.

The power ultimately lies in the consumer's hands, just like South Park's Cartman thinks. And with great power comes great responsibility.

(The rest of this essay assumes the reviewer is a reasonable, charitable, and kind person.)

Helping Everyone Out

Leaving as many honest, descriptive reviews as possible provides information for both the business and other potential customers to make decisions off of. Businesses can take the feedback and improve off of it, guarding against another potential review having the same piece of not-positive feedback. Customers can decide to not eat there, sending a silent signal to the business that they're doing something wrong. But what? Is it the prices? The dirty bathrooms? The fact that they require your phone number and spam you even though they literally call out your order number? How does the business know what exactly they're doing wrong?

The reviews! The businesses have to have feedback, preferably in the form of reviews, to know and improve on what they did wrong, and the only party that can give them that is the consumer.

Other businesses can also learn from reviews, both directly and via negativa. Business A can look at reviews of business B to figure out what they're doing wrong and fix it before it comes to bite them.

In the end, everyone is better off for it. Customers get better businesses and businesses get more customers because they're now better businesses. The cycle repeats itself until we're all eating a three-star Michelin restaurants and experiencing top-notch service at all bicycle shops.

Rating Businesses

I'm still slightly undecided on how to rate businesses. Do you rate them relative to others in their class (e.g., steakhouse vs. steakhouse, but not steakhouse vs. taco joint)? Do you aim to form a bell curve? Are they actually normally distributed? Is five stars the default, with anything less than the expected level of service or quality of product resulting in stars being removed?

In the end, I think you have to rate on an absolute scale (which should roughly turn into a bell curve, although maybe not entirely centered). The New York Times food reviewer Pete Wells has a nice system that helps him rate the restaurants he visited:

1. How delicious is it?
2. How well do they do the thing they're trying to do?

But that's just food. What about for all businesses, like a bicycle shop or hair salon or law office? I choose a weighted factor approach of:

• Job Quality (70%): This is the reason the business exists. A bicycle shop exists to sell and repair bicycles. If they did a kickass job, regardless of other factors, then the review should primarily reflect that. This includes things like speed, price, etc. If the job was slow compared to what was advertised or the quality did not meet the price paid, then that is poor quality. (These things should obviously be known or estimated before agreeing to start the job so there aren't any surprises or disappointments.)
• Service (20%): Did you enjoy doing business with them? Did it make you want to come back? Job quality can only compensate for poor service so much.
• Vibes (10%): Are the vibes cool? Do you like what they're doing and want to support them?

These weights may vary person-to-person, but I'd argue not by much. If they do, the priorities are probably wrong.

Structure of Good and Bad Reviews

How a review is structured matters because you get about five words. The important points should be up front with the minor points at the end.

Excellent experiences that are worthy of four or five stars should start positive in order to reinforce what the business is doing well and serve as a quick snippet for why others should come here. Any minor negative points should be at the end.

Here are two examples of five-star reviews for NMP Cafe, one high-quaity and one low-quality:

• HQ (5 stars): Delicious coffee (I had the latte), kind staff, and a cozy atmosphere that's great for both working and socializing. Music was a tad loud for my taste, but others didn't seem to have a problem with it.
• LQ (5 stars): Fine coffee shop. Music loud.

Poor experiences should start negative in order to directly explain what the business is doing poorly and serve as a quick snippet for why others should not come here. Positive points should come after.

Here are two examples of two-star reviews for NMP Burgers, one high-quaity and one low-quality:

• HQ (2 stars): Burger topping bar had flies buzzing around and was generally dirty. Cashier grabbed inside of cup with fingers. Burgers and fries were otherwise good.
• LQ (2 stars): Unhygienic food storage and staff practices. Food otherwise good.

All this said, leaving an X-star-only rating with no text is still better than nothing because it's some information. The owner may even be able to tie it back to the reviewer and learn from it.

In-Person Reviews

In-person, so effectively private, reviews should become more normalized. (These are in addition to online, public reviews.)

Opening up a real-time dialogue line between the customer and business rep allows for more effective communication to be had through answering questions, clarifications, etc. And there shouldn't be any awkwardness! The customer is essentially giving the rep a chance to do better and make even more money from happier future customers!

My approach in the few times I've done this is to politely ask for a manager, start with a simple "hey, I'd like to give you some polite feedback on X" (and maybe make it clear I'm not looking for a free anything), then kindly explain my position. They've always been outwardly receptive and appreciative of the chance to listen and talk. Experiences may vary.

Go Earn Your Google Points

Do it for your family, friends, and neighbors. Do it for the business owners that want to do better. Do it for the guy who was gonna experience a nasty meal, but because of your review—yes, your review—didn't. Do it for the business owners who are constantly asking for feedback on their product and the experience because they're struggling, but never get anything. Do it for the chance to become an influencer or food critic. Do it for the clout. Do it for your future self.

See Also

• Social Defection: Rules separate us from the animals
• The tyranny of the “best” - be wary of what strictly dominates