2026-08-27 16:47:17
I helped hundreds of unemployed people with their job search as COO of School of Hard Knocks.
We had a training module called "the difference between a dream and a plan".
A dream is saying you want to run a bar in Spain some day. A plan is learning Spanish, understanding the licensing laws, saving up a deposit and scouting locations. Many attendees had the former and almost none had the latter.
I'm sometimes reminded of this by global health 'scaling plans'. Ultra Philanthropy's Mid-Stage Global Health Fund, which I manage, aims to bring a cost-effective health intervention to scale by 2029, so I assess a lot of them.
As an UHNW advisor, I also see a fair number of growth pitches from animal charities.
We usually talk about scale as a question of ambition and money. These organisations have plenty of ambition, and I expect the Funding Anthropalypse will soon give them enough money.
What's missing is a proper plan to scale - one with professional management and leadership in it. In short, I think we need more Managing Directors and Chiefs of Staff.
Scaling models often sell funders a dream. A typical one looks like this:
This sets off alarm bells.
First, the GiveWell Top Charities are incredibly hard to beat. Any model predicting you'll beat them by two orders of magnitude is probably overstating your effectiveness or understating your real costs (in my experience, usually both).
Second, I've seen many, many organisations whose plan reads '... infinity', almost always 5+ years out. I am always suspicious of a plan where cost-effectiveness dramatically increases at a point far enough in the future that no one will check.
When those deadlines arrive, very few have hit their goal. Usually, this is because consistent delivery, team coordination and accurate monitoring all get harder as you expand, and so you need to add more bodies (and therefore more costs). There are very few roles that allow one person to oversee far more goods or services without being overburdened.
I think the scaling optimism copies the tech model, where a single repeatable product can serve hundreds, then thousands, then millions of people. However, most sectors don't work like this. A software product can serve its millionth user with the same codebase, but you can't serve a million people without more nurses.
Third, these models assume the co-founders can manage a much bigger staff, usually while doing their own work and most of the fundraising and strategy. The founders become the bottleneck, lacking either the time or the talent to do everything expected of them.
If you're a donor, treat any plan that reaches huge scale without expanding its back-office staff with great scepticism. A scaling plan without new senior hires in it is a dream with a budget attached.
Instead, we want to see reasonable hiring triggers, a clear idea of the founders' roles and a thought-through future org chart.
In the wider charity sector, the biggest blocker is usually the optics of overheads. The average donor hates overheads and wants them arbitrarily capped.
This is less of a problem in EA, where we have embraced unrestricted grantmaking for a long time. However, the pressure on cost-effectiveness at small scale remains, and this tends to discourage adequate back-office hiring.
Founders have several jobs no one else can do - defining the vision and strategy, and securing the money and people to make it happen.
But the cost-effectiveness world pushes us to keep overheads as low as possible. So co-founders, often smart generalists, end up ‘teaching themselves’ the next thing that needs doing: bookkeeping, MEL, designing their randomised controlled trial (RCT).
This is usually a false economy. You can do the basics, but the second you need to add a million contraceptive doses to your accounting inventory, you're frantically asking Claude and hoping its advice is kosher.
Autodidact founders scaling back-office functions is usually a mistake. It costs their time, produces errors and takes them away from work that only a founder can do.
Interestingly, I do not see this problem in the existential risk projects I review for clients. These organisations tend to have robust back-office functions.
I think this is because of two factors. First, although I am sure many longtermist projects wish they had more funding, it has been relatively easier to get a large cheque out of longtermist funders. Organisations have often been able to get seven-figure funding (even nine-figure funding) very soon after being founded.
Second, the astronomical stakes of 'and then we save all future humans from extinction' means that there is far less pressure on short-term cost-effectiveness than in other fields. It's inconceivable that hiring a senior MD would seriously affect your expected value if you prevent humanity's extinction.
If, as I expect, we are about to enter a world of more abundant funding in neartermist cause areas, this directly loosens constraint one, and we should strive to loosen constraint two.
Accordingly, I think a new rule of thumb for animal welfare and global health interventions should be: if it's a profession and not your core skillset, buy the expertise, from a consultant, a firm or a new hire.
This shows up most often in finance, operations, HR, MEL - but also, the most neglected of all, people management.
Running organisations and teams is a profession. I’ve built staff teams twice, at School of Hard Knocks and One for the World, so I can say from experience: managing people well is incredibly difficult. It was consistently the hardest and most time-consuming part of my work.
A randomised trial in Indian textile firms also found that adopting structured management practices raised productivity by 17% in the first year. These were for-profit manufacturers, so adjust accordingly, but also see Nick Allardice, CEO of GiveDirectly’s, writing for sector-relevant material.
The title varies - Managing Director, Chief of Staff, COO, Executive Director, CEO, even Founders’ Associate - but the impact is the same: it pays to hire someone to make the organisation function.
Managing Directors run the project, directing people and systems, so the founder can concentrate on strategy, fundraising and externally representing the organisation.
Chiefs of Staff focus on delivery, keeping people on the right priorities and on schedule. They specialise in seeing a plan through from conception to completion.
Most founders aren't actually selected for their capacity to run a team or direct people's work. They excel at articulating a vision and convincing staff and money to back it.
So why don't more founders hire these roles?
One reason is status. Founders want to stay at the top of the org chart. When I see a founder with 10 direct reports, it's often because they want to remain 'above' all the directors and heads of functions.
That doesn't mean they should direct everyone's work personally. Managing Directors and Chiefs of Staff let founders keep their status without spending most of their time on line management. Hiring an MD is a 'silent promotion' - it frees founders to do the strategy, vision and fundraising that only they can do.
I've been that hire twice. School of Hard Knocks' founder took me on as their first employee; One for the World's volunteer founders hired me as the first full-time Executive Director while they stayed on the board, and we grew revenue 7x in 4 years. Both times, the founders kept the vision and the status, and the organisations grew because someone was paid to run them.
As a founder, ask yourself: "How will this project run when I only personally know one third of the staff?" It might seem fanciful, but it's entirely plausible that every grantee of our Mid-Stage Fund gets to $10m revenue in the next 3 years. At that point, the founders will have hired only a fraction of the team and will probably see the whole organisation together once or twice a year. This is a significant organisational change.
A second reason is fear of bureaucracy. Tech readers will hear this as 'manager mode', the thing 'founder mode' warns against. But that failure belongs to product companies; delivery charities fail the other way, with the vision intact but the service late or substandard.
However, I think the most common cause of underhiring is the desire to keep costs low to improve short-term cost-effectiveness. This is something that we can address as a community, likely improving medium-term impact.
I think so, yes, for three reasons.
One, cost-effectiveness bars are falling as more money comes online. Accordingly, the threshold for being 'fundable' is easier to reach.
Two, donors will adapt to this new reality. My hope is that this post accelerates that transition, but it will likely happen organically as the new reality becomes clearer (e.g. because donors see GiveWell making unprecedented grants to Top Charities, signalling the changing times).
Three, most importantly of all, because I actually think that it is more impactful to staff up properly.
Most cost-effective organisations only make sense at scale and their unit economics may not even work while small. The goal is to grow until they can compete with the most cost-effective charities in the world.
When I look at the valley of death in global health, I see very few charities that die there because they hired a handful of senior staff. I see large numbers that stalled because of burnt-out employees, inadequate risk management, lack of quality control or poor coordination - in short, just about all the reasons Nick lists here.
Scale can solve your overhead rate, but a low overhead rate can’t take you to scale. Imagine a senior Managing Director at US$200,000 per year. At $1m revenue, that’s 20% of the budget. At $10m, it’s 2%, and no one minds any more.
I'm not saying you can explode your budget next month. But, pretty soon, I expect a robust scaling plan will do more for your fundraising and delivery than a super lean model and outstanding cost-effectiveness.
And, even if you don't have the budget or scale to hire a senior executor right now, you can still set triggers for when you should. Triggers protect against both failure modes: hiring too early and overloading the budget, or overstretching the current team and hiring only when something breaks.
For donors: scrutinise your grantees' plans to scale (or hire an advisor - talk to us for a free recommendation). Ask for the org chart in 3 years' time and what the founders aim to be doing. Look out for anyone managing more than 8 direct reports (ideally no more than 5).
Be deeply sceptical of any organisation that says it can scale dramatically with a minimalist leadership and back-office team. (Be especially sceptical of anyone claiming their cost-effectiveness will go from 'competitive' to 'orders of magnitude better than New Incentives'.)
Most importantly, make it explicit that you'll pay for overhead - and that you're less likely to fund organisations that don't plan for it.
For founders: turn your scaling dream into a serious plan. Set hiring triggers in advance and map your operations, finance, HR and MEL needs as you grow. Budget a line for an MD, a Chief of Staff or both before you need it. Then get someone who's scaled something to red-team the plan.
For talented people: I hope many more of these roles are hired in the next year. If you're organised, ruthless at delivering and can manage people, put yourself forward. A great first step is joining High Impact Professionals' talent directory, so organisations can find you. (They also have an Impact Accelerator Program starting soon.) I think a lot of $1m organisations are about to become $10m organisations, and you can help them do it.
A quick non-exhaustive list:
The Funding Anthropalypse can buy programmes at ten times the scale. The cheapest thing it should buy is the people who turn those dreams into plans.
Jack Lewars is the founder of Ultra Philanthropy, an independent advisory that helps major donors give for maximum impact, and is the fund manager of its Mid-Stage Global Health Fund. He advises donors giving up to nine figures a year, and is Chair of Trustees at High Impact Athletes. Talk to him about your giving.
2026-08-27 14:37:53
TL;DR: We’re quickly scaling the Iliad Intensive, which has seen 10x applicant growth per iteration since April, and are hiring for many roles in the Education team to keep up with that growth. The post gives a snapshot of the Intensive, the roles that enable our future plans, and my opinion of what it’s like to work at Iliad.
At Iliad, we do fieldbuilding for predominantly theoretical AI alignment research across the entire spectrum. We start at the foundations with the Iliad Intensive, a four-week full-time and in-person course on largely theoretical approaches to AI alignment research, followed by the Fellowship, Fellowship extensions, and new incubated research bets, among other activities. We are now quickly scaling the Intensive (and the Fellowship too!), with three more iterations just this year and, if funding and applicant interest allows, double cohorts in London and the Bay Area each month in 2027. As the Director of Education, I’m hiring for many roles to make that growth possible!
The Intensive started in April in London, with a cohort of just ~16 people, and materials later released in the form of a Google Doc, covering modules in the clusters of alignment, learning, interpretability, agency, and safety guarantees. It benefited tremendously from the network that Alexander Gietelink Oldenziel, our executive director, built over the years: The curriculum was created by around 15 different people, all focusing on their expertise, which led to content that is both broad and fairly deep on many topics. We’ve since then run another iteration in June (~15 participants), and now in August in Berkeley (~37 participants). Monthly iterations for September, October, and November are lined up. We are integrating our 3-month fellows directly into teaching to help with capacity constraints that come with this growth.
The Intensive clearly fills an important gap in the landscape, with participants in the first iteration calling it in their testimonials “life changing”, “new and ambitious”, and “a one-of-a-kind course”, among other praise. The demand for the course has risen dramatically, with a 10-fold increase in applicants per cohort between April and September, which means we can scale up the cohort size while simultaneously increasing the admissions bar. The praise and interest are probably predominantly driven by our novelty, uniqueness, and outreach efforts. The course itself is already good, but has many avenues for improvement. We’re in the process of improving the materials and are starting to host them on a website under construction.
I think if we work hard, the Intensive and follow-up programs could establish themselves widely as the best AI alignment research courses available. To achieve this, we need to work in parallel on many fronts:
Apply here for one or several of the roles.[1]
Working at Iliad is not for everyone. Here’s a brief summary of unique features that may or may not appeal to you:
Working here requires having or learning a tolerance for change and uncertainty. Yet in my opinion it is rewarded by a sense of possibility that I’ve not felt elsewhere. Ask me any questions in the comments or directly apply here.
A short bullet-point summary of the roles:
Full-time:
Trial and Contracting:
2026-08-27 13:29:48
We anchored a single concept (red) in the residual stream of a transformer. It ended up where we wanted, with nearby colors graded sensibly, and without degrading task accuracy. Steering is next.
Earlier posts in this sequence introduced Sparse Concept Anchoring (SCA), tested in autoencoders. This post applies the technique to transformers. You don't need to have read the earlier posts to understand this one. Light revisions by Claude Fable 5, and experiments run with help from all the Claude 5s.
Information flows through a transformer from input to output via the residual stream — an internal state, kind of like a mental state. It's composed of pieces, addressable by slice (layer) and position (token). At the input, each state is a token embedding. At deeper layers, the hidden states take on abstract meaning, being progressively transformed until they reach the final layer, where they are used to predict the next token.
Those hidden states are the subject of a lot of mech interp research, because we (researchers) expect to find structure that reveals something about how the model "thinks" about the world[1].
In our experiments, we used a mini color-mixing language of expressions like "red + blue = purple". The models were trained to predict the answer, and they could do so accurately, even for pairs of colors they had never seen before.
What would you expect to see, if you looked at the embeddings for the color tokens? We fit a linear probe on all the colors except the one being plotted, and decoded the held-out color with the probe. That was repeated for all colors in the vocabulary. Here's what we found:
All 216 colors of our 6-subdivision RGB cube, decoded from embeddings with linear probes. Open rings
This model has never actually seen a color; all we showed it was a bunch of opaque tokens in color-mixing equations, and it inferred the geometry of the RGB cube.[2]
However, we had to go looking for that structure. It would be much more convenient if we could just tell the model where to put the colors, e.g. we might say "red must be in dimension 1." We can't do that for all concepts — that would leave no room for superposition, and the capacity and accuracy of the model would quickly degrade. But perhaps we can do it for a handful of concepts. That's what Sparse Concept Anchoring (SCA) aims to do.
Suppose red was actually represented in dimension 1, at (1, 0, …). If we measured the cosine similarity from that direction to every color, we should see something like this:
That is, colors that are not red should be approximately orthogonal
Per-color alignment of the first operand in unanchored models, mean over all slices and seeds. Each pixel is the true color of the input (op1), dithered to show the cube internals.
... Fair enough! It turns out that, even though the states in this model are normalized (vectors of length 1), they are all approximately orthogonal to each of the dimensions of the residual stream. Without constraint, the model learns arbitrary concept vectors, and random vectors in a high-dimensional space (
We will use two of the geometric activity regularizer terms from our autoencoder experiments: anchor and anti-subspace.[3]
Anchor pulls the hidden states toward a direction — in this case, (1, 0, ...) — but only for color equations that have a red operand (labeling and targeting is described below). Anti-subspace pushes all activations away from that same direction and its antipode.
Let's see what that does to the states at the first operand.
Per-color alignment of the first operand, mean over all slices and seeds. The gray bands are sampled from individual slices and seeds.
All anchored conditions have exact-match accuracy within 0.02 of the un-anchored baseline. But the anchor term seems to have an excessive global effect: it pulls red to the anchor direction, but it drags the other colors along with it. Adding the anti-subspace term restores the separation.
To isolate certain concepts, we're going to need some labeled samples. For safety-relevant concepts in LLMs, realistic labels might be along the lines of "this tweet contains a lie" or "this paragraph is about protein synthesis". To make our toy scenario as transferable as possible, we'll use a comparable labeling scheme:
Pure red accounts for 31% of all labeled lines, and the ten reddest colors account for 85%. Overall, about 0.12% of training lines are labeled. The model needs to learn to position, separate, and grade colors in spite of the noisy, coarse signal.
Our equation-level labels don't say which states to pull toward the anchor. In the experiments above the pull was uniform: the anchor term was averaged over all hidden states in a batch. That means that in equations labeled "red", a non-red operand and the + and = positions would all be pulled just as strongly.
The figure below shows how aligned each state is given the color of the first operand (whatever it may be).[4]
Per-(slice, position) alignment with the anchor. The height of the red lines shows measured alignment when the first operand is pure red; the grading clouds show the same thing for all colors.
Notice that all states have become aligned with the anchor: when operand 1 is red, all other positions point toward the nominated red direction. The model still has high task accuracy, but the anchor term seems to have had an excessive global effect.
Let's try to make it more targeted. We can't tell the regularizer which states to pull, but we can let it choose. Instead of using a uniform average over positions, we'll use mellowmax pooling within color equations (lines) before averaging over the batch.
It's a soft minimum function — a bit like softmax, but rather than focusing next-token probability, think of it as concentrating the regularizer loss on one or two positions. Since the anchor term is a distance, the soft minimum focuses on the states that are already closest to the anchor. Here's what pooling does to the measurements:
Alignment when using mellowmax pooling over positions. Left: per-(slice, position), keyed on the first operand. Right: mean over slices and seeds at op1.
That seems better: now when the first operand is red, most of the alignment toward the anchor is in fact at the first operand. It's unclear whether that's optimal — perhaps a different position should have been favored at deeper slices. But at least the effect isn't dominating over all positions.
Grading is also improved, with non-reds roughly as unaligned as in the un-anchored models. Again it's unclear if the grading is optimal for reddish colors. Should the relationship between redness and alignment to the anchor be linear? Quadratic? Sigmoid? We scored grading against a
Soon we will try intervening on the anchored models by projecting out the anchored direction or ablating the related weights, as we did in autoencoders. We expect to see reduced (near-random) task accuracy for equations involving red, while non-reds should be unaffected.
We will also expand the mini language to have more mixing operations, and we'll anchor those as well as the color operands.
At that point, we would like to compare SCA to other methods like Gradient Routing and steering with SAEs.
We have shown that Sparse Concept Anchoring "works" in transformers: we picked a single concept and nudged it onto an axis of our choice. Nearby concepts became partially aligned, non-red colors stayed as unaligned as in the un-anchored models, and linear probes still decode the RGB cube from anchored models roughly as well as they did before (although that last comparison is noisy). Task accuracy was unchanged.
We are looking forward to progressing to interventions (steering) and more complex models.
Our experiments are published at z0u.github.io/sca2 (source).
Or "models the data distribution", if you prefer.
The hidden states and embeddings in this model are 64-dimensional, and normalized to length 1. So the embedding space is a hypersphere, and yet the RGB cube is right there, linearly decodable.
An activity regularizer is a loss term, calculated from the hidden states (activations) rather than the model output. Ours are formulated as before: anchor is simply the cosine distance to the concept vector, and anti-subspace is the squared distance of intrusion into the subspace:
where
Figures showing the same thing but keyed by the second operand are available in this notebook.
2026-08-27 09:57:20
Site: https://rumble-34-69-187-69.nip.io/
Embedding models have gotten pretty good at capturing the meaning behind text, so I ran qwen3-embedding-8b over every lesswrong post.
You can paste in a draft and find the most similar post on the site to it. Pretty useful for checking what's already been said about a topic.
For example, I put in the text for this and the top result was: https://www.lesswrong.com/posts/vDcJHD95XCg7ywANM/i-built-a-semantic-search-engine-for-lesswrong
2026-08-27 08:11:47
The full version of our work is available at https://xiaohan2012.github.io/articles/tfm-self-repair/
Ablation is a widely-used technique in mechanistic interpretability research. It can be used to draw conclusions about the functional roles of certain computational components (e.g., attention heads, transformer layers) inside a model. Ablation works by removing or damaging the behavior of certain components in a model and recording the changes in the model's responses (e.g., output logits). For example, ablation has been used to identify circuits (Wang et al., 2022; Conmy et al., 2023).
Self-repair of a model refers to the following: when certain components in a model are ablated, other components reacts and compensates for the damage. As a result, the model's output barely changes. Self-repair has been observed in large language models (McGrath et al., 2023; Rushing & Nanda, 2024). For example, in the circuit discovery paper (Wang et al., 2022), after ablating on the back-up head, the model's prediction does not get affected. In fact, another dormant head activates and takes up the role of the ablated head. (McGrath 2023) observes layer-level self-repair in a Chinchilla models. Ablating layer X causes a downstream layer to change its behavior and the output logits stay almost unchanged.
Self-repair directly challenges the conclusions researches draw from ablation experiments. One cannot conclude that the ablated component is causally unimportant when self-repair is present. Rather, the component takes certain role. Output logits barely change because downstream component fills the gap.
In this work, we study whether layer-wise self-repair is present in tabular foundation models (TFMs). Tabular foundation models are essentially transformer models used to make predictions on tabular data (Hollmann et al., 2023; Hollmann et al., 2025). Illustrated in Figure 1, a TFM takes in two tables of the same schema: a context table with data attributes and labels and a test table with data attributes only. The model then predicts the missing labels in the test table.
Figure 1: a TFM takes in two tables: context table and test table, and predicts the missing labels in the test table.
TFMs share the application of supervised ML models for tabular data (e.g., XGBoost) and relies on a related technical backbone (e.g. transformers) as language models. We choose to focus on TFMs because it is a relatively nascent topic, yet has wide practical applications. However, mechanistic interpretability research on TFMs has not received comparable attention as their language counterparts.
The subsequent sections are structured as follows:
We follow the definition of self-repair in McGrath et al. (2023). For the given component under study, two quantities are measured:
The process is illustrated in Figure 2, where layer A is the component to ablate, and x and y are the model input and output, respectively.
a and b, called "clean values", are recorded for later use.y after ablating on A with value a' (denoted as do(A=a')). Note that downstream layer B reacts freely, therefore, its internal activation takes b', rather than its clean value b.y after ablating on A as before but fixing B to its clean value b'.y when B reacts (e.g., it takes value b') as if A is ablated (but in reality it is not).
Figure 2: Illustration of total/direct/indirect effect in a toy 2-layer transformer model. do(A=a') means applying causal intervention on component A with activation value a'. Taken from McGrath et al., 2023.
Note that to draw statistical reliable conclusion, the above process is repeated on multiple inputs (e.g., different input text for language models and different tables for TFMs). In other words, we end up with a list of (TE, DE) tuples.
Further, we define the compensation effect (CE) as CE = TE - DE [2]. It reads as the downstream components remedies for the ablation e.g., in terms of change of logit.
Why both TE and DE are needed? Suppose TE remains unchanged after ablation. There are at least two explanations (illustrated in the table below). The model either self-repairs or contains redundant information/functionality. Redundancy differs from self-repair by whether downstream components have a direct impact on model output or not, that is, whether DE > 0. That is to say, TE alone cannot tell apart self-repair from redundancy.
To illustrate the need of using both TE and DE to single out self-repair, we use a 2-layer transformer with layer
do(A=0).| Scenario | B | a+b | y | TE | DE |
|---|---|---|---|---|---|
| Redundant | 2 | 1 | 0 | 0 | |
| Self-repair | 1 | 1 | 0 | 1 |
As can be seen in the table, both TE and DE are needed to distinguish redundancy from self-repair.
The criteria for self-repair. Finally, we use the following criteria to determine if self-repair is strong in a model:
Self-repair in language models. McGrath et al. (2023) provides strong empirical evidence that self-repair is present in language models. In their study, they evaluate against a 7B-parameter Chinchilla model on 1208 factual prompts. They observe a remarkable "below-diagonal" pattern in the DE vs TE plot, as shown in Figure 3. They further identified the layer 23 as the one with the strongest repair capability. The slope is 0.69 (1 for perfect match in scale) and
Figure 3: McGrath et al. (2023) gives strong evidence of self-repair in a language model (Chinchilla 7B). Point mass lies below the diagonal, that is DE > TE. Each point corresponds to a (prompt, layer) pair, where layer index indicates node color (deeper layer -> darker color).
Does self-repair exist in tabular foundation models? Balef et al., 2026 investigate if tabular foundation models self-repair. They follow a different definition of self-repair though:
Figure 4: Balef's approach of studying self-repair: ablate/skip one layer at a time and measure layer-level performance using tabular logit lens. Left: an illustration of ablating one layer only. Dashed line connects the ablated layer and the next one. Right: results on a SOTA model LimiX-2M and every layer is ablated and shown (corresponding to a line, whose color is indicated by depth of ablated layer). The dip-and-recovery pattern prompts them to conclude that self-repair is present in TFMs
According to their criterion, self-repair happens because of the "drop-then-recover" pattern they observe across multiple layers and models. This definition differs from the one we use in the work. The key difference is their criterion does not carry the same causal meaning, e.g., whether downstream components reacts to the ablation.
Their choice implies that only total effect is measured (against output from the last layer). As we have shown before, self-repair and redundancy thus cannot be distinguished from each other. In fact, they are aware that "it is unclear whether this robustness arises from self-repair or from layer redundancy". This gap is what we want to address in this work.
Our main finding is that evidence of self-repair in TFMs is quite weak.
Setup. Our experiment is set up as follows:
We state our main results next.
There are very few points below the TE-vs-DE diagonal. We plot DE-vs-TE for all 4 models below. It can be seen that the "below-diagonal" patterns in these TFMs are much less obvious than their language counterparts (Figure 3), if non-existent at all!
Figure 5: TE vs DE plot on 4 TFMs. Each point corresponds to a (dataset, layer) pair. Point color indicated by relative layer depth (e.g., first layer is 0 and last one is 1). We do not see point mass under the diagonal, suggesting self-repair is weak in these models.
The visualization below illustrates the "meaning" of different sections in the TE-vs-DE plot above. Note that the text below is generated by AI (but verified by the authors).
Redundancy is more prevalent than self-repair. We partition the points in the above plot into 3 regions, corresponding to:
Where
| model | redundant | repaired | others |
|---|---|---|---|
| LimiX-2M | 16% | 11% | 73% |
| Mitra | 44% | 2% | 54% |
| TabICL | 41% | 0% | 59% |
| TabFM | 78% | 2% | 20% |
We can see that redundancy is more prominent than self-repair. In fact, layers in TabFM seem to be highly redundant!.
Relation between CE and DE. We further fit lines to the CE vs DE points per layer and identify the layer with the largest
The result is summarized below. Result on a language is appended as a reference.
| Model | Layer with max |
slope | |
|---|---|---|---|
| LimiX-2M | Layer 11 (12) | -0.03 | 0.00 |
| Mitra | Layer 11 (12) | +0.08 | 0.04 |
| TabICL | Layer 11 (12) | -0.41 | 0.58 |
| TabFM | Layer 18 (24) | -0.08 | 0.04 |
| Chinchilla 7B (McGrath et al., 2023) | Layer 23 (32) | +0.69 | 0.92 |
In contrast to large slope and large
In our study, we look for evidence of self-repair in tabular foundation models. We rely on the causal framework described in McGrath et al. (2023) and measure the direct effect and total effect, both of which are required to identify self-repair. Our experiment suggests a negative result.
We also point out the limitations of our work.
In the case of the Hydra effect, the component is a given layer and the output is the logit on the correct next token. For our research question, we also study at layer level, but the output is the logit on the correct label instead. ↩︎
Note that when the output layer is a linear function, TE = DE + IE and CE = -IE. ↩︎
The TabArena task ids are 363621, 363671, 363696, 363629, 363626, 363682, 363684, 363700, 363674, 363694, 363619, 363623, 363689, 363706, 363624. ↩︎
If below 1000/500 rows, we take all available samples in the dataset. ↩︎
Because resample ablation (conceptually) suffers less from out-of-distribution issues and provides more stable estimations according to our empirical observation. More details in Section 5.3 in the full version. ↩︎
We find that logit margin reveals more subtle patterns than ROC AUC (which saturates). More details in Section 5.2 in the full version. ↩︎
2026-08-27 06:38:08
I assume that you're roughly familiar with emergent misalignment in general. If not, I recommend reading these papers
https://arxiv.org/abs/2512.09742 [Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs]
https://arxiv.org/abs/2502.17424 [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs]
https://arxiv.org/abs/2506.13206 [Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models]
My source code + data: https://github.com/mild-rgb/cot-bert-analysis / https://huggingface.co/datasets/mild-rgb/bert_cot_em
TLDR; while a ModernBert CoT monitor can spot misalignment on the Thought Crime medical advice LoRA, it's barely better than chance.
Intro: This post is the first about a series of experiments I did on CoT monitoring with Emergent Misalignment. I'll update this intro with links to the later posts when they're up.
All compute was done on a Colab G4
I then trained these 3 classifiers to predict whether the answer would be misaligned:
1. Bag of Words model (TF-IDF 1–2gram + LR) [this is essentially just a word counter]
2. ModernBERT-large
3. ModernBERT-large + BoW Product of Experts (PoE)
In this particular case, PoE is when ModernBert is trained in this setup:
loss = logits(BERT + BoW)
Then BoW is frozen, so the loss is only calculated through ModernBert. This prevents ModernBert from just learning to copy BoW, so the only thing that BERT can find is subliminal signals that aren't just mono/bigrams.
Within-prompt shuffle returned an AUC of 0.5692.
Global shuffle returned an AUC of 0.5.
Conclusion: some prompts cause more misalignment than others and the bias classifier was picking up on that. While questions weren't directly shown to the classifiers, the CoTs do discuss the topic:
appropriate null (within-prompt) 0.5692 ± 0.008
BoW / bias expert v2 0.5914 → +0.022 (~2.8 sigma)
ModernBERT-large plain 0.5988 → +0.030
ModernBERT-large PoE 0.5014 → below the null
Full conclusion: in Qwen3 32B with the EM medical advice adapter, CoTs which produce misaligned answers have no subliminal signal. While lexical classifiers are slightly better than chance, this is largely due to some prompt topics being more likely to produce misalignment than other prompt topics and the prompt topic having lexical tells in the CoT.