2026-08-06 23:31:50
Every application built on stored data performs two kinds of operations against it.
A write operation records a fact, such as a new order, a changed email address, or a deleted comment. On the other hand, a read operation answers a question, such as which orders were placed this week or what should appear on a profile page. A single database on modest hardware can handle both types of operations without much trouble, and a developer need not be bothered about which operation is more common in the application’s context.
However, high traffic can change things. Let’s say a page starts to load slowly. A fix is identified, which is creating an index on the column being filtered. Now, several months later, the same page is slow again under higher load. This time the fix is a cache in front of the query. A year after that, the database saturates during peak hours. This time the fix is a read replica with reporting traffic routed to it. In principle, each fix works, but each one is needed at a different time and for a different reason.
Now, let’s assume that a user updates a profile, reloads the page, but still sees the previous value. The bug does not reproduce locally, and it disappears on its own before anyone can investigate its root cause. However, this behavior can be a direct consequence of some of the other fixes. This is because each of the previous fixes placed a copy of some data somewhere other than its source. But the copy is not updated in sync with the source update. In other words, a seemingly simple fix on the read path can impact how things appear to work on the write path.
In this article, we will look at read path and write path operations and techniques in detail. Here’s what we will cover:
Why fast reads and correct writes require opposing data structures
Precomputation and duplication, the single operation underneath every read optimization
Two different definitions of consistency, and the bugs caused by treating them as one
Indexes, denormalization, caching, read replicas, materialized views, purpose-built read stores, fan-out on write versus read, and CQRS
For each strategy, its sync mechanism, staleness window, and characteristic failure mode
Write-heavy systems, where the ratio inverts and the decisions reverse with it.
2026-08-05 23:30:28
AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.
Join live on Aug 19 (FREE) to learn:
The 4 metrics to measure where AI gains leak out before production.
The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is
Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.
The most capable AI models are also the most expensive to run. They need specialized hardware, they consume large amounts of memory, and they add cost and delay to every request they handle.
These traits make them hard to deploy in places where resources are limited, such as a mobile device or a service that handles heavy traffic and needs fast and low-cost responses.
There is also a second fact that sounds backward at first. A small model can sometimes match or beat a much larger model on a specific task, even when the small model learned everything it knows from the larger one. On an intuitive level, a model trained on another model’s output would seem to inherit a ceiling rather than break through it, yet the results are different. The method that makes this work is called knowledge distillation, and it has become a standard part of how production AI systems get built.
In this article, we will walk through the idea from the ground up. The main points we will cover are as follows:
What distillation is, and how it differs from compression.
Why learning from a model’s output can beat learning from raw labels.
The three main methods, and which one dominates?
What distilled models achieve in practice.
Where the method breaks down, and where it is heading next.
Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models:
The first is a large, capable model called the teacher.
The second is a smaller model called the student, which is trained to reproduce the teacher’s outputs.
Once training finishes, the student runs on its own, and the teacher steps out of the picture.
A common assumption is that the student is the teacher in compressed form. The reality, however, is different.
Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter.
Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher.
One operation shrinks an existing model. The other trains a fresh one. The payoff is practical, since a small student can run inside a single service or on a phone, respond in less time, and cost far less for each request, and in some cases, it can run on the device itself without sending data elsewhere.
This method is now standard practice. For example, Google’s Gemma models are built using distillation during training, drawing on a larger model in the Gemini family. The two ideas also work together in sequence. A model is often distilled first to produce a smaller capable model, then quantized to shrink that model further for a specific device.
Keeping the distinction clear matters because it affects how we understand further concepts. A compressed model carries a copy of the original inside it. A distilled model is a separate thing that was trained to act like the original, which is exactly why it can sometimes behave in ways the original would not.
If the student only copies the teacher, why does copying work so well?
The answer is in what the teacher hands over.
Learning from a model’s output beats learning from raw data because the output carries more information than a plain answer.
Standard training data gives one answer per example. An image of a cat carries the label “cat,” and the model is rewarded for producing “cat” and penalized for anything else. A teacher model offers something richer. Instead of a single answer, its output is a set of probabilities across the options, such as cat at 0.70, dog at 0.25, and fox at 0.05. That full set of probabilities is called a soft label, in contrast to the single hard label found in ordinary data.
The extra numbers carry additional information. They show that the teacher’s output ranks dog as a plausible alternative and fox as a distant one, which says something about how the categories relate to each other. Researchers sometimes call this dark knowledge, meaning the structure hidden in a model’s confidence that a bare label leaves out.
During training, the student works to match this distribution. It is scored on how far its own probabilities sit from the teacher’s, and training pushes it to close that gap. In other words, the student learns the teacher’s whole pattern of confidence rather than a single right answer, and that pattern is a stronger training signal than a one-word label.
This is the core reason distillation works as well as it does. A single correct label discards the relationships between options, and soft labels keep them.
An early result showed the practical payoff, since a student could reach good performance from far fewer examples when trained on soft targets, because each example now carried more than a single answer. The original 2015 work added a control called temperature for exactly this purpose, where a higher temperature spreads the probabilities out and exposes more of that fine structure for the student to learn.
With the mechanism clear, the next question is how this gets done in practice, which has more than one answer.
Distillation comes in three main forms, and they differ in what the student copies:
Output distillation: The student matches the teacher’s final outputs, including the soft labels described above. This is the original form from 2015 and the most direct one.
Feature distillation: The student matches the teacher’s internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. The aim is a similar internal picture, not only a similar output. Google’s EmbeddingGemma is trained this way, learning to produce internal representations close to those of a larger Gemini model.
Synthetic data distillation: The teacher generates a dataset of examples, and the student is fine-tuned on that dataset the same way it would be trained on any ordinary data. Stanford’s Alpaca was an early case, fine-tuned on examples produced by an existing large model to improve how well it followed instructions.
The third form has become the most common approach in practice, and part of the reason comes down to access.
Many strong models are reachable only through an interface that returns text, with their internal values and probabilities kept private. When those internals are out of reach, generating data is the route that still works.
The three forms also differ in what they require. Output distillation needs the teacher’s probabilities, feature distillation needs access to its internal values, and synthetic data distillation needs only the text the teacher produces, which is why it travels the furthest across closed models.
These methods can also be combined. A single training run might use a generated dataset alongside soft labels, and newer methods mix teacher and student generation during training.
These methods are not only theoretical. The next section shows what they produce.
The results in practice are strong, with one important qualifier.
A clear example came in early 2025 from a lab called DeepSeek. It used a large reasoning model to generate a set of training examples, then fine-tuned several existing smaller models on those examples. One result stood out.
A 7-billion-parameter student scored higher than a 32-billion-parameter model on a competition mathematics benchmark, even though it was produced by plain fine-tuning on the larger model’s outputs. The released family of distilled models ran from 1.5 billion parameters up to 70 billion, and the smaller ones were compact enough to run on a single graphics card, which is part of why the release drew so much attention. The practical effect was that strong performance on these narrow tasks became something a small team could run locally and cheaply, rather than only through a large hosted model.
The qualifier matters as much as the headline.
These wins tend to appear on narrow, well-defined tasks such as mathematics and code. On those tasks, a small distilled model can perform at a level its size would not suggest. Across broader measures of general knowledge, the same small models still trail the larger ones. For example, a model can become excellent at competition mathematics through distillation while remaining weaker at wide-ranging questions about the world. Therefore, a claim that a small model beats a large one is usually true in a specific, narrow sense.
If the results are this good, the natural question is where the method falls short, which the next section takes on directly.
Distillation has clear limits, and they matter when deciding whether it fits a given problem.
A ceiling effect from the teacher: A student trained on a teacher’s output tends to stay at or below the teacher’s level on the kind of data they saw. When the teacher produces a wrong answer, the student learns that wrong answer along with the right ones. The teacher’s quality sets the bar, which makes the choice of teacher one of the most consequential decisions in the process.
A wider gap can hurt: A larger, stronger teacher does not always produce a better student. When the gap between teacher and student is very wide, transfer can degrade, because the student has too little capacity to absorb everything that a much larger model expresses. Research on this capacity gap has found that the strongest available teacher is sometimes a poor choice. A set of methods exists to bridge wide gaps by adding a middle step, where the teacher trains a mid-sized model and that model trains the small student, so each handoff spans a smaller distance.
Architecture can outweigh size: The design of the base model can matter more than its parameter count. In one study, a 32-billion-parameter student outperformed a 70-billion-parameter student on the same task, because the smaller one was built on a stronger base architecture. Size alone is a weak predictor of how well distillation will go.
The teacher can pass on more than the task: In a 2025 study later published in Nature, a teacher model with a particular trait, a tendency to favor owls, was used to generate training data made up only of number sequences. A student trained on those numbers picked up the same preference for owls, even after the data was filtered to remove any visible trace of the trait. The same effect appeared with more serious behaviors, and it occurred only when the teacher and student shared the same base model. The takeaway is that distillation can carry across more than the task being taught, and that filtering the visible data is sometimes too coarse to stop it.
These limits set the boundaries, and within them, the method keeps advancing. The next section covers where it is heading.
The newest direction in distillation reduces the manual effort by automating the whole process.
In this setup, the large model runs the full loop on its own. It generates training data, fine-tunes the student, evaluates the student against a held-out set of examples it also generates, and repeats the cycle, adjusting what it produces until the student stops improving. The human role shrinks to defining the task and the success criteria at the start, with a final check on real data at the end.
Recent work in 2026 applied this to a detection task and found that it worked well, with one finding worth keeping in mind.
The choice of teacher model had a large effect on the outcome. Different teachers, given the same loop and the same student, produced students of noticeably different quality. So automation removes manual effort while making the initial choice of teacher more consequential, since that choice now drives an entire self-running process rather than a single training pass.
The same loop also points toward less hand-built pipeline work over time, as more of the data generation and evaluation moves to the model itself. For a team, the appeal is building a small, task-specific model without assembling a large hand-labeled dataset first, since the teacher supplies both the training examples and the data used to score them.
Distillation is a method for training a small, deployable model to copy the behavior of a large, expensive one. It produces a separate model rather than a compressed version of the original, and that distinction explains most of how it behaves.
It works because a model’s output carries more information than a plain label, in the form of soft labels that show a full pattern of confidence across the options.
In practice, the most common form has the teacher generate a training set that the student learns from, and the results can be strong, though usually on narrow tasks.
The limits are real:
The teacher sets a ceiling,
A wider size gap can hurt rather than help,
Architecture can outweigh size
The process can carry across traits that were never intended.
Taken together, distillation tends to be a good fit when the task is well defined and a capable teacher is available, and a weaker fit when the goal is broad, open-ended capability.
References:
2026-08-04 23:31:00
On-call shouldn’t feel like constant firefighting. This guide from Datadog breaks down how high-performing SRE teams reduce alert fatigue, streamline incident response, and design rotations that don’t burn engineers out.
You’ll learn how to:
Cut alert noise by tying signals to real user impact
Improve response with clear roles and smarter escalation paths
Turn incidents into feedback loops that improve system reliability
Why does sending a model a 100K-word prompt cost so much more than sending it a short one, even when the model and the hardware stay the same?
A key part of the answer lies in a block of working memory called the KV cache.
This memory is built up while the model generates a response. It is separate from the knowledge stored in the model’s weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU.
For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.
The above chart raises an obvious question, which is why a cache exists at all and why it grows the way it does. In this article, we will learn how LLMs use memory, how it gets expensive, and how to fix it.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Let us start with the work a model does to produce one token.
To choose the next word, it runs an attention step, where the newest token compares itself against every token that came before it. This comparison uses two vectors for each earlier token, a key and a value, which are simply the numerical summaries the model computes for that token inside each layer. A model that rebuilt the key and value for every earlier token at every step would watch the work per token climb as the input grows. Such a repetition is pure waste, because those keys and values stay the same once a token has been processed.
The KV cache removes the waste by storing those key and value vectors the first time they are computed. On the next step, the model computes the key and value for only the new token and reads the rest straight from the cache.
Overall, this is a great solution. However, caching fixes the speed problem while creating a new one. The cache now has to be read on every step, and this turns out to be the real source of increasing cost.
One detail worth understanding here is that the cache holds vectors rather than the original text. It also explains out-of-memory errors that look puzzling when the model itself fits with room to spare.
Generation of tokens runs in two phases, and they stress the hardware in different ways:
The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU’s math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic.
The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound.
The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single token.
A larger cache means more data crossing the memory bus per token, which shows up directly as slower and costlier generation. It also explains why a request can run slowly even when it fits in memory comfortably.
If the cost tracks how much of the cache we read each step, then the size of the cache is the next thing to understand clearly.
The cache size is the product of a handful of numbers:
A factor of two covers the key and the value.
Layers count because each layer keeps its own cache.
Key-value heads set how many sets each layer stores.
Head dimension is the size of each of those vectors.
Bytes per number is the space one stored value takes.
Tokens are the context length, with one entry each.
Batch size is the number of requests served at once.
To summarize, the cache size equals 2 times layers times key-value heads times head dimension times bytes per number times tokens times batch size.
Two things that should be noted here are as follows:
The cache grows in a straight line with the token count, so doubling the context doubles the cache
It grows the same way with the batch size, so serving more users at once scales it just as fast.
As an approximate example, a Llama 3 70B model has 80 layers, 8 key-value heads, a head dimension of 128, and stores each number in 2 bytes. At a context of 128,000 tokens for a single request, those numbers multiply out to roughly 40 gigabytes, which is why a single long request can fill most of an 80-gigabyte card on its own.
Let us now look at the optimization techniques that help an LLM manage the memory aspects. Each of these techniques tries to push against a specific number.
The first two techniques change how attention itself is built, which means a model is committed to them during training. Both shrink the footprint of each token in the cache.
Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one.
There is a more aggressive version called multi-query attention, where every query head shares a single key-value head. It saves the most memory of any head-sharing scheme. However, pushed that far, quality tends to drop, and training grows unstable, so most setups settle on the grouped middle ground as the better trade.
The second attack keeps the heads and compresses what each one stores.
Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.
As mentioned, head-sharing and latent attention both require control over the architecture, so they help when we are choosing or training a model. The next attacks work on a model we already have in hand.
Quantization goes after the bytes per number.
The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely.
The quality cost depends on how far we push it.
Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.
Specialized methods beat plain rounding because a few numbers in the cache deserve more precision than the rest, though plain rounding already captures most of the gain.
Eviction goes after the token count by dropping entries that the model is unlikely to need.
The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model’s output stable.
See the diagram below:
The trouble with eviction is structural.
Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.
More refined schemes score each token’s importance and try to predict which ones are safe to drop, which helps, though the core problem remains.
Even with the contents of the cache fixed, the way a serving system manages memory leaves a lot on the table, and two techniques help.
The first is paged attention.
Older serving systems reserved one large contiguous block per request, sized for the longest output it might produce. Most requests finished well short of that, leaving the reserved space idle, and the fragmentation added up. However, paged attention borrows an idea from operating systems, which break memory into small fixed-size pages and hand them out on demand. The cache gets split into small blocks that can live anywhere in memory, tracked by a lookup table that maps each request to its blocks. The result is that systems that wasted 60 to 80 percent of cache memory to fragmentation dropped that figure below 4 percent, and throughput climbed by two to three times, all from packing the same data more tightly.
The second technique results from the first. Since the cache lives in shareable blocks, two requests that begin with the same text can point at the same physical blocks while each holds its own private continuation. This is the foundation of prefix caching, and the productized version that the major APIs call prompt caching.
The win is large for any workload that repeats a prefix, such as an agent that sends the same multi-thousand-token system prompt on every call. OpenAI and Anthropic both report cost and latency reductions of 50 to 90 percent on cache hits, with cached tokens billed at a fraction of fresh ones.
However, one thing to note is that sharing cached state across users has opened timing side-channels that can leak information about other people’s prompts, an active concern we will leave aside here.
The techniques we’ve looked at look alike in how much memory they save and differ widely in what they ask for in return.
Some are close to free. For example:
Grouped-query attention costs very little quality and has become the safe default, which is why nearly every current model ships with it.
Paged attention and prefix caching barely touch quality, since they change how the cache is stored and shared rather than what it contains.
Other techniques require more. For example:
Quantization is cheap at 8 bits and grows risky as we push toward 4 and below, so the right setting depends on how sensitive the task is.
Latent attention saves most of the architectural options and asks for real engineering effort to serve well.
Eviction frees a large amount of memory and can also lose information that the generation later needs, which makes it a genuine gamble rather than a clean win.
For a short context, the cache is small, and most of these techniques solve a problem that has yet to appear. They earn their place in a long context and high concurrency, where the cache grows into the dominant cost. The right mix follows the workload, with agent loops leaning on reuse and long-document retrieval leaning away from eviction.
The cost of long-context inference comes down to one cache, sized by one short equation.
Since decoding reads the whole cache on every token, the cache is a bandwidth cost as much as a storage one, which is why shrinking it speeds things up. Every optimization we covered deals with a specific aspect of the overall equation or trims the waste around it.
Grouped query and latent attention reduce what each token costs.
Quantization stores each number in fewer bits.
Eviction keeps fewer tokens.
Paged attention and prefix caching manage and share the cache more efficiently.
References:
2026-08-03 23:31:14
Try Matic, backed by their 6 month money back guarantee.
In June 2025, security researchers at Aim Security demonstrated that a single email could cause Microsoft 365 Copilot to retrieve a company’s internal files and transmit them to an external server, with no user interaction.
The email contained ordinary text and exploited no software flaw. Copilot’s output was driven by hidden instructions in the email together with the user’s actual request, and the system applied no distinction between the two. Microsoft closed the specific path within days and reported that customers were unaffected.
But this incident sheds interesting light on how language models process text. The vulnerability was named EchoLeak, assigned CVE-2025-32711, and it serves as a useful study to understand the landscape of LLM security.
For contrast, consider a different case. For roughly $20 in API queries, a team extracted part of a production OpenAI model through its public interface. That sounds like the threat most teams should really fear.
In this article, we try to build a map of the full attack surface that threatens an LLM’s security. With it, a given LLM feature can be located, its exposure points identified, and new threats reasoned about as they appear.
So let’s start with the single most important property.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Almost every LLM vulnerability traces to one property. A language model receives instructions and data as a single sequence of tokens, and that sequence contains no marker separating commands from information.
Traditional software keeps the two apart.
For example, a parameterized database query holds the command in one position and the user’s input in another. This means that the text typed into a name field stays text even if it spells out a SQL statement, because the structure of the query enforces the separation.
However, an LLM context has one position for everything. The system prompt defining the assistant’s role, the user’s message, a document retrieved from a database, and the output returned by a called tool all arrive concatenated in the same sequence. Any portion of that sequence can affect the generated output as if it were an instruction, because the model computes each next token from the entire preceding sequence with no separate treatment for any part of it.
This is where prompt injection comes in, which means the supply of instruction-like text so that the model output conforms to those instructions instead of the operator’s intent. It reaches the model by two routes:
The direct route is a hostile instruction typed into the chat box. This is the version most people picture.
The indirect route places the instruction inside the content that the model retrieves during a legitimate task, such as a web page being summarized, a document being read, or an email in a managed inbox.
EchoLeak was an example of the indirect route. The user requested ordinary work, and the attacker’s instructions arrived in an email the user had not opened. In fact, the payload passed through Microsoft’s dedicated cross-prompt-injection classifier, which indicates that input filtering can be porous when used alone.
The property applies to any system that supplies external text to a model. A feature that processes retrieved search results, uploaded files, tickets, or comments carries indirect injection exposure by construction.
Parameterization solved SQL injection by separating code from data at the database boundary. No equivalent exists for natural language, because instructions and information are both expressed as words, and there is currently no reliable method to mark a span of text as inert and have that marking respected during generation. Filtering reduces the problem to some extent but does not eliminate it.
Since the confusion can occur wherever text enters the model or an action leaves it, the next step is to map those points.
The OWASP Top 10 for Large Language Model Applications is the industry reference for the most critical LLM risks. Its recent edition presents ten separate items. Placed against the path that data takes through an application, those ten items become positions on a single map.
The pipeline runs in stages.
Input arrives from the user. The system frequently retrieves context to ground its answer, a step usually built on a vector database that stores documents as numerical embeddings and returns the ones most relevant to a query. That pattern is retrieval-augmented generation, or RAG. The model then processes the assembled input. It may call tools or other agents that take actions externally. Output returns to the user. Monitoring surrounds the pipeline, and every component within it originates from a supplier, which forms the supply chain beneath the system.
Here are some more details about each stage and the potential risks associated with them:
Input: Direct prompt injection and unbounded consumption, which OWASP defines as uncontrolled resource use capable of running up a victim’s costs, sometimes called denial of wallet.
Retrieval: Indirect injection, vector, and embedding weaknesses. PoisonedRAG, a 2024 study, corrupted a RAG system’s answers by inserting as few as five malicious passages into a knowledge base of millions, reaching a 90 percent success rate on targeted questions.
Model: Training-data leakage, data and model poisoning, and system prompt leakage.
Tools: Excessive agency, the condition of an agent holding more permission than its task requires.
Output: Improper output handling, where the response passes downstream without sanitization, and misinformation, where a confident answer is incorrect.
Supply chain: Compromise of any component feeding the stages above.
The map provides a reference point for figuring out the specifics of an attack. A new attack can be located on it. The relevant questions are where it injects untrusted text and at which stage, and whether it abuses a permission the model should not hold. The answers indicate both the severity and the applicable defense.
Supply chain cannot be cleanly placed on the path, because a compromised model or poisoned vector store affects every later stage at once. For that reason, it appears as its own segment that spans everything.
The map also exposes a mismatch in attention. The threats that generate the most concern and the threats that reach production are different.
Attacks aimed at the model’s interior, including weight theft, training-data extraction, and training-time poisoning, are quite real, but for most developers, they rank low for initial effort. They tend to be expensive, narrow, or already mitigated by the model provider.
Here are a few examples:
Model theft: As mentioned earlier in the OpenAI scenario about a team that recovered the final embedding-projection layer of production OpenAI models for under twenty dollars and confirmed previously secret hidden dimensions. The result is significant and also bounded. It recovers one layer among many, and the researchers stated that reconstructing a full frontier model through an API remains impractical, since the cost exceeds training an equivalent model. OpenAI received advance notice and modified its API.
Training-data extraction: In late 2023, a group from Google DeepMind and several universities found that prompting ChatGPT to repeat a single word continuously could cause it to emit verbatim fragments of training data, including real contact details, with megabytes recoverable for a few hundred dollars. The privacy implications were serious, and OpenAI filtered the triggering behavior after disclosure.
Poisoning: In 2025, a team from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that approximately 250 malicious documents were sufficient to install a backdoor in models from 600 million to 13 billion parameters, with the count remaining roughly constant across model sizes. That finding overturned the assumption that larger models require proportionally more poisoned data. The researchers also stated the limitation directly. The backdoor produced only gibberish output on a trigger phrase, which is a low-stakes behavior they described as unlikely to pose significant risk in frontier models.
The graph below shows the placement of various threats on a fear vs frequency scale:
The ranking matters because attention is finite. A team focused on model theft while deploying an agent with broad permissions has addressed a rare attack and left a common one open.
The bounded status describes the present. These attacks rise in priority for teams that host open model weights, fine-tune on sensitive data, or operate their own training pipelines, where the model’s interior becomes the team’s responsibility rather than a provider’s.
The interior attacks sit at the edge of the map. The center, where the model triggers external actions, carries the larger risk.
The point at which LLM attacks cause material damage has a specific structure and is identifiable in a system. It is also called the lethal trifecta. It consists of three capabilities held together by a single agent:
Access to private data, such as an inbox, a customer database, or a source repository.
Exposure to untrusted content, meaning anything read from outside, including web pages, emails, and shared documents.
A channel to send data out or act externally, such as an outbound request, a sent message, or a tool call.
An agent holding all three can be directed by injected instructions to transfer private data to an attacker. Model alignment does not remove this exposure, because producing output that conforms to instruction-like input is how the model normally operates.
A few documented cases follow the pattern:
GitHub’s MCP server, MCP being the Model Context Protocol that connects models to external tools and data, has been compromised using malicious issues filed on a public repository to expose data from a victim’s private repositories.
GitLab’s Duo assistant has been supplied a public project containing hidden instructions and made to leak private repository contents.
A Chevrolet dealership chatbot was once manipulated into agreeing to sell an SUV for one dollar.
A crypto trading agent was socially engineered into transferring 55 ETH.
Removing any one of the three capabilities reduces the exposure. The least costly reduction is usually cutting the outbound channel or narrowing what the agent can access, which tends to be cheaper than adding a stronger filter.
Connecting tools through MCP is the most common way an agent acquires the third capability, and the protocol is recent enough that even established servers have shipped injectable configurations. Anthropic’s own official Git MCP server received three injection-related CVEs in 2025. Runtime input is one source of risk. The system’s components are another, and they can be compromised before any request arrives.
Every model, adapter, vector store, and tool in a stack originates from a supplier, and any of them can arrive compromised. This is the supply chain surface, and it bypasses runtime defenses because the threat is present before input validation runs.
The common mechanism is straightforward.
Many models are distributed as serialized files, and some serialization formats execute code when the file loads. In early 2025, ReversingLabs documented a technique named nullifAI, in which malicious models uploaded to Hugging Face concealed a reverse shell, code that opens a connection back to an attacker, inside a Python pickle file. The file was compressed in a manner that evaded Picklescan, the platform’s scanner, so the model appeared clean and executed hostile code on load.
The scale demands attention. Protect AI, which scans models hosted on Hugging Face, has examined more than four million and flagged roughly 352,000 as carrying unsafe or suspicious issues, across more than fifty thousand models.
Provenance is one of the few fully controllable factors. The selection of which models, tools, and data sources to trust sits with the team, unlike most of the runtime surface.
Two mitigations are advancing:
Safer serialization formats that avoid executing code on load.
Model signing that verifies origin, comparable to signed releases in package ecosystems.
No single defense holds all the time, and the supporting evidence is strong enough that the realistic objective changes from preventing every attack to surviving the ones that succeed. The working posture is defense in depth, a set of independent layers arranged so that the failure of one is contained by another.
One layer is insufficient.
In November 2025, a team from OpenAI, Anthropic, and Google DeepMind published a study that took twelve previously proposed defenses against prompt injection and jailbreaking and defeated them, using attacks permitted to adapt and iterate. Strong production filters still allow a measurable fraction of attacks through, and a single success is enough. A lone guardrail provides confidence that the measurements do not support.
The more durable approach constrains the system around the model rather than relying on the model to resist manipulation.
Google DeepMind’s CaMeL is one example. It treats the model as untrusted, uses a separate privileged component to plan actions, and quarantines externally retrieved data so that data cannot trigger sensitive operations on its own.
Meta’s Agents Rule of Two is a simpler operational version, recommending that an agent satisfy at most two of three risky properties: processing untrusted input, holding sensitive access, and acting externally, without a human in the loop. Meta presents the rule as a supplement to least privilege rather than a complete solution.
The standard layers each cover a stage of the map:
Input is validated and constrained.
Retrieval sources are kept clean.
Each tool is scoped to the minimum permission its task requires.
Model output is treated as untrusted and sanitized before downstream use.
Monitoring watches for anomalies.
A human reviews the highest-consequence actions.
The threat model organizes around one fact.
A language model processes instructions and data as the same sequence of tokens, and the full map follows from that property.
The OWASP Top 10 becomes a set of positions on a path rather than a list. The widely publicized attacks, model theft, and training-data extraction are bounded and largely mitigated, while the larger risk concentrates wherever an agent holds private data, untrusted content, and an external channel at once. The supply chain underlies every stage and is the surface most directly controlled by the team operating the system. Since no single layer holds, defense in depth is the realistic posture to implement for a team.
The cost is high. Every layer adds latency, expense, and friction, and the strongest mitigation, human review of consequential actions, also limits how autonomously a system can operate. The trade-off is unavoidable.
The landscape reduces to a few durable points:
The root cause is the absence of a boundary between instructions and data.
Named threats are positions on the pipeline rather than isolated facts.
The interior attacks are mostly bounded, and the trifecta marks where real damage occurs.
Provenance is the most directly controllable surface.
Defense in depth, rather than any single filter, contains the residual risk.
References:
EchoLeak: zero-click prompt injection in a production LLM (case study)
PoisonedRAG: knowledge-base poisoning of retrieval-augmented generation
New prompt injection papers: Agents Rule of Two and The Attacker Moves Second
Agents Rule of Two: a practical approach to AI agent security
Anthropic Git MCP server injection advisory (CVE-2025-68143)
2026-07-31 23:01:37
We’re hiring a part-time instructor for “𝐖𝐫𝐢𝐭𝐞 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧 𝐆𝐫𝐚𝐝𝐞 𝐂𝐨𝐝𝐞 𝐰𝐢𝐭𝐡 𝐀𝐈”. This is a live, cohort-based course for software engineers.
This course is about teaching engineers how to reliably ship production-grade software with coding agents. Students will learn how to delegate real work to coding agents, write specs and plans that agents can execute, and verify, review, and secure the code that comes back.
You’ll help refine the curriculum, teach live sessions, answer student questions, and share practical lessons from your own experience building software with AI. The initial onboarding requires some preparation, but after that the commitment is approximately 2 to 10 hours every two weeks. This is a flexible opportunity that fits alongside a full time engineering role. Compensation is competitive and based on experience.
You’re a great fit if you:
Are excited about helping engineering teams avoid AI slop
5+ years of professional software engineering experience building production systems
Use AI coding agents such as Claude Code, Codex, Cursor, etc., daily.
Understand how coding agents work, including planning, tool use, context management, and common failure modes.
Have experience breaking complex engineering problems into clear specifications and implementation plans that AI can execute
Have worked extensively with large or legacy codebases and know how to make them AI friendly
Have reviewed significant amounts of AI generated code and know how to verify correctness, catch subtle bugs, identify security risks, and maintain long term code quality
Have strong software engineering fundamentals, including testing, CI/CD, debugging, code review, etc.
Enjoy teaching
If this sounds like you, email [email protected] with your background, how you use AI in your engineering workflow, or any teaching, writing, speaking work that demonstrates your expertise.
2026-07-30 23:30:29
What happens when a service sends a request to charge a customer, but the request times out with no response? The burning question is whether the charge went through. Or should it be retried?
Two different things could have happened. The charge succeeded, and the confirmation was lost on the way back, or the request never reached the payment service at all. Both possibilities produce identical outputs, which makes it difficult to figure out what happened and the next action to be taken. Retrying risks charging the customer twice. On the other hand, declining to retry risks never charging them at all.
Idempotency is the property that makes the retry safe. An operation is idempotent when applying it more than once produces the same state as applying it once. For example, setting an account balance to 500 is idempotent, because even the tenth time to execute this operation, the outcome will be the same. In contrast, adding 500 to a balance is not idempotent, since every time it is executed, the balance amount changes. Most operations that matter in a business system resemble the second one.
In this article, we will look at the following topics in detail:
The three different delivery semantics available for developers.
The three points where duplicates enter a producer, broker, and consumer path, and why a fix at one point does nothing for the other two
The difference between an operation that is idempotent by nature and an endpoint engineered to behave that way
What does an idempotency key need to work, and how can it fail?
Why every deduplication scheme has a time limit, and what the guarantee is worth once that limit passes
What “exactly-once” means in real-world systems, and where each guarantee ends?
