2026-08-26 23:30:34
Every agent already runs a loop. Loop engineering adds a loop around the agent itself, enabling it to evaluate its output, try again when the work falls short, and refine its instructions when the same mistakes recur. Today, you perform that role: reviewing the work, diagnosing what went wrong, and prompting the agent again. This article shows how to automate that process with a working example, while exploring where human judgment still belongs.
A 70-billion-parameter model requires reading roughly 140 GBs of weights out of the GPU memory. On a modern data center GPU, this transfer can take tens of milliseconds. The actual calculation applied to these weights takes a fraction of that time. This means that the processor’s math units are unused for most of the time taken by the token generation step.
Speculative decoding is a technique that converts this unused capacity into output. A second, much smaller model produces several candidate tokens in advance. The large model evaluates all of them in a single forward pass instead of one pass per token, resulting in 2-3 times faster generation. To make things better, the text produced remains statistically identical to the output of the large model running alone.
In this article, we will look at how speculative decoding works. Here’s what we will cover:
Why token generation runs one step at a time
What a GPU spends its time on during generation
How several candidate tokens are evaluated in a single pass
The accept and reject loop, and what happens when a candidate is wrong
Why output quality is preserved exactly
Acceptance rate, and why it varies by workload
The four places a draft can come from
When speculative decoding stops helping
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Text generation works one token at a time.
The model reads everything produced so far, computes a probability distribution over its vocabulary, selects the next token, appends that token to the input, and repeats the cycle. Each cycle is called a forward pass, and every forward pass runs the input through all layers of the model.
For example, token 50 depends on token 49 being present in the input, and token 49 depends on token 48, and so on. Computing them simultaneously would break the dependency chain that makes the output coherent.
The implication is that a 500-token response requires 500 sequential forward passes, each one completing before the next begins. Since the duration of a single pass depends on the size of the model, the total generation time equals the number of output tokens multiplied by the time per forward pass.
This explains why the response speed stays roughly steady whether the answer is a short factual reply or a long block of code, because the per-token cost stays the same either way. It also explains why a larger model produces text more slowly on identical hardware.
Modern inference systems use a KV cache, which stores the attention state for tokens already processed so that each new pass only computes attention for the newest position. This cuts the work done inside each pass to a large extent, though the requirement for one pass per token still remains.
Since the number of passes is fixed by how much text we want, it leaves the second half of the equation. What does a single forward pass actually spend its time doing?
To put it simply, a forward pass spends most of its duration moving data rather than performing arithmetic calculations.
Model weights live in the GPU memory, usually called VRAM. To compute anything with those weights, the GPU has to transfer them into the compute units where the multiplication happens. For a 70-billion-parameter model stored at 16-bit precision, this transfer amounts to roughly 140 GBs for every single token.
The arithmetic performed on those 140 GBs is quite small by comparison. One token means one narrow vector flowing through each weight matrix. The GPU loads an enormous matrix out of memory, multiplies it against that vector, discards it, and loads the next one.
The consequence is that during prompt processing, compute utilization is around 90 to 95 percent. However, during token generation, it falls to somewhere between 20 and 40 percent. The math units are unused for most of every step while the memory bus runs near capacity.
The difference is driven by how much work each weight read supports:
Prompt processing reads the weights once and applies them to thousands of input tokens simultaneously.
Token generation reads the same weights and applies them to exactly one token.
This is capacity that has already been paid for, but underutilized.
But why does this matter practically?
A GPU with higher memory bandwidth improves generation speed more than one with more raw compute.
However, spare capacity only helps if there is useful work to put into it. The question is whether a single forward pass can produce more than one token’s worth of output.
A single forward pass can evaluate many positions at once.
Transformers process an entire sequence in parallel. When we feed in a sequence of tokens, the model computes a next-token prediction at every position in that sequence during the same pass. For example, a five-token input produces five predictions.
These predictions stay valid because of causal masking. Inside the attention mechanism, position 5 can access positions 1 through 5 while positions 6 and beyond are masked out, and position 3 can access only positions 1 through 3. Each position is therefore conditioned on exactly the tokens preceding it, identical to the conditioning it would have received had we generated the sequence one step at a time.
This is the property that makes prompt processing fast. A 2,000-token prompt runs through the model in one pass rather than 2,000, because all 2,000 positions are computed together.
When applied to verification, the consequence is direct. For example, if we append four candidate tokens to the context and run one forward pass, we receive the model’s own prediction at each of those four positions.
One thing to understand here is that verification and generation are basically the same operation. The target model performs identical work at each position. The cost savings comes from performing that work across several positions in a single pass instead of across one position in each of several passes.
The complete loop combines a fast source of candidate tokens with the batched evaluation described above.
This setup uses two models:
The large model we want output from is called the target model
Running alongside it is a much smaller draft model with 10 to 20 times fewer parameters. It is usually drawn from the same family and uses the same tokenizer.
Each round has three steps:
The draft model produces K candidate tokens through its own serial loop. Those passes are sequential as well, though each one costs a small fraction of a target model pass.
The candidates are appended to the context. The target model evaluates the extended sequence in one forward pass.
Working left to right, each candidate is compared against the target model’s prediction at that position. The matching candidates are kept, and as soon as the first mismatch appears, the remaining candidates are discarded.
The mismatch point plays an important role in this. The verification pass already computed the target model’s prediction at that position, so that token gets used directly. We keep the matching prefix and receive one correct token at no additional cost.
This property places a bound on the downside.
In the worst case, all four candidates might fail to match, but we would still have the one token the target model produced at the first position, which is exactly what plain decoding would have delivered from one forward pass. The wasted effort amounts to just the draft model’s compute and some extra effort in the verification pass. Both are drawn from otherwise available capacity. In a typical case where two of four candidates match, we get to keep two plus the free token, giving three tokens from one target model pass.
Draft length K is a tunable value, which is commonly set between 3 and 5. Larger values raise the ceiling on savings, since a fully accepted draft of eight saves more than a fully accepted draft of three. However, larger values also reduce the odds that later candidates survive, because the draft model conditions on its own unverified output as it moves forward. Past a certain point, the additional candidates get discarded often enough that the extra work outweighs the benefit.
The question at this point is whether the resulting text is still the text the target model would have produced.
Speculative decoding produces text with the same statistical properties as the target model running alone. This is enforced by means of the acceptance rule.
Under greedy decoding, where we always take the highest-probability token, the rule is quite direct. A candidate is kept when it matches the target model’s top choice at that position, and dropped when it fails to match.
However, sampling takes more care, since tokens get picked with some randomness. Both models produce a full set of probabilities across the vocabulary at each position. The rule compares the two sets:
When the target model gave the candidate at least as much probability as the draft model did, the candidate is kept.
When the target model gave it less, the candidate is kept part of the time. This is in proportion to how far apart the two numbers were.
When a candidate is dropped, the replacement gets picked from an adjusted set of probabilities, with the draft model’s own scores subtracted out first.
This last step is critical. If we add up both paths, the candidates kept and the candidates replaced, the odds of any particular token appearing depend exactly on the target model’s own odds for it. This is regardless of what the draft model suggested.
There are two qualifications to this:
Matching odds still allow different wording, since sampling stays random either way. Running the same prompt twice can give varied text in both setups.
Computers store these numbers with limited precision, so rounding can flip the winner when two tokens sit almost exactly tied.
The size of the speed increase in this approach is governed by the acceptance rate, which is largely a property of the workload.
Acceptance rate is the fraction of candidate tokens the target model keeps, and acceptance length is the average number confirmed per verification pass, including the free token at the end.
Different workload types produce different results:
Structured and repetitive output produces high acceptance. For example, code generation, summarization, extraction, and retrieval-augmented answers reuse large amounts of text from the input, which makes the next token easy to predict from a small model.
Open-ended output produces low acceptance. Creative writing and open conversation generate text with genuine variety, where a small model diverges from a large one far more often.
Sampling temperature contributes as well. Higher temperature flattens the probability distribution, which increases mismatches between the two models and pushes acceptance down. If the acceptance falls below roughly 50%, the additional work outweighs the savings.
For reference, DeepSeek reported acceptance rates between 80 and 90 percent for the second predicted token in production serving of DeepSeek-V3, which translated to roughly 1.8x generation throughput.
The practical implication is that two teams can deploy the same configuration on the same hardware and get different results, because their users are asking different questions. Ultimately, acceptance depends on the quality of the candidates, which brings us to where candidates come from.
The key question while choosing candidate or draft sources is where to obtain fast predictions cheaply. There are four answers in common use:
A separate small model: The original approach pairs the target with a smaller sibling, so a 1B model drafting for a 13B target, or a 3B to 8B model drafting for a 70B target. Same family and identical tokenizer are requirements in this approach. The cost is a second checkpoint to deploy and version, plus VRAM that comes out of the KV cache budget, which reduces how many concurrent requests the server can hold.
Extra prediction heads on the target model: Lightweight output heads predict tokens two or three positions ahead using the target model’s internal representations. DeepSeek-V3 trained these during pretraining to improve model quality, then reused them at inference as the draft source. The cost is training, which puts this option out of reach unless we control the model.
A cheaper version of the same model: The draft runs the same weights under a reduced compute budget through quantization, layer skipping, or a compressed KV cache. For example, QuantSpec uses 4-bit weights and a 4-bit KV cache for drafting while verification runs at higher precision, reporting speedups above 1.78x with acceptance above 90 percent. The cost is implementation complexity, since draft and target share hardware and cache structures.
A search over existing text: This approach scans the prompt and previous output for a recent matching sequence, then proposes whatever followed it last time. Memory cost is zero, and a single model is involved. It contributes only when output repeats input, where it reaches 2x to 4x on tasks like document editing and summarization.
Selecting an option depends on the deployment. Also, tokenizer compatibility constrains pairing more tightly than model quality does. A stronger small model with a different vocabulary is unusable as a draft source without additional machinery. All four options depend on spare compute being available. This condition holds under some serving loads better than others.
The increase in speed also depends on the operating regime rather than on the technique alone. The gains shrink as server load rises.
Speculative decoding spends compute capacity that would otherwise go unassigned. When a server handles a single request, this capacity is genuinely available. As concurrent requests accumulate, the same weight read operation serves many requests at once, and the compute units approach saturation. Verification work has to compete with real requests.
One systematic evaluation reported up to 1.96x on a 70B model at batch size 1, declining to 1.21x at batch size 128. Under higher concurrency, the technique can fall below baseline throughput, at which point enabling it costs more than the benefits.
Serving systems try to handle this in different ways. For example, vLLM exposes a flag that disables speculation above a configurable batch size. It supports dynamic adjustment where draft length shrinks as concurrency rises and reaches zero under heavy load. The control signals routine operational tuning rather than an edge case.
Another boundary is that the time to first token stays roughly the same, since speculative decoding applies to generation rather than prompt processing. Therefore, workloads with long prompts and short outputs have relatively little to gain.
DeepSeek documented the tradeoff, describing multi-token prediction as slightly reducing throughput while significantly improving end-to-end generation latency.
Speculative decoding rearranges when the processing happens rather than reducing how much the target model performs. Here are some key points we have understood:
Token generation is slow because every token requires reading the full set of model weights out of memory, while the arithmetic applied to those weights is comparatively small.
A transformer computes a prediction at every position in one pass, which makes evaluating several candidate tokens cost about the same as evaluating one.
A rejected draft truncates rather than wastes, since the verification pass supplies a correct token at the mismatch position regardless.
Output quality is preserved by the acceptance rule itself, so it holds without tuning.
The size of the gain depends on how predictable the output is and how much spare compute the server has available.
The variants differ in where predictions come from and what that source costs.
2026-08-25 23:31:09
Securing AI usually requires hardcoding permissions into application code for every MCP server, or using static API tokens for all-or-nothing access (and hoping your LLM doesn’t drift from intended actions).
Teleport eliminates these problems with zero-code MCP integration that applies the same zero trust security principles you use for human engineers:
Least privilege access control that denies new tools by default
Just-in-time (JIT) access requests for high-risk tools
Logs for every action – with full audit and identity context
Zero trust agent access to MCP servers, databases, and Kubernetes clusters
No need to write authorization code, rewrite MCP servers, or limit agent work.
When an AI model handles a difficult question, it produces three separate pieces of text:
The first is the answer displayed on screen.
The second is a shorter block, usually labeled thinking or reasoning, that appears while the answer is being assembled.
The third is the model’s full reasoning process, which is never displayed.
The second piece of text is a summary of the third. It is generated separately and shown in place of the original reasoning process. The full reasoning runs longer. It also contains material that the summary leaves out. Most major providers withhold it. However, in place of the complete process, an encrypted version is shared with the client during the conversation.
In August 2026, a team at MATS Research, the ELLIS Institute Tübingen, and the Max Planck Institute for Intelligent Systems wanted to test whether the encrypted reasoning blocks that Anthropic, OpenAI, and Google hand back to clients actually keep that reasoning private. They showed that the blocks can be replayed into a cheaper model in the same family, which will then print the hidden reasoning in plaintext. In other words, the AI model’s thoughts are stolen, exposing information that should ideally be hidden.
In this article, we will cover what the researchers found out:
Reasoning traces, and how they differ from answers and summaries
Why providers withhold reasoning
The storage problem that creates, and the two ways to solve it
What the encrypted block contains, and what it authenticates
The three forms of compatibility that follow
The extraction method and its verification
The four attack vectors
Findings from a scan of published session logs
The proposed fixes, and the limit that remains
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Modern frontier models first generate an extended internal sequence of text before producing a visible answer. For example, if we ask a model to solve a hard mathematics question, the model may generate two thousand words of exploration, dead ends, and corrections, which ultimately helps it write a clean two hundred-word answer. This longer sequence of exploration is the reasoning trace, which is also known as the chain of thought.
The trace holds intermediate hypotheses that the model may have tried and abandoned. It contains the raw output of any tools that were called, the user’s data as the model processed it, and whatever contextual secrets were present in the session. Unsurprisingly, this trace is much denser and more revealing than the final polished output.
For example, if we ask a coding agent to remove hardcoded credentials from a code repository, the agent has to read those credentials to do the work. In other words, the credentials pass through the reasoning trace before any answer is produced.
Why do the model providers hide these reasoning traces?
There are two separate motivations:
The first is commercial. A competitor can collect a large number of traces from a strong model to create training material for building a cheaper imitation of the strong model. This is because the final answer just provides the endpoint of a computation, whereas a trace provides the methodology behind the answer.
The second is safety-related. A model sometimes has to generate reasoning about a harmful topic to refuse. The filtering that turns a completed trace into a safe visible answer runs after the trace already exists. Publishing the trace skips that filter and lets that information be seen by the user.
Withholding a trace from reaching the user does not mean that the trace is not needed. In a multi-turn conversation, the reasoning from an earlier turn has to be available on the next turn. For example, a request to a model API carries no memory of what came before. The continuity we see in a chat window is reconstructed by the client resending the full history with every message. However, the server is stateless, which means it stores nothing between requests.
This leaves us with two options:
The first option keeps the state on the server. The provider stores the trace in its own database, returns a meaningless identifier to the client, and looks it up when the next message arrives. This is straightforward but expensive because it means storing state for every conversation from every user of a global service.
The second option encrypts the trace and returns it to the client, which stores the block and sends it back with each subsequent request. In this case, the provider stores nothing.
Providers like OpenAI, Anthropic, and Google chose the second option. Confidentiality keeps competitors from reading the trace, integrity prevents an altered trace from being fed back, and statelessness removes the storage cost. As you can see, two of those are security goals, and the third is a cost goal.
So what is actually inside the block that gets sent back to the client?
It consists of a long string of base64 text, which is a way of writing binary data using ordinary letters and digits so that it survives inside JSON. Once decoded, it is an AEAD envelope. AEAD stands for Authenticated Encryption with Associated Data, and it accomplishes two tasks at once: hiding the content and proving that the content was not altered. The associated data portion holds extra fields that stay readable while remaining tamper-protected.
The envelope carries a header. Depending on the provider, it can include the model name, block type, version, and key identifier. Alongside that is a nonce, which is a random value used once per encryption so that identical content produces different-looking output each time, an authentication tag, and the ciphertext. The field carrying all of this is called signature at Anthropic, encrypted_content at OpenAI, and thinkingSignature at Google.
Authentication here proves that the contents came from the provider and were not modified afterward. The model name and version are covered by that proof. However, the account that generated the block and the conversation it belonged to are absent from the authenticated fields entirely. To make this clear, since no provider has published a description of the scheme, the researchers inferred this mainly from observable behaviour. The evidence points toward a single global key used across an entire ecosystem.
If the authenticated fields provide no information about the origin, a valid block stays valid everywhere. The researchers have described three forms of this, each more permissive than the last:
Cross-session compatibility means a block can be replayed out of order, and blocks from earlier sessions work in new ones. This supports editing conversation history and trimming long sessions to fit a context window.
Cross-user compatibility means a block produced in one account is accepted when submitted from another.
Cross-model compatibility means a block from one model is accepted by a different one, which supports switching models mid-conversation and automatic rerouting.
The researchers tested every source and target combination available in July 2026. Some of the findings were as follows:
Claude accepted almost every combination, the exception being Fable 5, whose blocks were accepted only by Fable 5.
GPT was organised by generation. The GPT-5.6 series accepted blocks from all earlier generations, while older models accepted only their own.
Gemini accepted every combination across every generation.
Cross-model compatibility carries a lot of importance because protection inside a model family is uneven.
Flagship models such as Claude Opus 4.8 and GPT-5.6 Sol receive anti-distillation training aimed at preventing disclosure of their own reasoning. It is present behind input and output filters that check for verbatim matches. However, smaller models in the same family, such as Claude Haiku 4.5 and GPT-5.6 Luna, are optimised for cost and speed. As a result, they receive far less of that training. In other words, cross-model compatibility means the smaller model accepts blocks produced by the larger one.
The information extraction method is a direct result of this gap in training. Here’s what can happen:
A question is asked of the strong model, and the response returns a visible answer and an encrypted reasoning block.
The answer is discarded, and the block is kept.
Next, a fresh conversation is opened with a weaker model in the same family.
The reasoning block is placed into it as prior context, and the request asks for the attached reasoning to be transcribed.
The weaker model outputs the stronger model’s reasoning in plaintext.
In other words, the strong model was queried once, with an ordinary question, and was never asked to disclose anything. Therefore, its refusal training was never engaged, and its output filter received only a benign answer.
The researchers describe the weaker model as a fuzzy decoder, since it generates an approximation rather than performing exact decryption.
This results in a verification problem. Without access to the original trace, how do you establish that a reconstruction is faithful rather than a plausible invention? The researchers used billing records. API responses report the number of reasoning tokens consumed, and that figure is exact because the charge depends on it. Re-encoding the recovered text and counting its tokens produces a number that should match. Across 120 programming problems, the two counts tracked closely. It was roughly one to one for Claude.
The difficulty of this approach varied by provider. A single fixed prompt worked for Claude, while GPT required up to 50 candidate extractions per block and output chunked below roughly 50 tokens to avoid a rejection triggered by verbatim reproduction.
The research paper talks about four consequences of this gap. These are split based on who produced the original block.
Two of them use blocks the attacker generated:
Distillation: It involves training a copycat model on a target’s visible answers. Traces make it stronger, because a trace supplies the problem decomposition and the intermediate steps. For reference, earlier work using approximate traces raised the MATH500 accuracy of a fine-tuned model from 68.4 percent to 76.0 percent over answer-only training. Decoding 10,000 traces at Claude Haiku 4.5 pricing costs roughly $720, and where blocks come from public datasets, the frontier model is never queried.
Jailbreaking: Models are trained to withhold harmful content from visible output. However, they are largely not trained to avoid generating reasoning about harmful topics, since constraining trace content is believed to degrade the usefulness of traces for safety monitoring. In one demonstration, a prompt drew out extended reasoning about vehicle theft while the visible answer stayed within a responsible write-up aimed at manufacturers. The recovered trace contained specific vulnerable makes and model years.
The other two vectors use blocks produced by other people.
Developers publish agent session logs routinely, for reproducibility or by committing them accidentally, and they sanitise the visible text before publishing. However, they cannot sanitise the encrypted blocks, because they cannot read them either.
The researchers collected 6,708 public agent trajectories from GitHub and Hugging Face and decoded 315,320 reasoning blocks, filtering the results through an automated pipeline to remove placeholders and benchmark fixtures. Here are some findings:
From genuine user sessions, 62 API keys, 33 passwords, 24 access tokens, 7 private keys, and 30 personal email addresses
Across all sources, 1,028 blocks holding at least one confirmed leak
328 of the 6,708 sessions leaking at least one item
Sanitisation operates on plaintext only, so even if every user in that sample had scrubbed their visible text perfectly, all 62 API keys would have remained in the reasoning blocks. This was a non-exhaustive scan of public traces, and logs held locally or produced by live services are likely to contain more.
The fourth vector is prompt injection. Long agentic runs are expensive to repeat. Therefore, it makes sense to resume a published run. Encrypted blocks make that possible.
A block containing an instruction can be planted in a shared trace. When the session resumes, that block is processed as prior context. In one demonstration, a block carrying an instruction to upload PowerPoint files to an external address was ported into a conversation with GPT-5.6 Sol and paired with an unrelated slide-editing request. The resulting script added the requested slide and uploaded the presentation.
Recovering traces at this scale brought out findings unrelated to the vulnerability itself.
Summaries and traces sometimes diverge. On an AIME 2025 mathematics problem, the summary returned by the API described methodical work through the geometry using the law of cosines. The recovered trace opened by stating the answer from memory and then working backward toward a justification, with several wrong guesses along the way. The researchers did not measure how often this occurs. However, they note that a smaller model summarising a larger one will compress the original.
Recovered GPT traces are frequently not fluent English. They are made up of compressed telegraphic notes with articles dropped and grammar abandoned. This carries a direct consequence for oversight, because trace monitoring depends on traces being legible.
The research paper ultimately makes one strong point, which is the cause of the overall problem. The envelope we talked about earlier authenticates the content of a reasoning block, but it does not authenticate the context in which that block was produced or replayed.
The cheapest fix for this problem should address cross-user replay. This involves embedding an account identifier inside the authenticated data at issuance, allowing the API to compare it against the authenticated caller and reject any mismatch. This closes the attack vector entirely without the need for server-side storage.
Cross-session binding is harder, because some genuine features depend on the same portability. Users can fork conversations, compact old turns out of long sessions, and downgrade models mid-conversation. Binding each block to the complete transcript would break all three. The proposal is a hash chain tying each block to its session and to the fingerprint of its predecessor, combined with a Merkle tree that retains only root fingerprints once older blocks are pruned. This approach preserves ordering cheaply while still allowing integrity checks over any surviving stretch.
Blocks already published present a separate problem, since they were signed under a key encoding neither user nor session. The only retroactive remedy is to rotate those keys and refuse to decode anything signed under a retired key identifier. This also invalidates legitimate continuations of old sessions.
Other measures include moving to server-side storage entirely, configuring API gateways to reject envelopes from a different model version than the one queried, and training models to decline transcription requests regardless of framing.
We’ve now understood the key points in the research paper. Here are some of the main takeaways:
This wasn’t a failure of cryptographic techniques. The guarantee made by the envelope remained throughout the flow. The missing piece was a binding between a block and the context that produced it, which is a decision about which fields go into the authenticated portion.
The summary displayed alongside an answer is a separate artifact from the trace it describes, and the two can diverge.
Sanitising a session log reaches the plaintext only. Encrypted blocks have to be removed rather than cleaned, since the person publishing them cannot inspect what they contain.
The security of a model family depends on its least protected member. Anti-distillation training on a flagship model provides limited benefits while a cheaper sibling model accepts the same blocks.
References:
2026-08-24 23:31:01
Give an agent an access token and it spreads: into the context window, into tool call logs, into notes it keeps between steps. Each copy works from anywhere, long after the fact.
Relay keeps the credential at WorkOS. Your agent names the user, WorkOS attaches that token, refreshes it, and releases it only to allowlisted hosts. A hijacked agent session is a live process you can kill.
The gap between code that executes fine and code that is actually safe to trust is growing wider pretty fast. For many years, writing code was the slow, expensive step, whereas reviewing it was a smaller task at the end. With the rise of AI-assisted coding, this balance is shifting.
AI tools can now create a working function in seconds and a full feature in minutes. Teams are able to write more machine-generated code every month. In other words, producing code is now fast and relatively easy, whereas code verification is the harder part. A reviewer still has to read the change, understand it, and decide whether it belongs in production. In fact, more code written simply means more code that should be verified.
We recently got a chance to speak with Andrea Malagodi, the CTO of Sonar (the company that has built some of the most used code verification software). He provided deep insights into code verification, especially in the context of AI and how Sonar is adapting to the recent changes.
In this article, we will look at how code verification works, why the rise of AI-generated code puts more pressure on it, along with the extremely useful insights from Andrea on what the future may look like.
The shift with regard to code generation and verification is quite visible when we look at the data. One of the clearest signals comes from Google’s DORA research, a long-running study of how thousands of teams build and ship software. Their recent work found that as teams adopt more AI, delivery stability dipped. Trust in AI-generated code stayed low, with well over a third of developers reporting little confidence in what these tools produced [2]. In other words, more speed in writing code brought more pressure further down the line.
A controlled trial from the research group METR gives a similar indication. Its participants were experienced open-source developers working on their own mature projects, and each task was randomly assigned to allow or disallow AI tools. The developers expected AI to speed them up by roughly a quarter.
However, the result showed a totally different picture. AI-assisted tasks took about 19 percent longer [3]. Moreover, this happened after the developers internally believed that the AI helped them be more productive. Turns out, a lot of extra time went into prompting, waiting, reading the output, and correcting it. To be fair, the same team later reported a more confusing follow-up signal. This was partly because developers preferred to keep their AI tools [4].
Nevertheless, if we consider these results together, it is evident that while AI definitely increases the amount of code written, it also leads to more verification work down the line.
So let us first understand what code verification actually means.
Code verification is the umbrella term for every check that ensures whether a piece of code is correct, safe, and maintainable enough to ship to production. In other words, it is the work of earning enough trust to put a change in front of real users. The key term to note here is “earning”. This is because trust arrives in degrees. It is built up one check at a time, rather than granted in a single stroke.
Think of a task of drafting a contract. Writing the words is one part of this task. However, the review, the legal checks, and the signatures are what transform those words into something people can actually rely on. Writing code works in a similar way. The moment a piece of code leaves an editor and is committed to a code repository, it carries an implicit claim about the functionality. Code verification is the process through which that claim gets tested until a team feels safe to use that code in a real production environment.
Some domains push this stage to its limit through rigorous formal verification. In such domains, engineers have to mathematically prove that the code being deployed matches a precise specification. Such a process is standard for critical stuff like flight-control systems and kernels, where a single defect can risk lives. However, for most software, having a similar approach can cost far more than it returns. Therefore, teams opt for lighter checks that are arranged in layers.
Taking the layered analogy further, we can imagine code verification as a stack of filters. As you can see, each filter catches a certain type of problem.
At the top of the stack, we have the cheapest checks. For example:
Type Checker: It confirms whether the values moving through your code are the exact type each operation expects. This way it can catch a whole class of mistakes before the code even runs.
Linter: It scans for suspicious patterns and style problems.
These types of checks can run in an instant and cost almost nothing. Below this stack, we have tests.
A unit test runs a small piece of code with known inputs and confirms whether it returns the expected output. Tests can catch behavioral mistakes in the code that a type checker cannot detect. This is because a piece of code can have perfectly valid types while still computing the wrong answer. For example, consider a simple function that is supposed to add two numbers, but instead multiplies them. In such a case, the type checker and linter would not point out any issues. Only a unit test that compares the result of the operation against a known answer will reveal the mistake.
Below the layer of tests, we have the human review filter. This is basically the case where another developer goes through the change and judges whether it fits the system, solves the right problem, and is readable. This layer catches what machines can miss, such as a solution that works yet takes an approach the team standards don’t recommend.
Beneath all of these layers is the production monitoring setup. The job of this system is to observe the code under real traffic and flag problems that may have passed through every earlier layer.
Real-world filter stacks can also hold more layers than this. This may include security scanners and dependency checks. However, the point is that each filter covers a specific weakness in the one above it. This is why serious teams run several such layers in a specific order before releasing any code into production.
The filters in that stack fall into two families:
Static Analysis: The filters in this family check the source without executing it, which makes this type of analysis fast and broad. With static analysis, we are able to scan an entire codebase in one pass. Type checkers and linters belong here. The tradeoff is that real behavior at runtime remains partly out of view. Therefore, static analysis can sometimes raise an alarm about a problem that might not exist during live conditions.
Dynamic Analysis: The filters in this family run the code with real inputs and observe the result. Tests belong here. This family relies on checking actual behavior, but is limited by paths that are exercised. A test suite that only runs the happy path cannot detect a crash that might be waiting to happen on an empty input.
Despite the extensive coverage, a clean scan and a green test suite together can still leave gaps. This is why code verification relies on many filters working together to be effective.
A tempting conclusion we might make is that more checking is always better. However, there is a tradeoff at the heart of code verification. Every filter can make two kinds of mistakes:
False Positive: This means flagging something as a problem when the code is actually fine.
False Negative: This means staying quiet while a real bug slips through.
If we tune a tool to catch every possible issue, it can flood the developers with false alarms. However, if we tune it to stay quiet unless it is certain, it can start to miss real defects. The two aspects pull against each other.
False alarms carry a pretty steep cost. When a tool raises frequent false alarms, developers start ignoring it. However, this habit can be catastrophic. Even an occasional real warning can get waved away with the rest. Research on static analysis tools describes this pattern, where high false-positive rates erode trust until teams switch the tool off or don’t pay attention to its warnings. However, doing so reopens the door to the very bugs the tool was meant to stop [7].
This is why a good code verification setup cares as much about signal quality as about coverage. Andrea from Sonar described the balancing act as almost a CAP theorem for code verification. This was built from the classic idea that you can push some properties only at the expense of others. The three competing priorities in the case of code verification are speed, accuracy, and coverage, and no tool fully wins all three. Andrea’s team tries to keep the focus on humans with a simple rule that a finding a developer can act on is worth raising.
The positioning of the filter can also change the cost associated with a mistake. This brings us to the overall setup of the code verification pipeline.
Filters run at different moments in the lifecycle of a change request. The timing determines their cost. If we spread the moments out in order, we get a pipeline. A change begins in the developer’s editor, moves to a set of automated checks that fire the moment code is committed, then to review, then to a merge, then out to deployment and live monitoring.
The same check becomes more expensive the later it runs. This is because more work piles on top of the mistake as development progresses. For example, catching a flaw in the editor may cost a brief moment of the author’s attention. However, catching that same flaw after it reaches production can result in an incident or a rollback. There might also be user impact. This is the real meaning of the phrase “shift left”: moving checks earlier in the pipeline so problems surface while they remain cheap to fix.
Often, vendors might try to market this approach with precise multipliers. You might come across claims that a bug costs ten times more at each stage. While the exact numbers deserve skepticism, the important takeaway is that the general direction of pushing checks earlier in the pipeline is beneficial from a cost point of view.
This stack of filters was built for a setup where developers wrote most of the code. As we have seen, this assumption has weakened over the last few years with the rise of AI-coding tools. This change has an impact on every layer in two distinct ways.
The first pressure is volume.
When an agent writes a thousand lines in the time a person once wrote a hundred, the review burden increases dramatically. There is also a subtle effect on batch size, meaning the amount of change bundled into a single review. AI-based coding tools tend to produce larger changes, which are harder to review. This is because attention spreads thin across a big diff and small mistakes can slip through more easily. Andrea mentioned this failure mode with a line many developers will be familiar with. The reviewer who faces a 5,000-line pull request types “looks good to me,” and figures that things will anyway surface during production.
The second pressure concerns the kind of mistakes that AI can make. A study across more than a hundred models tested the security of AI-generated code and found that it introduced a known security flaw in roughly 45 percent of cases [5].
Over the same period, while these models have become far better at producing code that runs cleanly, their security checks have remained mostly flat. In other words, AI has improved sharply at making the code work, but only a little at making that code truly safe. If anything, the gap between the two aspects has widened. A separate analysis of millions of code changes has also found rising duplication and falling reuse [6].
When there is more code than developers can carefully read and analyze, the natural approach is to hand over some of the code verification to machines. This is why AI-driven code review has gained real momentum over the past few years.
An AI code reviewer offers three main advantages:
Speed: It scans a change the moment it appears, before a human has time to look.
Coverage: It catches a meaningful share of bugs and security issues early.
Consistency: It applies the same standards across every change and every team member. Due to the probabilistic nature of AI code review, maintaining consistency can be challenging. It’s important to have multiple layers of review that include both AI-driven tools and other deterministic algorithmic tools.
This review process can also run inside the agent’s own loop. The agent writes a draft, the reviewer flags problems, and the agent corrects them before a human developer ever gets a chance to look at the code. This tightens the feedback loop and clears routine work off a human reviewer’s plate, ultimately helping teams handle larger volumes of generated code.
See the diagram below:
However, this approach also has a risk. A reviewer model built from the same kind of model as the code creator tends to work with the same assumptions. It would therefore have the same blind spots. When both the writing of the code and reviewing it depend on similar training and similar patterns, the reviewer can confirm that the code looks right. The key question about whether the code does what was actually intended stays irrelevant. In other words, two similar models can resemble one opinion stated twice more than two completely independent checks.
Consider an agent that turns a ticket into a function. An AI reviewer scans it and reports the code as clean. The code compiles, runs, and matches common patterns. Whether it does what the ticket truly meant cannot be answered by pattern-matching alone. This is where different views exist:
Some argue that the models have grown capable enough to reduce or even remove the human review step.
Others hold that people remain essential for judgment about architecture, context, and accountability.
Both camps make a fair case. The reasonable answer today is that it depends on what you are shipping and the cost of a potential mistake.
Let us now see how these ideas assemble into a real workflow.
Everything starts with the context. Most engineering happens in brownfield code, meaning large existing codebases with history and quirks, rather than greenfield projects that have started from scratch. An agent looking into that code without guidance works out the layout on its own, but it does so differently each time. Andrea called that inconsistency “a box of chocolates”, where the result you get back varies from one run to the next and from one developer to the next.
A mature setup handles this by feeding the agent a shared and consistent picture up front. This includes the real architecture, the coding guidelines for the language, and rules that try to capture the intended design. Three loops take care of the verification part. Here’s a brief breakdown of the loops:
Agentic Loop - Where agents iteratively build: It optimizes code generated within the agentic sandbox and improves agent effectiveness. It also reduces token costs, improves output quality, and reduces risk.
CI verification loop - The validation pipeline for all code: Deals with code review, zero-trust, multi-layered verification, and quality gate at sandbox exit. It also merges fixes at high velocity and volume with confidence.
Code maintenance loop - Background remediation of tech debt: It continuously patrols to address legacy issues in the background agentically. Cleaner code makes it easier for coding agents to work efficiently.
Andrea also helped sketch what a mature setup in 2026 might look like. In the case of Sonar, the components of the modern stack are as follows:
Verification Engine: It consists of thousands of rules doing automated code analysis for reliability, maintainability, and security across more than 40 programming languages, frameworks, and IaC technologies. Sonar’s offerings, SonarQube (what it’s most known for) and Sonar Vortex (a new solution), provide this in the agentic loop.
AI Code Review: A newer capability built from AI, that came through Sonar’s acquisition of Gitar. It uses carefully written instructions rather than fixed rules. Its real value is making each finding explainable to the reviewer, so that a 5,000-line change becomes something a developer can reason about. This sits in the CI verification loop.
Remediation Agent: Aimed at the existing backlog, working through old issues progressively to clean up history rather than only guarding new code. This covers the code maintenance loop.
Other companies are more or less converging on a similar idea.
There is also a big advantage to keeping the code clean. The team at Sonar measured what happens when AI-generated code, which is often tangled and rather dense, is left to evolve across many developer sessions over half a year or more. They found that messy code ultimately starts to cost more tokens to work with. This is because the AI model needs to spend more effort understanding it every single time there is a change.
One more aspect sits at the very front of this debate. It concerns secrets, meaning credentials like API keys and passwords. The danger with secrets is rarely intentional sabotage. Most of the time, someone pastes code or loads a configuration file into an AI session. The secret then rides along with the change, and it is usually caught too late, once it has already become part of a commit or a log file.
The fix for this is to run a scanner right at the terminal, before the developer pastes the code. In other words, the goal would be to stop it as early as the process allows. Sonar calls this approach “starting left”. This is one step earlier than the familiar shift left that we talked about. Andrea’s advice is that every developer should run a guard like this to ensure that the secrets remain safe.
All these points lead to one important practical question.
How much code verification does a given change actually need?
The answer is that it depends on what a specific failure would cost. Determining the cost is the real skill that requires insight. For example, a typo on a marketing page and a bug in a payment system deserve very different scrutiny. A developer can probably fix the typo on a marketing page in a minute without much effort. However, the bug in the payment system can move money to the wrong place, break trust, and trigger an unwanted news item.
Mature teams treat verification depth as a dial dependent upon the risk factor. Low-risk changes pass through with light automated checking. However, high-risk changes often get routed to human eyes and undergo heavier scrutiny.
Deciding the exact position of where the line should be drawn is a judgment call that should be made by the team. For example, a comma can be the difference between a working operating system and a crash, so the risk appetite has to be chosen in a deliberate manner. There is no fixed rule that can be applied to all situations.
Agents take a change as far toward a clean and verified state as they can on their own, while staying within guardrails. They can merge low-risk work automatically while routing riskier stuff to a human developer. However, setting these tiers properly so that a change is routed to the right level of checking is an emerging practice that is only going to get more important with time.
The center of gravity in software development is shifting. Writing code has now become the faster activity. On the other hand, verifying that same code, confirming that it is correct, secure, and worthy of real users, is where the effort seems to be increasing.
As we have seen, code verification works as a stack of filters where each filter trades a bit of cost for a bit of confidence. Each filter covers a weakness in the one above it. Those filters divide into static and dynamic families. Every filter balances false alarms against missed bugs, and moving filters earlier keeps their mistakes cheap.
The flood of AI-generated code has an impact on all of it, raising both the volume and the risk. Also, the tempting shortcut of letting AI review for AI has a real catch, since a machine checking a machine can agree that code looks fine while the more important checks get ignored.
The human side to this shift is that as writing code gets cheap, the developer’s work becomes even more important. Developers need to spend more time orchestrating agents by providing instructions. They might have to focus more on the older and harder problem of knowing what to build at all. Cheaper code generation provides more room for spending time on that kind of judgment rather than removing the need for it.
References
CrowdStrike outage: We finally know what caused it and how much it cost
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
We are Changing our Developer Productivity Experiment Design
AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones
FP-Predictor: False Positive Prediction for Static Analysis Reports
2026-08-22 23:31:34
Many teams over-provision containers, underuse spot instances, and have no visibility into which pods are burning budget. Get the eBook from Datadog, which covers five practical optimizations for Kubernetes and ECS environments with specific techniques your team can apply today.
You’ll learn how to:
Pinpoint idle containers, over-provisioned pods, and unused clusters draining your cloud budget.
Right-size CPU and memory with resource requests, limits, and automated cost recommendations.
Cut costs up to 90% with spot instances and savings plans and know exactly when to use each
This week’s system design refresher:
Ollama vs vLLM vs SGLang
How does Claude’s text watermark work?
Top 12 Agent Skills You Should Know
Git Workflow: Essential Commands
Apache Kafka vs. RabbitMQ
To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. The diagram below shows the differences and the main techniques behind each engine.
Ollama: A local user calls the OpenAI-compatible API, and requests line up in a FIFO queue. Then Ollama runs a pre-quantized GGUF model, a compressed format it pulls, and the response comes back to the user.
Ollama is best for local dev, prototyping, and laptop-scale hardware.
vLLM: Many users hit the server at once, and continuous batching slots new requests into the running batch instead of making them wait for it to finish. PagedAttention stores the KV cache, the memory a model keeps for tokens it has already processed.
vLLM is best for high-traffic serving, max GPU utilization, and thousands of concurrent requests.
SGLang: Agents and multi-turn chats send requests whose prompts overlap heavily. A prefix-aware scheduler routes them through the RadixAttention cache, a radix tree that reuses every shared prefix instead of recomputing it.
SGLang is best for AI agents and tool loops, multi-turn chats, and JSON/regex outputs.
Anthropic recently shared their intent to watermark text so they can identify AI-generated text. This post is based on my understanding of how it works.
LLMs produce text word by word. At each step, they generate probabilities for the next likely word. Instead of sampling randomly from those words, the watermarking trick changes which words are allowed to be picked.
How to watermark a response?
Step 1: The model produces probabilities for the next word.
Step 2: Normally a random number generator picks one of the good candidates. With watermarking, a keyed function takes a secret key plus the previous few words and decides which candidates are valid to pick from.
Step 3: This repeats for the whole response. Places where there are multiple plausible choices carry the watermark signal.
How to detect a watermarked text?
Step 1: For any candidate word in the text, we check whether it is a valid choice based on the secret key and the few preceding words. If the word is valid, that is counted as a match.
Step 2: Run this across the entire text. Watermarked text matches far more often. The overall match rate can be treated as an AI-generated score.
I’m personally getting quite annoyed by the false negatives from all these AI text detection techniques, especially for technical writing.
What's your thoughts on AI text detection? Do you think AI text detection is useful, or will it create more problems?
Agent skills are instructions and scripts that teach your LLM agent a new skill. The diagram below shows the 12 most-starred skill repos on GitHub as of August 2026.
Superpowers (obra/superpowers): This skill makes your agent plan before it writes code.
skills (mattpocock/skills): Matt Pocock's personal skill set makes your agent challenge your plan first. This is useful as agents can sometimes be too soft.
andrej-karpathy-skills: Multica AI distilled Karpathy's advice on AI coding pitfalls into one skill.
everything-claude-code: Skills that help you set up your coding agent. This is useful when you are starting Claude Code from scratch.
skills (anthropics/skills): This is Anthropic's official skills. It makes your agent capable of creating outputs like Word or PDF files.
ui-ux-pro-max-skill: This has instructions that teach your agent how to prevent AI-like designs.
caveman: Julius Brussee's skill makes your agent reply in short caveman speak.
ponytail: Dietrich Gebert's skill teaches your agent how to write code that is simple and clean.
agent-skills: Google's Addy Osmani included production-grade engineering practices in a skill
graphify (safishamsi/graphify): This skill converts a codebase into a knowledge graph, so an agent can navigate easier.
Understand-Anything: Egonex AI converts a codebase into visual maps to explore.
impeccable (pbakaus/impeccable): This skill makes an agent better at UI polish.
Over to you: Which skill would you add to this list?
Git has a lot of commands. Most workflows use a fraction of them. The part that causes problems isn’t the commands themselves, it’s not knowing where your code sits after running one.
Working directory, staging area, local repo, remote repo. Each command moves code between these. Here’s what each one does.
Saving Your Work: “git add” moves files from your working directory to the staging area. “git commit” saves those staged files to your local repository. “git push” uploads your commits to the remote repository
Getting a Project: “git clone” pulls down the entire remote repository to your machine. “git checkout” switches you to a specific branch.
Syncing Changes: “git fetch” downloads updates from remote but doesn’t change your files. “git merge” integrates those changes. “git pull” does both at once.
The Safety Net: “git stash” is your undo button. It temporarily saves your uncommitted changes so you can switch contexts without losing work. “git stash apply” brings them back. “git stash pop” brings them back and deletes the stash.
Kafka and RabbitMQ both handle messages, but they solve fundamentally different problems. Understanding the difference matters when designing distributed systems.
Kafka is a distributed log. Producers append messages to partitions. Those messages stick around based on retention policy, not because someone consumed them. Consumers pull messages at their own pace using offsets. You can rewind, replay, reprocess everything. It is designed for high throughput event streaming where multiple consumers need the same data independently.
RabbitMQ is a message broker. Producers publish messages to exchanges. Those exchanges route to queues based on binding keys and patterns (direct, topic, fanout). Messages get pushed to consumers and then deleted once acknowledged. It is built for task distribution and traditional messaging workflows.
The common mistake is using Kafka like a queue or RabbitMQ like an event log. They’re different tools built for different use cases.
Over to you: If you had to explain when NOT to use Kafka, what would you say?
2026-08-20 23:32:18
A schema change is usually one of the most difficult types of change for a software system. However, it might look quite small and simple in review. For example, it might be something as simple as a column being renamed, or a new field being added to a particular event, or a response payload dropping a field that was not being used.
To make matters more complicated, the migration goes smoothly and cleanly during the staging phase, but unrelated services and components start failing as soon as the change is deployed to production. On investigation, it is found that nothing was wrong with the migration itself. But it took effect while two versions of the application were still running against the same database, and only one of those versions referenced the modified schema.
This situation is common with schema-related changes. It is also not limited to the deployment window. For example, rows written years ago can get produced by application code that has since been replaced. The messages sitting in a queue were published before the current version of the consumer was written. Mobile app versions from eighteen months back are still installed on real devices and still calling the API. In each case, data written under a particular schema version is read under a different version, resulting in multiple issues.
In this article, we will look at schema evolution and strategies for the same. Here’s what we will cover:
Why more than one schema version is always in play at the same time
Backward and forward compatibility
Which changes break consumers, which do not, and the qualifiers that decide it
Expand and contract migrations
Schema registries and their use
How the same problem differs across databases, APIs, and event streams
Versioning strategies and deprecation timelines
2026-08-19 23:31:18
Turning new models into systems that work inside real customer operations is still hard.
That gap is creating demand for engineers who can move between code, customer context, and production outcomes. Enter: the forward deployed engineer.
The free State of FDE Jobs 2026 report maps the emerging labor market around this work.
Imagine an AI-based retrieval system pointed at five years of your team’s engineering documents, including design docs, incident postmortems, and architecture decision records. Someone asks which service owns the payments retry logic, and a pretty accurate and well-cited answer is provided by the system. However, when someone asks which failure causes recur most often across all the postmortems, the quality of the answer goes down.
Depending on the setup, the response might list a handful of incidents that happen to use the word recurring, but we don’t get any idea of the underlying pattern from the answer. In other words, the reason for asking the question is not fulfilled.
Both questions can look similar from the outside. Architecturally, however, they are opposites:
The first has an answer that can be found in a specific document, which is precisely what similarity search was built for.
The second has an answer that shows up only after the entire collection has been surveyed and understood. This requires a completely different retrieval mechanism.
GraphRAG was designed to handle the second kind of questions, and we are going to learn more about it in this article. Here’s what we will cover:
How standard RAG retrieval works, and where it reaches its limit
Knowledge graphs, and how one gets built from ordinary documents
The GraphRAG indexing pipeline
Community detection and hierarchical summaries
Local search and global search
Cost, latency, and maintenance tradeoffs
When standard RAG remains the better option
Agentic RAG
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Standard RAG (Retrieval Augmented Generation) depends on a compact pipeline.
We take a collection of documents, slice each one into chunks of a few hundred to a few thousand tokens, and pass every chunk through an embedding model. The embedding model returns a vector, which is basically a long list of numbers standing in for the meaning of that text. Chunks with related meanings produce vectors that sit close together in the same numeric space. All of those vectors go into a vector index.
At query time, the same treatment applies to the question. The question also becomes a vector, the index returns the handful of chunk vectors closest to it, and the original text of those chunks gets placed into the prompt alongside the question. The language model then generates an answer from the supplied text.
The whole design rests on one simple assumption, which is that text answering a question would resemble that question. For a large share of queries, this assumption holds up well. For example, a question like “Which service owns the payments retry logic” contains the same vocabulary as the architecture decision record where that ownership was recorded. The vectors land near each other, retrieval returns the right document, and the citation points somewhere a reader can actually verify.
This assumption about questions being similar to the answers holds for a specific class of questions, but it is by no means a universal thing.
Microsoft’s GraphRAG documentation distinguishes local queries from global queries. A local query has an answer that resembles the query and lives inside a small number of text regions, which covers most who, what, when, and where questions. A global query requires reasoning across large portions of a dataset, or across all of it.
Our two example questions from earlier land on opposite sides of that line. The question “Which service owns the retry logic” is local. However, the question “Which failure causes recur most often across all postmortems” is global.
The reason the answer for the second one goes down in quality is that the phrase “recur most often” produces a vector, and the index returns whatever appears nearest to it. Across a corpus of incident reports, the nearest neighbours will be documents using words like recurring or frequent, which is a coincidence of vocabulary. However, the real answer to the question exists across two hundred documents as a distribution, which spans the corpus rather than occupying one retrievable location.
A reasonable objection at this point is that modern context windows are large enough to sidestep the problem entirely. Microsoft tested exactly that, comparing GraphRAG against vector retrieval pulling in 8,000 and then 64,000 tokens of context. However, on global questions, the larger window left the gap open on comprehensiveness, diversity, and quality of supporting source material.
This outcome is usually labelled as a hallucination problem. What actually happens is that retrieval returns material with little bearing on the question, and the model produces fluent text from it.
Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.
More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.
Join us for a FREE webinar on Sep 2 to see:
Where teams get stuck on the AI maturity curve and why common fixes fall short
How a context layer solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context layer
If you want to maximize the value you get from AI agents, this one is worth your time.
Crossing this boundary in terms of the quality of answers requires recording how documents relate to one another, instead of treating each chunk as an independent unit of text. A knowledge graph is one way to record it.
A knowledge graph stores two kinds of things:
Entities are the nouns a corpus talks about, such as people, services, teams, incidents, and decisions.
Relationships are the typed connections between those entities.
Both carry a plain-text description.
Take one sentence from an incident postmortem: “The checkout service began returning timeouts after the payments team deployed the new retry handler on March 3.” Extraction over that sentence produces entities for the checkout service, the payments team, and the retry handler, along with relationships recording that the team deployed the handler and that the deployment preceded the timeouts.
Once thousands of sentences have each contributed nodes and edges, paths appear that no single document contains. An engineer may be namednamed in one design doc, a service is named in a second, an incident is described in a third, and the path from that engineer to that incident runs through both intermediate nodes.
LinkedIn’s customer service team published results from this approach at SIGIR in 2024. Their support tickets had been stored as plain text, which discarded the internal structure of each ticket along with the connections between tickets. Rebuilding retrieval around a knowledge graph that preserved both improved mean reciprocal rank by 77.6 percent, and median per-issue resolution time dropped 28.6 percent in production. Similarly, Neo4j’s documentation separates the lexical graph, which links documents to their chunks, from the entity graph, which links the things those documents describe.
Most GraphRAG systems build both and query across them.
Building that graph from raw documents is a pipeline, and most of its cost concentrates in a single stage.
For reference, Microsoft’s documented indexing workflow runs through six phases:
Documents are sliced into text units, the same chunking step standard RAG performs.
A language model processes each text unit and extracts entities carrying a title, type, and description, along with relationships carrying a source, target, and description.
Entities sharing a title and type are merged across text units, and their descriptions collect into an array. A second language model pass compresses each array into one description. Relationships receive the same treatment.
Claim extraction runs optionally, producing time-bound factual statements about entities.
The assembled entity graph is clustered into a community hierarchy.
Community reports are generated, and text units, entity descriptions, and report contents are embedded into a vector store.
The merge step accounts for a lot of the expense. For example, a service mentioned across two hundred documents produces two hundred separate descriptions during extraction. Every one of those has to be reconciled into a single coherent description before the graph becomes usable.
Every extracted entity, relationship, and claim retains a pointer back to the text unit it came from. This pointer is what allows a generated answer to cite a specific paragraph in a specific document.
Also, two language model passes over an entire corpus add a substantial amount of inference. Microsoft’s documentation estimates graph extraction at roughly 75 percent of total indexing cost. Lastly, extraction quality also depends on prompts tuned to the domain.
As a separate point, FastGraphRAG replaces the language model in extraction with traditional NLP, treating noun phrases as entities and co-occurrence within a chunk as a relationship. Indexing becomes far cheaper, and the resulting graph carries considerably more noise
A graph of entities and relationships answers connection questions well. However, answering whole-collection questions requires one more layer on top of it.
GraphRAG runs hierarchical Leiden clustering across the entity graph. The algorithm recursively partitions the graph into clusters, called communities, and keeps subdividing until communities fall below a size threshold. The output is a hierarchy with several levels.
These levels behave like a resolution control over the same underlying graph. Level 0 contains a small number of broad communities, each covering a large region. Deeper levels contain many more communities, each covering a narrower region. A single payments community at level 0 might split into separate communities for retry behaviour, settlement, and fraud checks two levels further down.
For every community at every level, a language model generates a community report. Each report contains an overview of that community along with its key entities, relationships, and claims. Reports are then summarized again into shorthand versions for compact use at query time.
This is the step that makes whole-collection questions answerable. A summary of what a cluster of documents collectively says gets written during indexing, well before anyone asks about it. When a global question arrives, the material required to answer it already exists as text.
Which particular level supplies the reports is a decision with real consequences. Microsoft’s documentation states that response quality is heavily influenced by that choice. Lower levels produce more thorough answers because their reports carry more detail, but they also cost more time and more tokens because there are many more reports to process.
With a graph and a hierarchy of reports sitting on disk, retrieval can follow two structurally different paths. GraphRAG supports both.
Local search begins by matching the query against entity description embeddings, which produces a set of entry-point entities. From each entry point, expansion proceeds along five directions in parallel:
Text units that mention the entity.
Community reports that contain it.
Neighbouring entities connected to it.
The relationships forming those connections.
Covariates, meaning any extracted claims attached to it.
Each of those candidate sets is ranked and filtered independently. The survivors are packed into a single context window of predefined size. The expansion is bounded, and the ranking is explicit, which makes local search closer to a structured gather-and-rank operation than to open-ended pathfinding across the graph.
Global search leaves the entity graph untouched. Community reports from a chosen hierarchy level are split into batches, and those batches are shuffled so that batch ordering stays randomized. A map stage runs each batch through a language model and produces an intermediate answer where every point carries a numerical importance rating. A reduce stage then collects the highest-rated points across all batches and generates the final answer from them.
The mapping back to our payments questions can be made clearer now. The question: “Which service owns the retry logic” names an entity, so local search locates it and expands around it. Also, the question “Which failure causes recur most often” names no entity in particular, so global search aggregates across pre-written reports covering the whole corpus.
See the diagram below:
A third mode, DRIFT search, blends the two. It starts by comparing the query against the most relevant community reports to produce a broad initial answer along with follow-up questions, runs local search against those follow-ups, and returns a hierarchy of questions and answers ranked by relevance.
GraphRAG also ships a basic search mode, which is plain top-k vector retrieval, for queries where that remains the appropriate tool. When an answer comes back broad and shallow, or narrow and precise, the query mode usually explains it.
[Diagram 6. Local search and global search side by side] Left panel traces the query to matched entities, then the five parallel expansion streams, then per-stream ranking and filtering, then a single assembled context window. Right panel traces the query alongside shuffled community report batches, into parallel map calls producing rated intermediate answers, then filtering, then the reduce call producing the final answer.
What we have looked at so far are the various capabilities associated with GraphRAG. However, cost determines whether a specific capability is worth acquiring for a given system.
Standard RAG has a modest cost at both ends, with one embedding pass at index time and one nearest-neighbour lookup per query. In contrast, GraphRAG redistributes the spending considerably. Index time absorbs two language model passes over the corpus plus report generation for every community at every level. Query time then splits, with local search running cheaply against a prepared context window, and global search running a language model across many report batches for a single question.
The index is also a derived artifact. New documents arriving means extraction, clustering, and summarization run again over the affected material, and the community hierarchy itself can shift as the graph grows. For a corpus that changes daily, this becomes an ongoing operational commitment.
Microsoft’s own follow-up work addressed the cost directly. For example, LazyGraphRAG builds its index using NLP rather than a language model, skips summarization entirely, and defers all language model work to query time. In this case, indexing cost matches vector RAG and lands at 0.1 percent of full GraphRAG. Global-query quality stays comparable to global search while query cost drops by more than a factor of 700.
Microsoft still argues against making every deployment LazyGraphRAG. Their stated reasoning is that the pre-built entity, relationship, and community summaries carry value beyond question answering, because people read and share those reports directly.
Two findings from Microsoft’s own evaluations are as follows:
Vector RAG remains the stronger option for local queries, where the answer resembles the question and sits in a specific region of text.
GraphRAG’s measured advantage lies in comprehensiveness, diversity, and supporting source material. On faithfulness, it scored at a similar level to baseline RAG.
The second point matters whenever someone asks whether GraphRAG reduces hallucination. The evidence supports better coverage and better sourcing, but it stops short of claiming better factual accuracy per individual claim.
Since different question types favour different retrieval strategies, committing to one strategy when a system is built gives up the others.
Agentic RAG helps formulate a response to that constraint.
A language model classifies the incoming query, selects a retrieval strategy, executes it, and synthesizes the result. Available strategies might include vector search for local questions, global search for corpus-wide questions, a SQL query for structured data, and web search for anything current.
LlamaIndex documents a two-layer version of this:
A composite retriever selects which index to query, guided by a description supplied for each index.
Within the selected index, an auto-routed mode then selects which retrieval method applies to that specific query.
In other words, routing decisions happen at both layers.
The approach carries its own costs. It adds a language model call ahead of retrieval, which increases both latency and per-query spend. Routing errors also produce a debugging problem, because a poor answer can come from a perfectly good retrieval running under the wrong strategy.
The overall progression from basic RAG through advanced RAG to GraphRAG and then agentic RAG describes a sequence of decisions about when a system commits to a retrieval strategy. It works well as a ladder where each step outperforms the one below it.
In this article, we’ve gone deep into GraphRAG and understood it in detail. Here are the key learning points to remember:
Local queries have answers that resemble the question and sit in a small number of text regions, while global queries require reasoning across large portions of a collection.
Similarity search returns whichever chunks sit nearest the query vector, so global questions tend to retrieve vocabulary matches instead of the underlying pattern.
Larger context windows leave that gap open. Vector retrieval with 64,000 tokens still trailed on global questions in Microsoft’s testing.
A knowledge graph stores entities and typed relationships with descriptions, preserving connections that plain chunking discards.
GraphRAG indexing runs two language model passes over the corpus, one to extract entities and relationships and one to merge their descriptions.
Graph extraction accounts for roughly 75 percent of indexing cost, making it the first stage to examine when reducing spend.
Hierarchical Leiden clustering produces communities at several levels of resolution over the same entity graph.
A community report is generated for every community at every level, so summaries of what the corpus collectively says exist before any question arrives.
Local search expands from matched entities and ranks the results into one context window, while global search runs map-reduce across community reports.
The index is derived and perishable. LazyGraphRAG responds by moving language model work to query time, cutting indexing cost to 0.1 percent of full GraphRAG.
Vector RAG remains stronger for local queries, and agentic retrieval selects a strategy per query rather than committing to one when the system is built.