2026-09-16 23:31:42
Your agent returns something odd. Was it the prompt, a tool call that timed out, or a response your code could not parse? Without traces, you are guessing.
In this hands-on workshop, Serge from Sentry instruments three agents with Sentry Agent Tracing: a chatbot in an ecommerce store, a custom Slack agent, and a GitHub Action that reviews PRs. You will see how to catch bad tool calls and unexpected output, plus how to track token spend and performance across every agent you run.
Imagine a scenario where an employee is stranded at an airport after a cancelled flight. Before booking a hotel, they ask the company’s AI assistant a simple question: “Can I expense a hotel if my flight gets cancelled?”
The company has thousands of documents covering travel, expenses, insurance, employee benefits, and regional policies. Somewhere inside them, a paragraph states that accommodation costs caused by involuntary travel disruption are reimbursable, subject to certain conditions.
For an LLM chatbot, finding that piece of information is harder than it appears. The question mentions a “cancelled flight,” while the policy refers to “involuntary travel disruption.” Other documents discuss hotels but apply to different countries. An older policy may be present that contains a reimbursement limit that has since changed.
In such a case, the LLM must find information that matches the question’s meaning, belongs to the correct policy, and remains valid today. Only then can it write a useful answer that solves the user’s problem.
This is known as the retrieval problem behind many LLM applications. In this article, we are going to look at how LLMs can find a needle in a haystack. Here’s what we will cover:
An LLM needs evidence to answer questions
How documents are turned into searchable passages
How LLMs find the meaning behind different words
How close is close enough
Why searching every passage is too expensive
Following connections to the the right neighborhood
How much searching is enough
What happens when the answer changes
A language model does not automatically know what is inside a company’s private documents. An application must provide that information, either by including documents directly in its input or by retrieving relevant passages when a question arrives.
For a large collection, retrieval offers a way to select a manageable amount of information. An embedding model converts the question into a numerical representation. A search system then uses that representation to find promising passages. The application supplies the original text to the LLM.
The model can then use those passages to explain the policy and cite the exact sources where it got the information from. This pattern is called retrieval-augmented generation, or RAG. A vector database supports the retrieval part by storing and searching numerical representations of content.
This division of responsibilities is really useful. If the application retrieves an outdated policy, even a capable language model will produce an outdated answer. If it retrieves a general hotel-booking rule while missing the cancellation exception, the answer may sound reasonably correct while overlooking the important condition.
Reliable answers therefore depend on what happens before generation. This means we have to make the document collection searchable at the right level of detail.
A travel handbook can cover flights, accommodation, meals, approvals, and insurance. However, treating the entire handbook as a single searchable item produces a broad representation that can obscure a specific rule.
Instead, the application divides documents into smaller units called chunks. One chunk might describe hotel reimbursement, while another explains approval requirements. Search can then identify a particular passage instead of merely identifying the handbook that contains it.
Chunking creates a balance between precision and context. A small chunk may focus tightly on the question but leave out an exception. A large chunk may preserve the exception while including several unrelated policies. For example, suppose a passage says, “Accommodation expenses are reimbursable following a cancellation.” And the next sentence adds, “This applies only when accommodation is not provided by the airline.” Separating those sentences could cause the assistant to give an incorrect answer despite having the relevant text.
A useful chunk therefore preserves a complete idea wherever possible. Section headings and limited overlap between neighboring chunks can help retain context. The goal is to create passages that remain understandable when retrieved independently.
For our example question by the employee, the ideal searchable unit contains the reimbursement rule, its conditions, and enough identifying context to establish which policy it belongs to.
Embeddings make it possible to search for related meaning even when the wording differs.
An embedding model takes a passage and produces a vector: a list of numbers, often containing hundreds or thousands of decimal values. Think of it as a map of text. Passages with related meanings occupy nearby positions. For example, a question about hotel expenses after a cancelled flight should appear closer to a travel-disruption policy than to instructions for resetting a password.
The real representation has many more dimensions than a physical map. Individual dimensions also don’t have simple labels such as “hotel,” “flight,” or “reimbursement.” Meaning is represented through patterns across the complete vector.
The trick is that the application embeds the question into the same space as the document chunks. This way, it can compare the question’s vector with stored vectors to identify nearby passages. Also, the query and document embeddings must come from compatible encoders. However, matching vector lengths alone doesn’t make two models compatible.
Each searchable record also needs a connection to the original text. The vector helps locate a passage, but the LLM needs the words themselves to interpret the rule. Therefore, a record should connect an embedding with a chunk identifier, the passage text, or its location. Also, metadata such as document ID, section, effective date, and version are present. These fields can be stored together or across connected storage systems. Structured identifiers make it possible to retrieve and maintain all chunks belonging to a document.
The search system needs a precise definition of “nearby.” This definition comes from a distance or similarity metric, which compares two vectors and assigns a score. There are different metrics around this:
Cosine similarity compares their directions while ignoring their lengths.
Euclidean distance measures the straight-line distance between their endpoints, so vector length can influence the result.
Dot product reflects both alignment and length.
Normalization rescales vectors to length one. When both query and document vectors are normalized, dot product equals cosine similarity. Euclidean distance then produces the same ranking, although its scores differ. Dot product also works with unnormalized vectors when that matches the embedding model’s design.
The choice of the metric should follow the embedding model’s intended use. Choosing a metric simply because it is popular can change the ranking in unintended ways.
A similarity score should also be interpreted carefully. A score of 0.85 does not mean that a passage has an 85% probability of answering the question correctly. It describes a mathematical relationship between representations.
The passage may discuss the right subject while stating the wrong regional policy. It may be outdated or omit an exception. Similarity provides evidence of relevance, but additional checks are needed before that evidence becomes an answer.
Searching every vector is straightforward, but the work grows with the collection.
A flat index compares the query with every eligible vector and returns those with the best scores. For a fixed number of dimensions, the comparison work grows roughly in proportion to the vector count. This is the meaning of O(n). If the collection doubles, the number of vector comparisons doubles. Longer vectors also require more work per comparison.
Flat search produces exact nearest neighbors under the selected metric. “Exact” describes the numerical search result. It doesn’t guarantee that those neighbors contain the correct answer.
On the other hand, an Inverted-File index reduces this work by organizing vectors into groups. During construction, clustering identifies representative centers and assigns vectors to nearby groups. A query can then select promising groups and search their contents. The parameter commonly called nprobe controls how many groups are examined. However, searching more groups generally improves recall but also increases work.
Imagine one million passages divided into 1000 groups. Searching 10 groups might involve roughly 10K passages rather than the entire million. Of course, real groups are uneven, and selecting them also has a cost. The main tradeoff is that a useful passage can belong to a group that is skipped by the search. These groups are mathematical neighborhoods, not tidy subject folders. A travel-disruption rule might sit near insurance documents rather than ordinary expense policies.
IVF therefore introduces approximation. It saves work by accepting some risk of missing the nearest vectors.
HNSW (Hierarchical Navigable Small World) avoids exhaustive comparisons by building routes between vectors. Its structure is a graph. In other words, points are connected by links.
Each point represents a vector. The links provide routes through the collection. HNSW organizes these connections into layers, with sparse upper layers and a detailed bottom layer containing all vectors.
In this approach, search starts near the top and moves toward points closer to the query. It then descends through the layers, progressively refining the search. At the bottom, it explores a broader set of nearby candidates to select the results.
You can think of this navigation like travelling through a road network. Major routes help reach the right area, while local roads help locate a specific destination. HNSW uses this broad-to-detailed pattern to avoid visiting every point.
Like the Inverted-File index, it performs approximate nearest-neighbor search. However, selected routes can miss a true nearest neighbor. Its benefit is that useful results can often be found with substantially fewer comparisons.
There is no fixed collection size at which flat search must give way to Inverted-File or HNSW. Hardware, vector dimensions, query volume, memory, filtering, and latency requirements all influence the choice. HNSW is a strong candidate when memory permits, but we cannot say that it is automatically the best option for every workload.
A collection searched a few times per day creates a different problem from one serving thousands of simultaneous users. Vector count alone cannot capture that difference.
Approximate search creates a measurable tradeoff between speed and recall.
Let’s say an exact search identifies the 10 nearest vectors. An approximate search returns eight of those same vectors and two others. This measures how closely the approximate search reproduces exact nearest-neighbor results. It doesn’t measure whether the returned passages really answer the employee’s question. Both index recall and actual evidence relevance need evaluation.
HNSW has several settings that have an impact on this tradeoff:
M controls graph connectivity. Higher values generally provide more routes, improving recall while increasing memory use and construction work.
ef_construction controls how broadly the algorithm searches for suitable connections during insertion. A larger value generally produces a better graph but takes longer to build.
ef_search controls the breadth of candidate exploration during a query. Increasing it generally improves recall while increasing search latency. It isn’t the number of results returned or a fixed count of visited points.
The application may need five passages while exploring a much larger candidate set to find them. Result count and search effort are separate decisions.
For tuning, we should use representative questions and actual performance measurements. If a broader search consistently finds a previously missed cancellation exception, the added latency may be worthwhile. However, if answers don’t improve, changing the setting merely adds work.
Similarity must also be combined with rules about which documents are eligible.
For example, our employee needs the policy for their country and business unit, with an effective date that covers the journey. A highly similar passage from another region cannot help answer their query.
We use metadata filtering to define the eligible subset. The request becomes “find the most similar passages among current policies for this employee’s region.”
Pre-filtering identifies eligible records before similarity ranking.
Post-filtering retrieves similarity candidates first and then removes records that fail the conditions.
If only two of the first 20 candidates qualify, post-filtering cannot supply 5 eligible results from that batch. Retrieving additional candidates may help, but requires more work.
Nevertheless, pre-filtering is not automatically faster. The outcome depends on how selective the filter is and how the search engine combines filtering with its index. Graph search adds another complication. Disallowed points may still provide useful navigation routes toward allowed points. Blocking every such point during traversal can make eligible neighbors harder to reach.
Search engines can address this through filtering-aware graph structures, traversal strategies, or an exact scan when the eligible subset is small. Filtering can therefore be integrated into search rather than occurring entirely before or after it.
The collection must remain correct as its documents change. For example, let’s say the company raises its hotel reimbursement limit from ₹5,000 to ₹7,000. If both versions remain eligible for current-policy searches, the assistant may retrieve either figure or receive contradictory evidence.
Changes to embedded text require new embeddings. However, an update doesn’t always require rebuilding every chunk. Unchanged chunks may be reusable, and stable identifiers can support replacing existing records.
A metadata-only change can often be applied without re-embedding, provided that metadata was not a part of the text used to create the vector. Some vector databases can expose separate operations for changing vectors and metadata.
For our policy example, a sensible design can prepare the new version’s chunks, verify their availability, and then change which version is eligible for current searches. Older versions can remain accessible for historical questions.
That transition requires some sort of coordination. Deleting old chunks first can create a temporary gap. Inserting new chunks first can create temporary duplication. Version identifiers and explicit active-version rules help make the change predictable. Lastly, changing the embedding model requires similar planning but on a much larger scale. Existing documents generally need embeddings in the new model’s space, and queries must use the matching representation.
The final retrieval step turns promising matches into evidence the LLM can use. For example, a vector search might return 30 candidate passages. A reranker can compare each passage’s text with the question and select the most useful few.
Hybrid search adds another source of candidates by combining semantic retrieval with keyword-based retrieval. Embeddings help connect “cancelled flight” with “travel disruption,” while keyword search can preserve exact matches for policy identifiers, names, or unusual technical terms. Reranking can refine the combined results.
The application then supplies the selected text and source details to the LLM. For the stranded employee, this should include the current reimbursement rule, the relevant conditions, and enough context to explain how the policy applies to their situation.
The search must also accommodate the scenario of an unanswered question. Every collection has nearest vectors, even when none might contain useful information. Returning the closest passage doesn’t mean that an answer exists. For example, if the retrieved documents discuss ordinary hotel bookings but say nothing about cancellations, the assistant should explain clearly that the available policy details don’t have a clear answer.
Finding a useful passage among thousands of documents requires several parts of an LLM application to work together. Documents first become smaller, meaningful chunks. Embeddings represent those chunks as vectors, allowing the search system to connect a question with passages that express related ideas, even when their wording differs.
As the collection grows, indexes make that search more efficient. Flat search compares every eligible vector, while IVF and HNSW reduce the work through grouping or graph navigation. These approaches introduce tradeoffs between speed, memory, and recall that need to be measured against real questions.
Similarity alone, however, cannot establish whether a passage is suitable evidence. Metadata filters help select the correct region, document type, or policy version. Careful updates keep outdated and duplicate passages from appearing in current searches. Hybrid search and reranking can further improve the evidence selected for the LLM.
The final answer depends on the quality of this entire process. For the employee stranded at the airport, success means finding the current reimbursement rule, preserving its conditions, and explaining it clearly.
2026-09-16 03:31:01
We’re relaunching Build with Claude Code, a 2-day intensive cohort-based course taught by John Kim, who has trained hundreds of engineers at Meta to use Claude Code in real production workflows.
The course kicks off on September 16th, and enrollment closes in 24 hours. If you’ve been thinking about leveling up how you and your team work with Claude Code, this is the moment.
A few things you’ll learn:
The agentic loop, context engineering, and memory layers that make Claude Code useful for real projects
How to build with Claude Code Skills, MCPs, and hooks to give Claude the tools and feedback loops it needs to self correct
Parallel development with Git worktrees, subagents, and agent teams
A capstone project where you ship something real on your own stack
The course includes live sessions, assignments, and office hours, so there’s plenty of room to ask questions and get unstuck.
If you want to learn everything from the fundamentals of Claude Code to advanced production workflows, including working with large codebases, this could be a great way to level up.
2026-09-15 23:31:12
Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.
More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.
Join us for a FREE webinar on Sep 23 to see:
Where teams get stuck on the AI maturity curve and why common fixes fall short
How a context layer solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context layer
If you want to maximize the value you get from AI agents, this one is worth your time.
An LLM can analyze a 100-page document, follow a complicated programming discussion, and refer back to something discussed several messages ago. However, as soon as we open a new chat where the earlier conversation is not present, it immediately forgets everything. It appears that LLMs have the memory of a goldfish.
This observation is indeed true. LLMs usually have no personal or persistent memory of previous interactions. So, how is it able to refer to what we’ve said in the past?
In truth, an LLM doesn’t remember a conversation like human beings do. However, it receives information about the conversation so far with each new message that we send to the LLM. All of this is handled by the application that is built around the model and not by the model itself. For example, a chat application might store messages, maintain summaries of earlier discussions, retrieve relevant memories, and maintain a user profile. It can then place some of that information in front of the model as needed. From the user’s perspective, the model appears to remember. But technically, the surrounding application is doing most of the remembering.
This difference between the model and the application around the model is the key to understanding LLM memory.
This design has important consequences. As a conversation grows, the application must keep processing more text, thereby increasing cost and latency. Eventually, the conversation ends up turning too large to fit inside the model’s context window. At that point, older information must be removed, summarized, or stored somewhere else.
In this article, we will learn how LLMs handle memory so that they are useful to end users in performing complex tasks that require conversation and holding context.
In the context of LLMs, the word “memory” is used for several different things that should not be confused. Let’s look at each type in detail.
During training, an LLM learns patterns from enormous amounts of data. These patterns are encoded in billions of numerical values called parameters or weights.
This is why a model can explain JavaScript, recognize a common historical event, or write an email without receiving that knowledge in the current prompt.
But this is not personal memory. If a user tells the model, “My preferred programming language is TypeScript,” a normal API response does not rewrite the model’s weights. The base model does not permanently learn that fact from the conversation.
The model’s temporary working memory is its context window. This contains everything the model can consider while generating its current response.
It can include:
System and developer instructions
The current user message
Previous messages in the conversation
Retrieved documents
Tool descriptions and tool results
Saved user preferences
Summaries of older conversations
Space required for the model’s output
The context window is closer to a desk than a human memory. The model can work with whatever documents have been placed on the desk. Once the desk is cleared, the model cannot recover those documents unless the application places them there again.
Persistent memory normally lives outside the model in a database, file, vector store, or profile service. When the model needs that information, the application retrieves it and inserts it into the current context.
Therefore, the model doesn’t possess persistent memory. It receives persistent information from another system.
Shipping agents to production is the easy part. Keeping them reliable, governable, and improving over time is where most enterprise AI programs stall.
How do top teams do it? They use an Agentic Operating Model (AOM), a step-by-step framework for aligning people, process, and technology so enterprise agents improve as they scale.
In LangChain’s latest guide, you’ll learn:
Why AI agents don’t break like traditional software
The engineering stack that covers the entire agent lifecycle
Shifting from “build and deploy” to “operate and continuously improve”
Consider the simplest possible API request:
{
“messages”: [
{
“role”: “user”,
“content”: “My name is Adam.”
}
]
}The model might answer: “Nice to meet you, Adam.”
Suppose the next request contains only this:
{
“messages”: [
{
“role”: “user”,
“content”: “What is my name?”
}
]
}The model can’t reliably answer because the second request doesn’t include the name. The earlier request has already finished. And there is no private diary that exists inside the model that is kept updated.
For the conversation to work, the application must send the earlier exchange again:
{
“messages”: [
{
“role”: “user”,
“content”: “My name is Adam.”
},
{
“role”: “assistant”,
“content”: “Nice to meet you, Adam.”
},
{
“role”: “user”,
“content”: “What is my name?”
}
]
}Now the model answers “Adam” because the name is visible in the current input.
Most model providers have a stateless Messages API that requires the conversation history to be supplied for a multi-turn conversation.
A new model invocation doesn’t typically start without any knowledge. The model still has several things:
Its trained weights
Its general language abilities
Knowledge acquired during training
Safety and behavioral instructions supplied by the platform
Everything included in the current context
What it lacks is an automatically updated memory of this particular user or conversation.
A better way to put it is that every response is generated from the model’s existing weights plus the context made available for that response. Information from the conversation history is not available unless the surrounding system carries it forward.
Some APIs offer server-managed conversation state. In such a case, we don’t need to resend every message manually. We can provide a conversation identifier, and the server locates the previous messages based on that identifier to reconstruct the necessary context. This makes the API easier to use, but it doesn’t mean that the model has developed a personal memory.
Suppose a conversation contains five user messages and five assistant responses.
When the user sends the next message, the application may construct an input that contains the following details:
System instructions
User message 1
Assistant response 1
User message 2
Assistant response 2
User message 3
Assistant response 3
User message 4
Assistant response 4
User message 5
Assistant response 5
New user messageThe model receives all of this as one large input. For example, in the earlier conversation, the user may have mentioned a database problem, selected PostgreSQL, and asked for TypeScript examples. Therefore, it can continue the answers based on this path in a more natural manner.
To the user, this might feel like the model has remembered things from earlier. However, from the perspective of the model, those details are simply present in the text that is being processed right now. We can’t simply call it “fake memory” because there is real continuity at the application level. A better term for this is reconstructed memory or context-based memory.
A context window of an LLM is measured in tokens. Tokens are small pieces of text. A short word may be one token, while a longer or unusual word may be split into several tokens.
The context window normally contains more than the visible conversation. It may also contain hidden instructions, tool definitions, search results, documents, and space for the answer. Imagine a hypothetical model with a 100K-token context window. An application might need to fit the following into that space:
System instructions: 3,000 tokens
Tool definitions: 8,000 tokens
Conversation history: 55,000 tokens
Retrieved documents: 20,000 tokens
Current question: 1,000 tokens
Remaining response budget: 13,000 tokens
Once the available space is exhausted, the application can’t just keep adding information indefinitely. It must remove, compress, or replace something.
A large context window also doesn’t guarantee perfect recall. As the amount of information grows, it can become harder for the model to differentiate important facts from irrelevant, repetitive, or contradictory material. This degradation is also known as “context rot”. We need to select useful context even if we have a very large context window at our disposal. Therefore, the context window is both a capacity limit and an attention-management problem.
LLM APIs commonly charge for the number of input and output tokens processed. Let’s say each completed conversational round adds approximately 1000 tokens.
Request 1 processes approximately 1,000 tokens.
Request 2 processes approximately 2,000 tokens.
Request 3 processes approximately 3,000 tokens.
Request 10 processes approximately 10,000 tokens.
Across those ten requests, the application has processed approximately 55K tokens of input.
The visible conversation contains only about 10,000 tokens, but earlier parts have been processed repeatedly. A long system prompt, large tool definitions, and retrieved documents can increase this cost even further.
Longer contexts also increase latency. This is because more information needs to be processed before the model starts to answer.
Techniques like prompt caching can help alleviate costs, but they don’t create memory.
Providers can cache a repeated prefix such as the system prompt and previous conversation history. When the next request begins with the same content, the provider may reuse previously computed information. This can reduce cost and latency. However, cached tokens consume part of the context window. Caching merely changes how efficiently repeated context is processed. It doesn’t give the model unlimited memory.
Prompt caching is therefore an optimization, not a memory architecture.
There is no single universal behavior that LLMs perform when the context window fills up. Depending on the API and application, several things may happen.
The API may reject the request because it is too large.
The application may remove the oldest messages.
A chat product may manage history using a rolling window.
The system may replace old material with a compact summary.
Some APIs also provide server-side compaction mechanisms.
Compaction is carrying forward important state in a smaller representation so that long-running interactions continue with lower context usage. This means a very long chat may not contain every original sentence in its active context. The model may instead receive something like:
Conversation summary:
The user is building an invoice service in TypeScript.
PostgreSQL was selected as the database.
The API uses Express and Prisma.
The current problem concerns duplicate invoice creation.
Previous attempts involving application-level checks failed under concurrency.
The summary preserves the central state while discarding greetings, repeated explanations, abandoned ideas, and low-value details. The tradeoff is that summarization is lossy. A small detail that seemed unimportant during summarization may become important later.
Real applications usually combine several techniques to extend memory instead of relying on one. Let’s look at a few techniques in detail.
A sliding window keeps only the most recent portion of the conversation. When new messages arrive, the oldest ones are removed.
For example, the application might always retain:
The system instructions
The latest 20 conversational turns
The current user message
This approach is simple, fast, and predictable. It works well when recent messages matter much more than older ones.
Its weakness is that once an old fact leaves the context window, it disappears. If the user mentioned an important requirement 30 turns earlier, the model may no longer be able to address it.
Summarization periodically compresses older messages into a much shorter description. The summary stays in the context while the original messages are removed.
A common structure is:
System instructions
Conversation summary
Recent unsummarized messages
Current user message
This helps preserve the general direction of the conversation much more efficiently than retaining every original message.
However, a summary is more of an interpretation. It omits nuance, simplifies uncertainty, or accidentally turns an assumption into a fact. If we repeatedly summarize previous summaries, it can gradually distort the meaning of the conversation, much like repeatedly copying a photocopy. Therefore, it’s a much better approach to store important facts separately rather than trusting them to be retained in a simple narrative summary.
Instead of remembering the conversation as prose, the application can extract specific facts into structured fields. For example:
{
“preferred_language”: “TypeScript”,
“database”: “PostgreSQL”,
“framework”: “Express”,
“current_project”: “invoice service”,
“confirmed_decisions”: [
“Use optimistic concurrency control”,
“Do not introduce Redis”
]
}This is more reliable than searching through a long summary when the application needs exact project state.
Structured memory is really useful for the following types of information:
User preferences
Names and identifiers
Confirmed technical decisions
Current tasks
Workflow status
Dates and deadlines
Product configuration
However, structured entity is doesn’t work well for subtle, narrative information that cannot be expressed neatly into predefined fields.
In this approach, a vector store supports semantic retrieval. Instead of putting the entire conversation history into every request, the application divides past conversations into small pieces and creates an embedding for each piece.
An embedding is a numerical representation of meaning. This means that texts concerning similar ideas receive similar representations even when they don’t use exactly the same words.
For example, let’s say the user asks: “Why did we reject Redis for the invoice service?” The memory system searches for past passages semantically related to “Redis,” “invoice service”, and “rejected architecture decisions.”
It may retrieve information such as “we decided against Redis because the deployment environment does not provide a managed Redis service, and PostgreSQL advisory locks already cover the required coordination.”
Only the retrieved passage is added to the current context. Here’s how the process works roughly:
Store selected pieces of earlier conversations.
Convert those pieces into embeddings.
Convert the new question into an embedding.
Find semantically similar memories.
Filter and rank the candidates.
Insert the best candidates into the model’s prompt.
This doesn’t enlarge the context window. It selects which memories deserve space inside it.
However, vector retrieval can also sometimes fail. It may retrieve something similar but irrelevant. It can miss a memory because it was phrased strangely. It can also come up with an outdated decision. Metadata such as user ID, project ID, date, and memory type is therefore essential to make sense of this data.
A long-term profile stores durable facts that may be useful across many conversations. For example, this could include things like:
The user generally prefers beginner-friendly technical explanations.
Examples should use TypeScript where practical.
Explanations should use complete paragraphs rather than fragmented bullets.
A profile should contain stable preferences, not every passing statement. For example, “I am testing Python today” is probably session information. On the other hand, “I use Python for all data projects” may be a durable preference if repeatedly confirmed.
We also need to update profiles. This is because preferences can change over time. Good systems attach timestamps, sources, and sometimes confidence scores to memory entries.
Cross-session memory means that information survives after one chat ends and can influence another chat.
A typical implementation works like this:
During or after a conversation, a memory process identifies potentially durable information.
It stores that information in a user-level, project-level, or organization-level memory store.
A future conversation starts.
The application selects memories relevant to the new conversation.
Those memories are inserted into the new conversation’s context.
The model generates its response using the supplied memories.
The important step is number five. The new model call still needs the remembered information placed into its current context.
In this article, we’ve looked at LLM memory in detail. The key takeaways are as follows:
Model weights contain general knowledge learned during training.
The context window contains information available for the current response.
Conversation storage holds previous messages.
Long-term memory stores selected facts, preferences, and past events.
The memory manager decides what to retrieve and place on the desk.
It can be said that LLMs, along with the application surrounding them, don’t have the memory of a goldfish. They can process an enormous amount of information at once. But they don’t automatically carry personal experiences from one call to the next.
What appears to be memory is a carefully constructed system of context reconstruction, summarization, retrieval, and persistent storage. The quality of an LLM application’s memory depends at least as much on the surrounding architecture as it does on the model itself.
2026-09-14 23:31:02
If you’re shipping AI agents without offline validation, your users are the ones doing the testing. Get the practical framework for evaluating production-grade AI agents before they hit production.
Get the guide to learn how to:
Build annotated test datasets that cover core use cases, edge cases, and adversarial inputs
Design deterministic and LLM-as-a-judge evaluators that reflect real business impact
Trace multi-agent workflows end-to-end during experimentation to catch failures before users do
Prevent model drift by keeping your offline test environment aligned with production
Large Language Models (LLMs) are also software systems just like any other software system we may have encountered. But we cannot test an LLM the same way as an ordinary software system.
For example, a normal function in a software application can receive two numbers and always return the same number as the total. However, when we ask the same question twice to an LLM, it will most likely produce two differently worded answers. Both of those answers may be acceptable. But it makes evaluation tricky. To evaluate an LLM, we have to measure whether the application continues to behave properly across many situations.
“LLM-as-a-Judge” is one part of this evaluation process. It involves using one language model to assess the output generated by another language model. But a judge model isn’t enough on its own. A healthy LLM evaluation system combines several ingredients such as conventional software tests, carefully curated examples, automated checks, model-based judging, human review, and production monitoring.
In this article, we are going to look at the process of LLM evaluation in detail. Here’s what we will cover:
What does it mean for an LLM application to be healthy?
Why are ordinary tests not sufficient for LLMs?
The basic LLM evaluation loop
Golden datasets for repeatable tests of LLM behavior.
Automated metrics: Fast but limited checks
What does LLM-as-a-judge mean?
Different ways a judge model can evaluate an answer
Human evaluation and calibration
The evaluation stack
When can we call an LLM healthy?
The answer is quite simple. An LLM is deemed healthy if it consistently generates useful results while remaining within acceptable limits for accuracy, safety, speed, reliability, and cost. For example, consider a customer-support assistant. We cannot say it is healthy by a single question such as “Did it return the correct output?”
We need to consider several different questions:
Did it understand what the customer was asking?
Was the answer factually correct based on the company documentation?
Did it answer the entire question?
Did it follow the required tone and format?
Did it avoid inventing policies that do not exist?
Did it refuse requests that it should not answer?
Did it respond within an acceptable amount of time?
Did the request cost an acceptable amount to process?
As we can see, these questions are all related to different dimensions of quality, where each dimension is important. We can have an assistant that is friendly and relevant, but it might generate factually incorrect answers. It might be accurate but so verbose that users can’t simply find the answer. It might produce excellent answers but take 30 seconds for every request. Therefore, while evaluating an LLM, we must measure multiple aspects of the system.
We also need to differentiate between the health of the LLM and the application. We can’t simply label the model as the source of all problems. The problems can be caused by various sources. For example, we might have an incorrect prompt, missing documents, poor retrieval logic, incorrect tool calls, stale data, or a change in the surrounding code.
Traditional software tests usually depend on deterministic behavior. If a function receives a known input, the test expects a specific output. For example:
Input: add(2, 3)
Expected output: 5This type of test works only because there is one exact correct answer. However, an LLM’s task is different. For example, consider the question “Explain why a password reset link may expire”
One possible response might begin by talking about security. Another response might first explain the expiration period. The way the sentences are written may be totally different even though the underlying answers are correct.
Three properties make LLM evaluation tricky.
First, the outputs from an LLM are not deterministic. This is because LLMs generate answers using a probability-based approach. We can use the same prompt and model, but the answers may be different. Increasing the model’s temperature only increases the variation. But even if we set a low temperature, it doesn’t make every model and infra combination perfectly repeatable.
Even an exact string comparison can mark many valid answers as failures. For example, the sentences “The payment was declined because the card expired” and “The card’s expiration date caused the payment to fail” convey the same information.
Second, quality is multidimensional and partly subjective. We can’t have a universally correct level of detail, tone, or organization of the answer. A response appropriate for a developer may simply confuse the customer. A short answer may work well in a support chat. But a longer answer might be more suitable in documentation.
This doesn’t imply that quality is impossible to measure. It just means that we need to first define the desired qualities. For example, having a goal like “give a good answer” is too vague to test. A much better option could be something like “Answer the question directly, use only the supplied policy, and explain all required steps”.
Third, the correctness also depends on the context. For example, the answer to “Can this order be refunded?” depends on multiple data points such as order date, product type, account status, region, and current refund policy. A particular response might be correct for one customer and wrong for another.
To evaluate properly, we must therefore include the context available to the model. We must be able to test whether the answer is correct relative to that context.
Of course, traditional testing is still needed for the deterministic parts of an LLM application. For example, things like JSON parsing, permission checks, calculations, database operations, API contracts, and tool execution should continue to rely on the usual unit and integration tests.
A practical LLM evaluation system is built around a repeated process that works as follows:
Collect representative test cases.
Run the application on those cases.
Inspect the resulting answers with several evaluation methods.
Compare the results with the current production version.
Block or investigate changes that cause an important regression.
Monitor real production traffic for problems that the test set missed.
Add newly discovered failures back into the test set.
See the diagram below:
The last step is super important. We need to evolve the useful evaluation dataset by adding more examples. Such a dataset grows as real users reveal unexpected questions, ambiguous instructions, document formats, and new ways the system can fail.
A golden dataset is a carefully curated collection of possible inputs and information about what a good response to such an input should contain. Think of it as something that plays a role similar to a unit-testing suite. But unlike unit tests, we don’t always check the answers based on exact equality.
For example, a test case for a refund assistant might contain the following details:
This type of approach is much more useful than providing one perfectly written reference answer. This is because there may be many reasonable ways to express an answer. The important point is that the LLM makes the correct decision while following the stated constraints.
We should create a strong golden dataset with more than just easy and ordinary requests. It should ideally include the following:
Common requests that represent most real traffic.
Important cases where an incorrect answer would cause significant harm.
Ambiguous questions that require clarification.
Questions for which the answer is absent from the supplied information.
Malicious or irrelevant instructions embedded inside retrieved documents.
Very short, very long, poorly written, and multilingual inputs where relevant.
Previous production failures.
Boundary cases, such as a return made exactly on the final allowed day.
We should also divide the dataset into groups. A development set can be used while prompts are in the improvement phase. However, a separate holdout set should remain less visible during the development phase. Otherwise, the prompt may gradually be tailored to known examples without becoming better for real users.
We should not consider golden datasets as static input-output pairs. Depending on the application, a test case may contain an input, source documents, expected facts, forbidden claims, acceptable tool calls, a scoring rubric, and an optional reference response.
Despite the non-deterministic nature, automated metrics are also useful in testing LLMs. However, automated metrics use code rather than another person or model to inspect an answer. They are also cheap, fast, and repeatable. Therefore, they are useful for running large numbers of tests.
Exact match checks help where the output has to precisely match an expected value. It works well for constrained extraction tasks such as returning a country code, classification label, or database identifier. But it works poorly for open-ended natural-language answers.
Regular expressions and schema validators help check whether an output follows a required structure. For example, an application may require valid JSON containing fields such as customer_id, issue_type, and priority. In such a case, a validator can reliably check whether those fields exist and whether their values use the expected types.
Programmatic checks can help verify URLs, citations, numerical ranges, required phrases, banned phrases, word limits, or whether specific product IDs exist in a database.
BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) compare the words or word sequences in a generated answer with those in a reference answer. BLEU became popular for machine translation, while ROUGE has often been used for summarization. They can be useful for certain large-scale comparisons. But they mainly deal with measuring textual overlap. We can have a response with a different arrangement of words that still preserves the meaning.
Other semantic similarity metrics compare the meanings of responses through embeddings or specialized evaluation models. These metrics are more flexible than word overlap, but we still cannot treat similarity the same as correctness. An incorrect answer about the right subject can still be semantically similar to the reference answer.
The main limitation is that automated metrics generally measure a narrow, observable property. They should be used for things that code can check reliably.
LLM-as-a-Judge involves sending one language model’s answer to another language model and asking that model to evaluate it based on specific criteria.
For example, consider an RAG assistant that answers a question using retrieved company documents. The judge may receive:
The original question.
The documents supplied to the assistant.
The assistant’s answer.
The evaluation rubric.
An optional reference answer.
The judge LLM can then assess qualities such as relevance, factual support, completeness, clarity, and compliance with instructions. For reference, a simplified judge prompt might say:
Evaluate the response using only the supplied policy.
Score each category from 1 to 5:
Accuracy:
Does every factual claim agree with the policy?
Completeness:
Does the response answer every part of the question?
Relevance:
Does it remain focused on the customer’s request?
Instruction compliance:
Does it avoid making promises not supported by the policy?
Return JSON with the scores, a short explanation, and the exact unsupported claims, if any.We can store the output from the judge model to compare with previous evaluation runs.
This approach is far more flexible than exact matching because the judge can recognize that differently worded responses express the same idea. It can also identify subtle failures such as answering only half of a question or introducing a claim that might not be present in the evidence.
There are several common methods for judging an answer. Let us look at each of them in detail.
In this approach, the judge model assigns scores such as 1 to 5 for different quality dimensions. This results in numbers that are easier to track over time as we refine the model under testing.
The main drawback of this approach is that the meaning of a score may be inconsistent. We need to define every score and its rubric properly. For example, consider the following scores:
5: Every factual claim is supported by the provided documents.
4: The main answer is supported, with one insignificant unsupported detail.
3: The answer is mostly correct but contains a meaningful unsupported claim.
2: Several important claims are unsupported or incorrect.
1: The central conclusion contradicts the documents.
The judge model can also decide whether an output meets a minimum standard. We can use it for deployment gates, especially when the requirement is crystal clear. For example, “the answer must not contradict policy, expose personal information, or produce invalid instructions.”
A pass-or-fail result is simple, but it hides smaller changes. A score can decline from excellent to barely acceptable without crossing the failure boundary.
In this method, the judge model receives two answers and decides which one is better. For example, it may compare the response from the current production prompt with the response from a proposed prompt.
Pairwise comparison is often easier than assigning an absolute score. We can decide whether answer A is more useful than answer B more consistently than deciding whether one answer deserves a score of 3 or 4 out of 5.
To make this method better, the order of the answers should sometimes be reversed. Judge models can have a position bias and may start favoring whichever response appears first or second.
Instead of merely producing a score, the judge model can also identify specific problems.
The judge model can list unsupported claims, unanswered parts of the question, irrelevant sections, or violated instructions. This is quite useful during the development phase because it explains why a score has changed from one value to another.
Despite all the advancements in testing, the ultimate test of a model’s ability depends on human evaluation.
Human evaluation involves people reviewing model outputs using the same rubric. This is why domain experts are really important when the correctness of an answer depends on legal, medical, financial, scientific, or company-specific knowledge.
However, since humans are expensive and slow, they cannot inspect every answer. Their best role is often to calibrate things. To make this work, human reviewers rate a representative sample. The results are compared with the ratings of a judge LLM.
For example, let’s say experts mark 100 answers as pass or fail. The same answers are evaluated by the judge model. The exercise reveals how often the judge agrees with the experts, which types of failure it misses, and whether its passing threshold needs adjustment.
Human reviewers can also disagree with one another. This usually means that the rubric is a little ambiguous or that the task is genuinely subjective. We should, therefore, measure reviewer agreement before treating human scores as a perfect reference point.
Human evaluation for LLMs remains highly important for:
Creating and validating the first rubric.
Reviewing high-risk failures.
Evaluating new kinds of requests.
Checking whether automated scores reflect real usefulness.
Investigating disagreement between different evaluators.
Periodically auditing the judge for drift.
Having gone through all the methods of LLM evaluation, we can now look at the overall evaluation stack.
We should understand this proposed evaluation stack as a set of complementary layers rather than a strict ladder where one layer replaces another.
At the bottom are conventional software tests. These are used to verify deterministic components such as permissions, tool schemas, database writes, and calculations.
Golden datasets provide repeatable scenarios on which the entire LLM application can be tested.
Automated checks inspect properties that can be measured reliably, including valid JSON, required fields, exact extracted values, citation structure, and latency.
LLM judges evaluate qualities that require the model to interpret meaning. We can use it to assess whether an answer is relevant, complete, supported by evidence, and compliant with a detailed rubric.
Human reviewers calibrate the judge and assess cases where the consequences or subjectivity are too great to rely entirely on automation.
Production monitoring completes the stack. A test set can never anticipate every real request, so production signals are needed to reveal emerging failures.
We should not treat LLM evaluation as a search for a perfect accuracy number. It is a system for building confidence in a model.
Golden datasets make critical situations repeatable. Automated metrics quickly check narrow and objective properties. LLM judges assess meaning according to a rubric. Human reviewers calibrate those judges and handle difficult decisions.
The most reliable question is therefore not simply, “Is the model healthy?”
It is the idea that across the situations that matter, does the complete application stay accurate, useful, safe, reliable, fast, and affordable? Also, is there enough evidence to detect when that changes?
This is the real purpose of LLM evaluation.
2026-09-12 23:30:46
Goldfish get a bad rap. Turns out they actually remember things for months, not seconds. Your agent, on the other hand, forgets everything the second the context window fills up.
So really, your agent has worse memory than a goldfish. That’s the bar. It’s easy to clear with the right infrastructure.
This series builds a context aware app on Redis Iris in five 15 minute sessions, live, one gap at a time:
Session 1: Search (Sept. 16)
Session 2: Agent memory (Sept. 23)
Session 3: Context retrieving (Sept. 30)
Session 4: Semantic caching (Oct. 7)
Session 5: Fully built context aware apps + use cases (Oct. 14)
This week’s system design refresher:
Why Does Git Revert Cause Conflicts?
12 Claude Code Features Every Engineer Should Know
Symmetric vs. Asymmetric Encryption
7 Key Load Balancer Use Cases
How can Cache Systems go wrong?
Launching ByteByteGo Live
git revert looks straightforward until it throws a conflict. Here’s why that happens.
What git revert actually does: Unlike reset, a revert doesn’t rewrite history. Instead, it creates a new commit that undoes the changes from an earlier one. This keeps your history clean, traceable, and safe for shared branches.
Why revert conflicts happen: Conflicts appear when a later commit changed the same lines as the commit you’re trying to undo.
Example in the diagram:
Commit C2 added a feature
Commit C3 changed those same lines
Reverting C2 now collides with changes from C3
Git can’t know which version is correct, so a revert conflict is triggered.
How to resolve it:
1. Run git revert C2
2. Git pauses when it hits the conflict
3. You manually fix the file
4. Stage it
5. Continue the revert
Git then creates a new commit that cleanly undoes C2 while keeping C3 intact.
Over to you: Have you ever hit a revert conflict at the worst possible moment? How did you resolve it?
CLAUDE. md: A project memory file to define custom rules and conventions. Claude reads at the start of every session.
Permissions: Control which tools Claude can and can’t use.
Plan Mode: Claude plans before it acts. You can review them before any code changes.
Checkpoints: Automatic snapshots of your project to revert to if something goes wrong.
Skills: Reusable instruction files Claude follows automatically.
Hooks: Run custom shell scripts on lifecycle events like PreToolUse or PostToolUse.
MCP: Connect Claude to any external tools like databases and third-party services.
Plugins: Extend Claude with third-party integrations containing skills, MCPs, and hooks.
Context: Feed Claude what it needs and manage the current context window with /context.
Slash Commands: Create shortcuts for tasks you run often. Type / and pick from your saved commands.
Compaction: Compress long conversations to save tokens.
Subagents: Spawn parallel agents for complex tasks. Divide large multi-step workflows and run them simultaneously.
Over to you: Which Claude Code feature do you use the most? Any features you wish were on this list?
Symmetric and asymmetric encryption often get explained together, but they solve very different problems.
Symmetric encryption uses a single shared key. The same key encrypts and decrypts the data. It’s fast, efficient, and ideal for large amounts of data. That’s why it’s used for things like encrypting files, database records, and message payloads.
The catch is key distribution, both parties must already have the secret, and sharing it securely is hard.
Asymmetric encryption uses a key pair. A public key that can be shared with anyone, and a private key that stays secret. Data encrypted with the public key can only be decrypted with the private key.
This removes the need for secure key sharing upfront, but it comes at a cost. It’s slower and computationally expensive, which makes it impractical for encrypting large payloads.
That’s why asymmetric encryption is usually used for identity, authentication, and key exchange, not bulk data.
Over to you: What’s the most common misunderstanding you’ve seen about encryption in system design?
Traffic Distribution: Load Balancers help evenly distribute traffic among multiple server instances.
SSL Termination: Load Balancers can offload the responsibility of SSL termination from the backend servers, thereby reducing their workload.
Session Persistence: Load Balancers ensure that all requests from a user hit the same instance to maintain session persistence.
High Availability: Improves the system’s availability by rerouting traffic away from failed or unhealthy servers to healthy ones.
Scalability: Load Balancers facilitate horizontal scaling when additional instances are added to the server pool to handle increased traffic.
DDoS Mitigation: Load Balancers can help mitigate the impact of DDoS attacks by rate limiting requests or distributing them across a wider surface.
Health Monitoring: Load Balancers also monitor the health and performance of server instances and remove failed or unhealthy servers from the pool.
Over to you: Which other load balancer use case will you add to the list?
The diagram below shows 4 typical cases where caches can go wrong and their solutions.
Thunder herd problem
This happens when a large number of keys in the cache expire at the same time. Then the query requests directly hit the database, which overloads the database.
There are two ways to mitigate this issue: one is to avoid setting the same expiry time for the keys, adding a random number in the configuration; the other is to allow only the core business data to hit the database and prevent non-core data to access the database until the cache is back up.
Cache penetration
This happens when the key doesn’t exist in the cache or the database. The application cannot retrieve relevant data from the database to update the cache. This problem creates a lot of pressure on both the cache and the database.
To solve this, there are two suggestions. One is to cache a null value for non-existent keys, avoiding hitting the database. The other is to use a bloom filter to check the key existence first, and if the key doesn’t exist, we can avoid hitting the database.
Cache breakdown
This is similar to the thunder herd problem. It happens when a hot key expires. A large number of requests hit the database.
Since the hot keys take up 80% of the queries, we do not set an expiration time for them.
Cache crash
This happens when the cache is down and all the requests go to the database.
There are two ways to solve this problem. One is to set up a circuit breaker, and when the cache is down, the application services cannot visit the cache or the database. The other is to set up a cluster for the cache to improve cache availability.
Over to you: Have you met any of these issues in production?
Most online courses never get finished (~4% completion). Live cohorts get ~40%, roughly 10x higher. Live courses are the only courses people actually finish.
So we’re launching ByteByteGo Live. The lineup:
Build with Claude Code (John Kim, Senior Staff Engineer @ Meta, starting in a few days)
Build Production Grade AI Systems (Tanya Roosta, Director @ AMD, PhD UC Berkeley)
AI Engineering Fundamentals (Ali Aminian, Google, bestselling author)
AI Evals in Practice (Manjeet Singh, Senior Director @ Salesforce)
AI Cost Optimization (Jeremy Hintz, Engineering Lead @ Meta)
Rebuild YouTube with AI (Mikhail Sychev, Staff Software Engineer @ Google)
Trust-optimized AI Development (Kent Beck, Creator of TDD)
One membership includes every live course we run over the next 12 months, including new ones added along the way.
2026-09-11 23:32:16
Most online courses never get finished (~4% completion). Live cohorts get ~40%, roughly 10x higher. Live courses are the only courses people actually finish.
So we’re launching ByteByteGo Live. The lineup:
Build with Claude Code (John Kim, Senior Staff Engineer @ Meta, starting in a few days)
Build Production Grade AI Systems (Tanya Roosta, Director @ AMD, PhD UC Berkeley)
AI Engineering Fundamentals (Ali Aminian, Google, bestselling author)
AI Evals in Practice (Manjeet Singh, Senior Director @ Salesforce)
AI Cost Optimization (Jeremy Hintz, Engineering Lead @ Meta)
Rebuild YouTube with AI (Mikhail Sychev, Staff Software Engineer @ Google)
Trust-optimized AI Development (Kent Beck, Creator of TDD)
One membership includes every live course we run over the next 12 months, including new ones added along the way.