MoreRSS

site iconByteByteGoModify

System design and interviewing experts, authors of best-selling books, offer newsletters and courses.
Please copy the RSS to your reader, or quickly subscribe to:

Inoreader Feedly Follow Feedbin Local Reader

Rss preview of Blog of ByteByteGo

Why Thinking More Can Make an LLM Dumb

2026-11-18 00:31:11

How much of your week is spent answering questions? (Sponsored)

New hires, support, PMs, and your coding agents all need to know how the product works and why it works that way. Today, you’re the one pointing them in the right direction.

Unblocked works out what’s current and relevant across your code, PRs, tickets, Slack, and production data, so:

  • Everyone gets cited answers in Slack without pulling in an engineer

  • When the answer turns into a fix, a cloud agent drafts the PR

  • Your agents generate code that follows past decisions, no steering needed

See how it works


When you ask someone to double-check an answer, you usually expect that you’ll get a better answer.

For example, when we look at an invoice the second time, we might find out that an item was counted twice. However, the opposite experience is also quite common. You might revisit a straightforward question on an exam and convince yourself that there’s a catch. In this case, you might end up changing an answer that was correct the very first time.

A reasoning language model can do both. It might use extra time to catch a mistake, or it can also use that time to develop a wrong interpretation. Both scenarios come from a basic feature of language models. Once a model writes down an idea, you can build on that idea a few steps later. But whether it improves the answer or gets things wrong depends on the original question.

In this article, we’ll try to understand why thinking longer can make an LLM dumber. Here’s what we’ll cover:

  • How a language model builds on its answers

  • What happens before a reasoning model sends a reply

  • The advantage of extra time on a question

  • How to teach a model to use extra steps

  • When the model has already done enough

  • How an extra step turns a right answer into a wrong one

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

How a Language Model Builds on its Own Words

An LLM generates text in small pieces called tokens. You can think of a token as a word, part of a word, or a symbol. When the model selects its next token, it takes the original question into consideration along with the text it has already generated. This means that whatever the model writes near the start of an answer can have a big impact on the calculations it does multiple sentences later.

Let’s say you have a shop that raises the price of a $100 item by 20% and then reduces the new price by 20%. What’s the final price? Without too much thinking, you might be tempted to say $100. This is because the percentage values look the same, and the words “raises” and “reduces” seem to cancel each other out.

But as you might’ve guessed, such an answer would be incorrect. The final price is $96. And the best way to get to this answer in a systematic manner is to write down the intermediate answer. Or the answer to the first calculation, which should be $120. This makes it easy to calculate the price after a 20% reduction.

An LLM can also benefit from the same sequence of checking the intermediate results. Once it has found out that the new price is $120, it can easily calculate the price after the discount. You can see that the question has been broken into smaller ones and carried the result of the first answer into the second. Such a sequence of steps is also known as a chain of thought. In this example, the most useful part of the chain is a single number that makes it easy to handle the next calculation.

What Happens Before a Reasoning Model Replies?

An ordinary LLM can provide a fully worked solution when you ask it to explain a calculation step by step. However, reasoning models such as OpenAI’s o1 were specifically trained to generate intermediate work before generating a final answer. Even if you ask for just the final price, a reasoning model can work through the price change before returning the final answer.

Most LLMs use a type of neural network called a Transformer. One of the abilities of the Transformer-based model is to connect the text that it is currently generating with relevant parts of the text that was generated earlier. This basic design approach can handle both a quick reply and a longer calculation. With their model, DeepSeek also showed how further training could teach an existing Transformer-based model to use this ability of intermediate results much more effectively.

You can think of the intermediate results working as a sort of scratchpad. Let’s say you give a model 3 different calendars of people and ask for the earliest half-hour when everyone is free. The final answer might be “Tuesday at 11.” But before generating that answer, the model is going to compare the calendars, find out that an earlier slot works for only two people, and then move on to the next available slot. The answer might take just a few words. But the work that happens behind the scenes is much more than that.

The tokens that are used for this work are often called reasoning tokens or thinking tokens. Many model providers keep the raw reasoning of their models private and might just display a summary so that the user might think of the work that was done in the background as mainly a pause of some sort. Sometimes a reasoning trace of a model might look like a conversation between two people because one sentence proposes an idea and a later one objects to it. A single model might be generating both sentences.

The Advantage of Extra Time on a Difficult Question

There are two main times when you can spend computing power on a model.

First is at the time of training, when the model adjusts its internal settings based on examples or feedback. These changes can have a big impact on how the model answers future questions.

Second is at the time of inference. At that time, the trained model uses the settings made at the time of training to answer the question in front of it. If you spend more computing power on that individual request, it is called inference-time scaling. Also known as test-time scaling.

Let’s say you have a piece of code that’s supposed to process every item in a list. It numbers the items starting from one and keeps running while the current position is less than the total number of items in the list. Now, imagine that a user complains that the last item always goes missing. A model might immediately think that the stopping condition of the loop might have a problem. That’s a pretty reasonable guess, and there is a simple way to test it.

With three items, the loop goes to positions one and two. When it goes to position three, the condition checks whether 3 is less than 3. The answer is false, and so the loop stops before processing the third item. Simply the act of following the code through a small example can turn a possible suspicion into a final explanation. It also shows that making the loop run when the two numbers are equal would fix this bug.

This is one way to spend extra time. Within a single attempt, you can have each step build on what the previous step found out. Another approach is to come up with several separate solutions and compare their final answers. However, separate attempts can help more each attempt can make different mistakes. In contrast, a longer individual attempt is much more beneficial when you let the model follow a promising idea long enough to verify it.

How To Teach A Model to Use Extra Steps

How can a model learn that checking its work can pay off?

For reference, DeepSeek explored this with their R1-Zero model. The team started with a pretrained model that had already learned patterns of language, code, and mathematics. This was followed by the use of reinforcement learning. The model attempted problems, received scores for its responses, and gradually adjusted what kind of responses it was likely to generate.

Some problems benefit from this approach because you can check their answers automatically. For example, in the case of a math question with a known result, a program can easily compare the model’s answers with an answer key. For a coding problem, you can have tests that show what works and what doesn’t. A different solution might pass both cases. But the tests can differentiate between the two even if both come with convincing explanations.

DeepSeek used a training method called Group Relative Policy Optimization (GRPO).

The idea behind this method is that for a question, the model generates a group of attempts. Some might solve the problem directly. Some might fix an earlier mistake. Others might give the wrong answer. The training procedure can score each attempt and compare its score with those of the other attempts. Once that’s done, you can adjust the model to favor attempts that work better than their peers.

There are two jobs over here.

One is where you grade an actual attempt. The other is when you estimate what score an attempt might ordinarily get so that the training procedure has something to compare things to.

Some reinforcement learning methods also train a separate model, known as the critic model, to provide that estimate. GRPO gets its comparison from the group’s own scores. The R1-Zero model made use of these rules to grade correctness and to check whether the response followed the proper format. This allowed the system to recognize a successful attempt even when nobody had written out the particular route the model took to solve it.

During training, DeepSeek looked at an intermediate version of the R1-Zero model working on an equation that had square roots. The model squared both sides and began developing a more complicated expression. However, in the middle, it stopped that approach and came back to the original equation to reconsider what it had done. Partway through, it interrupted that approach, returned to the original equation, and reconsidered its steps. The team thought of this as an “aha moment.”

The interesting bit for them was where the decision to go back had come from. The team had given a problem, a place where the model could write the working solution, and a way to score the result. This had left the choice of strategy to the model. Yet at a particular point in the calculation, the model came up with a response that considered one of its own steps worth looking at again.

You can probably see why this behavior can be so useful over many practice problems.

An attempt that continues after making an early mistake might carry that mistake all the way to the answer. An attempt that reconsiders the questionable step has a good chance to recover and get a better score. The feedback applies to the attempt as a whole. Therefore, you can also have a lucky guess that also earns credits. However, across many different problems, such a strategy has more advantages.

This was DeepSeek’s experiment in going directly from a pretrained model to reinforcement learning. For the released DeepSeek-R1, the team also used examples. R1-Zero sometimes switched languages or produced solutions that were hard to follow. Therefore, the team began R1 with supervised fine-tuning, which meant training on examples of desired responses.

When the Model Has Already Done Enough

The first form of overthinking is easy to overlook because the user still receives a correct answer.

However, in a study of reasoning models, researchers showed what can happen when one model was asked to calculate “2 + 3”. It arrived at five in its first solution, then continued until its response contained thirteen solutions to the same question. Nearly all of that text appeared after the calculation had already been done correctly.

Why would a model keep going even when it had the correct answer?

Think about how an attempt is scored. If you have two responses that use the required format and both end in 5, a checker dealing with correctness can give them the same score. You might have one response that stops after a short calculation. The other response might count objects, reverse the order of addition. Both arrive at the right answer. The score offers little guidance about which response made better use of the user’s time.

If you have a check that confirms an answer, it can still be valuable. Let’s say you calculate the total for an order and then compare the items that were counted with the original order form. Even if you confirm the total, that comparison could have shown that an item was missing. If you simply add the numbers from our own incomplete list a second time, you are much less likely to catch that particular error. What matters is whether the second pass tests something the first one could have missed.

This shows why knowing when to stop is a separate skill. A user with an answer key can look back and point to the first correct result. However, a model that answers a new question usually has to decide if another check is worth it before it knows how the answer will be judged. On a difficult problem, if the model stops at the first plausible result, it can be quite costly. But on an easy one, the habit of checking again can produce a long wait for an answer, which is unnecessary.

How an Extra Step Turns a Right Answer Into a Wrong One

The “2 +3” example shows the impact of wasted effort. However, you can also run into a different problem when the extra reasoning brings in an incorrect assumption. For example, let’s say you have a question about the train departure time printed on the timetable. A train is scheduled to leave at 9:00, but it usually leaves 10 minutes late.

The timetable gives the time as 9:00. Let’s say that the model identifies that answer immediately, but continues because the point about the delay sounds too deliberate to ignore. It might come to the conclusion that the train will probably leave at 9:10. It may even produce a sensible explanation about the difference between scheduled and actual departures. If it finally answers 9:10, the calculation is correct. But the model has gone from its core task of reporting what the timetable says to trying to predict when the train will leave.

As you can see, the original question is still available to the model. What has changed is the text it can use to form the final answer. Early in the response, simply telling the scheduled time from the timetable is the obvious candidate. Later, the model has also produced a second answer and multiple sentences that try to explain why that time might be important. While each sentence makes sense, the response drifts away from what was intended from the model. To a casual user, the answer might even appear dumb.

This is also important when looking at the relation between thinking time and accuracy. If longer answers contain more errors, one possibility is that the model wrote longer because those questions were harder to begin with. A better comparison is to give the same model the same questions with different amounts of reasoning capability. On difficult questions, the extra steps may catch mistakes. But on straightforward questions, they may repeat a simple conclusion or end up providing an irrelevant detail that’s enough to change the answer.

References:

How Instacart Uses LLMs to Make People Buy More

2026-11-17 00:30:56

Your AI agents shouldn’t have all your permissions (Sponsored)

Most agents run with the full access of whoever triggered them. WorkOS Airlock is the authorization layer for agentic software. You write policies, and Airlock checks every agent call at runtime, then allows it, denies it or sends it to a person for approval.

Agents never see credentials. A risky action, like a refund over your limit, goes to a teammate to approve in Slack. Airlock works with coding agents, custom harnesses and any API or MCP server.

Learn more →


When people search for a product on an e-commerce platform, they rarely ask for it using the exact words that are present in the product catalog. For example, users might ask for something like “bread no gluten” or “x-large zip lock”.

A good search system is one that is able to look at these phrases and link them to the actual products the user might be interested in. This first step to do so is called query understanding (QU). The job of QU is to take the words the user has typed in a search box and find out useful information about the user’s shopping intentions.

In this article, we’ll look at how Instacart used LLMs to build a product search that could accurately understand a user’s shopping intention. Here’s what we’ll cover:

  • The problem with Instacart’s initial approach

  • How a general LLM becomes useful for Instacart

  • How Instacart made sense of the query category

  • How Instacart made sure that something is always suggested to the user

  • Extracting details from the query

  • How to make the query understanding process much faster

Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.

The Problem With Instacart’s Initial Approach

You can separate product search into 3 broad activities:

  • First, you have QU, which is meant to understand the user’s request.

  • Second is retrieval, where you find a set of relevant products.

  • Third, you have the ranking phase in which you decide the order in which each product is shown to the user.

As you can see, information created during the QU phase is the one that guides both retrieval and ranking. For example, let’s say you have a search query that says “bread no gluten”. It refers to bread with a gluten-free attribute. This search phrase gives a very important piece of information to the search setup if you don’t treat the phrase as three unrelated words.

At Instacart, the QU phase can produce three types of information. First is the product categories, in which you decide which sections of the catalog are important for answering the query. Second is alternative queries, which deal with other search phrases that might help find suitable products. Third is structured tags. This deals with the parts of the query that might help learn more about the product, its brand, or specific attributes.

Instacart already had machine learning systems that helped it understand queries. The problem was how to get it to work consistently for the enormous variety of things that people search for.

One way to solve this problem is to analyze search behaviour. But this behaviour provides imperfect training data. A supervised machine learning model learns from examples that are paired with desired answers. For classifying queries, you might connect a search phrase to the correct product category.

However, as you might have already noticed, it’s quite difficult to collect all these labels. Instacart can look at searches, clicks, and final purchases. But those actions don’t directly reveal whether the system understood the query in a correct manner. For example, you might have a user who searches for bread but ends up buying bananas. Now, this purchase doesn’t mean that “bread” should be classified as a query about fruits. Of course, behaviour can give some nice evidence, but a relation between a query and a purchase decision can be quite noisy.

You can use human reviewers to provide cleaner labels. But this also needs time and money. Also, rare queries have very little history. For example, the user might make a very specific query such as “2% reduced-fat ultra-pasteurized chocolate milk”. If you have a system that learns mostly from historical clicks and purchases, it might not have enough examples to deal with such phrases.

These uncommon searches are part of the long tail. While each such query may account for little traffic, together they might represent a large collection of customer requirements.

Broad queries also create a different type of problem. For example, let’s say you have a search phrase such as “healthy food”. It can point to so many categories, and the system has to handle ambiguity rather than force the query into a single narrow interpretation.

With separate models, you also end up with a lot of maintenance work. Instacart had to maintain different systems for different tasks. For example, they had a category classifier that used a FastText model. The query rewrite system mined patterns from user sessions. Each system needed its own data processing, training, and deployment requirements. This increased maintenance work. And even then, they made inconsistent interpretations of the same query.

Therefore, the Instacart engineering team wanted something that could improve their QU and also provide a more consistent foundation for taking care of the other responsibilities.

How a General LLM Becomes Useful for Instacart

A large language model (LLM) starts off with a broad knowledge of language and the general world. This makes it pretty good at making sense of queries it might have never seen before in the historical data.

For example, an LLM might already know that something like “Italian parsley” and “flat parsley” refer to the same thing, and “curly parsley” can act as a substitute. This is all part of the model’s general awareness that it got during training, and it doesn’t need to learn these things from a bunch of examples.

Of course, general knowledge doesn’t automatically make the LLM knowledgeable about Instacart’s product catalog and the behaviour of their customers. To bridge this gap, the Instacart engineering team used the following techniques:

First, you provide all the relevant information in the prompt to the LLM. This approach is known as context engineering. RAG is the primary method that is used over here. The system pulls in information such as the catalog records and historical patterns and puts that information along with the query.

To be clear, RAG here means that you try to give the model all the useful business information before asking the model to make a prediction.

Second, the system validates the model’s output. Instacart has implemented post-processing guardrails to check how suitable the output is and how compatible it is with their product catalog. You can have a model generate a plausible answer, but it still doesn’t fit the actual data of the business. Therefore, validation is a separate part of the system.

In the third technique, they fine-tune smaller models on their own examples. Fine-tuning means additional training that can change the parameters of the model. Instead of giving all the domain guidance using the prompt every single time, the team teaches the model to work on a particular task with high-quality examples.

How Instacart Knows What the Query Is About

Instacart has organized its catalog using a hierarchy of categories. For example, you have the following path:

Meat → Beef → Beef Ribs → Short Ribs

Each step in this category path makes things more specific. If the search system knows the entire path, it can easily identify the relevant part of the catalog.

The earlier version of the classifier treated these categories more like a flat collection of labels. For example, in the case of “buttermilk”, it might predict “Dairy” and “Milk” separately, where each label came up with a confidence score. While those predictions were also useful, they didn’t express the full relationship.

With the new approach, you combine historical evidence with LLM interpretation and also check the relevance of the product.

In this approach, the system first finds the leading categories that are associated with purchases following the query. These categories form the initial set of candidates. The advantage is that this gives the model a starting point linked with Instacart’s own data. Since typical purchase behaviour can be quite erratic, you need to assess the candidates even more.

In the next step, you ask the LLM to assess the candidates. The LLM re-ranks the candidate categories using the query and the Instacart context that was given to it. For example, “buttermilk” might map to Dairy followed by Milk and then Buttermilk. On the other hand, something like “vegan roast” might map to Meat followed by Meat Alternatives.

As you can see, this is how language understanding and knowledge of the world help understand a query whose wording doesn’t directly match a category name.

Next, they check whether the prediction is relevant. After the LLM makes a prediction, a guardrail compares the meaning of the original query with the predicted category path. This part is done using embeddings, which are numerical representations of text. If the query and the category path get a similarity score that falls below a specific threshold, the system throws away that pairing.

To summarize, the three stages have different responsibilities. Historical data is meant to propose candidates. The LLM’s job is to interpret them. And at the end, the guardrail filters irrelevant results.

How Instacart Makes The Queries Stronger

Even if you have a system that understands a query well enough, the original words that make up the query may retrieve very few useful results. This is where query rewriting helps.

Query rewriting generates alternative phrases that can help the search process find more candidates.

The older search system at Instacart did this using patterns found in user sessions. But it covered only about 50% of the search traffic. The Instacart engineering team initially tried asking an LLM to generate rewrites using a general prompt. But this didn’t work as expected because just asking the LLM to rewrite the query doesn’t tell the LLM what kind of alternative would be useful.

For example, let’s say you have the query “1% milk”. Rewriting might change the query to “one percent milk”. While the meaning stays the same, it does very little to help the user find an alternative product.

To make things more effective, Instacart separated rewriting into three tasks:

  • First, they use synonym rewrites that try to express the same request using different wording. “One percent milk” illustrates this relationship.

  • Next, you have broader rewrites that attempt to get rid of some specificity. For example, “milk” can be a broader rewrite of “1% milk”

  • Last, you have substitute rewrites that suggest a different product that might be a good alternative. For example, something like “2% milk” can act as a potential substitute.

You might think that these differences are pretty minute, but they change the search approach in different ways. When you broaden a query, you make the constraints less rigid. Making the suggestion of a substitute product changes the requested product. A synonym keeps the intended meaning intact.

Each type of rewrite gets a dedicated prompt with specific instructions and examples. They also give it reasoning guidance. Instacart then applies a bunch of post-processing checks and semantic filters.

This approach increased rewrite coverage to over 95%, with more than 90% precision across all three types.

Extracting Search Tags from the Query

The third task is semantic role labeling (SRL). SRL deals with taking out structured concepts such as the product, brand, and other features from a user’s shopping query.

For example, let’s say you have a query that says “2% chocolate milk”. It contains a product “milk”. It also talks about the fat content and flavor. If you represent these concepts explicitly, they become easier to use in the search process.

Instacart uses these tags for retrieval, ranking, advertising, and filters. The main challenge is to find the accurate tags without making every search slow and costly.

One way to do so is by processing frequent queries in advance. Search traffic has an uneven distribution. You might have a relatively small collection of frequent queries that take up a big share of the traffic. These are known as head queries. Beyond them, you can have a much larger collection of uncommon or previously unseen queries.

For these types of common queries, Instacart can generate tags ahead of time and save the results in a cache. Subsequent searches can use those tags immediately.

However, you cannot have the system precompute tags for every possible phrase. New combinations keep coming up, so caching needs a companion process that can deal with queries with no pre-existing result.

To handle this, Instacart used an offline teacher pipeline and a smaller model that runs at the time of live searches. The job of this offline teacher is to create high-quality examples. Here, “offline” means that this work takes place outside the live request flow that the user is waiting for. Due to this, the pipeline has more freedom to use a powerful model and richer context information.

For a common query, it can find relevant information from internal systems, provide that information to the LLM, generate tags, and validate the tags against the catalog. For example, consider the search phrase “verdant machine”. Without business context, an LLM might think of this phrase as talking about machinery. However, Instacart’s internal evidence connects this query with a smoothie brand called MuchPure and with other smoothie juice categories.

The output validation serves two purposes. It helps populate the cache for common queries. Also, it becomes part of a training dataset for the smaller model.

This is why the large-model pipeline is called a teacher. The outputs generated by this model become examples from which another model learns.

When a live query comes in, the system checks whether tags are already cached. If they are, it returns them. If they are missing, the query goes to the smaller, fine-tuned model.

The student has already been trained on examples generated by the teacher pipeline. It learns how queries should map to structured tags, including patterns present in Instacart’s domain context.

Teaching the Smaller Model

For its real-time SRL system, Instacart fine-tuned Llama-3 8B. This is a model with approximately 8 billion parameters.

For reference, parameters are the numerical values a model learns during training. Although 8 billion is still a pretty big number, this model is a lot smaller than the larger frontier model used by the teacher pipeline.

Instacart trained it using LoRA (Low-Rank Adaptation). In LoRA, you make a model learn a relatively small set of additional parameters while keeping the original model weights fixed during fine-tuning. This brings down the amount of training work you need to adapt the model.

The training examples all depend on the offline pipeline.

This process is a form of knowledge distillation. In this, you have a smaller model learning from outputs generated by a more capable teacher model. Based on their analysis, they found that the student generated fewer incorrect tags. But it also missed slightly more correct tags.

How Instacart Made the Model Fast For Live Search

For a high-traffic website like Instacart, you want the search system to respond quickly enough so that users don’t face long delays.

The Instacart engineering team targeted model-serving latency in the low 100s of milliseconds. Its initial setup took nearly 700 milliseconds on an A100 GPU.

Several changes made this possible.

One of them was merging the LoRA adapter into the base model. After training, the LoRA adjustments can be combined with the base model’s weights. This prevents the need to keep the adapter as a separate computation during inference. According to the Instacart engineering team, this adapter merge brought down latency by about 30%. After they combined this with an upgrade to H100 GPUs, they were able to achieve the 300-millisecond target.

Therefore, the total improvement came from both a change in how the model was served and faster hardware.

The team also tested FP8 quantization, which uses lower-precision numerical representations for model computation. This brought down the latency by another 10%, but recall declined slightly. Instacart chose to deploy the unquantized model because they wanted to give more priority to higher-quality results.

You can think of this as an engineering trade-off. Though a faster model was available, the team made this choice.

They also reduced how often live inference is needed. Caching had a large effect on the overall cost. According to their estimate, only 2% of queries required real-time model inference.

Therefore, the smaller model handled a limited amount of traffic, while cached tags served the common cases. Instacart also used GPU autoscaling to run fewer GPUs during quieter periods.

Conclusion

Instacart made an evaluation of the real-time SRL system through A/B testing. For the tail queries discussed we talked about earlier, it reported a 6% reduction in average scroll depth. You can interpret this as users finding the right items more quickly.

The system was also dealing with millions of cold-start queries each week. These were the queries for which the system didn’t have query-specific history or a precomputed answer. There was also a 50% reduction in user complaints related to poor search results for tail queries, with only a marginal latency increase.

As you can see, these stats connect the problems to the solutions. Better classification, rewrites, and tags help shoppers find relevant products more easily. As part of further developments, Instacart aims to understand a shopper’s broader journey. The current design lays a strong foundation for that direction.

References:

LAST CALL FOR ENROLLMENT: AI Engineering Fundamentals

2026-11-16 06:15:41

One final reminder: the AI Engineering Fundamentals cohort starts tomorrow, Monday, November 16, 5 PM PT.

Enrollment closes in less than 24 hours. Today is your last day to enroll!

Reserve your spot now

Join Ali Aminian, Google engineer, bestselling AI author, and Stanford course contributor, for live, hands-on sessions where you’ll learn how to:

  • Understand how LLMs work and choose the right models.

  • Build RAG pipelines that retrieve information from your own data and use it to generate answers.

  • Create AI agents that use tools, search the web, and complete multi-step tasks.

  • Build AI apps using reasoning models and advanced prompting techniques.

  • Put your skills into practice by building four real-world AI projects in four weeks.

If you’ve been thinking about building your AI engineering skills to future-proof your career, now’s the time.

Reserve your spot now

Hope to see you inside!

The ByteByteGo Team

EP233: Kubernetes Explained for Busy Engineers

2026-11-15 00:30:57

Workshop: How Anthropic Traces Agents with OpenTelemetry and Sentry (Sponsored)

Where did your agent’s tokens go?

CJ Avilla from Anthropic sent his Claude Platform OpenTelemetry data to Sentry. It turned up an old project that was still burning tokens.

Every retry costs tokens, and the cause can be a slow API or database, not the model. In this live workshop, you’ll see:

  • How to send Claude Platform OpenTelemetry (OTel) data to Sentry

  • A Claude agent’s failed tool calls, retries, and final success in one trace

  • Whether the fix belongs in the prompt, the tool, or the service behind it

Save your spot for November 17 at 10am PT.

Register now


This week’s system design refresher:

  • Kubernetes Explained for Busy Engineers

  • Single Agent vs. Multi-Agent Architecture

  • The Testing Pyramid

  • How LLMs Use AI Agents with Deep Research

  • Common types of AI Agents guardrails on production


Kubernetes Explained for Busy Engineers

Kubernetes is an orchestration platform that manages the deployment and maintenance of containerized applications, particularly on multiple machines.

Let’s see how kubernetes creates and keeps an application running.

  • You describe the desired state of your application in a deployment.yaml file.

  • The API server validates the deployment and stores this desired state in etcd. The etcd is a persistent database that stores the Kubernetes resources, like the number of replicas, existing pods and nodes, and their assignments.

  • The controller manager job is to run controllers and ensure that the desired state matches the actual state of the Kubernetes cluster. For example, when a Deployment requests four replicas, the controllers ensure that four Pods are created and maintained.

  • The scheduler decides where each pod is supposed to run. It assigns the pods to the worker nodes based on the available CPU and memory resources, placement rules, or any other constraints.

  • The worker nodes are the ones that run the assigned pods. Each worker node contains a container runtime that downloads the image and actually runs the container. The kubelet keeps track of the node-to-pod assignment and ensures they run properly.

  • Once the pods are running, the Kubernetes Service provides a stable endpoint to the pods.

The Ingress or Gateway then exposes that Service to the external users.


Single Agent vs. Multi-Agent Architecture

Some tasks need a single agent. Others need a whole team. Knowing the difference is the skill.

Single-agent system: One reasoning LLM that plans, picks a tool, and loops on its own until the task is done. Use a single agent when:

  • the task is a clear, linear sequence

  • one agent can hold the whole problem in its head

  • you want something simple to build and easy to debug

Multi-agent system: An orchestrator that splits a task into subtasks and routes each one to a specialized agent. Use multi-agent when:

  • subtasks can run in parallel

  • one agent writes and another independently verifies the work

  • the problem is too big for one agent to coordinate alone

Single agents are cheaper and easier to build, but they hit a ceiling on complex work.

Multi-agent systems are more capable and more reliable, but they add coordination cost.

Start with a single agent. Move to multi-agent only when context or reliability become the bottleneck.

Over to you: Are you running single-agent or multi-agent systems in production?


The Testing Pyramid

Testing is the backbone of reliable software. The Testing Pyramid is a widely accepted strategy for structuring tests into three key layers:

  • Unit Tests: These are the foundation of the pyramid. Unit tests are fast, isolated, and low-cost to write and maintain. They test individual functions, methods, or components.

  • Integration Tests: These tests validate interactions between components, such as APIs, databases, and external services. They are slower than unit tests and require more setup.

  • E2E Tests: These simulate real user flows from start to finish across the full system. They are expensive to write and maintain and tend to be slow to execute.

As you go up the pyramid, the cost of test development, execution, and maintenance increases.

Over to you: Which layer do you find most valuable in your testing strategy, and why?


How LLMs Use AI Agents with Deep Research

When you ask an LLM such as Claude, ChatGPT, or Gemini to do deep research on a complex topic, it’s not just one model doing all the work. It’s a coordinated system of specialized AI agents.

Here’s how it works:

Step 1: Understanding The Question and Making a Plan

It all starts with the query, something like “Analyze the competitive landscape of AI agents in 2026. The system doesn’t just dive in blindly. First, it may ask clarifying questions to understand exactly what is needed. Then, it generates a plan and breaks the big question down into smaller and manageable tasks.

Step 2: Sub-Agents Get to Work

Each small task gets assigned to a sub-agent, which is basically a mini AI worker with a specific job. For example, one sub-agent might be tasked with finding the latest Nvidia earnings. It figures out which tools to use, such as searching the web, browsing a specific page, or even run code to analyze data. All of this happens through a secure layer of APIs and services that connect the AI to the outside world.

Step 3: Putting it All Together

Once all the sub-agents finish their tasks, a Synthesizer Agent takes over. It aggregates everything, identifies key themes, plans an outline, and removes any redundant or duplicate information. At the same time, a Citation Agent makes sure every claim is linked back to its source and properly formatted. The end result is a polished, well-cited final output ready for use.

Over to you: Have you tried deep research in any LLM?


Common types of AI Agents guardrails on production

  • Input Screening: Every user request is scanned for prompt injection, sensitive data, and off-scope topics before the model sees it. Unsafe inputs return a fallback message.

  • Context Verification: Every user request is scanned for prompt injection, sensitive data, and off-scope topics before the model sees it. Unsafe inputs return a fallback message.

  • Response Generation: The LLM reasons only over verified context.

  • Output Validation: Every response is checked for groundedness, format, and safety before it leaves the system. Failures trigger up to two retries, then a safe fallback.

  • Operational Controls: A final layer enforces limits, logs every call, and routes low-confidence or high-risk actions to humans.

Reliable agents aren’t built on better prompts. They’re built on the guardrails wrapped around them.

Over to you: Which guardrail layer catches the most issues in your setup?

Free Live Masterclass: Building AI Agents That Work in the Real World

2026-11-14 01:04:49

An agent needs context to make decisions, tools to take action, guardrails to stay within bounds, and checks to verify the results.

How do these pieces fit together?

Join ByteByteGo’s FREE live masterclass: “Don’t Build AI Agents Without Learning These Techniques.”

📅 Sunday, Nov 15

🕛 12 PM PT · 1 hour

💻 Live on Zoom

Register for free

Top Types of Hidden Coupling and How to Deal With Them

2026-11-13 00:30:41

When you create an architecture diagram, you usually try to show the services, databases, queues, and the connections between these different components. With the help of these diagrams, developers can understand how a request goes through the system and how the data flows. We also get to know which components depend on other components.

However, these diagrams cannot show all the things that a component might depend on. For example, let’s say you have two services with separate code and API endpoints. At first glance, you might think that these services are independent. But imagine that whenever you change one service, you need to make a change to the other service as well. Or maybe both of these services talk to the same database whose capacity can be consumed by either service. Perhaps both of them rely on a common logging system.

You can think of these dependencies as hidden forms of coupling. If not handled properly, hidden coupling can have a big impact on the health of your system. Also, it can affect how you change things in your system, or how deployments are done. Or how recovery happens. In this article, we’ll look at the various forms of hidden coupling and how you can deal with them. Here’s what we will cover:

  • The various dimensions of coupling

  • Temporal coupling

  • Semantic coupling

  • Deployment coupling

  • Shared schemas and libraries

  • Observability coupling

  • Shared incidents

The Dimensions of Hidden Coupling

Read more