2026-10-28 23:31:05
Markets don’t wait. On a prediction market, prices move in milliseconds, and every trade has to land the moment it happens. At the Agentic Data Summit on December 9, Batu Balimtas, Member of Technical Staff at Kalshi, takes you behind the scenes in Engineering for the Moments That Move Markets: Kalshi and Redpanda. Hear how Kalshi’s data infrastructure keeps pace when it matters most, and what it takes to run it in production. Catch Kalshi live at 12:00 PM PT. The summit is free and virtual and runs 8:30 AM–2:00 PM PT, so you can join for the sessions that matter most to you.
An off-the-shelf large language model can explain why there’s an error in your code. It can suggest possible causes. Or it can perform tasks like writing a database query. However, all this would be at a general level based on the model’s training. You cannot expect it to write answers based on your system’s internal workings.
This is handled using tools. Each tool is built for a specific job. But you can end up with lots of tools, especially when working in an enterprise setup. This is the situation the Uber engineering team faced.
To deal with this, they built the MCP Gateway. You can think of it as a shared platform that lets different AI agents find which tools are present and use them to perform their job. In this article, we’ll look at how Uber built this gateway in detail. Here’s what we will cover:
What are tools and why AI agents need them
The role of MCP at Uber
How the Uber MCP gateway is designed
How the gateway finds out the available tools
How does the MCP gateway work
How the MCP gateway deals with permission and access
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
If you want a general-purpose LLM to write answers specifically for your current systems, you need to give it access to information about your system. For example, let’s say a developer asks an LLM-based AI assistant why XYZ service suddenly started failing. The answer is hidden in the system’s logs, deployments, metrics, and incidents. The LLM needs to access this information to be able to answer this question.
This is where we can create tools. You can think of a tool as an operation an AI application can execute. For example, you can have a tool to search logs. Retrieve incident details. Or check the status of a deployment. The model selects a tool and provides the necessary inputs. The application executes it and sends the result back to the model.
Many of these tools already exist as APIs. As you might already know, an API is an interface through which one program can request something from another program. For example, we can have a logging service that exposes an API. It takes the service name and time range as input and sends back all the log entries that fall within the range.
However, for an agent to use such a service, we need to describe what it does and what inputs it needs. This structured definition takes the form of a tool that the LLM application can use. Behind the scenes, the API performs the real search.
MCP stands for Model Context Protocol. With MCP, you get a standard way for AI applications to find out available external capabilities and talk to them. Uber uses MCP for its tool ecosystem.
We must know three important terms to understand the role of MCP in a better way.
First, you have the MCP client. This is the component that sits inside an AI application. It communicates using MCP and handles various interactions, such as requesting the available tools. Or making calls to the selected tools.
Second, there is the MCP server. It exposes the capabilities (or tools) through MCP. In Uber’s case, the server shows the available tools and accepts any requests made to those tools.
Third, you have the MCP tool. This is the individual operation exposed by a server. You can also think of them as capabilities. One server can provide multiple tools.
MCP requires you to define the tools properly. This definition is made up of information such as name, description, and input schema. A schema provides the structure of data. Things like which fields are present, which are mandatory, and what types of values they can accept. For example, you can have a schema where service_name has to be a string and limit is an integer.
MCP uses a structured way to format a request or its parameters. Also, the corresponding result or error. The format is usually JSON-RPC 2.0. This standardization makes it easy to integrate stuff. You can have the same MCP client talk to different MCP servers.
However, this standardization doesn’t come for free. Someone has to decide which tools exist and where they run. Also, you need to track the ownership of the tools and how they work. Initially, the engineering teams within Uber built their own MCP integrations. The autonomy was great at a small level. But as adoption increased, it was obvious that each team was solving similar problems. For example, exposing tools. Managing connections. Dealing with permissions. And so on. To make things even more difficult, finding out which tools were available was difficult. Everything was scattered.
On top of all this, Uber’s existing service landscape brought another challenge. All the services in Uber didn’t talk to each other in the same way. Some used HTTP. Some used gRPC. And others used TChannel. For example, you might have an HTTP API use JSON and a gRPC API use Protobuf messages. You cannot simply forward an MCP request across these services without modifying the request format.
Therefore, Uber needed a common layer that could connect MCP clients to the tools. They also wanted this layer to handle shared discovery, security, and operational support. This layer is known as the MCP Gateway.
The Uber MCP Gateway platform is divided into two main components: the MCP Registry and the Proxy Gateway. This separation helps Uber’s engineering team manage tool definitions and permissions independently.
Let’s look at both in detail.
You can think of the MCP Registry as the control plane. It holds the catalog of MCP servers and tools. You can record the information needed to discover the tools and learn more about them in the registry.
The reason they call it the control plane is that the registry also manages the configuration. For example, you can enable or disable the tool from the registry. It is also a shared source of truth.
This is the data plane. It handles the real MCP requests while agents are doing their work.
The gateway identifies the requested tool. It applies the necessary controls and sends the request to the right service. Also, it returns an MCP-compatible result.
You can’t manually create MCP integrations for 1000s of services. It’s a lot of work, and there are chances of errors. Uber has automated much of this work by using AutoCrawler.
AutoCrawler finds out existing APIs and MCP servers. It generates or retrieves their tool definitions and registers them in the MCP registry.
On a technical level, AutoCrawler is a distributed workflow system. It’s powered by Cadence. In this setup, you have a scheduled job start workflow that looks for any new services, APIs, or schema changes.
But how does AutoCrawler discover new things? It has two main discovery pathways:
Many APIs use an Internal Definition Language (IDL) to describe their APIs.
An IDL is a machine-readable description of a service interface. You can use it to specify operation names, request fields, and documentation comments. For example, you may have an API definition describe an operation named GetDeploymentStatus. It might take a deployment ID as input and return a status and timestamp.
AutoCrawler can read these definitions and then go through several steps.
First, it creates an MCP server representation for the service. It might also update an existing representation. Next, it pulls out method names, request and response structures. Then, it translates whatever it has found into tool schemas that MCP can make sense of.
AutoCrawler also uses an LLM to come up with better tool descriptions from the details it has extracted from the service. Finally, it stores these tool definitions in the registry.
As you can see, AutoCrawler is a good example of Uber’s no-code approach. Existing service contracts have enough information for the platform to build the MCP interface.
Some services directly implement MCP. At Uber, these are called native MCP servers. These servers already have tool definitions. So less work for AutoCrawler to get those definitions.
Each native server emits a heartbeat metric. It signals the presence and readiness of the server. AutoCrawler keeps track of these signals. When it finds a server, it calls the server’s listTools operation. Once it finds the tools, it creates a proxy representation in the registry for all the tools and their schemas.
As you can see, both discovery pathways end up in the same catalog.
Uber made a design choice that newly discovered servers and tools start in a disabled state.
This means that automatic discovery only makes the capability visible to the platform. After this, the team that owns the service should review and enable the service to make it available for the agents. The choice to enable is an ownership decision. It’s better that the service team has the opportunity to check whether the tool description is correct.
The owner team can review the tool definitions and update them using the registry’s interface. Every modification is a new diff that should be approved. Only approved changes are deployed. Also, you can have owners roll back to the previous version if needed.
Once you have saved the tool configurations in the registry and have marked them as available, the data plane consumes these tool details to refresh its in-memory state. This happens at a fixed periodic interval.
The gateway uses the mappings at the time of request processing. The periodic refresh process makes sure that any new changes can take effect without the need to restart the service. Or redeploy it.
The gateway takes these configuration details and uses them to create virtual MCP servers. You can think of a virtual server as a logical MCP interface that exists inside the gateway. It’s made up of a collection of tools along with an endpoint for clients to reach those tools. For example, you can have a service called “logging” exposed on the endpoint “logging/mcp”.
To use the tool, the clients talk to this endpoint as an MCP server. You have a handler within the gateway that knows which tools are part of that server and how to reach the service that backs the tool.
To understand things in a better way, it’s good to follow the path of a request through the gateway.
Let’s say you have an AI agent that calls a tool backed by a gRPC service
First, the MCP request hits the gateway. The gateway finds the corresponding virtual server and the tool that is requested.
Next, the server handler uses an in-memory mapping to find out the downstream destination and operation. In this case, the downstream service is the service to which the gateway will send the request.
The handler converts the tool’s JSON arguments into a form that the service expects. For example, in the case of a gRPC operation, you have to serialize the request into Protobuf bytes.
Moving further, the service runs the operation and sends back the response.
Finally, the gateway converts the response into an MCP-compatible result and ships it back to the caller.
As you can guess, the same approach works for other types of services. For example, an HTTP backend receives an HTTP request. Or a TChannel service uses the TChannel or Thrift path.
You can consider the gateway as a shared place to enforce access rules for all the tools.
The Uber engineering team uses their internal Access Control System to apply access rules and policies. These policies are based on the caller. You can have a human caller. A service. Or an AI agent. You can configure policies at the server level. Also, you can go as far as setting up specific policies for specific tools if needed. So the system provides a lot of flexibility.
This means that the platform supports authorization at the tool level. For example, you can have an incident management server expose tools for reading incidents and changing incident status. You can have a policy grant different access to those operations even though they are part of the same server.
The gateway can also redact responses for personally identifiable information (PII). Or any other sensitive data that may be present in the tool’s response.
For 3rd-party services such as Jira and Google, you need to have an additional identity check. This is because an internal Uber credential is different from a credential that might be accepted by a 3rd-party system. The gateway sends the caller’s internal user token to this service as part of its process flow. The 3rd-party service exchanges that token for the external auth token. Then, it sends the request to the external MCP server with that auth token.
As you can see, the responsibilities are divided. The gateway handles common access and execution details. The 3rd-party service takes care of the external stuff.
The huge catalog of tools running at Uber can create a context problem for the MCP gateway.
You can have a gateway make 1000s of tools available, but an AI agent needs to have an efficient way to select the useful tools. This is because each tool definition takes up space in the model’s context window. You can think of this context window as the space available as part of the model’s input. This window has a limit. You also need it for other things like system instructions, conversation history, and results coming back from the tools.
If you have an application that loads tool descriptions and schemas for 1000s of tools in advance, they can take up a huge part of the context. Also, the amount of input that the model has to process shoots up. The same goes for tool responses. An API may send back a detailed object even when the task can be accomplished with just two fields.
To deal with these problems, Uber made three additions to the gateway:
Omni MCP helps agents discover capabilities gradually. It’s a single proxy server that lets MCP clients access any of the gateway’s servers with gradual discovery. It provides tools like “discover_server”, “discover_tools”, “get_tool_schema”, and so on.
Response Projection reduces the fields that are sent back in tool results. It’s a GraphQL-like pattern for MCP tools. You inject a new field in the tool request that tells the gateway which fields should be part of the response. The gateway removes the other fields.
Code Mode lets coding agents work with results using shell commands and files.
Building the MCP Gateway has changed how AI agents operate at Uber in a big way.
It began as a fragmentation problem. You had dozens of teams independently wiring up MCP integrations. There was inconsistent tooling. No shared security guarantees. Also, duplicated infrastructure. Now, everything is a unified platform that can be scaled. Any team can plug into this platform within minutes.
The core philosophy that drove Uber’s team was that existing APIs are the fastest way to let AI agents access the tools. They didn’t want the teams to rewrite their services to make them fit for the gateway. The MCP gateway does the heavy lifting.
The key takeaway for developers is that the hardest part of building agentic systems is not the AI. You should worry more about building the right connective tissue. Things like discovery, security, and reliability are very important in a production environment.
References:
2026-10-27 23:31:04
Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.
But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.
Join us live on Oct 28 (FREE) to see:
The 8-levels of AI maturity and where teams get stuck
How a context engine curates context and resolves conflicts before your agent writes a line of code
Live demo: token savings in the same coding task, and use cases beyond code gen
Bonus: 2 open source tools you can use today.
A large language model (LLM) is a neural network. You can use it to process and generate language. How the model behaves is dependent on the numerical values you teach it during training. These numerical values are known as parameters.
Apple fits a 3-billion-parameter model on your iPhone. What this means is that the model has roughly 3 billion numerical values that it needs to work properly. These values influence how the model makes sense of its input and how it produces output.
How is Apple able to fit such a model on an iPhone without any adverse impact on the device?
In this article, we’ll try to answer this question. Here’s what we will cover:
The problems of running an LLM on a phone
How Apple makes the weights cheaper to store
How Apple made repeated calculations faster
How adapters help features share the foundation
How Apple optimizes the hardware to run the models
How does the system decide when to use the cloud
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Running an LLM is also called inference. In this stage, the LLM uses its parameters to process a request.
Before processing some text, the LLM has to divide it into tokens. You can think of a token as a word, part of a word, or even punctuation. The model takes these tokens and generates more tokens as part of the response. Within the model, these tokens pass through a bunch of processing layers.
The first problem is about storing the parameters.
At 16-bit precision, the 3 billion parameters we talked about earlier can occupy almost 6GB. But that’s not all. You need space for the input prompt, calculations, and temporary results that the model needs. Lastly, you also need to run the OS and other apps using the same memory. As you can see, storage is a big decision.
The second problem is about moving the weights quickly from memory to the processor. During text generation, the processor has to repeatedly fetch numerical data from memory. Even if you have a powerful processor, it might spend a lot of time just waiting for that data to be fetched from memory.
The Apple engineering team also had to deal with the challenges regarding memory capacity and memory bandwidth while trying to make their LLM run on the iPhone. On top of this, a mobile phone also has to take care of battery consumption, overheating, and other apps trying to use the same resources. You don’t want the LLM to make the rest of the device unresponsive.
The most direct way of reducing the size of the model is to use fewer bits to store each weight. This technique is known as quantization.
You can think of quantization as a way to represent numerical values using a smaller set of possibilities. Sure, you lose some numerical detail in the process. But the model can still perform its tasks at a pretty good level. The main trade-off is to select a representation approach that sits at the sweet spot for memory savings and quality. See the diagram below.
The Apple engineering team uses a technique known as palettization. In this technique, you group similar weight values. A small lookup table stores representative values, and individual weights store short references to that table. For example, you can have a 2-bit reference select from four entries in a lookup table.
In the earlier model version, the Apple engineering team combined 2-bit and 4-bit representations. It has an average of 3.7 bits per weight. This meant that the 3 billion weights could be stored in approximately 1.39 GB. A pretty good optimization.
They mixed different precisions because different parts of a model can easily handle different amounts of numerical error. You need to spend more bits on sensitive components so that quality doesn’t go down. Other components can be compressed more.
The later model Apple released makes the compression even more aggressive. The main decoder weights use 2 bits per weight. The embedding table uses 4 bits per weight. To make this compression work, Apple uses quantization-aware training. In this approach, you simulate the kind of restrictions that quantization will impose. Then, you adjust the model’s weights with those restrictions in place so that it can learn to work under them.
The weight storage is only one part of the overall memory problem. After you have successfully compressed the model, the model still needs memory to process a request.
Let’s say you have a model that has already read a paragraph of text and generated the beginning of a response. When the model tries to predict the next token, it needs to use information from earlier tokens. However, if you have to recalculate every intermediate result, it would be a big waste of computational resources.
A KV cache (key-value cache) stores these intermediate results. You can think of this cache as a way to reuse calculations that have already been done. However, the KV cache has a trade-off. As you can see, the cache takes up memory. The more text you have in a request, the more information you need to store in the cache. Even if you have a model whose weights fit comfortably within the memory, it can still face memory issues when processing a document with lots of text.
This means you have two targets to optimize. First, you can compress weights to reduce the space required by the model. Second, you can try to reduce the size of the KV cache.
Apple performs the cache optimization through a technique known as cross-layer KV sharing. In this technique, you need to divide the model into two blocks with a 5:3 depth ratio. Layers in the second block reuse keys and values produced by the final layer of the first block. By using this approach, Apple has brought down the KV-cache memory usage by 37.5%.
You can use compression to reduce the size and cost of one model. But often, you may need many copies of that model with slightly different capabilities.
A simple approach would be to create one model for summarization. One for rewriting. Another for extracting information. And so on. But all these models would have a great deal of overlapping language capability. You are basically duplicating all of that capability.
An adapter can help you get rid of this duplication. You can think of an adapter as a small collection of parameters that changes how a base model works for a specific task. For example, let’s say you have a feature that writes concise notification summaries. Another rewrites longer passages into a different style. Both of these features can share the same base LLM. No need for duplication.
This technique is known as LoRA (Low-Rank Adaptation). While training an adapter, you fix the base model’s weights. Only the adapter weights are modified. This is why it becomes cheaper to specialize a model as compared to distributing a separate new model.
Apple applies adapters to various features such as attention matrices, attention-output projections, and feed-forward layers. Its rank-16 adapters use 16-bit values and occupy approx tens of MBs. They can load, cache, or swap these adapters as needed.
Even if the model is small, you need an efficient hardware implementation to run the model at the best of its ability.
Apple iPhones have several kinds of processors to take care of this. You have the CPU that handles all the general-purpose work. Then, you have the GPU that can deal with many numerical operations in parallel. The Apple Neural Engine (ANE) can run machine-learning workloads efficiently.
However, you can’t just use an accelerator and think that the model will run efficiently. You still need to select the right data layouts and split large arrays into manageable pieces. Also, you need to minimize memory copies. These techniques have an impact on speed as well as memory use.
Even after making all these optimizations, the model has limits.
In the case of Apple, its on-device model is quite capable of tasks like summarization, extraction, and classification. However, Apple Intelligence checks whether a request can be processed on the device. However, for more complex work, it can send the request to Private Cloud Compute (PCC).
PCC gives you a protected environment within which only authorized software can process the data. It also restricts any external parties from seeing the data.
Apple uses an encrypted network connection to protect information as it travels to the cloud. Your device encrypts the request for validated PCC nodes. Any infrastructure that is outside this boundary doesn’t have the keys to decrypt the request and read its contents. The data can be read only when it crosses into the authorized boundary. The goal is to make sure that even Apple’s staff and admin systems cannot access what’s inside this request.
When the request finally reaches the cloud, a separate server model takes care of the request. Apple uses mixture-of-experts computation and parallel processing tracks. This lets a large model choose the best computational components to distribute work.
Apple has been making a lot of new changes to its model.
In 2026, they launched the AFM 3 Core. It’s a new generation of its 3-billion-parameter model. Also, AFM 3 Core Advanced. It’s a sparse model with 20 billion total parameters. But it activates only 1 to 4 billion parameters at a time, based on the capabilities of the Apple Silicon systems.
In Core Advanced, the full model is stored on flash storage. Only expert weights are loaded into RAM. It selects weights based on the prompt. No need to change experts from the flash drive for every token. Apple has also expanded the cloud architecture. You can now run PCC workloads on NVIDIA GPUs in Google Cloud.
The main design principle that Apple follows is that the model’s capability should be based on the resources that are available at runtime. All of the techniques we’ve looked at come together to give the best experience to the users.
References:
2026-10-27 04:24:39
We’ll stop selling ByteByteGo Live annual memberships on October 28 at 2 PM PT. After that, courses will be sold individually.
I wanted to explain why, and give you a chance to join before the change.
When we launched ByteByteGo Live (our cohort-based learning platform), we thought one membership covering every live course would make learning from industry experts simpler and more affordable.
But the membership offer has caused confusion. The enrollment has been lower than we hoped. So we’re simplifying things: choose the course you want to take, and pay for that course only.
For the next two days, you can still purchase the annual membership.
From the day you join, you can enroll in every live course we offer. Our current lineup includes:
AI Cost Optimization, taught by Jeremy Hintz, Engineering Lead at Meta.
Build Production Grade AI Systems, taught by Tanya Roosta, Director of AI at AMD.
AI Engineering Fundamentals, taught by Ali Aminian, bestselling author of books on machine learning and generative AI and contributor to AI courses at Stanford.
Build with Claude Code, taught by John Kim, Senior Staff Engineer at Meta.
AI Evals in Practice: For Engineers & PMs, taught by Manjeet Singh, Senior Director of AI Product Management at Salesforce.
Trust-optimized AI Development, taught by Kent Beck, creator of Extreme Programming and pioneer of test-driven development.
Rebuild YouTube with AI, taught by Mikhail Sychev, former YouTube tech lead at Google.
You’ll also get access to every new live course we add during your 12-month membership.
The annual membership is 78% below the combined listed value of our current courses.
Existing memberships, including those purchased before the cutoff, will keep the same access and benefits for their membership term.
If you are ready to build and ship real AI projects before the end of 2026, join now to access every live course.
Membership sales close on October 28 at 2 PM PT.
This change only affects our live, cohort-based courses. Your ByteByteGo newsletter subscription isn’t changing.
Thank you for learning with us.
Alex Xu
Co-founder, ByteByteGo
2026-10-26 23:30:38
Datadog’s free eBook covers what actually separates the agents that ship from the pilots that stall. Spoiler: it’s not the model. It’s durable execution, memory management, evaluation pipelines, and observability.
You’ll learn how to:
Identify the four failure modes that kill agent programs before they ship
Build evaluation pipelines that let you trust and govern agents at production scale
Instrument agent workflows with the observability needed to debug non-deterministic behavior in production
Measure the leading indicators that prove agent ROI to your team and leadership
Salesforce agents have completed 7 billion Agentic Work Units (AWUs), the unit the company uses to count work done by AI agents, across Agentforce and Slack. Of those, 3.2 billion came in its second quarter alone.
But getting agents to follow instructions consistently was a real challenge. The first version of Agentforce, Salesforce’s platform for building and running agents, put all instructions into one large prompt. As the rules piled up, the model began skipping some important ones. For customers using agents to interview job candidates or handle patient data, that was unacceptable.
Salesforce moved to a new design, a multi-layer system that combines deterministic code with reasoning models for maximum reliability. To understand how this works, we met two of the leaders behind it at Dreamforce 2026: Muralidhar Krishnaprasad (MK), President and CTO at Salesforce, and Khushwant Singh, SVP of Product Management for the AI application development platform. Thanks to MK and Khushwant for sharing the details with us.
In this article, you’ll learn:
Why a naive design that relies on prompting is not reliable
How to combine deterministic code with probabilistic reasoners to increase reliability
How to build a data layer that supports identity resolution and links records from many systems into one unified profile
How to build an efficient context engine
How Agent Identity gives an agent its own permissions
How AIforce and the Headless Toolkit let agents in tools such as Slack and Claude reach Salesforce data and actions
The simplest way to build an agent is to write everything into a single prompt. That is how most teams build their first agent. Imagine you are building a support agent for a healthcare company. You can write the business rules as natural language instructions (e.g., only discuss the patient’s own records). Then add a description of available tools (e.g., “get scheduled appointments”). You also include other relevant info like patient data. The agent then includes all of this in the LLM’s context window. That’s the whole design. We just hope that the LLM does the right thing.
In this naive design, the model is expected to handle four things at once: 1) enforce the rules, 2) find who the customer is across different systems, 3) include the data the model may need, and 4) ensure the agent does what it is allowed to do.
The first version of Agentforce handled everything this way. According to MK, the team kept adding instructions such as only do this, only do that, and first get all the appointments. An interesting observation from the team was that this worked fine for up to eight instructions. But beyond eight, the model would discard instructions.
But why would a model discard an instruction? To a model, an instruction is not necessarily a rule. It is just some text in the context. There is no guarantee that an instruction will actually be followed. Besides that, instruction following will also get worse as we add more instructions to the context. This is not specific to Salesforce. A benchmark called IFScale tested 20 models with up to 500 simple instructions in one prompt to measure instruction following, and even the best model followed only 68% of them at the full 500.
So a prompt only tries to shape the behavior of the LLM. It cannot enforce that behavior. Let’s see what Salesforce built around the model to fix this.
Salesforce’s new design handles reliability by taking the four jobs out of the prompt and having dedicated systems to handle each. To see where these systems are, let’s review the current Salesforce platform stack. The four main layers in the stack, from bottom to top, are:
Data layer: Turns a company’s data sources into one consistent view. Salesforce calls this Data 360.
Application layer: The business applications for sales, service, and marketing, which supply the actions an agent can take. Salesforce calls this Customer 360.
Agent layer: Where agents are defined and run. Salesforce calls this Agentforce.
Surface layer: The places where a person talks to the agent, such as a Claude chat or Slack.
Instead of putting everything in a prompt and expecting the model to handle every job, each job is handled separately by a system in one of these layers. For example, in the Agentforce layer, a scripting language called Agent Script ensures to enforce the rules. In the data layer, identity resolution is responsible for finding a unified profile of the customer. Agent Context Engine is a separate system that decides which data reaches the model. The model just handles steps that require reasoning.
In the rest of this section, we go through the four systems built to handle each job one by one. To make it concrete, we follow the demo MK shared with us. The company is a simulated healthcare provider called Cumulus and the patient is Harper Clark, but according to MK everything behind it runs in production.
The first job that should be handled outside of the prompt is to enforce the rules. Many checks have to happen in every session turn, and they can be enforced deterministically. It is neither efficient nor reliable to expect the model to handle them. Another system can enforce them more reliably by just running code. That is the job of Agent Script, the language Salesforce built for describing agent behavior. According to MK, the team started working on it last year, after the first version of Agentforce ran into the instruction limit. It is open source and looks almost like Python [x].
How does it work? You just write the agent’s behavior in an Agent Script file. That file is compiled into a structured specification called the Agent Graph, and the Agent Graph is executed by the Atlas Reasoning Engine, the runtime behind Agentforce [x]. The engine runs the code parts and calls the model when reasoning is needed.
To better understand, compare this with SQL triggers. A trigger is code that the database runs automatically before or after updating a row. The application that issues the update cannot skip it. Agent Script follows a similar pattern.
Let’s compare the naive design with Agent Script using a concrete example. Imagine Harper asks a question like “When is my upcoming appointment with Dr. Perl?” In the naive design, the prompt says “always fetch the appointments first” and we hope the model would follow. In Agent Script, the lookup is a line of code in the before_reasoning block of the medical appointments subagent. Because of that, it always runs reliably.
Putting the pieces together, here is what happens when Harper asks her question:
The model picks the medical appointments subagent.
The before_reasoning block runs (the appointment lookup executes as code)
The engine builds the prompt.
The model receives the prompt and writes the answer.
Lastly, the after_reasoning block executes. It may also send the conversation to a different subagent if needed.
This separation between deterministic tasks and reasoning parts allows the system to stay reliable. The model only focuses on reasoning over the context prepared by the code, not things like fetching appointments or what should happen after the reply.
While deterministic instructions improve reliability, they should not be used everywhere. The best practice is to use code for the parts we want predictable behavior, such as conditions, and use natural language for the parts that need actual reasoning.
The second system handles “who the customer is.” Harper’s appointment question may look like a simple lookup, but the appointment data may not be in the CRM (customer relationship management) system at all. It may be in Snowflake, a cloud data warehouse. Harper may also exist many times, as shown below.
One approach is to hand the model different lists of records and ask which ones are the same person. But that becomes a source of unreliability. Instead, a data layer, in three steps, can handle this.
Step 1: Reach the data. The data an agent needs is usually not in one place. The data layer connects to various sources and reads from them directly. This way, the agent has an updated view of the records. To show this, MK changed the insurance plan on Harper’s record in Snowflake and the change was reflected immediately.
Step 2: Harmonize. Different sources may have inconsistent views of the same record. For example, one system may store a full name in a single field while another splits it into two columns. The data layer maps each source onto a shared data model with common objects such as person, lead, and maybe account. According to MK, this mapping is automatic and works across thousands of sources.
Step 3: Resolve identities. This step handles situations where records from different sources describe the same person. In Data 360, this is done by match rules that an admin can configure. A rule can be an exact match (such as the same email address), a fuzzy rule that allows small differences, or a normalized rule, which cleans up a field before comparing it. This way, two phone numbers written in different formats will match. Note that the source records are never changed, and the output is a unified profile that links back to each of them.
With these steps, when Harper’s appointments are needed, it is obtained from the unified profile, instead of the model figuring out who Harper Clark is.
The third job is to decide what data should go into the prompt. If we put everything into a prompt, it adds to the latency and may bury the relevant facts. A context management system has to sit between the data and the prompt and decide what a question needs. That is the job of the Agent Context Engine (ACE), a new component of Data 360.
The context engine handles a request in six steps:
Resolve: Identify the user, the agent, the purpose, and the entities the question is about.
Plan: Choose where to look for that information (e.g., relational queries, the profile, search indexes)
Reconcile: Rank findings by some metric (e.g., relevance, authority, freshness)
Govern: Apply rules (e.g., access, masking, consent)
Compile: Create the context that includes working state, memory, etc.
Learn: Decide what from this interaction should be remembered or updated.
MK compared this with the naive design of an insurance support agent. In the naive version, with no context management, the question “show me details of my insurance plan” made the agent pull data from various sources, leading to 124,000 tokens. With the context engine, the same question ended up with 9,000 tokens. After a follow-up about moving to Florida, the naive session had used about 300,000 tokens in two questions, while the session with the context engine used 65,000 after a third step to process an enrollment. The gap between the two designs grows as the conversation continues, which shows the importance of having context management in place.
So how does the context engine pick what to send?
There are different implementations in practice. One is to hold the customer’s profile as a graph in memory, where the nodes are things like the person or the plan. Then we select only the nodes that relate to the question. Documents can be filtered in a similar way. When Harper asks “What lifestyle is best for me?” three indexes that match exact words, meaning, and linked entities (an approach known as GraphRAG) select relevant passages out of hundreds of documents.
The context engine can also handle memory. For example, after the Florida question, it can store a note that this customer may be moving to Florida. MK pointed out that this is a memory, not yet a fact. It became a fact only when the customer went ahead with the enrollment, and only approved facts like this are promoted into long-term memory. Either way, the model only ever sees a small, current slice of what the company knows about the customer.
The last job is to ensure the agent does what it is allowed to do. In most agent setups today, the agent acts as the user. Imagine Alex is a sales rep who connects an agent to Salesforce. The agent gets Alex’s credentials and can do everything Alex can do.
This setup is not ideal because with Alex’s full credentials, any mistake it makes becomes a real problem. A human with broad permissions is at least limited by how fast they can work. An agent is not. It can make a mistake and repeat it many times. According to Khushwant, what customers want is: “This agent is working on behalf of Alex, but I don’t want the agent to have the same rights as Alex.”
The fix is to give the agent an identity of its own. Salesforce already controls what users can do with permission sets. A new system, Agent Identity, lets an admin attach rules and permissions to an agent too, so each agent ends up having a unique identity with unique permissions.
With Agent Identity, when an agent authenticates, there are two tokens, one for the agent and one for the user it is acting for. Before an operation runs, Salesforce checks the user’s permissions, the agent’s permissions, and the data policies. The operation runs only if none of them deny it.
So far, we have looked at what happens inside a reliable agent platform. But users using an agent may work from somewhere else, like Slack or Claude. Some companies build their agents on other platforms entirely. Customers generally want Salesforce to meet their users where they are.
The way to do this is to separate the platform from its own interface. Software built this way is called headless. Instead of running on its own user interface, it exposes its functions through APIs to make it available to other surfaces. To make this happen, Salesforce developed a live interface layer, called AIforce, that brings the platform’s data, workflows, and logic to wherever people and agents already work.
For an outside agent to use a headless platform, it needs a few things. The first is APIs, the functions the platform exposes. The second is skills. An API tells an agent what it can call, but not which calls a task needs or in what order. A skill fills this gap by listing the APIs, their parameters, and the order of the steps. The third is MCP (Model Context Protocol), an open standard where a server publishes a list of tools and agents can call them.
Salesforce developed the set of building blocks behind AIforce, named Headless Toolkit, that provides all of the above. Salesforce’s APIs already serve 9.6 billion calls a day, and its skills are published in a public GitHub repository called sf-skills.
A separate challenge is tool count. MCP makes it easy to expose tools to an agent, but every tool the agent can call is typically loaded into its context window as a definition. It may contain things like a name, a description, and a parameter schema. Those definitions are sent each time a request is sent to the model, which is costly. When there are hundreds of tools, they will consume a large share of the context with definitions the agent will rarely use.
A common solution is to create a small index of tools instead of all tools. The agent first searches for what it needs, then reads the details of what it found, and finally runs it. The Salesforce MCP Server works this way with just four tools.
Suppose you are in Claude with the Salesforce plugin, and you ask for the 20 best opportunities from the last 100 days. The request goes through three steps:
Claude calls “discover” with its interpretation of your request. The server runs a semantic search over an index of available API and skill, and returns a short ranked list.
Claude calls “describe” on the best match. The server returns the skill’s contract, which lists the APIs, the parameters, the dependencies, and the order of the steps.
Claude calls dispatch. The server routes the call to the right endpoint, checks the user’s access, and returns the result.
This way, the agent’s context holds only four tool definitions instead of containing tool definitions for all available tools.
Everything in this article is a piece of scaffolding around the model. Today those pieces live in different products. The next step, according to both MK and Khushwant, is to make them one system. That is Enterprise AI Harness. A single place that enterprises can flexibly use. They plug in their agents and their data, and get eval, tracing, and other functionalities, applied to all of them at once, whether the agents run on Agentforce or somewhere else.
2026-10-24 23:30:37
Long-running agents need endurance.
They must endure waiting for users, and weather tool failures. They must take each new jump in model intelligence in stride.
Inngest makes long-running agents durable, observable and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
This week’s system design refresher:
Top 5 AI Building Blocks Every Engineer Should Know (Youtube Video)
RAG vs Agentic RAG
OAuth (authorization) + OIDC (authentication layer)
Understanding the Linux Directory Structure
RAG is a technique that combines information retrieval with an LLM to generate answers. Here is how RAG works at a high level:
A user asks a question to the application.
The retrieval system fetches the relevant information from a knowledge base.
The application adds that information to the prompt.
The LLM uses that context to generate an answer.
RAG usually follows a predefined retrieval process. The quality of its results depends on the accuracy of the retrieved context.
Agentic RAG follows a different approach. Instead of retrieving once, an agent can repeatedly fetch and evaluate information until it has enough evidence to generate an answer. Here is how it typically works:
An agent selects the right tools based on the task.
Those tools can search documents, APIs, databases, or the web.
The fetched evidence is then checked if it is sufficient for the task.
If it is not, the agent can adjust its search or choose a different tool.
Once sufficient evidence is achieved, LLM uses that context to generate an answer.
Open source is the foundation of nearly everything we build, and the maintainers behind it make it possible. The good news: the movement to support them is growing.
Two years ago, the Open Source Pledge set out to make paying maintainers the norm. Today, 37 member companies have paid $8.4M to maintainers and foundations, and every new member adds to that momentum.
See how Sentry put $1M to work this year, or join the Pledge and be part of what’s next.
If you want to show Google Calendar events inside your Slack channel, it will require permission to read your calendar. How would you let Slack access your calendar without giving Slack your Google password?
This is where OAuth 2.0 comes into the picture. Slack basically never sees your Google password. It lets an app access a protected resource on your behalf without knowing your credentials.
Basically OAuth 2.0 handles authorization. In Slack’s case, it will send you to Google’s authorization server, where you will login and give Slack permission to access your calendar. Google gives you the authorization code, and Slack will exchange that code for access_token.
If Slack also wants to authenticate the identity of the person requesting the access to Google Calendar, you will need OIDC for that. The OIDC handles authentication and sits on top of OAuth 2.0.
OAuth answers what you’re allowed to touch, while OIDC answers who you are. In addition to the access_token returned by the authorization server, this time, it will also send an id_token, a signed JWT which basically tells Slack who just signed in.
The linux directory structure starts with the root directory "/". Let’s navigate to the directory structure of linux.
The kernel image and bootloader files are stored in the "boot" directory. Your hard disk partitions and device related files are located in the “dev” directory.
All your programs and libraries are stored in "bin", "sbin", "usr", and "lib" directories. Like, your installed software is located in the "usr" directory. Ever wondered where ls, cp, and cat commands are? "bin" is the directory that holds those essential commands.
System-wide configuration goes in "etc", and "var" holds data that changes as the system runs, which is where you'll find your logs.
If you have a USB drive, it gets mounted in “bin” folder. For any external drive that you want to mount temporarily on your linux machine, "media" is the directory for it. If you have multiple user accounts, "home" is the directory for each user in your linux system.
2026-10-22 23:30:47
In a distributed system, you have components running in separate processes or machines. These components could be application services, database replicas, or even background workers.
Since in distributed systems, we divide the system into so many components, it’s pretty common that one component may need to talk to another component to get something done. It may also have to wait for the other component’s response. You can think of coordination as this talking between components. For example, you may have one service waiting for another service to approve a request. Or a database waits for acknowledgment from its replicas. Or a worker waits for a lock.
In this article, we’ll look at why coordination is a challenge for distributed systems. Here’s what we will cover.
Why coordination becomes a challenge
How coordination increases in a system
How to reduce coordination
Strategies to handle coordination
