Agents are API clients with no compiler
Strip away the framing and a tool-using agent is a client of whatever APIs sit behind its tools. A create_refund tool wraps a payments endpoint. A search_tickets tool wraps a support system's search API. The model never sees the HTTP call; it sees a name, a description and a JSON Schema for the arguments, and then it sees whatever text the tool returns.
A conventional client has guard rails that agents lack. A typed SDK fails to compile when a field is renamed. A strict deserializer throws when a type changes. A test suite pinned to a version fails in CI. An agent has none of these by default. Its contract with the API is a paragraph of natural language and a schema that someone wrote once and may never have updated.
That makes every breaking change in an upstream API a potential agent bug, and it makes a whole class of changes that are technically non-breaking for code (a new enum value, a changed default, a reworded error message) potentially breaking for a model that reasons over the text.
Two ways agents fail when an API drifts
In practice, drift shows up in two shapes, and neither of them looks like a stack trace.
Quiet failure
The tool call returns a 400 or an empty result. A human client would surface the error. An agent often reads the error body, decides the task is impossible or the data does not exist, and tells the user something plausible. "I could not find any open invoices for that customer" is a perfectly fluent answer to a request that failed because a query parameter was renamed from customer to customer_id.
Confident wrongness
Worse is the case where the call succeeds. The API switched an amount from major to minor currency units, or changed the default sort from newest to oldest, or started returning a paginated subset where it used to return everything. The response is valid JSON, the schema matches, and the model reasons correctly over incorrect premises. It refunds 100 times the intended amount, summarises the oldest tickets as the most recent, or reports a total that only covers the first page.
Both failure modes have the same root cause: the tool definition the model trusts no longer describes the API it is calling, and nothing in the loop checks.
Why this is worse than the same change in ordinary code
- Errors become prose. Agents are designed to recover from failure, so they turn hard errors into soft explanations. Monitoring that alerts on exceptions sees nothing.
- Behavior is not repeatable. The same drift can produce a correct-looking answer on one run and a wrong one on the next, which makes it hard to reproduce from a bug report.
- The blast radius includes writes. Agents are increasingly given tools that create, update and delete. A misread response feeds directly into the next mutating call.
- Ownership is diffuse. The API team, the team that wrote the tool wrapper and the team that owns the prompt are often different people, and none of them sees the full chain.
Treat tool definitions as versioned contracts
The first fix is organisational more than technical: a tool definition is an API contract and should be managed like one. Pin each tool to an explicit upstream API version rather than "latest", the same way you would pin an SDK. If the provider supports date-based or header-based versioning, send the version header from the tool wrapper so the provider cannot move it for you.
Keep the tool schema in the same repository as the wrapper code, review changes to it like code, and give it its own version number. A minimal shape looks like this:
{
"name": "list_invoices",
"version": "3",
"upstream": { "api": "billing", "version": "2026-06-01" },
"description": "List invoices for a customer. Amounts are integers in minor units (cents).",
"parameters": {
"type": "object",
"properties": {
"customer_id": { "type": "string" },
"status": { "type": "string", "enum": ["draft", "open", "paid", "void"] }
},
"required": ["customer_id"]
}
}Notice that the description states units explicitly. Semantics that a human developer would learn from documentation have to live in the tool description, because that is the only documentation the model reads. When a provider publishes a deprecation or sunset notice, it should map to a specific tool version you need to update.
Check schemas in both directions
A tool sits between two schemas: the arguments the model produces and the response the API returns. Validate both at runtime, not just at design time.
- Validate model output before calling the API. Reject arguments that do not match the tool schema and return a structured error to the model. Most agent frameworks support this; make sure it is turned on and strict.
- Validate API responses before returning them to the model. Parse the response against the schema you expect for the pinned version. If a field is missing, a type changed or an unknown enum value appears, fail loudly to your own monitoring instead of handing the model ambiguous data.
- Diff the upstream spec in CI. If the provider publishes an OpenAPI document, run an OpenAPI diff against the version your tools were written for, and fail the build when an endpoint behind a tool changes in a breaking way.
Replay real tool traffic before you upgrade
Schema checks catch contract breaks. They do not catch behavioral breaks, where the shape is identical and the meaning moved. For those, the reliable technique is the same one that works for any API client: record real requests and responses at the tool boundary, then replay them against the new API version or the new tool wrapper and diff the full responses.
Because agent runs are nondeterministic, replay at the tool layer rather than the conversation layer. You want to know whether list_invoices with these exact arguments returns the same data under the new version, independent of what the model would have done with it. If the outputs diverge, you have found the change before your agent finds it in production.
This is the problem Releo is built around: detect the upstream change, update every affected caller, and verify the update by replaying recorded traffic so a divergence blocks the merge. Whether or not you use a tool for it, the principle holds: an agent should never be the first client to discover that an API changed.
Frequently asked questions
What is schema drift in AI agents?
Schema drift is when the API behind an agent tool changes its request or response shape, or its meaning, while the tool definition the model relies on stays the same. The model keeps calling the tool as described and misinterprets what comes back.
Should agent tools always use the latest API version?
No. Pin each tool to an explicit upstream version and upgrade deliberately, with schema checks and replayed traffic as evidence that behavior is unchanged. Floating on the latest version means the provider can change your agent without a code review.
Do evals catch API changes that affect agents?
Only partially. End-to-end evals run against live or mocked APIs; mocks freeze old behavior, and live runs are noisy. Tool-level response validation and replay give a deterministic signal that evals can then build on.