LLM tool-calling errors are rarely one bug. A request can fail because the model selected the wrong tool, emitted invalid arguments, omitted a required call, repeated an action, or lost the connection between a call and its result. Even a syntactically valid call can be dangerous if the application executes it twice.
The production fix is to treat tool use as a stateful protocol, not as trusted model output. Classify failures, constrain the schema, validate before execution, make side effects idempotent, and preserve enough sanitized evidence to reproduce the incident. This guide provides that workflow without assuming that one provider’s message format applies to every API.
Verified and updated: 2026-07-24. Provider-specific behavior in this article was checked against the official documentation linked below. Recheck those pages before publication because tool schemas, API fields, and supported modes can change.
1. Classify the tool-call failure before changing the prompt
Start with the layer that broke. “The agent failed” is not a useful incident category because it mixes model behavior, protocol handling, business validation, and downstream execution.
Use a failure taxonomy like this:
| Failure class | Observable symptom | First place to inspect |
|---|---|---|
| Selection | Wrong tool, no tool, or a tool when none was needed | Active tool set, descriptions, tool-choice mode |
| Syntax | Arguments are incomplete JSON or cannot be decoded | Streaming assembler, parser, SDK version |
| Schema | JSON parses but has missing, extra, or mistyped fields | Tool schema and validation report |
| Semantic | Fields pass schema but are unsafe or nonsensical | Business rules, authorization, current state |
| Correlation | A result is attached to the wrong call | Call ID, turn state, parallel-call handling |
| Execution | Tool times out, rejects, or partially succeeds | Downstream service and timeout policy |
| Orchestration | Missing result, duplicate call, or loop never terminates | State machine, retry scope, completion rules |
Record the earliest failed invariant. A network timeout may be transient; an impossible schema is deterministic. Retrying the latter without changing the input wastes tokens and can amplify side effects.
Preserve a sanitized replay bundle: provider and model ID, request time, schema and prompt versions, tool-choice setting, call ID, raw argument text, validation errors, execution status, retries, and terminal outcome.
2. Make tool schemas easier for models to use correctly
A schema is both a machine contract and part of the model’s instructions. A technically valid but ambiguous schema can shift errors from JSON parsing into tool selection or business logic.
Prefer narrow tools with explicit names. create_support_ticket is easier to distinguish than manage_record. Describe when the tool should be used, when it should not be used, what each field means, and what the tool does not return. Anthropic’s official guidance puts particular emphasis on detailed descriptions and recommends explaining tool purpose, usage conditions, parameter behavior, and limitations (Anthropic: Define tools, verified 2026-07-23).
Give every parameter a concrete type and a domain-specific description. Use enums for small closed sets, distinguish machine identifiers from display names, and state units or formats explicitly. Avoid unnecessary nesting and fields that accept several unrelated representations. Represent optionality explicitly and reject unknown properties when the provider’s supported schema subset permits it.
JSON Schema defines a vocabulary for describing and validating the structure of JSON data, but providers can support different subsets or impose additional rules. Pin the dialect or provider contract used by each tool rather than assuming that every valid JSON Schema keyword is accepted (JSON Schema: Draft 2020-12 Core, verified 2026-07-24). For a focused comparison of schema-constrained output modes, see JSON mode vs. structured outputs.
Schema-constrained generation reduces structural errors, but it does not prove that an action is correct or authorized. OpenAI documents that strict: true makes function calls adhere to the declared schema, with requirements including additionalProperties: false on objects and all properties marked as required; optional values can be represented with a nullable type (OpenAI: Function calling, verified 2026-07-23). Anthropic also documents strict tool use, while Google exposes validated function-calling modes for supported APIs. These are provider features, not replacements for application validation.
Version schemas as production interfaces. Store the schema version in each trace and provide compatibility logic for in-flight conversations.
3. Validate arguments before any tool executes
Never pass model-produced arguments directly into a side-effecting function. Put a validation gateway between the model response and the executor.
A practical gateway performs five checks in order:
- Protocol check: the response contains a complete call with a supported type, tool name, and call identifier.
- Parse check: streamed argument fragments have finished and decode as one JSON value.
- Schema check: types, required properties, enums, lengths, and unknown fields satisfy the registered schema version.
- Business-rule check: referenced records exist, dates are allowed, amounts are within policy, and transitions are valid for current state.
- Authorization check: the current user and workflow are permitted to perform this specific action on this specific resource.
Google’s function-calling documentation explicitly describes execution as the application’s responsibility and recommends validating function calls before execution (Google: Function calling with the Gemini API, verified 2026-07-23). That boundary is important: the model proposes an action; your application decides whether and how it can run.
Return validation errors as structured, bounded feedback: a stable error code, invalid field path, short explanation, and whether correction is allowed. Never return stack traces, database details, secrets, or the entire rejected payload. Stop when the same normalized error repeats. Normalize input only when the conversion is unambiguous; if human confirmation is required, create a pending action instead of executing first.
For high-impact tools, apply the threat-model and approval controls in the MCP security checklist even if the integration does not use MCP.
4. Handle missing, duplicate, and parallel calls explicitly
A missing call is not always an API error. Decide which tasks require, allow, or prohibit tools, then enforce that policy in the orchestrator instead of relying only on prompt wording.
If a required call is missing, issue one bounded repair turn that states the unmet requirement and allowed tools. If it remains missing, stop with a typed REQUIRED_TOOL_MISSING outcome.
Duplicate calls need a different defense. Assign every proposed action an idempotency key derived from workflow ID, logical step, tool name, and canonicalized arguments. Store execution state before invoking the downstream system:
proposed -> validated -> executing -> succeeded | failed | outcome_unknown
If the same key appears again after success, return the stored result instead of executing again. If a timeout leaves the outcome unknown, reconcile with the downstream service using its operation ID or idempotency support before retrying.
Parallel calls must remain separate from duplicate calls. OpenAI documents multiple function calls with a call_id for each result and provides parallel_tool_calls: false when at most one call is wanted (OpenAI: Function calling, verified 2026-07-23). Google also documents multiple calls and call identifiers for results (Google: Function calling with the Gemini API, verified 2026-07-23).
Execute calls in parallel only when they are independent. Two reads may be safe; “create invoice” and “charge invoice” have an ordering dependency. Model dependencies as a graph or force sequential tool use. Always correlate results by call ID, never by array position or tool name alone.
5. Retry without repeating side effects
Distinguish model retries, API transport retries, downstream tool retries, and workflow resumes. They are not interchangeable: a model retry can produce new arguments, a tool retry can repeat a purchase or message, and a workflow retry can replay earlier actions.
Mark operations as read-only, idempotent write, or non-idempotent write. Read-only calls can usually use bounded exponential backoff with jitter. Idempotent writes require a stable downstream idempotency key. Non-idempotent writes should not be retried automatically unless the executor can first prove the earlier attempt did not commit. RFC 9110 defines an idempotent request by whether repeated identical requests have the same intended server effect as one request, and cautions against automatically retrying non-idempotent requests without proof that the operation is safe to repeat or was not applied (IETF RFC 9110, section 9.2.2, verified 2026-07-24).
Use this table as a conservative starting policy, then tighten it for the consequences of each tool:
| Operation state | Automatic retry | Required evidence before proceeding |
|---|---|---|
| Validation failed deterministically | 0 | Change the arguments, schema, or policy |
| Read-only call failed transiently | Bounded | Classified transient error and remaining retry budget |
| Idempotent write failed transiently | Bounded | Stable idempotency key and downstream support |
| Non-idempotent write returned a definite failure before commit | Usually 0 | Executor proves no state change occurred |
| Write outcome is unknown after timeout | 0 | Reconciliation proves success or non-commit |
| Same normalized model error repeats | 0 further repairs | Human review or a changed contract |
“Bounded” should be an explicit per-workflow number and deadline, not an open-ended loop. A practical initial policy is one model repair for a correctable validation error; tool retry counts depend on the downstream service’s semantics and latency budget.
Use a retry budget per logical task, counting model repairs, rate-limit retries, tool retries, and failover attempts. On exhaustion, terminate with the latest known state and a safe recovery instruction. The LLM fallback strategy explains why retry and provider failover should remain separate controls.
Do not feed an opaque exception back to the model. Translate failures into a small taxonomy such as INVALID_ARGUMENT, NOT_AUTHORIZED, RESOURCE_NOT_FOUND, CONFLICT, TEMPORARY_UNAVAILABLE, and OUTCOME_UNKNOWN. Only the errors that the model can safely correct should trigger another model turn.
6. Log enough to debug without leaking secrets
Tool traces can contain prompts, tokens, personal data, internal URLs, and private tool results. Logging everything creates a second sensitive data store.
Use an allowlist for logged fields. Keep identifiers and diagnostics needed for correlation, while redacting or hashing sensitive values before the event leaves the execution boundary. A useful event contains:
- Trace, workflow, turn, and call IDs
- Provider, model ID, prompt version, and schema version
- Tool name and argument field names
- Sanitized validation errors
- Start time, duration, timeout stage, and retry count
- Idempotency key or a non-reversible fingerprint
- Execution outcome and whether the result was returned to the model
Do not log authorization headers, cookies, raw secrets, full files, or unrestricted tool output. Record schema-aware summaries such as length, enum value, record count, or salted fingerprint. Apply retention and access controls, and test redaction with adversarial samples. Track schema validity, business-rule rejections, duplicate suppression, tool success, unknown outcomes, retries, and end-to-end completion by tool and schema version.
7. Run a production tool-calling test checklist
Before release, test the protocol and the dangerous edges, not only the happy path.
- Every tool has a distinct purpose, explicit non-use cases, and a versioned schema.
- Valid calls pass both schema and business validation.
- Missing required fields, extra fields, wrong enums, and oversized strings fail before execution.
- Incomplete streamed arguments are never executed.
- Unknown tool names and unknown call IDs terminate safely.
- Required-tool, optional-tool, and prohibited-tool tasks behave differently by policy.
- Multiple independent calls retain correct call-to-result correlation.
- Dependent calls execute in the declared order.
- Replayed successful calls return stored results instead of repeating side effects.
- Timeouts produce
outcome_unknownuntil reconciliation proves success or failure. - Retry budgets stop deterministic loops and retry storms.
- Authorization is checked at execution time, not inferred from the model’s choice.
- Logs redact secrets and sensitive tool output.
- Model, SDK, prompt, and schema upgrades run against a frozen regression set.
- A kill switch can disable a tool without redeploying the entire application.
Include conflicting instructions, prompt injection in tool output, Unicode edge cases, stale IDs, repeated delivery events, and downstream success followed by a lost response. Score terminal state and side effects, not plausible prose.
Use a release gate that fails on any unauthorized execution, duplicate side effect, mis-correlated result, or secret leak. Quality thresholds for lower-impact selection errors can be workload-specific, but safety invariants should be absolute. The approach in building a self-hosted evaluation gate can help keep this suite reproducible.
The central rule is simple: a tool call is a proposal carried over a protocol. Production reliability comes from validating that proposal, executing it through a durable state machine, and proving what happened before trying again.
FAQ
Why does a tool call fail even when the JSON is valid?
Valid JSON proves only that the text can be parsed. The call can still violate the tool schema, business rules, authorization policy, or current resource state. Validate those layers separately and log the earliest failed invariant.
Should I retry malformed tool arguments automatically?
Allow at most a bounded repair turn when the error is specific and safely correctable. Do not retry unchanged input, and stop when the same normalized error repeats. Never execute partially streamed or unvalidated arguments.
Does strict schema mode make tool calls safe?
No. Strict modes can reduce structural mismatches, but they do not establish intent, authorization, semantic correctness, or idempotency. Keep application-side validation and execution controls.
How do I prevent the same tool from running twice?
Create a stable idempotency key for the logical action, persist execution state before the side effect, and return the stored result when a completed key is replayed. If the earlier outcome is unknown, reconcile it before any retry.
What should I log for a tool-calling incident?
Keep trace and call IDs, provider and model ID, prompt and schema versions, sanitized validation errors, timing, retry count, idempotency fingerprint, and terminal outcome. Redact secrets and minimize raw arguments and tool output.
Primary sources
- OpenAI: Function calling — verified 2026-07-23
- Anthropic: Define tools — verified 2026-07-23
- Google: Function calling with the Gemini API — verified 2026-07-23
- JSON Schema: Draft 2020-12 Core — verified 2026-07-24
- IETF RFC 9110, section 9.2.2 — verified 2026-07-24