You upgraded to the most capable model in your stack and it started inventing API endpoints that do not exist. Not because you misconfigured anything. Because you made it smarter.
Yin et al., "The Reasoning Trap" (accepted to ACL 2026 Main, revised April 2026) establishes this as causal, not correlational: reinforcement learning that improves reasoning performance raises tool hallucination proportionally across every RL method and base model they tested. Training a model on GSM8K mathematics, where no tools appear at all, still amplifies subsequent tool hallucination. The mechanism is in the weights, not in the prompts.
Your system prompt cannot fix this. Here is what can.
The numbers behind the tradeoff are not subtle
The paper benchmarks models on SimpleToolHalluBench, which isolates two failure modes worth naming precisely because both need separate instrumentation:
- NTA (No-Tool-Available): the agent must recognize that no appropriate tool exists and abstain. Invoking anything is wrong.
- DT (Distractor-Tool): a plausible-looking but wrong tool is available. The agent must reject it, not select it.
The hallucination rates across model pairs tell the story plainly.
| Model | Configuration | NTA Rate | DT Rate |
|---|---|---|---|
| Qwen2.5-7B | Base | 34.8% | 54.7% |
| DeepSeek-R1-Distill-Qwen-7B | RL-enhanced | 74.3% | 78.7% |
| Qwen3-8B | Thinking disabled | 4.1% | 36.2% |
| Qwen3-8B | Thinking enabled | 5.4% | 56.8% |
The Qwen3-8B row is the clearest single demonstration. Flipping on native thinking mode is one switch. NTA barely moves (4.1% to 5.4%). But distractor-tool hallucination nearly doubles (36.2% to 56.8%). The model became more likely to reach for a wrong tool every time a reasoning chain fired.
The DeepSeek comparison is starker. Moving from the base Qwen2.5-7B to its RL-distilled variant: NTA jumps from 34.8% to 74.3%, DT from 54.7% to 78.7%. Capability roughly doubled on task benchmarks. Tool reliability was cut in half.
Prompt engineering moves the needle by rounding error
The paper ran mitigation experiments on ReCall-7B and the results deserve their own table before anyone in your org argues that better instructions are the answer.
| Mitigation | NTA Rate | DT Rate | Task Reward |
|---|---|---|---|
| Baseline | 90.2% | 100.0% | 0.45 |
| + Prompt engineering | 87.5% | 98.9% | 0.44 |
| + DPO alignment | 55.8% | 71.4% | 0.34 |
Prompt engineering reduces NTA by 2.7 points. DT drops by 1.1 points. Both numbers are within measurement noise. The system prompt is not moving anything material.
DPO alignment is a real intervention: NTA falls 34 points, DT falls 29 points. But task reward drops from 0.45 to 0.34, a 24% degradation in the capability you were trying to preserve. It is not a calibration dial. It is a trade where you give back the capability gain to buy back the reliability.
The mechanistic finding explains why prompts are inert. Yin et al. show via Centered Kernel Alignment (CKA) analysis that RL training leaves in-distribution task representations stable (CKA above 0.9) while tool-related representations collapse in middle network layers (CKA below 0.75). Hallucinations then accumulate through late-layer residual streams, where the discrimination scores between correct and hallucinated responses peak above 0.14. There is no single "tool gating" component a system prompt can reach. The drift is distributed across network depth.
Vinay's failure mode taxonomy (arXiv 2511.19933, 15 documented failure modes across real-world LLM systems) frames the same observation from a systems perspective: incorrect tool invocation is a first-class system-engineering failure mode, not a sub-case of generic hallucination. The implication is the same from both research traditions. This is structural. Fix the architecture.
Capability partitioning is the architectural answer
The authors of "The Reasoning Trap" call for novel training objectives. That is a research programme measured in quarters. The architectural response is available today.
The fix is to stop asking the same model to both reason about what to do and execute tool calls. Those two responsibilities have opposite requirements under RL training: reasoning improves with capability enhancement, tool dispatch degrades. Merge them in one model and you cannot optimize both.
Xu et al., "Reducing Tool Hallucination via Reliability Alignment" (arXiv 2412.04141) arrived at the same partition from a different direction in December 2024. Their reliability alignment approach trains a model to distinguish "should I use a tool?" from "execute the call." That separation is the embryonic form of what production systems now need explicitly.
The partition in concrete terms:
Reasoning layer: A high-capability, RL-optimized model. Its job is to interpret the task, plan the steps, and evaluate whether a tool call is warranted. Elevated DT and NTA hallucination risk is acceptable here because this layer does not dispatch tool calls directly. It emits structured intent.
Constrained executor: A smaller, reliability-aligned model, or better still a deterministic registry router. Its action space is bounded by a verified tool schema it cannot escape. It receives structured intent from the reasoning layer and resolves it against the registry. If no match exists, it returns a structured no-match signal rather than fabricating a call.
The concreteness matters. "Constrained" does not mean "prompted to be careful." It means:
- JSON schema validation before every dispatch
- Deterministic registry resolution (no fuzzy string matching against tool names)
- Hard
no-matchpath that escalates rather than inventing
When the reasoning layer's confidence on a registry match is low, that is a product decision about escalation: route to human review, fall back to safe-mode abstention, or retry with a narrowed tool set. None of those options involve the model inventing a tool name.
This partition also maps directly to the orchestration fault taxonomy in our earlier post on production failures: the most damaging faults are the ones where the planner picks the wrong tool and every downstream step inherits the bad plan. A constrained executor that cannot pick a wrong tool by construction eliminates that entire propagation pathway.
Tool-fidelity evaluation is a different metric from task completion
Your current eval dashboard almost certainly shows task completion rates. It almost certainly does not show NTA rates or DT rates. Those are invisible at the task level and only visible at the span level, which means you are shipping systems with unknown tool reliability into production.
Madvil et al., "Holistic Evaluation and Failure Diagnosis of AI Agents" (Deepchecks, arXiv 2605.14865) builds the case quantitatively. A framework that combines agent-level assessment with bottom-up span-level analysis gives GPT-5.4 2.8x better localization accuracy on GAIA benchmark versus monolithic evaluation. On longer SWE-bench traces, the gap widens to 12x. Monolithic pass/fail evaluation is not just incomplete; it is actively misleading on the metrics that govern whether your tool-calling agent is reliable.
The paper's central observation lands squarely:
— Madvil et al. — Holistic Evaluation and Failure Diagnosis of AI Agents (2026)Methodology rather than model capability represents the primary bottleneck.
What a tool-fidelity harness measures that task completion does not:
- Tool selection accuracy: did the model choose the right tool when one was available?
- NTA abstention rate: did the model abstain correctly when no tool fit?
- DT rejection rate: did the model reject the plausible-but-wrong tool?
- Invocation ordering correctness: did multi-tool sequences execute in valid dependency order?
SimpleToolHalluBench is a usable starting point for the NTA and DT dimensions specifically. The benchmark is constructed from 296 tools with query generation via ChatGPT-4o, which gives enough coverage to serve as a CI gate on any model you are considering for RL enhancement.
Add NTA rate and DT rate as gates in your CI pipeline. A model that improves pass@1 while degrading either metric is not an upgrade for tool-heavy workloads.
Right model for this layer, not best model for the task
The question your architecture review should be asking is not "what is the best model for this agent?" It is "what is the right model for each layer of this agent?"
For the reasoning layer: highest-capability RL-optimized model you can justify. The elevated hallucination risk is acceptable because this layer does not touch the tool dispatch surface. It plans.
For the executor layer: a smaller reliability-aligned model, or a deterministic registry router with no language model in the dispatch path at all. A JSON schema resolver is more reliable than any language model making the final dispatch decision, and it is faster and cheaper.
The cost implication is worth naming. Routing only planning through a frontier model and dispatch through a constrained layer cuts your per-invocation cost on the most frequent operation in an agentic system. Safer and more economical are not usually the same direction. Here they are.
This does not mean your system becomes cheap or simple. Capability partitioning adds architectural complexity: the reasoning layer and executor layer need a clean interface contract, the no-match escalation path needs a real product decision behind it, and the eval harness now needs to validate both layers independently. You are trading prompt-and-pray simplicity for a system you can actually debug when something goes wrong. That trade is worth making.
Add two metrics to your harness this week
NTA rate and DT rate are not in your current pass@1 dashboard. Both are exactly what your most capable model is quietly failing on every time its reasoning chain fires on a task with no matching tool.
Add both as CI gates before the next RL-enhanced model goes into production. SimpleToolHalluBench (arXiv 2510.22977) is the starting harness. Run it on your current model to get a baseline. Run it again on the next version that changes RL training. Any regression on NTA or DT is a signal the reasoning trap has activated, regardless of what the task completion number shows.
The structural fix takes longer. The metrics are available today.
References
- Yin et al. — The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination (ACL 2026)
- Yin et al. — The Reasoning Trap, full HTML (arXiv 2510.22977v1)
- Madvil et al. — Holistic Evaluation and Failure Diagnosis of AI Agents (Deepchecks, 2026)
- Vinay — Failure Modes in LLM Systems: A System-Level Taxonomy (arXiv 2511.19933)
- Xu et al. — Reducing Tool Hallucination via Reliability Alignment (arXiv 2412.04141)