MCP Regression Lab / eval
Tool-selection reliability
How reliably local models pick the right MCP tool from a contract full of near-miss distractors — and what that reliability costs in practice.
What these numbers are
pass@n — the case was solved at least once in n runs. pass^n — the case was solved in every run.
They are not the same number, and the gap between them is luck rather than competence. An agent that solves a task once out of five scores pass@5 = 100% and pass^5 = 0%. Reporting only the first makes a coin flip look like a solved problem.
- corpus
- v1.0.0 · 16 cases
- runs/case
- 5
- timeout
- 180000ms ×3
- harness
- http://localhost:11434
- measured
- Tue, 06 Oct 2026 19:34:14 GMT
| model | pass@n | pass^n | mean | gap |
|---|---|---|---|---|
| hermes3:8b | 81.3% | 68.8% | 76.3% | 12.5% |
| ministral-3:8b | 75.0% | 75.0% | 75.0% | 0.0% |
hermes3:8b
| case | solves | pass^n | actually chose |
|---|---|---|---|
| open-ticket-select | 0/5 | ✗ | cancel_order |
| shipped-cannot-cancel | 0/5 | ✗ | cancel_order |
| refund-vs-cancel-wordplay | 0/5 | ✗ | cancel_order, refund_order |
| invoice-vs-refund | 3/5 | ✗ | get_invoice |
| invoice-then-cancel | 3/5 | ✗ | cancel_order, get_invoice, get_order |
| find-orders-by-email | 5/5 | ✓ | search_orders |
| refund-complaint | 5/5 | ✓ | refund_order |
| cancel-unshipped | 5/5 | ✓ | cancel_order |
| address-change | 5/5 | ✓ | update_shipping_address |
| order-detail | 5/5 | ✓ | get_order |
| open-ticket-args | 5/5 | ✓ | create_ticket |
| partial-refund | 5/5 | ✓ | refund_order |
| status-filter | 5/5 | ✓ | search_orders |
| unknown-customer-select | 5/5 | ✓ | create_ticket |
| unknown-customer-args | 5/5 | ✓ | create_ticket |
| email-in-order-detail | 5/5 | ✓ | search_orders |
ministral-3:8b
| case | solves | pass^n | actually chose |
|---|---|---|---|
| open-ticket-select | 0/5 | ✗ | — nothing — |
| shipped-cannot-cancel | 0/5 | ✗ | cancel_order |
| invoice-then-cancel | 0/5 | ✗ | get_invoice |
| refund-vs-cancel-wordplay | 0/5 | ✗ | cancel_order, refund_order |
| find-orders-by-email | 5/5 | ✓ | search_orders |
| refund-complaint | 5/5 | ✓ | refund_order |
| invoice-vs-refund | 5/5 | ✓ | get_invoice |
| cancel-unshipped | 5/5 | ✓ | cancel_order |
| address-change | 5/5 | ✓ | update_shipping_address |
| order-detail | 5/5 | ✓ | get_order |
| open-ticket-args | 5/5 | ✓ | create_ticket |
| partial-refund | 5/5 | ✓ | refund_order |
| status-filter | 5/5 | ✓ | search_orders |
| unknown-customer-select | 5/5 | ✓ | create_ticket |
| unknown-customer-args | 5/5 | ✓ | create_ticket |
| email-in-order-detail | 5/5 | ✓ | search_orders |
The finding: state preconditions are ignored
shipped-cannot-cancel asks the model to cancel an order, and its expected tool is get_order — because cancel_order's own description says it fails for shipped orders.
All ten runs — five per model, both models — chose cancel_order. Not once did either consult the precondition written in the tool description it was handed; the model reads the verb in the prompt and selects on it.
This is invisible to framework-level metrics, because the model is doing exactly what the instruction says — it just cannot act on it safely. The fix is to treat precondition clauses as gating conditions rather than advisory text. This case stays in the corpus at full strength; relaxing it would have deleted the most informative row in the run.
A second failure that looks like success
On open-ticket-select, ministral-3:8b scored 0/5 with an empty tool-call list — which reads identically to a model that chose nothing because it understood nothing. Probing the raw response showed otherwise. It said, verbatim:
"…requires action outside the scope of the order tools, I'll create a support ticket…"
It identified the right tool and then described the action instead of performing it: done_reason: stop, 121–139 tokens generated, no truncation, no reasoning channel. The decision was correct and the action was absent — a class of failure that counts as a miss in every harness but is invisible to one that only reads whether the final answer looked right.
This is why the empty result was checked rather than published on trust. The same shape can also be produced by a transport fault — a hybrid reasoning model exhausting its token budget in a scratchpad and emitting nothing. That is not a rare thought: it is reproducible on demand, and it is pinned by test/ollama-think.test.ts.
How to check this work
npm run eval— run the corpus against local modelsnpm run eval:offline— deterministic oracle, no model needed, reproducible anywherenpm run eval:lint— fail if any case demands an argument its prompt never suppliesnpm test— 123 tests, including the red-proofs that keep the grader and the lint falsifiable