MCP Regression Lab

Docs

MCP Regression Lab answers one question before your users do: does today’s release of my MCP server still behave like the release I approved, from an agent’s point of view? It reports two signals separately: a deterministic contract diff and probabilistic golden tool-selection tests.

Quickstart

  1. Create an account and add your server: a remote streamable-HTTP MCP endpoint over https, plus a bearer token if it needs one.
  2. The first successful check becomes your approved baseline. Every later check is diffed against it.
  3. In Settings, point golden tests at the model your users’ agents use, then add a few golden cases on the server page.
  4. Create an API token and add the GitHub Action so every pull request is checked before it ships.

Baselines

A baseline is the snapshot of your tool contract you consider correct. Checks compare against it, not against the previous check, so a breaking change stays visible on every check until you fix it or accept a new baseline. Alerts fire once per new breaking change rather than on every check.

When a change is intentional, open the server page, review the diff with Compare with baseline, then Accept as baseline. Breaking changes are removed or renamed tools, removed or retyped arguments, arguments that became required, and new required arguments. Added tools, new optional arguments and description edits are reported as changes, because descriptions steer how models choose tools. Descriptions that read like prompt injection are flagged as warnings.

Golden tests

A golden case is a user prompt plus the tool a model should call, the arguments it must supply, and tools it must never call. Each check gives the model your current tool list and the prompt, and records what it picks. Your MCP server is never called. Each case runs three times and reports a pass rate, because models are not deterministic; a drop in pass rate is a behaviour regression even when the contract is unchanged.

Tests run against your own OpenAI-compatible endpoint (OpenAI, Groq, OpenRouter, Together, or a self-hosted model behind https), so you test the model your users actually use and keep control of the cost.

Writing good cases

CI

Create an API token under Settings and store it as the repository secret MCP_REGRESSION_LAB_TOKEN. The action runs a check through the hosted service, comments the result on the pull request, and fails the job on a breaking change or a golden regression.

name: MCP regression check
on:
  pull_request:
  workflow_dispatch:

jobs:
  mcp:
    runs-on: ubuntu-latest
    steps:
      - uses: BoozeLee/mcp-regression-lab@v1
        with:
          api-token: ${{ secrets.MCP_REGRESSION_LAB_TOKEN }}
          server-id: <server id from your dashboard URL>

Any other CI can call the endpoint directly. It returns 200 whenever the check ran; read ok, status, breaking and golden from the JSON body. It answers 401 for a missing or revoked token, 404 for a server you don’t own, and 429 for more than one check per server per minute.

curl -X POST \
  -H "Authorization: Bearer $MCP_REGRESSION_LAB_TOKEN" \
  https://mcp-regression-lab.vercel.app/api/v1/servers/<server-id>/checks

Security model