Sign inTest your prompt
Docs

Check a prompt or model change before you merge

A saved Prompt test holds your test cases, your checks and the answers your current prompt gave. This check runs a candidate (a changed prompt, a changed system message, a different model, or any mix) on the same cases with the same checks, compares it with those saved answers, and answers pass, review or fail. A job in your CI can call it on every pull request that touches the prompt.

What you need

  • A Spring Prompt account with a Prompt test that has been run at least once. A free trial account is enough to try it. No credit card needed. Start one here.
  • The test's id. It is the last part of the test's address in the app, and it is shown on the test's page under “Check a change before you merge”.
  • An account key from Connect your agent in the app (the same keys the MCP server takes). Keep it in your CI's secrets, never in the repo.

A GitHub Actions workflow

Save this as .github/workflows/prompt-check.yml, add your key as the repository secret SPRINGPROMPT_KEY, and set the test id and the path to your prompt file. It starts a check, waits for the verdict, and fails the job on fail.

name: Prompt check
on:
  pull_request:
    paths:
      - "prompts/support.txt"

jobs:
  check:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
      - name: Check the prompt against the saved test
        env:
          SPRINGPROMPT_KEY: ${{ secrets.SPRINGPROMPT_KEY }}
          TEST_ID: 00000000-0000-0000-0000-000000000000
          PROMPT_FILE: prompts/support.txt
          LABEL: "PR #${{ github.event.pull_request.number }}"
          API: https://springprompt.com/api/checks
        run: |
          set -euo pipefail
          auth="Authorization: Bearer $SPRINGPROMPT_KEY"
          body=$(jq -n --arg test "$TEST_ID" --arg label "$LABEL" --rawfile prompt "$PROMPT_FILE" \
            '{test_id: $test, prompt: $prompt, label: $label}')
          started=$(curl -sS -X POST "$API" -H "$auth" -H "Content-Type: application/json" --data "$body")
          id=$(echo "$started" | jq -r '.id // empty' 2>/dev/null || true)
          if [ -z "$id" ]; then
            echo "::error::The check did not start: $started"
            exit 1
          fi
          result="$started"
          for _ in $(seq 1 60); do
            [ "$(echo "$result" | jq -r '.status' 2>/dev/null || echo running)" = "running" ] || break
            sleep 2
            result=$(curl -sS "$API/$id?wait=20" -H "$auth" || echo '{"status":"running"}')
          done
          echo "$result" | jq '{status, verdict, reasons, numbers, cost_usd, url}'
          summary=$(echo "$result" | jq -r '.summary // .error // "It was still running when this job stopped waiting."')
          url=$(echo "$result" | jq -r '.url // empty')
          case "$(echo "$result" | jq -r '.verdict // empty')" in
            pass) echo "Passed. $summary $url" ;;
            review) echo "::warning::Review before you merge. $summary $url" ;;
            fail) echo "::error::$summary $url"; exit 1 ;;
            *) echo "::error::No verdict. $summary Run the job again."; exit 1 ;;
          esac

To check a model change instead, send "model": "the-new-model-id" in place of (or as well as) prompt. A model id is the maker and the model's name, such as maker/model-name. The list_models tool on our MCP server lists them, a check's answer names the ids it used, and a request with an id we do not know is refused with the closest ones.

Pull requests from forks cannot read repository secrets on GitHub, so the job will not start a check for them.

What the verdict means

  • fail: a must check that passed on the saved answers fails on the candidate, on any case. Or quality dropped by more than chance would explain.
  • review: something got worse, but by an amount that could be chance on this many cases. Or part of the candidate could not be answered or judged. The job above passes with a warning; a person should read the cases that changed.
  • pass: neither of those.

Quality is counted case by case. A case is worse when the candidate passes fewer of its checks than the saved answer did (must and stretch checks alike, and a rubric by its level), and better when it passes more. The split of worse against better cases is put to an exact sign test: a drop that chance would produce less than one time in twenty is a fail, and a smaller one is a review.

The answer also carries the numbers (cases compared, how many pass every must check before and after, cases better and worse), the checks that got worse, what the check cost, and a link to the test's page in the app, where the check is listed. The link opens for the account's owner once signed in.

The limits, plainly

  • It costs credit. A check runs every case once on one model and marks the answers, so it costs about what running your test on that one model costs. Starting a check tells you the most it could cost (most_it_can_cost_usd) and sets that aside; you are charged what it really cost. If it could cost more than the credit you have left, it is refused and nothing is spent.
  • There are spending limits on our side. At the moment one check can use up to $1.00 of credit, and all the checks on one account up to $5.00 in any 24 hours. A check that would go over is refused before it runs. You can set a lower limit for one check with max_spend_usd.
  • It takes as long as a test run. Usually one to three minutes; longer with many cases or a slow model. A check still going after 15 minutes is stopped with no verdict and no charge.
  • A small test cannot say much. With fewer than six cases that changed, a drop in quality is only ever a review. A must check that newly fails is a fail on a test of any size.
  • Models and judges vary. The same prompt can give a different answer on a second run, and a check that a model judges can come out differently on the same answer. So a fail on a judged must check is sometimes a false alarm: run the job again before you rewrite anything. Checks that are rules (length, a phrase, valid JSON) do not vary.
  • The baseline is saved, not rerun. The candidate is compared with the answers your test holds for its newest version. If those are old, the model behind them may have changed since. When you merge a change, add the new prompt as a version of the test in the app, so the next check is held to it. A test holds five versions; after that, start a new test from your best version and change the id in the workflow.
  • A deploy can interrupt a check. If our server restarts while a check is running, the check ends as interrupted with no verdict, and the credit set aside is returned. The job above fails with “No verdict”; run it again.
  • What we keep. The verdict, the numbers and the checks that got worse (with a short excerpt of the case and the reason) stay on the test, for the newest 30 checks. The candidate's prompt and its answers are not kept. As with any test, the prompt and cases are sent to the model you choose and to the judging model.

The two calls

Both take the account key as a bearer token: Authorization: Bearer YOUR_KEY.

POST /api/checks

Starts a check. Send JSON:

{
  "test_id": "the saved test's id",
  "prompt": "the candidate prompt, with the same {{variables}} as the test",
  "system": "the candidate system message",
  "model": "a different model id",
  "baseline_model": "which of the test's models to compare with, if it has several",
  "max_spend_usd": 0.5,
  "label": "PR #12"
}

Only test_id and one of prompt, system or model are needed. What you leave out stays as it is in the test. It answers 202 at once with the check's id, a poll address and most_it_can_cost_usd. If the candidate is the same as the saved prompt and model, it answers 200 with pass and runs nothing.

Refusals come back as {"error": "..."} with a plain reason: 401 without a valid key, 404 for a test that is not on the account, 409 while the test itself is running, 422 for a request that cannot be checked or would go over the limit for one check, 402 when credit is short, 429 at the daily limit, 503 if checks are switched off.

GET /api/checks/{id}

The check's state. Add ?wait=20 and it holds the request for up to 25 seconds while the check finishes. status is running, done, failed or interrupted; only done has a verdict.

{
  "id": "…",
  "status": "done",
  "verdict": "fail",
  "summary": "1 must check that passed on the baseline failed on the candidate, on 1 case.",
  "reasons": ["…"],
  "numbers": {"cases": 12, "baseline_passed": 11, "candidate_passed": 10,
              "cases_better": 0, "cases_worse": 1, "cases_same": 11, "p": 1.0,
              "must_regressions": 1, "could_not_judge": 0, "unanswered": 0, "not_compared": 0},
  "regressions": [{"case": 4, "input": "…", "check": "…", "was": "pass", "now": "fail", "detail": "…"}],
  "cost_usd": 0.04,
  "most_it_can_cost_usd": 0.31,
  "url": "https://springprompt.com/app/prompt/…?check=…#checks"
}

The numbers above show the shape of an answer; they are not from a real test.

From a coding agent

The same check is on our MCP server as two tools, check_change and get_check, with the same fields. Connect your agent to use them.

From your own code

The same key also starts tests, reads results and says which model each saved test points to, as plain JSON. The API docs list every call.