Docs

The API: test prompts and read the results from your own code

The API does what the app and our MCP server do, as plain JSON over HTTPS: start a test of a prompt on several models, read what it found, improve the prompt, and ask which model each of your saved tests points to. It is for servers, scripts and scheduled jobs. If you work through a coding agent, the guide for agents covers the same actions as MCP tools.

The typical use is a job that reads your saved tests' recommendations and keeps the model each feature of your product uses up to date. There is an example below.

Get a key

  1. Sign in and open Connect your agent. A free trial account is enough to try it. No credit card needed.
  2. Make a key. It is shown once. Keep it in your server's secrets, never in a repo or a prompt.
  3. Send it with every call as a bearer token: Authorization: Bearer YOUR_KEY.

The same keys work for the MCP server and for checks from a pull request. Removing a key on that page stops it at once.

The examples below read the key from the environment:

export SPRINGPROMPT_KEY="your key"

The endpoints

Every address starts with https://springprompt.com/api/v1. Calls that send data take a JSON object, and every answer is a JSON object. The answers shown here are examples of the shape: the model ids and the figures are made up.

CallWhat it doesCredit
GET /api/v1/modelsThe models a test can use, with list prices per million tokens, and the credit left on the account.Free
POST /api/v1/testsStarts a test of one prompt on several models and answers at once.Spends credit, never more than max_spend_usd by estimate
GET /api/v1/tests/{test_id}A test's status and, when it is done, what it found.Free
GET /api/v1/testsThe account's recent Prompt tests, newest first.Free
GET /api/v1/tests/{test_id}/recommendationWhich model one saved test's answers point to, read from the answers the test already holds, with what has changed since it last ran.Free: it runs nothing
GET /api/v1/recommendationsThe same for every saved test on the account: the model sheet as JSON.Free: it runs nothing
POST /api/v1/tests/{test_id}/improveRewrites a finished test's prompt from its failures and your ratings, adds it as a new version and tests it on the same cases, checks and models.Spends credit: about one more run of the test
GET /api/v1/judgesThe account's trained judges: each learned from one person's ratings and says how often it agreed with them on answers it had not seen.Free
POST /api/v1/judges/{judge_id}/scoreScores up to 50 answers with a trained judge: how likely the person it learned from is to call each one good, from 0 to 1.Spends a little credit

GET /api/v1/models

The models a test can use, with list prices per million tokens, and the credit left on the account. Free.

curl -sS "https://springprompt.com/api/v1/models" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 200 with credit_left_usd, the starting_mix, models with prices per million tokens, and a note on how many more there are.

{
  "credit_left_usd": 18.4,
  "starting_mix": [
    "maker-a/model-large",
    "maker-b/model-small"
  ],
  "models": [
    {
      "id": "maker-a/model-large",
      "name": "Model Large",
      "input_per_million": 3.0,
      "output_per_million": 15.0
    },
    {
      "id": "maker-b/model-small",
      "name": "Model Small",
      "input_per_million": 0.1,
      "output_per_million": 0.5
    }
  ],
  "note": "More models are available than are listed here."
}

POST /api/v1/tests

Starts a test of one prompt on several models and answers at once. Cases and checks are drafted for you if you leave them out. Models are left out, dearest first, to fit max_spend_usd. Spends credit, never more than max_spend_usd by estimate.

It takes: prompt (needed), system, name, cases, checks, models, max_spend_usd.

curl -sS -X POST "https://springprompt.com/api/v1/tests" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
  -H "Content-Type: application/json" \
  --data '{
  "name": "Support replies",
  "prompt": "Reply to this customer message: {{message}}",
  "system": "You are a support agent for a furniture shop.",
  "cases": [
    {
      "values": {
        "message": "Where is my order?"
      }
    },
    {
      "values": {
        "message": "The table arrived scratched."
      }
    }
  ],
  "checks": [
    "Answers the question asked",
    "Is polite"
  ],
  "models": [
    "maker-a/model-large",
    "maker-b/model-small"
  ],
  "max_spend_usd": 0.5
}'

It answers 202 with test_id, status, url, how many cases and checks it will use, the models, any left_out_to_fit_limit, and most_it_can_cost_usd.

{
  "keys_removed": 0,
  "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "status": "queued",
  "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "cases": 2,
  "checks": 2,
  "models": [
    "maker-a/model-large",
    "maker-b/model-small"
  ],
  "left_out_to_fit_limit": [],
  "most_it_can_cost_usd": 0.31,
  "next": "GET /api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b?wait=25 until status is done or failed. Tests usually take one to three minutes."
}

GET /api/v1/tests/{test_id}

A test's status and, when it is done, what it found. With ?wait= the request is held for up to 25 seconds while the test finishes; call it again until status is done or failed. Free.

curl -sS "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b?wait=25" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 200 with status, url, versions, results (each model's pass rate, likely range, cost per 1,000 answers and speed) and spent_usd. When done, also text and pick (the verdict), failures and cascades.

{
  "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "name": "Support replies",
  "status": "done",
  "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "versions": [
    {
      "id": "v1",
      "note": "",
      "system": "You are a support agent for a furniture shop.",
      "prompt": "Reply to this customer message: {{message}}"
    }
  ],
  "results": [
    {
      "version": "v1",
      "model": "maker-b/model-small",
      "name": "Model Small",
      "passed_percent": 91.7,
      "checks_percent": 95.8,
      "passed": 11,
      "of": 12,
      "likely_passed_percent": [
        65,
        99
      ],
      "cost_per_1000_answers": 0.42,
      "cost_per_1000_usable_answers": 0.46,
      "median_seconds": 1.2,
      "errors": 0
    }
  ],
  "spent_usd": 0.07,
  "text": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers.",
  "pick": "maker-b/model-small",
  "failures": [],
  "cascades": [],
  "next": "Rate some answers yourself, because passing checks isn't the same as being good."
}

GET /api/v1/tests

The account's recent Prompt tests, newest first. Free.

curl -sS "https://springprompt.com/api/v1/tests" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 200 with tests, each with a test_id, name, status, date and link.

{
  "tests": [
    {
      "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
      "name": "Support replies",
      "status": "done",
      "created_at": "2026-11-04T09:10:02+00:00",
      "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b"
    }
  ]
}

GET /api/v1/tests/{test_id}/recommendation

Which model one saved test's answers point to, read from the answers the test already holds, with what has changed since it last ran. Free: it runs nothing.

curl -sS "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b/recommendation" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 200 with recommended_model (or null), whether it passes, its pass_rate and cost_per_1000_usd, the runner_up, checked_at, whether it is stale and the changes since, changed_at and previous_model if a recheck moved it, and the url.

{
  "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "name": "Support replies",
  "status": "done",
  "version": "v2",
  "recommended_model": "maker-b/model-small",
  "recommended_model_name": "Model Small",
  "passes": true,
  "pass_rate": 91.7,
  "passed": 11,
  "cases": 12,
  "cost_per_1000_usd": 0.42,
  "cost_per_1000_usable_usd": 0.46,
  "runner_up": {
    "model": "maker-a/model-large",
    "name": "Model Large",
    "pass_rate": 100.0,
    "passed": 12,
    "cases": 12,
    "cost_per_1000_usd": 6.1,
    "cost_per_1000_usable_usd": 6.1
  },
  "models_compared": 3,
  "reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
  "note": "",
  "checked_at": "2026-11-04T09:12:44+00:00",
  "stale": true,
  "changes": [
    {
      "kind": "price",
      "text": "Model Small's list price has gone down since this test last ran."
    }
  ],
  "changed_at": "2026-11-04T09:12:44+00:00",
  "previous_model": "maker-a/model-large",
  "recheck": "on",
  "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b"
}

GET /api/v1/recommendations

The same for every saved test on the account: the model sheet as JSON. Free: it runs nothing.

curl -sS "https://springprompt.com/api/v1/recommendations" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 200 with recommendations, one for each saved test, each with the fields of the call above.

{
  "recommendations": [
    {
      "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
      "name": "Support replies",
      "status": "done",
      "version": "v2",
      "recommended_model": "maker-b/model-small",
      "recommended_model_name": "Model Small",
      "passes": true,
      "pass_rate": 91.7,
      "passed": 11,
      "cases": 12,
      "cost_per_1000_usd": 0.42,
      "cost_per_1000_usable_usd": 0.46,
      "runner_up": {
        "model": "maker-a/model-large",
        "name": "Model Large",
        "pass_rate": 100.0,
        "passed": 12,
        "cases": 12,
        "cost_per_1000_usd": 6.1,
        "cost_per_1000_usable_usd": 6.1
      },
      "models_compared": 3,
      "reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
      "note": "",
      "checked_at": "2026-11-04T09:12:44+00:00",
      "stale": true,
      "changes": [
        {
          "kind": "price",
          "text": "Model Small's list price has gone down since this test last ran."
        }
      ],
      "changed_at": "2026-11-04T09:12:44+00:00",
      "previous_model": "maker-a/model-large",
      "recheck": "on",
      "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b"
    }
  ]
}

POST /api/v1/tests/{test_id}/improve

Rewrites a finished test's prompt from its failures and your ratings, adds it as a new version and tests it on the same cases, checks and models. Spends credit: about one more run of the test.

curl -sS -X POST "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b/improve" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 202 with test_id, the new version, status, the changes made, and the rewritten prompt and system.

{
  "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "version": "v2",
  "status": "queued",
  "changes": [
    "Says to answer the question in the first sentence"
  ],
  "prompt": "Reply to this customer message: {{message}}\nAnswer the question in your first sentence.",
  "system": "You are a support agent for a furniture shop.",
  "next": "GET /api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b?wait=25 until status is done."
}

GET /api/v1/judges

The account's trained judges: each learned from one person's ratings and says how often it agreed with them on answers it had not seen. Free.

curl -sS "https://springprompt.com/api/v1/judges" -H "Authorization: Bearer $SPRINGPROMPT_KEY"

It answers 200 with judges, each with an id, name and agreement.

{
  "judges": [
    {
      "id": "judge_3f9a",
      "name": "Support tone",
      "labels": 60,
      "agreement": 0.86,
      "baseline": 0.62,
      "ready": true,
      "summary": "Agreed on 86% of answers it hadn't seen. Always giving the commoner verdict would get 62%. Trained on 60 ratings; a fast classifier."
    }
  ]
}

POST /api/v1/judges/{judge_id}/score

Scores up to 50 answers with a trained judge: how likely the person it learned from is to call each one good, from 0 to 1. Spends a little credit.

It takes: answers (needed).

curl -sS -X POST "https://springprompt.com/api/v1/judges/judge_3f9a/score" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
  -H "Content-Type: application/json" \
  --data '{
  "answers": [
    "Your order left our warehouse today.",
    "Check the website."
  ]
}'

It answers 200 with judge, its name, and good_probability, one number per answer.

{
  "judge": "Support tone",
  "good_probability": [
    0.93,
    0.12
  ]
}

The MCP tool behind each call has the same fields: list_models, test_prompt, get_test, list_tests, get_recommendation, list_recommendations, improve_prompt, list_judges, score_answers.

What “passes” and “recommended” mean

A recommendation is read from the answers a saved test already holds. Asking for one runs no model and spends no credit, so you can ask as often as you like. It is only as good as the test: a handful of cases cannot tell two close models apart, and it says so in note.

  • An answer passes when it came back without an error and met every must check of the test. A case that was run several times passes if it passed in most of its runs.
  • pass_rate is the share of the test's cases that passed, from 0 to 100, on the newest version of the prompt that has answers. passes is true when a model is recommended, and false when there is none to recommend; it does not mean every case passed, so read pass_rate for that.
  • recommended_model is the test's own pick, the one named at the top of its page: the model that passed the most cases (the cheapest of them if several tie), unless a model that costs less per usable answer shows no clear difference from it on these cases. Then it is the cheapest such model. “No clear difference” is an exact sign test on the cases where the two differ, and it is only applied with 6 cases or more; with fewer, the pick is simply the model that passed the most.
  • Cheapest means cost_per_1000_usable_usd: the cost of 1,000 answers divided by the pass rate, since failed answers are paid for too.
  • A model that passed no case is never recommended, and a test with no must check has no recommendation: recommended_model is null and note says why.
  • runner_up is what the same rule picks once the recommended model is left out.

stale is true when something has changed since the test last ran: a newer model in a line the test uses, a changed list price, or a model that is being retired. changes says what. The answers are then older than the market, and a rerun would say whether the recommendation still holds. changed_at and previous_model are set when a scheduled recheck moved the recommendation, and are null otherwise.

Benchmarks on this site compare models on fixed public tasks. A recommendation here is narrower: it is about your prompt and your cases, and it holds only for them.

Example: a job that keeps a per-feature model list up to date

Say each feature of your product has its own saved test, named after the feature, and your code reads the model for each feature from a file. This script writes that file from your tests' recommendations. It runs nothing and spends nothing, so it is safe to run every hour.

#!/bin/sh
# Writes models.json: {"Support replies": "maker-b/model-small", ...}
set -eu
out=$(curl -sS --fail "https://springprompt.com/api/v1/recommendations" \
  -H "Authorization: Bearer $SPRINGPROMPT_KEY")
echo "$out" | jq '[.recommendations[]
  | select(.recommended_model != null)
  | {key: .name, value: .recommended_model}] | from_entries' > models.json.new
mv models.json.new models.json
# Tests whose answers are older than a change in the market:
echo "$out" | jq -r '.recommendations[] | select(.stale) | "\(.name): \(.changes[0].text)"'

Run it from cron, for example at ten past every hour:

10 * * * * SPRINGPROMPT_KEY=... /opt/app/bin/update-models.sh

The script keeps the old file if a call fails, and leaves out a test that has no recommendation. A saved test is deleted 90 days after it was made, and its recommendation goes with it, so as written the script would then drop that feature from the file. A test with scheduled rechecks on is the exception: it is kept for as long as they stay on, and for 90 days after they go off. For a test your job depends on, switch them on, make the test again before it goes, or have your job keep the last model it knew. Have a person look at a change before it reaches production if the feature matters: previous_model and changed_at tell you when a recommendation moved, and reason says why the model was picked.

The recommendations only move when the tests are run again. You can rerun a test from its page when it is flagged, or switch on a scheduled recheck for it on the model sheet in the app: it reruns the test when something changes and once a week, within a spending limit (at the moment at most $2.00 a recheck and $5.00 on one account in any 24 hours), and stops if nobody has looked at the test for 3 rechecks in a row.

Webhooks

Instead of asking again and again, you can have us tell your server. Set one https address under Webhook on the keys page, and we post a small signed JSON event to it when a run of one of your tests ends, or when a run changes the model a test recommends. An account has one webhook.

Setting the address makes a signing secret, which is shown once. On the same page you can send a test event (its type is webhook.test), switch the webhook off and on, make a new secret, and see the last deliveries with the status your server answered.

The address has to start with https://, carry no user name or password, and belong to a host on the public internet. We refuse a host at a private, loopback or link-local address, check again every time we send, and do not follow redirects.

The events

TypeSent when
test.finishedA run of one of your Prompt tests has ended, done or failed, whoever started it: you in the app, a key over the API, a coding agent over MCP or a scheduled recheck.
recommendation.changedA run has ended and the test now recommends a different model from the one it recommended before that run. It is sent after the test.finished event for the same run. A test's first run does not send it, and neither does a run that failed.

The body is a JSON object. status is done or failed. started_by says what started the run: app, api, mcp, schedule. recommendation is exactly what GET /api/v1/tests/{id}/recommendation answers. As everywhere on this page, the model ids and the figures in the examples are made up.

An event never carries your prompts, test cases or answers. Treat it as a hint that something is ready: fetch the test from api_url with your key, and act on what that call answers.

test.finished

{
  "id": "evt_3b1f0c9d5e7a4f2c8d6b0a1e2f3c4d5e",
  "type": "test.finished",
  "created_at": "2026-11-04T09:12:45+00:00",
  "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "test_name": "Support replies",
  "status": "done",
  "started_by": "api",
  "api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "recommendation": {
    "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
    "name": "Support replies",
    "status": "done",
    "version": "v2",
    "recommended_model": "maker-b/model-small",
    "recommended_model_name": "Model Small",
    "passes": true,
    "pass_rate": 91.7,
    "passed": 11,
    "cases": 12,
    "cost_per_1000_usd": 0.42,
    "cost_per_1000_usable_usd": 0.46,
    "runner_up": {
      "model": "maker-a/model-large",
      "name": "Model Large",
      "pass_rate": 100.0,
      "passed": 12,
      "cases": 12,
      "cost_per_1000_usd": 6.1,
      "cost_per_1000_usable_usd": 6.1
    },
    "models_compared": 3,
    "reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
    "note": "",
    "checked_at": "2026-11-04T09:12:44+00:00",
    "stale": true,
    "changes": [
      {
        "kind": "price",
        "text": "Model Small's list price has gone down since this test last ran."
      }
    ],
    "changed_at": "2026-11-04T09:12:44+00:00",
    "previous_model": "maker-a/model-large",
    "recheck": "on",
    "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b"
  }
}

recommendation.changed

{
  "id": "evt_9a8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d",
  "type": "recommendation.changed",
  "created_at": "2026-11-04T09:12:45+00:00",
  "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "test_name": "Support replies",
  "status": "done",
  "started_by": "schedule",
  "api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
  "previous_model": "maker-a/model-large",
  "recommendation": {
    "test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
    "name": "Support replies",
    "status": "done",
    "version": "v2",
    "recommended_model": "maker-b/model-small",
    "recommended_model_name": "Model Small",
    "passes": true,
    "pass_rate": 91.7,
    "passed": 11,
    "cases": 12,
    "cost_per_1000_usd": 0.42,
    "cost_per_1000_usable_usd": 0.46,
    "runner_up": {
      "model": "maker-a/model-large",
      "name": "Model Large",
      "pass_rate": 100.0,
      "passed": 12,
      "cases": 12,
      "cost_per_1000_usd": 6.1,
      "cost_per_1000_usable_usd": 6.1
    },
    "models_compared": 3,
    "reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
    "note": "",
    "checked_at": "2026-11-04T09:12:44+00:00",
    "stale": true,
    "changes": [
      {
        "kind": "price",
        "text": "Model Small's list price has gone down since this test last ran."
      }
    ],
    "changed_at": "2026-11-04T09:12:44+00:00",
    "previous_model": "maker-a/model-large",
    "recheck": "on",
    "url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b"
  }
}

An event never carries your prompt, your cases or any answer. Treat it as a hint that something changed, and fetch api_url with your key for the rest. We may add fields and event types later, so ignore what your code does not know.

Check the signature

Each event comes with these headers:

X-SpringPrompt-EventThe event's type, the same as type in the body.
X-SpringPrompt-DeliveryThe delivery's id. It stays the same each time we try the same event again.
X-SpringPrompt-Signaturet=<unix seconds>,v1=<signature>, made new for each attempt.

The signature is the HMAC-SHA256, in hex, of the time, a full stop and the raw body, made with your signing secret as the key. Work it out over the bytes you received, before you parse the JSON, compare it without leaking timing, and refuse an event whose time is more than 5 minutes from now. In Python:

import hashlib, hmac, time

def verified(secret: str, raw_body: bytes, header: str) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(",") if "=" in p)
    try:
        at = int(parts.get("t", ""))
    except ValueError:
        return False
    if abs(time.time() - at) > 300:  # too old, or a clock far out
        return False
    signed = f"{at}.".encode() + raw_body
    want = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(want, parts.get("v1", ""))

In Node:

const crypto = require("node:crypto");

function verified(secret, rawBody, header) {
  const parts = Object.fromEntries(header.split(",").map((p) => p.split("=")));
  const at = Number(parts.t);
  if (!Number.isInteger(at) || Math.abs(Date.now() / 1000 - at) > 300) return false;
  const want = crypto.createHmac("sha256", secret).update(`${at}.`).update(rawBody).digest("hex");
  const got = Buffer.from(String(parts.v1 || ""));
  return got.length === want.length && crypto.timingSafeEqual(got, Buffer.from(want));
}

Keep the secret in your server's secrets. If it may have leaked, make a new one on the keys page: the old one stops signing at once.

Delivery, retries and repeats

  • Answer with any 2xx status within 10 seconds. Answer first and do the work afterwards: put the event on a queue, or note the test's id and fetch it later.
  • Any other answer, a redirect included, or no answer in time counts as a failure. We then send the event again after 1 minute, 5 minutes, 30 minutes, 2 hours and 6 hours: 6 attempts in all. After the last one we give up on that event.
  • The same event can reach you more than once, for example when your answer was lost on the way. X-SpringPrompt-Delivery and the id in the body stay the same each time, so keep the ids you have handled and skip the repeats. Events can also arrive out of order.
  • When 15 events in a row have been given up, we switch the webhook off and say why on the keys page. Events from the time it was off are not sent later.
  • Polling still works exactly as before, and it is the fallback: GET /api/v1/tests/{id}?wait=25 for one test and GET /api/v1/recommendations for all of them. A webhook is a hint, not a guarantee, so a slow job like the one above is a sensible safety net for an event that never arrived.
  • We read no more than the first few kilobytes of your answer and keep none of it. The keys page lists your deliveries for 30 days, without the events' contents.

Checks from a pull request

Two more calls take the same keys and sit outside /api/v1: POST /api/checks runs a changed prompt or a different model against a saved test and answers pass, review or fail, and GET /api/checks/{id} reads the verdict. They have their own spending limits (at the moment $1.00 a check and $5.00 on one account in any 24 hours). Check a prompt or model change before you merge has a workflow to copy, the fields and what the verdict means.

Limits

Reading is free. Starting a test, improving a prompt and scoring answers spend the account's credit on the model calls they make. Before anything runs, the server works out the most the call could cost and refuses it if that is over a limit or over the credit left, so a refused call spends nothing. You are charged what a test really cost, which a finished test reports as spent_usd.

Credit one key may use$5.00 in any 24 hours across calls to /api/v1, unless you set the key its own limit on the keys page (at most $50.00). A call is checked against the most it could cost before anything runs.
Credit for one test$1 unless the call sets max_spend_usd, and never more than $25.
Calls2,000 a day for each account, shared with the MCP tools.
Waiting on one readUp to 25 seconds.
Size of one request1 MB.
Models in one testUp to 6.
Cases in one testUp to 24.
KeysUp to 10 in use on one account.

Never put API keys, passwords or real customer data in a prompt or a case. A prompt and its cases are sent to the models you choose and to the model that marks the answers. If a key is found in what you send, it is removed before the test is saved, and keys_removed says how many were.

Errors

An error is a JSON object with one field, {"error": "a plain reason"}, and one of these statuses. Nothing is spent on a call that is refused.

400The body was not a JSON object.
401No key, or a key that has been removed.
402The account's plan does not include what was asked (the X-Plan-Needed header names the plan that does), or there is not enough credit for it.
404No such route, or no such test or judge on this account.
409The test is running. Wait for it to finish.
413The request was larger than the size limit.
422A field is missing or cannot be used. The reason says which.
429The day's calls are used up, or the call could take the key past its spending limit for 24 hours. Nothing was spent.

A test that could not finish is not an error of the call: reading it answers 200 with "status": "failed" and the reason in error, and nothing is charged for it.

What we keep

For each call we record, against your account, which endpoint was called and when, and if it was refused, why. Credit a call spends is recorded with the key that spent it, which is how a key's limit is kept. We do not keep the body of a request beyond what the action itself saves: a test you start is saved to your account like any other test, with its prompt, cases and answers. If you set a webhook, we store its address and signing secret, and keep a list of what we sent it for 30 days, without the events' contents. The privacy page has the full list.