Docs Live

Live: your app's real prompts, tested

Add two lines to your app. A small package copies a sample of the prompts your app sends and the answers it gets, and posts them to us in the background. We group them into prompts, write test cases like your real ones, and test them on other models: a first look once a prompt has 5 calls, the full test at 20, and again when a notable model comes out. When something does the job better, faster or cheaper, you hear about it with the cases that show it, and can copy the change straight into your coding agent. When nothing does, we say that too.

Your calls still go straight to your provider. Nothing in your app gets slower, we never see your API key, and if we are down your app does not notice.

Last updated 18 November 2026 as Markdown, to paste into your coding agent

Start

The quickest way

In your app's folder, run one of these:

npx springprompt init
python -m springprompt init

It opens Spring Prompt in your browser, where you sign in, pick or make a project and confirm “Connect this computer” after checking the code matches your terminal. The terminal then writes a new key to .env (and makes sure .env is in .gitignore), never into your code, and prints the two lines to add and where. It can send a test call too, so the set-up page shows your first call straight away. --project <id> picks a project in advance, --no-browser only prints the address, and --yes asks no questions.

It stops before making a key if .env is already committed to git, and asks before replacing a key that is already there. npx needs nothing installed first; for Python, pip install springprompt first.

By hand

  1. Sign in and open Live. Make a project for your app. You get a key starting spg_, shown once.
  2. Put it in your app's environment as SPRINGPROMPT_KEY. It can only add calls to that project: it cannot read anything or spend credit.
  3. Add the package and two lines where your app starts.

Python

pip install springprompt
import springprompt
springprompt.init()

It watches calls made with httpx or requests, which is what the OpenAI, Anthropic, Google and Mistral SDKs use. Python 3.9 or later. python -m springprompt test sends one made-up call so you can see it arrive.

Node

npm install springprompt
import { init } from "springprompt";
init();

It watches fetch, which the OpenAI, Anthropic and Google SDKs use. Node 18 or later. npx springprompt test sends one made-up call.

Serverless

A function that stops as soon as it returns may stop before the package has sent anything. Call springprompt.flush() (Python) or await flush() (Node) before returning.

Options

None are needed. The ones people use most are naming a prompt and removing more before anything leaves your app; every option is in the reference below.

# Python
with springprompt.prompt("ticket-triage"):
    client.chat.completions.create(...)
// Node
await prompt("ticket-triage", () => client.chat.completions.create({ ... }));

With no key set, init() does nothing, so laptops, tests and CI stay quiet without any change.

Privacy controls

What leaves your app is decided in your app, and what we keep is decided by your project's settings. Both can be tightened; neither needs a call to us.

Always removed, in your app

  • Every header, so your provider key never leaves your app, and anything after ? in the address.
  • Anything that looks like a key or token (provider keys, bearer tokens, our own spg_ keys), wherever it appears in a prompt or answer.
  • Images, audio and files: those calls are counted, not copied.

Masks

mask removes these in your app, before anything is sent. They work inside the text of the prompt and the answer only, so numbers elsewhere in the request, such as timestamps and token limits, are left alone.

MaskWhat it removesBecomes
"email"Email addresses.[email]
"card"Card numbers: 13 to 19 digits, with spaces or dashes, that pass the card checksum. Other long numbers, such as order numbers, are left.[card]
"ip"IPv4 addresses.[ip]
"phone"Runs of 7 or more digits with the usual spaces, dots, dashes, brackets and +. It also catches some dates and long ids, which is why it is only on if you ask for it.[phone]

Separately, we remove email addresses, card numbers and IP addresses again on our side before anything is kept, whatever your app sends (a project setting, on unless you switch it off), and phone numbers too if you tick that.

Your own redact function

redact is yours to write, for anything the masks don't know: customer and order numbers, names from your own records, internal codes. It is given the request body (JSON text) and the answer's text, after keys and masks are out, and returns them cleaned. It runs in your app, before sending. If it raises an error, that call is not sent at all.

# Python
import re
import springprompt

ORDER = re.compile(r"ORD-\d{6}")

def redact(text: str) -> str:
    return ORDER.sub("[order]", text)

springprompt.init(mask=["email", "phone"], redact=redact)
// Node
import { init } from "springprompt";

init({
  mask: ["email", "phone"],
  redact: (text) => text.replace(/ORD-\d{6}/g, "[order]"),
});

The request body is JSON, so change what is inside its strings and keep the rest as it is: a body that no longer reads as JSON is still sent, but its prompt cannot be worked out. The order is always: keys out, then masks, then your function.

On our side

  • Collecting can be switched off for a project; running apps stop sending within a minute.
  • The share of calls copied after the first 20 from each place in your code: one in ten unless you change it.
  • How long calls are kept: 30 days, or 7, or 90. Encrypted while kept. Deleting a project deletes everything in it at once.
  • What tests run on: made-up cases by default, so your users' messages never go to the models being compared. Testing on the real calls instead is a setting.

All of these are in the project's settings in the app. Every setting is listed below, and the privacy page says who receives what.

What you see

The Overview

Each prompt your app sends, named for you, with the model it uses, about what it costs a month, and an answer to three questions: better, faster or cheaper? Each answer links to the suggestion behind it, or says where the prompt is: collecting calls, testing, or “nothing better found” with how many models were tried. Above them: calls a day, spend a month, answers that pass your checks and typical answer time, over 30 days and against the week before.

The first look, at 5 calls

As soon as a prompt has 5 answered calls, it gets a first look, with no waiting for real traffic to build up:

  • Checks, drafted from your prompt. Yes-or-no questions a judge answers about each answer, the “must” ones deciding whether an answer is usable. You can check a few of the judge's verdicts: where it disagrees with you, the check is rewritten until it agrees, and the page says how often it agreed. That is how the checks are calibrated to your idea of a good answer.
  • Models, scored with how sure we are. Your model and the usual candidates answer 10 made-up cases written from your calls. Each gets a pass rate with its likely range and how sure that is (high, medium or low). A model is only called clearly better or worse when the ranges do not overlap. The ranges narrow when the full test runs at 20 calls, and with each test after it.
  • Ideas for the wording. A few specific ideas for tweaking your prompt, each aimed at a check your model failed. They are hypotheses to test, not a rewrite.

A first look runs by itself, even with automatic testing off, for a project's first 3 prompts, at most $0.60 of credit each. It can be switched off in the project's settings. After it, tests run on one click, or by themselves within a weekly budget you set: the full test at 20 calls, each notable new model on your prompts the week it comes out, and the named replacement when a model you use is being retired.

Suggestions

When a model or a new wording does a prompt's job better, faster or cheaper on cases it never saw, it becomes a suggestion, with the figures it was decided on and the cases behind it. Each suggestion has its own page, which the emails link to: the open suggestions down the side, and the words to paste into your coding agent, with a copy button (and one to copy them all). See Applying a change.

The morning report

A short email each morning, from 07:00 UTC: yesterday's calls and spend, what is worth changing (each with a link to copy it for your coding agent), the new models tried on your prompts, and prompts still collecting calls. A morning with no calls and nothing new sends nothing. In the project's settings you can have it on Mondays instead, or only when there is something to change. The same goes to a Slack or Discord channel if you give us its incoming-webhook address, and every email has a link to stop them.

What is sent, and what never is

For a sample of calls: the provider's address and path, the request body, the answer, its status, how long it took, the file and function in your code that made it, and the prompt's name if you gave one. For every call, sampled or not: a count, so volumes and cost estimates are real.

The sample is the first 20 calls from each place in your code, then one in ten. You can change the share, set a different one for any single place (0 to send nothing from it), or turn collecting off, in the project's settings; running apps pick it up within a minute.

Never sent:

  • Any header, so your provider key never leaves your app, and no query string.
  • Calls to anything that is not a model provider.
  • Images, audio and files. Those calls are counted, not copied. A tool that merely takes an image's address or a file as an argument is not media, and its calls are sent (package 0.2.0 on).
  • Bodies over 256 KB.
  • Anything that looks like a key or token. It is removed in your app before sending, and again by us before keeping.

What arrives is encrypted before it is stored, and deleted after 30 days unless you choose 7 or 90. Deleting a project deletes everything in it at once.

Tests run on made-up cases

Your real calls don't go into tests. One model reads a cleaned sample of a prompt's calls (keys, email addresses, phone and card numbers and IP addresses removed, whatever your settings) and writes new cases like them: the same kind of request, length, language and tone, with every name, number and detail made up. A made-up case that repeats a run of six words, a number of four or more digits, an email address or a code from a real one is dropped. Only the made-up cases go to the models being compared and the models that judge them, through OpenRouter, and each candidate is compared with the answer the model you use now gives to the same case.

You can test on the real calls instead, in the project's settings. That is closest to your traffic and compares each candidate with the answer your app really gave, but the calls then go to the models being compared and are kept with the test for 90 days.

Prompts that call tools

If your app offers the model tools, nothing more is needed: the tool definitions are in the request and the model's tool calls are in the answer, and both are read as they are, from OpenAI, Anthropic, Gemini and OpenAI-style providers, streamed or not.

We test the next step the model takes; we never run your tools. Each model is given the same conversation and the same tool definitions, once. What counts is whether it does what your app's model did: call the same tool with equivalent arguments, or answer in words where yours answered in words. Arguments worded differently, such as a search query, are read by a judge that says whether the two calls would do the same thing. A recommendation for such a prompt rests on that agreement, and says how often.

What this does not cover, plainly:

  • Anything after the next step. What your tools would return to a different call is something only your app knows, so we do not play a conversation forward, and we do not say a model is "better" at a tool prompt, only that it takes the same steps for less money or in less time.
  • With made-up cases (the default), only the first step of a conversation: a made-up message from the user, the same tools, and each model's step compared with your current model's. To test later steps, with earlier tool calls and their results in view, set the project to test on real calls. Then your own model is asked again too, which shows how often it repeats itself: no other model can be expected to agree more than that.
  • Calls that offer a provider's own tools, such as web search or code execution. Another model cannot be offered the same tool, so those calls are left out.
  • Calls over 256 KB, which are counted and not copied, like any other. A long conversation with large tool results can pass that.

Cost, with the prompt cache

If your app sends the same long instructions with every call, your provider reads most of them from its prompt cache and bills that part at a fraction of the list price. A comparison at list price alone would then pick the wrong cheapest model. So cost in Live is worked out from what your calls report: how many input tokens there were, and how many of them were read from the cache.

  • The model you use now: cached input at its cached price, the rest at list price. The prompt's page says what share was cached.
  • Each other model: the same share taken as cached, at that model's own cached price. Where we have no cached price for a model, none of its input is taken as cached, and the recommendation says its figure assumes no caching.
  • Tokens written to the cache are priced as ordinary input. Some providers charge a little more for those, so a prompt that writes to the cache often costs slightly more than we show.

The counts come from the usage in each answer, so there is nothing to switch on, and it works with every version of the packages. A streamed OpenAI answer only has usage if your app asks for it with stream_options: {"include_usage": true}; without it we have no token counts for that call.

Reference

init()

Python takes keyword arguments: springprompt.init(key=None, *, mask=..., ...). Node takes one object: init({ key, mask, ... }). Every option can be left out.

OptionDefaultWhat it does
keySPRINGPROMPT_KEYThe project's key. With none, the package does nothing at all.
masknoneAny of "email", "phone", "card", "ip", removed in your app. See Masks.
redactnoneYour own function over each request body and answer, run last. See redact.
hostsnoneMore addresses to watch, such as a LiteLLM proxy ("llm.internal:4000"). The main providers, OpenRouter and Azure OpenAI are watched already.
envSPRINGPROMPT_ENV, else "production"Which environment calls came from, so staging and production can be told apart.
releaseSPRINGPROMPT_RELEASE, or the commit Railway, Vercel, Render or Heroku setsWhich release calls came from, so a new version of a prompt can be matched to the deploy that shipped it.
share0.1The share of calls copied after the first 20 from each place in your code, 0 to 1. The project's setting, when you change it there, takes over.
placesnoneA share for one place, by a prompt's name or its file:function: {"triage": 0.02, "app/billing.py:summarise": 0}. It replaces share after the first 20, and 0 sends nothing from that place, not even the first 20. The project's settings can set the same; the lower of the two wins. From version 0.3.1.
urlSPRINGPROMPT_URL, else springprompt.comWhere calls are posted. Only for testing against another copy of Spring Prompt.
debugoffSay what the package does (Python writes to standard error, Node uses console.debug), to see which calls are watched and sent.

Functions

PythonNodeWhat it does
with springprompt.prompt("name"): or @springprompt.prompt("name")prompt("name", fn)Names the calls made inside it, so they group as one prompt. Works with async code.
springprompt.flush(timeout=2.0)await flush(2000)Sends what is waiting now. Call it at the end of a serverless handler.
springprompt.shutdown(timeout=2.0)await shutdown(2000)Sends what is waiting, then stops watching.
springprompt.hooked()(none)Which HTTP libraries are being watched: httpx, requests.

Commands

CommandWhat it does
npx springprompt init, python -m springprompt initConnects this app to a project and writes the key to .env. See The quickest way.
npx springprompt test, python -m springprompt testSends one made-up call with the key in the environment, to see it arrive on the set-up page.
npx springprompt status, python -m springprompt statusSays whether a key is set and where calls go.

Project settings

In the app, under Set up, for each project.

SettingDefaultWhat it does
CollectingonOff stops your app sending anything, within a minute.
Share of calls10%Of calls after the first 20 from each place in your code.
Keep calls for30 daysOr 7, or 90.
Remove before keepingemail, card, IPRemoved by us again before anything is kept; phone numbers too if ticked.
Test on real callsoffOff: tests run on made-up cases. On: on your real calls, kept with the test for 90 days.
First lookonEach new prompt tested at 5 calls, for the first 3 prompts, at most $0.60 each. See the first look.
Test without askingoffWith a weekly budget: the full test at 20 calls, new models as they come out, and replacements for retiring models.
Weekly check of real answersoffJudges a few of the answers your app really gave with each prompt's checks, and tells you if fewer pass.
Emailson, dailyThe morning report, on Mondays, or only when there is something to change; to the addresses you list.
Slack or DiscordnoneAn incoming-webhook address that gets the same news.

Rules for what we suggest

When you dismiss a suggestion, the reason you pick becomes a rule for the project:

  • Not this model: it is never suggested again for this project.
  • Not this provider: no model from that lab is suggested again.
  • Quality matters more than cost for this prompt: that prompt gets no more “cheaper” suggestions, only better and faster ones.

A cheaper model is only suggested when it saves at least 25% and does the job as well. A coding agent can read and set the same rules with our MCP server's set_rules tool (never_models, never_labs, quality_first and min_saving).

Any other language

The packages are only a convenient way to call one endpoint. From Go, Ruby, PHP, Java, C# or a no-code tool, post each call yourself: what you sent the provider and what came back. There is no schema of ours to learn.

curl https://springprompt.com/sdk/v1/calls \
  -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "provider": "openai",
  "path": "/v1/chat/completions",
  "request": {
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "system",
        "content": "Sort support tickets into billing, bug or other."
      },
      {
        "role": "user",
        "content": "I was charged twice this month."
      }
    ]
  },
  "answer": {
    "model": "gpt-4o-mini",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "billing"
        }
      }
    ],
    "usage": {
      "prompt_tokens": 31,
      "completion_tokens": 1
    }
  },
  "name": "ticket-triage",
  "ms": 640,
  "status": 200
}'
  • provider is one of openai, anthropic, google, openrouter, mistral, deepseek, xai, groq, together, fireworks, cerebras, azure. Or send host, such as api.openai.com.
  • request and answer are the provider's own JSON, or text: the raw text of a streamed answer is fine. If you only keep the final text of a stream, send answer_text instead of answer.
  • Optional: name, where (file and function), ms, status, id (a retried post with the same id is kept once).
  • Token counts are read from the usage in answer. If yours has none (you send answer_text), add "usage": {"input_tokens": 5210, "output_tokens": 48, "cached_tokens": 4950}, where input_tokens includes the cached part.
  • Tool definitions and tool calls need nothing extra: leave them in request and answer as the provider saw them.
  • One call, or {"calls": [...]} for up to 200 at once, 5 MB at most. Gzip is welcome.
  • If you send only some calls, say so with "share": 0.1 so volumes stay right.

Post after the answer has gone back to your user, never inside the request, never with the provider's headers, and never let a failed post fail the call. In Go, that looks like this:

// After the answer has gone back to your user. Never send the provider's headers.
func sendToSpringPrompt(provider, path string, request, answer []byte) {
	key := os.Getenv("SPRINGPROMPT_KEY")
	if key == "" {
		return
	}
	body, err := json.Marshal(map[string]any{
		"provider": provider,              // "openai", "anthropic", "google", ...
		"path":     path,                  // "/v1/chat/completions"
		"request":  json.RawMessage(request),
		"answer":   string(answer),        // JSON, or the raw text of a stream
	})
	if err != nil {
		return
	}
	go func() {
		req, err := http.NewRequest("POST", "https://springprompt.com/sdk/v1/calls", bytes.NewReader(body))
		if err != nil {
			return
		}
		req.Header.Set("Authorization", "Bearer "+key)
		req.Header.Set("Content-Type", "application/json")
		client := &http.Client{Timeout: 5 * time.Second}
		if resp, err := client.Do(req); err == nil {
			resp.Body.Close()
		}
	}()
}

The answer is 200 with the project's settings, 401 for a key we did not make, 403 for a removed key (stop sending), 400 for a batch we cannot read (drop it), and 429 or 503 to try again later, after any Retry-After.

Applying a change

Nothing changes in your app until you change it. Each suggestion says exactly what to edit: which call, in which file and function, and the new model or wording. On its page, “Copy for your coding agent” copies that as an instruction to paste into Claude Code, Codex or Cursor in your repo, with the check to run before merging; it works without our MCP server, which only lets the agent run the check itself. Mark it applied, or just ship it: once the new model or wording shows up in your app's calls, it closes by itself, and the prompt's page compares cost and speed before and after.

With a coding agent and our MCP server, the loop is: list_changes, edit the code as its apply says, run check_change on the prompt's real cases, and open the pull request only if the check passes. (A prompt that calls tools has no such check: its test covers the next step only, so try the whole flow before it ships.) Dismissing with a reason, such as “not this provider”, stops the same suggestion coming back.

With a coding agent

Paste this into Claude Code, Codex or Cursor in your repo. With our MCP server connected, the agent can make the project and check the first call by itself.

Add Spring Prompt Live to this app. It is a small package that copies a sample of the app's LLM calls (prompt and answer) to Spring Prompt in the background, so they can be tested on other models. Calls still go straight to the provider; the package never sees or sends provider keys or headers, and with no SPRINGPROMPT_KEY set it does nothing.

1. Get a key: call the Spring Prompt MCP tool create_project (or ask me to make a project at https://springprompt.com/app/live/setup). The key starts with spg_. It can only add calls to that project; it cannot read or spend anything. Do not commit it. Or run `npx springprompt init --no-browser` (or, once the Python package is installed, `python -m springprompt init --no-browser`) in the repo: it prints a link for me to confirm, then writes the key to .env and adds .env to .gitignore.
2. Install the package and add two lines where the app starts, before the first model call:
   Python: pip install springprompt (and add springprompt to the project's dependencies), then
       import springprompt
       springprompt.init()
   Node: npm install springprompt (it goes into package.json), then
       import { init } from "springprompt";
       init();
   In a serverless function (Lambda, Vercel, Cloudflare), also call flush() (Python: springprompt.flush(); Node: await flush()) before the handler returns.
3. Put SPRINGPROMPT_KEY in the app's environment where it runs (the hosting provider's variables or secrets, and .env for local runs if the app uses one and it is gitignored). Never in code or in the repo.
4. If one function sends several different prompts, name each with springprompt.prompt("name") (Python, as a with block or decorator) or prompt("name", fn) (Node). Optional.
5. Check it: run `python -m springprompt test` or `npx springprompt test` with the key set, then call the MCP tool setup_status, or tell me to open https://springprompt.com/app/live/setup.

Another language: post each call yourself to https://springprompt.com/sdk/v1/calls after the answer has gone back to the user, never inside the request, with `Authorization: Bearer $SPRINGPROMPT_KEY` and a JSON body of {"provider": ..., "path": ..., "request": <the body sent>, "answer": <the body received>}. Never send the provider's headers or key. A failed post must never fail the call. The full format is at https://springprompt.com/docs/live.

Change nothing else in the app.