Calibrating on your labels

When you deploy a decision question in production—such as routing support tickets, gating shell commands, or triaging failures—you want to automate as many decisions as possible while keeping wrong decisions below a fixed error budget (for example, target_error: 0.05 for ≥95% precision).

Instead of guessing a confidence threshold like 0.75, send 200 to 5,000 labeled examples from your own workload to POST /v1/calibrations.

Step 1: Prepare 200+ labeled examples

Each item in examples has the same state and questions structure as POST /v1/systemone, plus a label field on each question:

Question type	label type	Valid values
choice	string	Any option key present in criteria (e.g. "billing")
noul	boolean	true or false
score	integer	Level index 0 to len(criteria) - 1

Every distinct question signature (type + instructions + sorted option keys) requires at least 50 labeled examples to be fitted, and the total examples array must contain 200 to 5,000 examples.

Step 2: Fit the profile with POST /v1/calibrations

import os
import requests

# Load 250 labeled examples from your validation set
examples = [
    {
        "state": row["text"],
        "questions": {
            "intent": {
                "type": "choice",
                "instructions": "What does the customer need?",
                "criteria": {
                    "billing": "A charge, invoice, or refund issue",
                    "shipping": "Delivery status or tracking",
                    "account": "Login or profile settings",
                },
                "label": row["true_intent"],  # "billing" | "shipping" | "account"
            }
        },
    }
    for row in validation_rows
]

res = requests.post(
    "https://api.mouse.dev/v1/calibrations",
    headers={"Authorization": f"Bearer {os.environ['MOUSE_API_KEY']}"},
    json={
        "model": "mouse-preview-2026-10-13",
        "target_error": 0.05,
        "examples": examples,
    },
    timeout=30.0,
)
res.raise_for_status()
cal = res.json()

print("Signed profile:", cal["profile"])
print("Question report:", cal["questions"][0])

Inspecting the out-of-fold report

{
  "object": "calibration",
  "model": "mouse-preview-2026-10-13",
  "target_error": 0.05,
  "profile": "calp1.k1.eyJmaXRzIjpb...",
  "n_examples": 250,
  "questions": [
    {
      "signature": "4f8c12a90e3b7104",
      "type": "choice",
      "question_ids": ["intent"],
      "status": "fitted",
      "before": {
        "accuracy": 0.912,
        "ece": 0.031,
        "brier": 0.142,
        "nll": 0.264,
        "mean_confidence": 0.938,
        "coverage": 1.0,
        "cutoff": null,
        "n": 250
      },
      "after": {
        "accuracy": 0.912,
        "ece": 0.031,
        "brier": 0.142,
        "nll": 0.264,
        "mean_confidence": 0.938,
        "coverage": 0.864,
        "cutoff": 0.78,
        "n": 250,
        "coverage_at_target_error": 0.856,
        "error_when_covered": 0.0421
      }
    }
  ],
  "note": "'coverage_at_target_error' and 'error_when_covered' are 5-fold out-of-fold. Examples were not stored.",
  "usage": {
    "input_tokens": 14500,
    "state_tokens": 3500,
    "question_tokens": 7750,
    "framing_tokens": 3250,
    "image_tokens": 0,
    "cached_tokens": 0,
    "output_tokens": 0,
    "cost_usd": 0.00145
  }
}

Key metrics in after:

• cutoff (0.78): The fitted p_top threshold stored in your signed profile.
• coverage_at_target_error (0.856): Across 5 out-of-fold splits, 85.6% of examples had p_top >= cutoff and were answered automatically.
• error_when_covered (0.0421): Among those 85.6% automated examples, the out-of-fold error rate was 4.21% (within your 0.05 target).

Step 3: Pass calibration on POST /v1/systemone

Store the calp1. token in your environment or configuration (for example MOUSE_CALIBRATION_PROFILE) and pass it alongside the pinned model snapshot on /v1/systemone:

res = requests.post(
    "https://api.mouse.dev/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['MOUSE_API_KEY']}"},
    json={
        "model": "mouse-preview-2026-10-13",
        "calibration": cal["profile"],
        "state": "Where is my tracking link for order #1099?",
        "questions": {
            "intent": {
                "type": "choice",
                "instructions": "What does the customer need?",
                "criteria": {
                    "billing": "A charge, invoice, or refund issue",
                    "shipping": "Delivery status or tracking",
                    "account": "Login or profile settings",
                },
            }
        },
    },
).json()

intent = res["answers"]["intent"]
# intent["mouse"] == {"calibrated": True, "p_top": 0.94, "abstain": False}

if intent["mouse"]["abstain"]:
    route_to_human_queue(ticket)
else:
    route_to_department(intent["choice"], ticket)

Applying a calibration profile on /v1/systemone adds 0 tokens and $0 to your request cost.