Calibration and abstain

Zero-shot probabilities from any decision model can be over- or under-confident when your production traffic has different class priors or harder boundary cases than the model's training mix. Tuning confidence cutoffs by hand across dozens of questions is tedious and breaks whenever you change a rubric.

POST /v1/calibrations automates threshold fitting in a single stateless API call: send 200 to 5,000 labeled examples (with at least 50 examples per distinct question signature), specify your target_error (default 0.05, or 95% precision on answered items), and receive a signed calp1. profile token plus a 5-fold out-of-fold reliability report.

How calibration works

1. Question signature matching: Each question is hashed into a 16-character signature over (type, rendered_instructions, sorted_option_keys). Because option keys are sorted when computing the signature, a profile still matches even if option order changes between requests.
2. Temperature scaling: For each signature, a scalar temperature T is fitted by minimizing negative log-likelihood of your labels over the model's raw logits. Dividing the logits by T before the softmax makes the reported probabilities mean what they say on your traffic (an overconfident question gets T > 1 and flatter probabilities) without ever changing which option wins. after.temperature reports the fitted value and after.ece the calibration error that remains.
3. 5-fold out-of-fold cross-fitting: For each question signature with 50+ labeled examples, POST /v1/calibrations splits the scored examples into 5 folds. Each fold is evaluated using the temperature and confidence cutoff fitted on the other 4 folds:
   • The cutoff is the lowest scaled top probability (p_top) at which the empirical error rate among covered examples (p_top >= cutoff) is at most target_error.
   • after.coverage_at_target_error and after.error_when_covered report the honest out-of-fold fraction of traffic automated and the error rate on that automated slice.
4. Stateless HMAC-SHA256 profile (calp1.): The fitted temperatures, cutoffs, and the exact model snapshot ID (mouse-preview-2026-10-13) are serialized and signed with HMAC-SHA256 (calp1.<kid>.<base64url>). Neither your labeled examples nor the profile are stored on Mouse servers.

Why both a temperature and a cutoff: the temperature fixes what the number means; the cutoff fixes when to act on it. Fit once, then threshold on calibrated p_top (or on mouse.abstain) instead of a hard-coded 0.7 that was tuned against a different model's confidence scale.

Applying a profile on POST /v1/decisions

Pass the profile string as calibration on your /v1/decisions (or /v1/systemone) requests:

{
  "model": "mouse-preview-2026-10-13",
  "calibration": "calp1.k1.eyJmaXRzIjpb...",
  "state": "Customer says: my card was charged twice for order #1042.",
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "What does the customer need?",
      "criteria": {
        "billing": "A charge, invoice, or refund issue",
        "shipping": "Delivery status or tracking",
        "account": "Login or profile settings"
      }
    }
  }
}

When a question in /v1/systemone matches a signature in the profile:

Field	Uncalibrated (calibration omitted or question not in profile)	Calibrated (calibration matches question signature)
mouse.calibrated	false	true
probabilities, noul, score, confidence	Raw model distribution	Temperature-scaled distribution (same argmax)
mouse.p_top	Raw top probability ((\max_i p_i))	Calibrated top probability
mouse.abstain	null	true if p_top < cutoff, false if p_top >= cutoff
logits	Raw log-scores	Scaled log-scores (logits / T)

Your application code simply checks answer.mouse.abstain:

const intent = data.answers.intent;

if (intent.mouse.abstain) {
  // Below the fitted cutoff for target_error: escalate to human or reasoning model
  await escalateToHumanReview(ticket);
} else {
  // Guaranteed <= target_error on your validation distribution
  await routeAutomatically(intent.choice, ticket);
}

Model version binding

Because probability distributions can shift across model releases, every calp1. profile is cryptographically bound to the exact snapshot ID (mouse-preview-2026-10-13) it was fitted on.

• If you pass a profile fitted on mouse-preview-2026-10-13 to a request that resolves to a different model version, the API refuses the request with 409 profile_model_mismatch rather than silently applying stale cutoffs.
• Pin the exact snapshot ID (model: "mouse-preview-2026-10-13") in production alongside your calibration profile, and re-run POST /v1/calibrations on your validation set when upgrading to a newer snapshot.

Reliability curve and the Fair Coin / Fair Die sanity check

Practitioners evaluating decision models rightly test whether a model assigns uniform probability to symmetric random events (the "Fair Coin / Fair Die" check, where uncalibrated models in the wild have reported 86%–87% on a 50/50 coin flip) and how often reversing option order flips the winning choice.

Sanity check	Theoretical ideal	How mouse-1 handles it
Fair coin (heads vs tails)	50.0% / 50.0%	Evaluated via POST /v1/decisions; pass order_invariant: true to average forward and reversed option orderings in one call.
Fair 6-sided die (1..6)	16.7% per face	Uniform prior across symmetric options; confidence normalized to 0.0 at (1/k).
Option-order flip rate	0.0% under order_invariant: true	Setting order_invariant: true scores multi-option choice questions in both orders and averages the distributions, eliminating positional bias.
10-bin Expected Calibration Error (ECE)	0.0	Reported in before.ece and after.ece on every POST /v1/calibrations call over your own labeled examples.

Reproducibility note: We only publish empirical Fair Coin, Fair Die, and single-pass flip-rate numbers measured against the attached production checkpoint (reproducible via apps/decide/scripts/shadow_migrate.py), never hand-entered estimates.