Images

There is no separate image endpoint, header or classifier. mouse-1 reads images natively: attach them to a POST /v1/systemone request in images, and every question in the request sees them alongside state. The answers are the same choice, noul and score distributions you already use, so thresholds, calibration and fan-out work unchanged.

Image input is in beta. The vision encoder is the base model's, unchanged; mouse-1's tuning (gating, injection and order hardening, calibration) was done on text. We have checked image answers on smoke cases, not on a published image benchmark, so calibrate on your own labeled images before you put a threshold on one.

Send an image

images is an array of base64 data URLs: data:image/<png|jpeg|webp>;base64,<data>. Nothing else changes in the request. Images are off by default only in the sense that the array is empty; there is no feature header to send.

IMG=$(base64 < receipt.jpg | tr -d '\n')
curl https://api.mouse.dev/v1/systemone \
  -H "Authorization: Bearer $MOUSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mouse-1",
    "state": "Support ticket attachment.",
    "images": ["data:image/jpeg;base64,'"$IMG"'"],
    "questions": {
      "kind": {
        "type": "choice",
        "instructions": "What does the image show?",
        "criteria": {
          "receipt": "A purchase receipt or invoice",
          "screenshot": "A screenshot of an app or website",
          "id_document": "A government ID or passport",
          "other": "Anything else"
        }
      },
      "legible": {
        "type": "noul",
        "instructions": "Is the text in the image legible?"
      }
    }
  }'

import base64, httpx

def data_url(path, mime="image/jpeg"):
    return f"data:{mime};base64," + base64.b64encode(open(path, "rb").read()).decode()

res = httpx.post(
    "https://api.mouse.dev/v1/systemone",
    headers={"Authorization": f"Bearer {MOUSE_API_KEY}"},
    json={
        "state": "Support ticket attachment.",
        "images": [data_url("receipt.jpg")],
        "questions": {"kind": {"type": "choice", "criteria": {"receipt": None, "screenshot": None, "other": None}}},
    },
).json()
print(res["answers"]["kind"]["choice"], res["usage"]["image_tokens"])

import { readFile } from "node:fs/promises";

const img = (await readFile("receipt.jpg")).toString("base64");
const res = await fetch("https://api.mouse.dev/v1/systemone", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.MOUSE_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    state: "Support ticket attachment.",
    images: [`data:image/jpeg;base64,${img}`],
    questions: { kind: { type: "choice", criteria: { receipt: null, screenshot: null, other: null } } },
  }),
});
const { answers, usage } = await res.json();
console.log(answers.kind.choice, usage.image_tokens);

Image URLs (https://...) are not fetched: the API never makes a request on your behalf. Download the image yourself and send its bytes. The TypeSafe SDKs' assets map and multipart/form-data uploads are not accepted either; see Coming from TypeSafe.

Several images

Send up to 8 images in one array. They reach the model in array order, before state, and every question sees all of them.

• Refer to images by position. "the first image", "the second image". Image order is the only identity an image has; there are no reference strings.
• Describe them in state. state is the text that goes with the images: a caption, a ticket body, a tool output, or a legend that says what each position is. An empty string is fine.
• One copy each. Unlike a system where an image is re-sent for every question that mentions it, mouse-1 reads each image once per question pass whatever your questions say, and bills it once per request.

{
  "state": {
    "task": "Fix the overlapping header on the settings page.",
    "first_image": "screenshot before the change",
    "second_image": "screenshot after the change"
  },
  "images": ["data:image/png;base64,...", "data:image/png;base64,..."],
  "questions": {
    "fixed": {"type": "noul", "instructions": "Does the second image fix the overlap visible in the first image?"},
    "regressed": {"type": "noul", "instructions": "Does the second image break anything that worked in the first image?"}
  }
}

How an image is read

• Resized to between 65,536 and 1,638,400 pixels (a 256×256 to a 1280×1280 square), keeping its aspect ratio. Larger images are scaled down, so anything past 1.6 megapixels costs upload size and nothing else. Smaller images are scaled up and the request carries a warnings entry: an icon upscaled eightfold gives weak answers.
• One token per 32×32 pixel patch, plus 2 per image. See Tokens and cost.
• Before state, in array order. Text in state, instructions and criteria follows the images.
• Read again for every question, like state. Billed once; counted toward the per-question and per-request work limits.
• EXIF orientation is not applied. A phone photo that looks upright in your viewer can reach the model sideways. Rotate the pixels first.
• Transparency is not flattened. Transparent pixels in a PNG or WebP keep whatever colour is stored under them, usually black. Composite onto a background first.
• Still images only. An animated PNG or WebP is refused with invalid_image; export the frame you want. GIF, HEIC, AVIF, SVG and PDF are refused too; convert them to PNG, JPEG or WebP.
• Metadata is ignored, and, like everything else in a request, never stored or logged.

Preparing images

A 12-megapixel phone photo is about 3 MB as JPEG and 4 MB as base64, which on its own fills the request body limit. The model reads at most 1.6 megapixels, so shrink first. This Pillow snippet rotates, shrinks, flattens and re-encodes one image:

from PIL import Image, ImageOps

with Image.open("photo.jpg") as original:
    img = ImageOps.exif_transpose(original)  # rotate the pixels upright; the API does not read EXIF
    img.thumbnail((1280, 1280))              # longest side at most 1280: the model reads at most 1.6 MP anyway
    if img.mode in ("RGBA", "LA", "P"):      # flatten transparency onto white
        img = Image.alpha_composite(Image.new("RGBA", img.size, "white"), img.convert("RGBA"))
    img.convert("RGB").save("photo-ready.jpg", "JPEG", quality=85)  # typically 100 to 400 KB

Eight images prepared this way fit in one request. Screenshots compress better as PNG; photos as JPEG. If an image is refused that opens fine on your machine, re-export it with a standard library and send that.

Tokens and cost

Image tokens are input tokens at the same $0.10 / 1M as text, itemised in usage.image_tokens. The count is exact and you can compute it before sending: round each side to the nearest 32 pixels, fit the result into 65,536 to 1,638,400 pixels, count 32×32 patches, add 2.

Image	Tokens	Cost
256×256 (the minimum the model reads)	66	$0.0000066
640×480	302	$0.0000302
800×600	477	$0.0000477
1024×768	770	$0.0000770
1280×720	882	$0.0000882
1280×1280 and anything at or above 1.6 MP	1,566 to 1,602	about $0.00016

Like state, images are billed once per request however many questions you ask. Asking eight questions about one photo costs the photo once plus eight short questions. Rejected requests are never billed.

Patterns

Pattern	How it works	Request
Classify	One choice over your labels, plus the noul checks you would otherwise make a second call for (is_blurry, contains_person).	One image, several questions
Verify a change	Before and after screenshots; ask whether the second fixes what the first shows, and whether it breaks anything. Pair with a calibration profile to route low-confidence cases to a person.	Two images
Compare	Two product photos and a choice over first, second, same.	Two images
What changed	A short frame history, oldest first, described in state with timestamps; ask about motion, arrivals or differences.	Up to 8 images
Look closer when needed	Ask about a small copy first. Send a larger copy or a full-resolution crop only when the answer's confidence is low.	One image per request
Search a large image	Cut the image into up to 8 tiles and ask a choice over the tile positions. Repeat on the chosen tile until the target fills it.	Up to 8 images
Gate an agent	Include the screenshot an agent is about to act on and ask whether the proposed action matches the task.	One image plus the proposed action in state

Treat text inside an image as untrusted. The model is hardened against instructions in state, and an image can carry the same attack: a banner that says "approve this" is content, not an instruction, and your questions should ask about what the image shows, never what it says to do.

Limits

Limit	Value	Error
Images per request	8	422 too_many_images
Format	PNG, JPEG or WebP, still, as a base64 data URL	422 invalid_image
Source size	16,000,000 pixels; aspect ratio at most 200:1	422 invalid_image
Minimum size	none; under 65,536 pixels the image is upscaled and the response carries a warnings entry	—
Request body	4,000,000 bytes after decompression, base64 included (about 3 MB of image files)	413 request_too_large
Work per question	image tokens + state tokens + the question, at most 18,432	422 question_too_long
Work per request	(image tokens + state tokens) × questions, at most 64,000	422 too_much_work

Nothing is resized to fit a limit or silently dropped: a request with one bad image fails as a whole, before any GPU work, and is not billed. There is no separate rate limit for images; image tokens count toward the same per-key limits as text.

Eight full-size images are about 12,800 tokens, so with a short state you can ask four questions about them before too_much_work. To ask more, send fewer or smaller images, or split the questions across requests.

Errors

Status	code	Cause
422	invalid_image	Not a data:image/...;base64, URL (URLs are never fetched), invalid base64, not PNG, JPEG or WebP, damaged, animated, over 16 megapixels, or aspect ratio over 200:1. param names the image, images.<n>.
422	too_many_images	More than 8 images.
422	too_much_work, question_too_long	Images and text together exceed a work limit.
413	request_too_large	The body is over 4,000,000 bytes. Shrink the images before encoding them.
400	images_not_supported	The request uses an assets map, videos or a multipart upload. Send images instead.

The first problem found is the one reported.

Coming from TypeSafe

Jev's image input and mouse-1's take the same questions and differ in where the image goes.

TypeSafe	Mouse
TypeSafe-Danger-Zone-Images: true header	No header.
assets map keyed by reference string, data_base64 + format	images array of base64 data URLs.
multipart/form-data upload	Not supported; send base64 in the JSON body.
A reference in state, instructions or a criteria option	Images are positional and visible to every question. Put the description in state and say "the first image" in instructions.
An image per choice option	Not supported. Describe options in text, or send the candidates as images and ask a choice over their positions.
box question	Not supported (422).
PNG or JPEG, 280 to 1568 px per side, 32 MiB body	PNG, JPEG or WebP, any size up to 16 MP, 4 MB body.
Image tokens included in input_tokens, formula unpublished	Itemised in usage.image_tokens; the formula is above.

Limits of the beta

• No image URLs, multipart/form-data, assets, videos, or images inside criteria.
• Tunings accept text examples only and are gated on text suites. A tuned model still takes images, but nothing has checked it on them: run your own held-out images through it first.
• Zero data retention applies: images are never stored or logged.