Evaluations

An evaluation measures how well a collection answers a set of test questions. Supply the questions and the documents a correct answer would come from, and Captain runs each as a real query and scores where the right document landed.

Use it to pick query settings. The same questions run under up to eight named configurations at once, so the scorecards compare directly.

How it works

An eval is a paced background job. Each question is an ordinary query, issued a few at a time, so a large set neither holds a request open nor crowds out live traffic.

Write the case set

A case set is a file with one JSON object per line (the NDJSON format), at most 10,000 lines and 50 MB. Each line is one test question: the query text, and the files that a good search should return for it.

{"id": "auth-window", "query": "What language authorizes the merchant to charge the card on file?", "expected_files": ["payment-authorization-form.pdf"], "filters": {"policy_area": "payments"}}
{"id": "recurring-terms", "query": "Where are cancellation terms for recurring payments written?", "expected_files": ["f4a91c7e8b2d4a0c9e6f13b5a8d02744"]}
{"query": "Which spreadsheet rows list approval required for recurring payments?", "expected_files": ["billing-policy-controls.xlsx"]}

Writing good questions

• Phrase questions the way a user would, not by copying wording out of the document. Reusing the document’s own phrasing tests keyword matching rather than retrieval.
• Three to five hundred questions compare configurations reliably. A few dozen is a smoke test, and will not separate two close configurations.

Where errors surface

Two kinds of problem can occur in the file, and they fail at different moments.

Which documents count as correct

Before the eval starts, Captain looks up every entry in every question’s expected_files and turns it into a document in the collection:

1. If the entry matches a document id exactly, that document is used.
2. Otherwise it is matched against filenames. The comparison is exact and case-sensitive on the filename alone, so an entry of payments/form.pdf matches a document stored as form.pdf.

An entry matching zero documents, or more than one (two files called policy.pdf), cannot be scored. That question errors with GOLD_UNRESOLVED and is neither run nor billed; the rest still run. Fix it by naming the document by id, or renaming the duplicates, then create a new eval.

Resolved documents are recorded as expected_document_ids. If one is deleted before its question runs, that question is marked GOLD_MISSING: not run, not billed, not a miss.

Upload the case set

Uploading takes two requests. The first tells Captain the file’s exact size and gets back a URL plus the exact headers the upload must send.

curl -X POST \
  "https://api.captain.dev/v3/collections/customer_forms/evals/uploads" \
  -H "Authorization: Bearer $CAPTAIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "filename": "payments-eval.ndjson",
  "byte_size": 1842
}'

{
  "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
  "upload_url": "https://uploads.captainusercontent.com/evals/org_example/evu_019b4c2e7a3f70d2a91c4e8b6f1023aa.ndjson?sig=...",
  "expires_at": "2026-09-12T17:45:00Z",
  "headers": {
    "Content-Type": "application/x-ndjson",
    "Content-Length": "1842"
  },
  "byte_size": 1842,
  "filename": "payments-eval.ndjson",
  "collection_name": "customer_forms"
}

The second request sends the file’s bytes to that URL with exactly those headers. The API key is not sent here; the URL itself is the credential.

curl -X PUT "$UPLOAD_URL" \
  -H "Content-Type: application/x-ndjson" \
  -H "Content-Length: 1842" \
  --data-binary @payments-eval.ndjson

The URL works once, and it expires about fifteen minutes after being minted.

Queue the evaluation

The create request names the upload and lists the configurations to compare. Each configuration is the body of a Query v3 request without the query text, under a name of your choosing.

curl -X POST \
  "https://api.captain.dev/v3/collections/customer_forms/evals" \
  -H "Authorization: Bearer $CAPTAIN_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: eval-payments-2026-09-12-a" \
  -d '{
  "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa",
  "configs": [
    {"name": "baseline"},
    {"name": "rerank", "rerank": true},
    {"name": "rerank-deep", "rerank": {"enabled": true, "candidate_limit": 40}, "semantic_ratio": 0.35}
  ]
}'

Choosing configurations

Start by comparing a baseline with no settings, reranking on, and a deeper reranking pool. Then vary the keyword-to-semantic balance, exclude layout text like headers and footers, or add a metadata filter.

Up to eight configurations run per eval, and every question runs under each, so the scorecards compare directly.

The create response

The response comes back pending, echoing each configuration with its defaults filled in and a preview of the work:

{
  "eval_id": "eval_019b4c2f11d04c8e9a7b3d5c2e1840ff",
  "status": "pending",
  "preview": {
    "cases": 3,
    "configs": 3,
    "units": 9,
    "cases_error": 1,
    "billable_units": 6
  }
}

A unit is one question under one configuration, so units is questions times configurations. billable_units excludes questions whose expected documents did not resolve, and is the most the eval can cost.

Idempotency-Key is required and makes retrying safe: the same key and body within 24 hours returns the original eval as a 200, and the same key with a different body is refused with a 409. Each upload can be used once.

Poll for the result

Poll GET /v3/evals/{eval_id} until the status is terminal. It starts pending, waiting for one of your four running slots, moves to running, and ends in one of three states.

While the eval runs, the progress object reports the percentage finished, counts of scored queries, failed queries (query execution errors, plus questions never run because their configuration was stopped), and unscorable cases, plus an estimate of the remaining time. Scorecards and billing are filled in only once the eval is terminal.

curl -X GET \
  "https://api.captain.dev/v3/evals/eval_019b4c2f11d04c8e9a7b3d5c2e1840ff" \
  -H "Authorization: Bearer $CAPTAIN_API_KEY"

Read the scorecards

The eval returns one scorecard per configuration, keyed by the configuration’s name.

Read retrieval scores alongside execution-error and unscorable-case counts. Scores describe only the queries that were successfully scored. A scored query can be either a retrieval hit or a retrieval miss.

{
  "status": "scored",
  "question_set_size": 3,
  "scored": 2,
  "failed": 0,
  "error": 1,
  "recall_at_1": 0.5,
  "recall_at_3": 1.0,
  "recall_at_10": 1.0,
  "mrr": 0.75,
  "ndcg_at_10": 0.88,
  "latency_p50_ms": 412,
  "latency_p95_ms": 690,
  "document_count": 1284,
  "chunk_count": 19402
}

What the metrics mean

How scoring works

Scoring is per document, not per chunk. Captain walks the results in order, notes each document the first time one of its chunks appears, and stops at the configuration’s limit. The question’s rank is where the first expected document lands in that list. A document filling several slots counts once, at its best position, so returning many chunks per document costs nothing.

Scores describe the collection as it was when each question ran. An eval does not freeze the index.

The count fields

The three count fields always add up to question_set_size.

Query execution errors, timeouts, and unscorable cases are excluded from retrieval metrics; they are not counted as retrieval misses. A completed, scored query that does not retrieve any expected document within the configured result limit counts as a retrieval miss. A configuration is stopped in two cases: the query core rejects its settings outright, or it hits 20 consecutive query execution errors while other configurations keep scoring. Its scorecard reports CONFIG_FAILED with no metrics, its remaining questions close as CONFIG_FAILED without running and count under failed (unbilled), and the other configurations continue.

Per-question results

The items list holds one entry per question, in file order, paged with the items_limit and items_cursor query parameters (100 per page by default, 500 at most). Each item repeats the question and shows, under results, how it did under each configuration:

{
  "status": "scored",
  "hit": true,
  "rank": 2,
  "retrieved_document_ids": ["9f8e7d6c...", "a1b2c3d4...", "11223344..."],
  "latency_ms": 405,
  "error_code": null,
  "error_message": null
}

For a scored result, hit: true means at least one expected document was returned within the configured result limit; hit: false means a retrieval miss. A hit does not establish that every expected document, the necessary passage, or a correct generated answer was returned. rank says where the first expected document landed, and retrieved_document_ids what arrived instead on a miss. Comparing one question across two configurations is the quickest way to see what a setting changed.

Error codes

A question that errored before running has an empty results object and a case-level error code.

Reading back a single answer

Each result carries the query_id that produced it. Get Evaluation Answer returns the stored request and response for one question under one configuration, so any result traces back to the exact query sent and the exact results returned, long after the eval finished.

Limits and billing

Limits

These caps only ever apply to evals. They exist to keep an eval’s queries paced, so a large case set does not crowd out ordinary queries on the same collection.

Billing

Each unit, one question run under one configuration, is billed as one query, exactly as if the application had sent it to the query endpoint. Whether a query costs credits depends on the plan: see captain.dev/pricing, or the terms of the agreement for plans that bill differently.

Only units that actually ran are counted as queries. The billing object on a finished eval records the number counted, the number not counted, and any configurations that failed.

Queries are counted once, when the eval reaches a terminal state. A retried or replayed create never counts them twice.

What Captain keeps

An eval and its per-question results are kept until the collection is deleted, so runs months apart stay comparable. Re-run the same question set after re-indexing or changing settings, and the new scorecard shows whether retrieval improved.

Question text is stored, so keep secrets out of it.

Deleting a collection deletes its evals, and fetching one afterwards returns a 404. Pending and running evals stop, and only the units that already completed are counted as queries. Keep a copy of any scorecards you need after the collection is gone. Copying a collection does not copy its evals.

From the MCP server

The hosted MCP server calls this API directly, so you can run an eval without writing the requests. Three tools cover the three steps, and each needs a connection authenticated with an API key.

There is a second route, run client-side rather than as a Captain job. The captain_eval tool takes a question set built in the conversation, runs it through captain_search_v3 under each configuration, and scores them as a paired test, so it tells you whether a gain is real rather than merely larger.

Compare Query Configs covers building the question set, the candidate ladder, how many questions a decision needs, and the holdout discipline structural changes require.