> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true # Evaluations > Run a set of test questions against a collection under several query settings at once, and get a scorecard that shows which settings retrieve the right documents. ## Agent Quick Reference - Evaluations * **Mint an upload URL**: `POST /v3/collections/{collection_name}/evals/uploads` with `filename` (ends in `.ndjson`), `byte_size`, optional `content_type`. Returns `upload_id`, a single-use `upload_url`, and the exact `headers` the PUT must send. * **Upload**: `PUT {upload_url}` with the raw NDJSON bytes and exactly those headers. No `Authorization`. One PUT of exactly `byte_size` bytes; then the URL is consumed. * **Create**: `POST /v3/collections/{collection_name}/evals` with `Idempotency-Key` (required), `upload_id`, `configs[]` (1 to 8 named v3 query bodies minus `query`), optional `environment` label. `201` on accept, `200` on replay of the same key and body. * **Poll**: `GET /v3/evals/{eval_id}` until `status` is `completed`, `completed_with_errors`, or `failed`. `scorecards` and `billing` are empty until then. The eval runs asynchronously as a paced job, so a large case set does not hold the request open. * **NDJSON line**: `{"id": "...", "query": "...", "expected_files": ["form.pdf" | ""], "filters": {...}}`. `id` defaults to `case_{line}`; duplicate ids, blank lines, invalid JSON, or more than 10,000 lines reject the whole file with `400`. * **Expected documents**: each `expected_files` entry is matched by document id, then by exact filename; 0 or 2+ matches marks the case `error` / `GOLD_UNRESOLVED` (not queried, not billed). A document id can be the `document.id` from a query result or the id from the documents list; both resolve to the same document, and `retrieved_document_ids` in the results always use the form a query result reports. * **Scoring**: document level. Walk chunk results in rank order, keep each document the first time it appears, stop at `limit`; `rank` is the position in that list. `recall_at_1/3/10`, `mrr`, `ndcg_at_10` over scored units only. Failed queries count in `failed`, never as a miss. * **Billing**: one unit = one question under one config = one query. Whether a query costs credits depends on the plan (captain.dev/pricing). Scored units and `QUERY_TIMEOUT` units count as queries; `QUERY_ERROR`, `CONFIG_FAILED`, `GOLD_MISSING` and unstarted units do not. * **Caps**: 20 pending evals per organization (`429`), 4 running at once, 24 h max age, 10,000 lines, 8 configs, 80,000 units, 50 MB. * **Answers**: each per-config result carries a `query_id`. `GET /v3/evals/{eval_id}/answers/{case_id}/{config_name}` returns the stored `request` and `response` for one case under one config. * **Persisted**: evals and their per-question results are kept until the collection is deleted. Items keep `query`, `expected_files`, `filters` and `expected_document_ids`. Do not put secrets in questions. An evaluation measures how well a collection answers a set of test questions. Supply the questions and the documents a correct answer would come from, and Captain runs each as a real query and scores where the right document landed. Use it to pick query settings. The same questions run under up to eight named configurations at once, so the scorecards compare directly. ## How it works An eval is a paced background job. Each question is an ordinary query, issued a few at a time, so a large set neither holds a request open nor crowds out live traffic. ### Upload the case set Write one question per line as NDJSON, mint an upload URL, and PUT the bytes. See [Write the case set](#write-the-case-set) and [Upload the case set](#upload-the-case-set). ### Queue the evaluation Name the upload and list one to eight query configurations to compare. See [Queue the evaluation](#queue-the-evaluation). ### Poll until it finishes Fetch the eval until its status is terminal. See [Poll for the result](#poll-for-the-result). ### Read the scorecards One scorecard per configuration, plus a per-question breakdown. See [Read the scorecards](#read-the-scorecards). | Step | Endpoint | | ------------------------------------------------------- | ------------------------------------------------------ | | [Create an upload URL](/reference/evals/uploads/create) | `POST /v3/collections/{collection_name}/evals/uploads` | | [Upload the case set](#upload-the-case-set) | `PUT {upload_url}` | | [Queue the evaluation](/reference/evals/create) | `POST /v3/collections/{collection_name}/evals` | | [Poll for the result](/reference/evals/get) | `GET /v3/evals/{eval_id}` | ## Write the case set A case set is a file with one JSON object per line (the NDJSON format), at most 10,000 lines and 50 MB. Each line is one test question: the query text, and the files that a good search should return for it. ```json {"id": "auth-window", "query": "What language authorizes the merchant to charge the card on file?", "expected_files": ["payment-authorization-form.pdf"], "filters": {"policy_area": "payments"}} {"id": "recurring-terms", "query": "Where are cancellation terms for recurring payments written?", "expected_files": ["f4a91c7e8b2d4a0c9e6f13b5a8d02744"]} {"query": "Which spreadsheet rows list approval required for recurring payments?", "expected_files": ["billing-policy-controls.xlsx"]} ``` | Field | Required | Meaning | | ---------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | no | A name for the question, used to find it in the results. When omitted, Captain uses the line number: the third line above becomes `case_3`. Two lines with the same id reject the whole file. | | `query` | yes | The search query, written exactly as an application would send it to [Query v3](/reference/v3/query). | | `expected_files` | yes | One to fifty documents that should be retrieved for this question, each given as a filename or a document id (as shown in query results or in the documents list). | | `filters` | no | A metadata filter to apply to this question, in the same form [Query v3](/reference/v3/query) accepts. | ### Writing good questions * Phrase questions the way a user would, not by copying wording out of the document. Reusing the document's own phrasing tests keyword matching rather than retrieval. * Three to five hundred questions compare configurations reliably. A few dozen is a smoke test, and will not separate two close configurations. ### Where errors surface Two kinds of problem can occur in the file, and they fail at different moments. | Problem | Scope | Effect | | ---------------------------------------------------------------- | ------------------- | -------------------------------------------------------------------- | | Blank line, invalid JSON, duplicate `id`, more than 10,000 lines | The file as a whole | The create is rejected with `400` before anything runs | | Missing `query`, empty `expected_files`, an invalid filter | One line | That question is marked as an error; the rest of the file still runs | ### Which documents count as correct Before the eval starts, Captain looks up every entry in every question's `expected_files` and turns it into a document in the collection: 1. If the entry matches a document id exactly, that document is used. 2. Otherwise it is matched against filenames. The comparison is exact and case-sensitive on the filename alone, so an entry of `payments/form.pdf` matches a document stored as `form.pdf`. An entry matching zero documents, or more than one (two files called `policy.pdf`), cannot be scored. That question errors with `GOLD_UNRESOLVED` and is neither run nor billed; the rest still run. Fix it by naming the document by id, or renaming the duplicates, then create a new eval. Resolved documents are recorded as `expected_document_ids`. If one is deleted before its question runs, that question is marked `GOLD_MISSING`: not run, not billed, not a miss. ## Upload the case set Uploading takes two requests. The first tells Captain the file's exact size and gets back a URL plus the exact headers the upload must send. #### curl ```bash curl -X POST \ "https://api.filesearch.dev/v3/collections/customer_forms/evals/uploads" \ -H "Authorization: Bearer $CAPTAIN_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "filename": "payments-eval.ndjson", "byte_size": 1842 }' ``` #### Python ```python import os import requests BASE_URL = "https://api.filesearch.dev" API_KEY = "your_api_key" COLLECTION_NAME = "customer_forms" PATH = "payments-eval.ndjson" headers = { "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json", } response = requests.post( f"{BASE_URL}/v3/collections/{COLLECTION_NAME}/evals/uploads", headers=headers, json={"filename": PATH, "byte_size": os.path.getsize(PATH)}, timeout=60.0, ) upload = response.json() ``` #### TypeScript ```typescript import { statSync } from "fs"; const BASE_URL = "https://api.filesearch.dev"; const API_KEY = "your_api_key"; const COLLECTION_NAME = "customer_forms"; const PATH = "payments-eval.ndjson"; const response = await fetch( `${BASE_URL}/v3/collections/${COLLECTION_NAME}/evals/uploads`, { method: "POST", headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ filename: PATH, byte_size: statSync(PATH).size }), }, ); const upload = await response.json(); ``` **`Response (200)`** ```json title="Response (200)" { "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa", "upload_url": "https://uploads.captainusercontent.com/evals/org_example/evu_019b4c2e7a3f70d2a91c4e8b6f1023aa.ndjson?sig=...", "expires_at": "2026-09-12T17:45:00Z", "headers": { "Content-Type": "application/x-ndjson", "Content-Length": "1842" }, "byte_size": 1842, "filename": "payments-eval.ndjson", "collection_name": "customer_forms" } ``` The second request sends the file's bytes to that URL with exactly those headers. The API key is not sent here; the URL itself is the credential. #### curl ```bash curl -X PUT "$UPLOAD_URL" \ -H "Content-Type: application/x-ndjson" \ -H "Content-Length: 1842" \ --data-binary @payments-eval.ndjson ``` #### Python ```python with open(PATH, "rb") as f: requests.put(upload["upload_url"], headers=upload["headers"], data=f, timeout=300.0) ``` #### TypeScript ```typescript import { readFileSync } from "fs"; await fetch(upload.upload_url, { method: "PUT", headers: upload.headers, body: readFileSync(PATH), }); ``` The URL works once, and it expires about fifteen minutes after being minted. | Condition | Result | | ------------------------------------------------- | ----------------------------------------------------- | | A second upload to the same URL | `403` | | A size that does not match the minted `byte_size` | `400` | | A GET against the upload URL | The URL never serves the file back | | The URL expired before being used | The eval create is rejected, even if the file arrived | ## Queue the evaluation The create request names the upload and lists the configurations to compare. Each configuration is the body of a [Query v3](/reference/v3/query) request without the query text, under a name of your choosing. #### curl ```bash curl -X POST \ "https://api.filesearch.dev/v3/collections/customer_forms/evals" \ -H "Authorization: Bearer $CAPTAIN_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: eval-payments-2026-09-12-a" \ -d '{ "upload_id": "evu_019b4c2e7a3f70d2a91c4e8b6f1023aa", "configs": [ {"name": "baseline"}, {"name": "rerank", "rerank": true}, {"name": "rerank-deep", "rerank": {"enabled": true, "candidate_limit": 40}, "semantic_ratio": 0.35} ] }' ``` #### Python ```python response = requests.post( f"{BASE_URL}/v3/collections/{COLLECTION_NAME}/evals", headers={ "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json", "Idempotency-Key": "eval-payments-2026-09-12-a", }, json={ "upload_id": upload["upload_id"], "configs": [ {"name": "baseline"}, {"name": "rerank", "rerank": True}, {"name": "rerank-deep", "rerank": {"enabled": True, "candidate_limit": 40}, "semantic_ratio": 0.35}, ], }, timeout=60.0, ) eval_id = response.json()["eval_id"] ``` #### TypeScript ```typescript const response = await fetch( `${BASE_URL}/v3/collections/${COLLECTION_NAME}/evals`, { method: "POST", headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json", "Idempotency-Key": "eval-payments-2026-09-12-a", }, body: JSON.stringify({ upload_id: upload.upload_id, configs: [ { name: "baseline" }, { name: "rerank", rerank: true }, { name: "rerank-deep", rerank: { enabled: true, candidate_limit: 40 }, semantic_ratio: 0.35 }, ], }), }, ); const { eval_id } = await response.json(); ``` ### Choosing configurations Start by comparing a baseline with no settings, reranking on, and a deeper reranking pool. Then vary the keyword-to-semantic balance, exclude layout text like headers and footers, or add a metadata filter. | Field | Rule | | ---------------------- | ------------------------------------------------------------------------------------------------------------- | | `name` | Lowercase letters, digits, hyphens and underscores. Names key the scorecards. | | Omitted settings | Take the same default a live query would. | | An invalid setting | A configuration a live query would reject is rejected here too, with a `400` that names the configuration. | | Per-question `filters` | When a question carries its own filter and a configuration also has one, the question runs with both applied. | Up to eight configurations run per eval, and every question runs under each, so the scorecards compare directly. ### The create response The response comes back pending, echoing each configuration with its defaults filled in and a preview of the work: **`Response (201)`** ```json title="Response (201)" { "eval_id": "eval_019b4c2f11d04c8e9a7b3d5c2e1840ff", "status": "pending", "preview": { "cases": 3, "configs": 3, "units": 9, "cases_error": 1, "billable_units": 6 } } ``` A unit is one question under one configuration, so `units` is questions times configurations. `billable_units` excludes questions whose expected documents did not resolve, and is the most the eval can cost. `Idempotency-Key` is required and makes retrying safe: the same key and body within 24 hours returns the original eval as a `200`, and the same key with a different body is refused with a `409`. Each upload can be used once. ## Poll for the result Poll `GET /v3/evals/{eval_id}` until the status is terminal. It starts `pending`, waiting for one of your four running slots, moves to `running`, and ends in one of three states. | Terminal status | Meaning | | ----------------------- | ----------------------------------------------------------------------------------------------- | | `completed` | Every question ran and was scored under every configuration. | | `completed_with_errors` | Some questions or configurations errored, and scorecards exist for the configurations that ran. | | `failed` | Nothing to report. The error code and message on the eval say why. | While the eval runs, the `progress` object reports the percentage finished, counts of scored queries, failed queries (query execution errors, plus questions never run because their configuration was stopped), and unscorable cases, plus an estimate of the remaining time. Scorecards and billing are filled in only once the eval is terminal. #### curl ```bash curl -X GET \ "https://api.filesearch.dev/v3/evals/eval_019b4c2f11d04c8e9a7b3d5c2e1840ff" \ -H "Authorization: Bearer $CAPTAIN_API_KEY" ``` #### Python ```python import time while True: response = requests.get( f"{BASE_URL}/v3/evals/{eval_id}", headers={"Authorization": f"Bearer {API_KEY}"}, timeout=30.0, ) data = response.json() if data["status"] in ("completed", "completed_with_errors", "failed"): break time.sleep(10) for name, card in data["scorecards"].items(): print(name, card["recall_at_1"], card["mrr"], card["ndcg_at_10"]) ``` #### TypeScript ```typescript let data; while (true) { const response = await fetch(`${BASE_URL}/v3/evals/${eval_id}`, { headers: { Authorization: `Bearer ${API_KEY}` }, }); data = await response.json(); if (["completed", "completed_with_errors", "failed"].includes(data.status)) break; await new Promise((resolve) => setTimeout(resolve, 10_000)); } for (const [name, card] of Object.entries(data.scorecards)) { console.log(name, card.recall_at_1, card.mrr, card.ndcg_at_10); } ``` ## Read the scorecards The eval returns one scorecard per configuration, keyed by the configuration's name. Read retrieval scores alongside execution-error and unscorable-case counts. Scores describe only the queries that were successfully scored. A scored query can be either a retrieval hit or a retrieval miss. **`scorecards.baseline`** ```json title="scorecards.baseline" { "status": "scored", "question_set_size": 3, "scored": 2, "failed": 0, "error": 1, "recall_at_1": 0.5, "recall_at_3": 1.0, "recall_at_10": 1.0, "mrr": 0.75, "ndcg_at_10": 0.88, "latency_p50_ms": 412, "latency_p95_ms": 690, "document_count": 1284, "chunk_count": 19402 } ``` ### What the metrics mean | Metric | What it measures | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `recall_at_1`, `recall_at_3`, `recall_at_10` | The share of questions whose expected document appeared in the first one, three, or ten results. Recall at 10 answers "was the right document on the page at all"; recall at 1 answers "was it the top result". A configuration that lifts recall at 1 without hurting recall at 10 is ranking better, not just retrieving better. | | `mrr` | Mean reciprocal rank averages one divided by the rank of the expected document: position 1 scores 1, position 2 scores 0.5, position 4 scores 0.25, and a miss scores 0. It rewards putting the right document near the top, with diminishing credit the further down it appears. | | `ndcg_at_10` | Normalised discounted cumulative gain is the metric to read when questions list several expected documents. It gives credit for each expected document found in the top ten, more for the ones ranked higher, and scales the total so that a perfect ordering scores 1. With a single expected document per question it tracks MRR. | | `latency_p50_ms`, `latency_p95_ms` | The median and 95th-percentile time each query took, so a configuration that scores better can be weighed against what it costs in speed. | ### How scoring works Scoring is per document, not per chunk. Captain walks the results in order, notes each document the first time one of its chunks appears, and stops at the configuration's `limit`. The question's rank is where the first expected document lands in that list. A document filling several slots counts once, at its best position, so returning many chunks per document costs nothing. Scores describe the collection as it was when each question ran. An eval does not freeze the index. ### The count fields The three count fields always add up to `question_set_size`. | Field | Meaning | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `scored` | Questions whose queries completed and were scored, including retrieval misses. | | `failed` | Questions whose queries could not complete because of a timeout or a query execution error on Captain's side, plus questions never run because their configuration was stopped. These are excluded from retrieval metrics. | | `error` | Unscorable cases, such as questions with invalid inputs or expected documents that could not be resolved. These are excluded from retrieval metrics. | Query execution errors, timeouts, and unscorable cases are excluded from retrieval metrics; they are not counted as retrieval misses. A completed, scored query that does not retrieve any expected document within the configured result limit counts as a retrieval miss. A configuration is stopped in two cases: the query core rejects its settings outright, or it hits 20 consecutive query execution errors while other configurations keep scoring. Its scorecard reports `CONFIG_FAILED` with no metrics, its remaining questions close as `CONFIG_FAILED` without running and count under `failed` (unbilled), and the other configurations continue. ### Per-question results The `items` list holds one entry per question, in file order, paged with the `items_limit` and `items_cursor` query parameters (100 per page by default, 500 at most). Each item repeats the question and shows, under `results`, how it did under each configuration: **`items[0].results.baseline`** ```json title="items[0].results.baseline" { "status": "scored", "hit": true, "rank": 2, "retrieved_document_ids": ["9f8e7d6c...", "a1b2c3d4...", "11223344..."], "latency_ms": 405, "error_code": null, "error_message": null } ``` For a scored result, `hit: true` means at least one expected document was returned within the configured result limit; `hit: false` means a retrieval miss. A hit does not establish that every expected document, the necessary passage, or a correct generated answer was returned. `rank` says where the first expected document landed, and `retrieved_document_ids` what arrived instead on a miss. Comparing one question across two configurations is the quickest way to see what a setting changed. ### Error codes | Code | Level | Meaning | | ----------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GOLD_UNRESOLVED` | Case | An `expected_files` entry matched zero documents or more than one. The question is not run and not billed. | | `GOLD_MISSING` | Case, per configuration | A resolved expected document was deleted before the question ran. Not run, not billed, not a miss. | | `QUERY_TIMEOUT` | Query | The query ran but exceeded Captain's 30-second limit. Counted as a query. | | `QUERY_ERROR` | Query | The query failed on Captain's side. Not counted as a query. | | `CONFIG_FAILED` | Configuration | The query core rejected the configuration's settings, or 20 consecutive queries failed while other configurations kept scoring. The configuration was stopped and its remaining questions closed without running. The other configurations continue. | | `JOB_EXPIRED` | Eval | The eval passed its 24-hour maximum age. | A question that errored before running has an empty `results` object and a case-level error code. ### Reading back a single answer Each result carries the `query_id` that produced it. [Get Evaluation Answer](/reference/evals/answers) returns the stored `request` and `response` for one question under one configuration, so any result traces back to the exact query sent and the exact results returned, long after the eval finished. ## Limits and billing ### Limits | Limit | Value | | ------------------------------ | ------------------------------------------------------------------------------ | | Pending evals per organization | 20; further creates return `429` with a `Retry-After` header | | Running evals per organization | 4; further evals wait in the pending state | | Maximum age | 24 hours from creation, after which the eval fails with the code `JOB_EXPIRED` | | Questions per set | 10,000 | | Configurations per eval | 8 | | Units per eval | 80,000 | | Upload size | 50 MB | | Expected files per question | 50 | These caps only ever apply to evals. They exist to keep an eval's queries paced, so a large case set does not crowd out ordinary queries on the same collection. ### Billing Each unit, one question run under one configuration, is billed as one query, exactly as if the application had sent it to the query endpoint. Whether a query costs credits depends on the plan: see [captain.dev/pricing](https://captain.dev/pricing), or the terms of the agreement for plans that bill differently. Only units that actually ran are counted as queries. The `billing` object on a finished eval records the number counted, the number not counted, and any configurations that failed. | Outcome | Counted as a query | | ---------------------------------------------------------------- | ------------------ | | Scored, whether a hit or a miss | yes | | Timed out (the query ran but exceeded Captain's 30-second limit) | yes | | Failed on Captain's side | no | | Configuration stopped as failing | no | | Expected documents unresolved, missing, or the question invalid | no | | Never started (the eval failed, expired, or was cancelled first) | no | Queries are counted once, when the eval reaches a terminal state. A retried or replayed create never counts them twice. ## What Captain keeps An eval and its per-question results are kept until the collection is deleted, so runs months apart stay comparable. Re-run the same question set after re-indexing or changing settings, and the new scorecard shows whether retrieval improved. | Item | Retention | | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | The eval and its scorecards | Until the collection is deleted | | Per-question results, including each question's text, its `expected_files`, its filter, and the resolved document ids | Until the collection is deleted | | The uploaded NDJSON file | Seven days | | The text of retrieved chunks | Not stored; only their document ids | Question text is stored, so keep secrets out of it. Deleting a collection deletes its evals, and fetching one afterwards returns a `404`. Pending and running evals stop, and only the units that already completed are counted as queries. Keep a copy of any scorecards you need after the collection is gone. Copying a collection does not copy its evals. ## From the MCP server The hosted [MCP server](/guides/mcp) calls this API directly, so you can run an eval without writing the requests. Three tools cover the three steps, and each needs a connection authenticated with an API key. | Tool | What it does | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `captain_create_eval_upload` | Takes the case set as arguments (each case a query, its `expected_files`, and an optional filter), serialises it to NDJSON, mints the upload for the collection, and performs the PUT, returning the `upload_id` to run against. | | `captain_run_eval` | Queues the uploaded case set under one to eight named v3 configurations and returns the `eval_id` with a preview of the unit count. When no idempotency key is passed, one is derived from the collection, the upload, and the configurations, so a retried call replays the same eval instead of billing a second one. | | `captain_get_eval_results` | Polls one eval: status and a terminal flag, progress, the scorecards per configuration, billing, and a compact result per case (hit, rank, latency, `query_id`, error code), paged by cursor. With `include_answers`, the stored request and top results for up to 20 scored cases on the page. | There is a second route, run client-side rather than as a Captain job. The `captain_eval` tool takes a question set built in the conversation, runs it through `captain_search_v3` under each configuration, and scores them as a paired test, so it tells you whether a gain is real rather than merely larger. | Route | Best for | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | This API | A large case set of known expected documents, run as a paced job, with every answer kept and readable afterwards. | | `captain_eval` over MCP | Building a question set from the collection itself and comparing configurations interactively, with a significance test on the difference. | [Compare Query Configs](/guides/mcp/evaluate-and-tune) covers building the question set, the candidate ladder, how many questions a decision needs, and the holdout discipline structural changes require.