> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true

# Evaluation API

* **Evaluate retrieval as a job**: upload up to 10,000 queries with the documents each should retrieve, then [Queue Evaluation](/reference/evals/create) runs them under one to eight named [v3 query](/reference/v3/query) configurations side by side. The eval runs as a paced job and keeps results, questions, configurations and every answer.
* **Scorecards per configuration**: [Get Evaluation Results](/reference/evals/get) returns recall at 1, 3 and 10, MRR, nDCG at 10 and latency percentiles per config, plus a result for every case under every config. Scoring is document level, like the MCP `captain_eval` tool.
* **Billed as queries**: each question under one configuration is one query at the plan's query price. Only queries that ran count; Captain-side failures and unresolvable expected documents do not. The `201` response previews the maximum before anything runs.
* **Every answer readable**: each result carries a `query_id`, and `GET /v3/evals/{eval_id}/answers/{case_id}/{config_name}` returns the stored `request` and `response` for one case under one configuration.
* Case-set format, gold resolution, limits and persistence: [Evaluations](/guides/evaluations).

Same day: [Evaluation Tools over MCP](/changelog/2026-09-16-mcp-eval-tools)