Evaluation Tools over MCP

Evaluate retrieval as a job: upload up to 10,000 queries with the documents each should retrieve, then Queue Evaluation runs them under one to eight named v3 query configurations side by side. The eval runs as a paced job and keeps results, questions, configurations and every answer.
Scorecards per configuration: Get Evaluation Results returns recall at 1, 3 and 10, MRR, nDCG at 10 and latency percentiles per config, plus a result for every case under every config. Scoring is document level, like the MCP captain_eval tool.
Billed as queries: each question under one configuration is one query at the plan’s query price. Only queries that ran count; Captain-side failures and unresolvable expected documents do not. The 201 response previews the maximum before anything runs.
Every answer readable: each result carries a query_id, and GET /v3/evals/{eval_id}/answers/{case_id}/{config_name} returns the stored request and response for one case under one configuration.
Case-set format, gold resolution, limits and persistence: Evaluations.

Run an eval over MCP: the hosted MCP server adds captain_create_eval_upload, captain_run_eval and captain_get_eval_results, the three steps of the Evaluation API: upload up to 10,000 questions, queue them under one to eight named v3 query configurations, poll for scorecards. They need an API-key connection, since the Evaluation API accepts API keys only in v1.
Retries replay instead of rerunning: captain_run_eval derives an idempotency key from the collection, upload and configurations, so a repeated call returns the original eval instead of billing a second one.
Results sized for a conversation: captain_get_eval_results returns status, progress, per-configuration scorecards, billing and a compact row per case (hit, rank, latency, query_id, error code), paged by cursor; include_answers adds the stored request and top results for up to 20 cases. captain_eval stays for quick client-side comparisons. The hosted server is now at 70 tools.