Evaluate & Tune Through MCP

The Captain CLI runs an evaluation loop against your own files and tunes retrieval toward a target. This guide walks through the same loop by hand from an MCP client, so you can see exactly what is being measured, run it against a collection the CLI does not manage, or adapt it to your own question set.

You need the MCP server connected and a collection with documents in it. Everything below uses the MCP tools; the equivalent REST endpoints are linked where they matter.

What you are measuring

A v3 query runs two retrieval legs, keyword and semantic, blended by semantic_ratio, then optionally reranks the candidate pool. Every setting the evaluation tunes is a request parameter: semantic_ratio, rerank and its candidate_limit, exclude_chunk_types, limit, filter, and boost. Changing them never touches the index, so a candidate configuration costs one query per question to test.

Changes that mutate the collection (processing type, chunk metadata, relations) are the second, larger lever. They are covered at the end, with the extra discipline they need.

The loop

Map the collection

Call captain_list_documents for the collection, then captain_list_chunks for each document with limit: 5. Keep each chunk’s chunk_id, document_id, and text.

This pool of sampled chunks is the ceiling on your question count: one question per chunk. Two sampling rules worth following:

Take chunks from as many documents as possible before taking more chunks from one document. Questions from the same document are correlated and buy less information.
Skip chunks whose chunk_type is page_header, page_footer, footnote, or heading, and chunks shorter than about 200 characters. A question written from a footer is a bad question.

Write one question per chunk

For each sampled chunk, have a model write one question that the chunk answers, using this instruction:

You write search evaluation questions. Given one chunk of text from a real
document, write ONE question a real user of this document collection would
plausibly ask, that this chunk answers. Do not reuse distinctive words or
phrases from the chunk; paraphrase the underlying need in different language,
the way someone who has not read this exact passage would phrase it. Reply
with ONLY the question, no preamble, no quotes.

The paraphrase rule is the whole trick. A question that copies the chunk’s wording is a keyword test, not a retrieval test. Spot-check twenty questions against their chunks before trusting the set.

Store each question with its ground truth:

1 {
2   "id": "q-0042",
3   "question": "What happens to a borrower's rate when the index resets mid-quarter?",
4   "groundTruth": {
5     "chunkIds": ["doc_9f3a...:17"],
6     "documentIds": ["doc_9f3a..."]
7   }
8 }

Because the question was generated from the chunk, relevance is known without human grading.

Run a configuration

For every question, call captain_search_v3 with the question text, the candidate configuration, and limit: 10. Record the ranked document_id and chunk_id of each result plus the latency.

Score at document level: the ground-truth document counts as a hit at its best rank in the top ten. Chunk-level matching is stricter and useful once the document-level score is high.

A question whose query fails (timeout, 5xx) is excluded from the score and counted separately. It is never scored as a miss.

Score

Metric	What it answers	Weight in the composite
recall@10	Was the document found at all	0.40
recall@3	Was it found early enough for an agent that reads the top three	0.25
MRR	When found, was it first	0.20
nDCG@10	Is the ordering right, only meaningful with multi-hop questions	0.15

With one relevant document per question, recall@k is hit rate at k, and nDCG carries the same information as MRR, so leave it out until you add questions with more than one relevant chunk. F1 is deliberately absent: with one relevant document, precision@10 cannot exceed 0.1, so F1 grades a flawless pipeline as failing.

Record latency p50 and p95 alongside the scores. Reranking costs roughly 200 ms; a deeper candidate_limit costs more.

Try the candidate ladder

Run the same question set under each configuration, in the same session, against the same index:

Candidate	Config	Fixes
Baseline	`{}`	Reference point
Rerank on	`{ "rerank": true }`	Found but not early: recall@3, MRR
Deeper pool	`{ "rerank": { "enabled": true, "candidate_limit": 50 } }`	The right chunk was outside the default pool
Drop layout noise	`{ "rerank": true, "exclude_chunk_types": ["page_header", "page_footer", "footnote"] }`	Headers and footers outranking body text

Then move semantic_ratio in 0.2 steps from the default 0.5. Lower it for corpora full of exact strings (part numbers, error codes, tickers, gene symbols); raise it when callers phrase things differently from the documents.

Pick a winner honestly

Two rules keep the comparison fair:

Compare over the intersection. Recompute every candidate’s score over the questions that succeeded in every run. A candidate that drops a hard question to a query failure looks better than one that answers it and scores a real miss.
A round with failures cannot win. Fix the failures or exclude those questions everywhere before declaring a result.

Then ask whether the difference is real. The questions are the same across configs, so this is a paired comparison. Count the discordant questions: b where A hit and B missed, c where B hit and A missed. With 25 or more discordant pairs, (|b - c| - 1)^2 / (b + c) above 3.84 means significant at p = 0.05. Below 25, use an exact binomial test on b against c, or bootstrap the difference by resampling questions.

Rule of thumb: at 400 questions, two configs need to differ by about 3 points of recall@10 before you should believe it; at 100 questions, about 6.

How many questions

The CLI derives its default from the collection size. For a proportion at 95% confidence and a 5-point margin, n0 = 385. With N sampled chunks available, n = n0 / (1 + (n0 - 1) / N), and if that covers 80% or more of the pool, use the whole pool.

Sampled chunks	Questions
40	40 (whole pool, no sampling error)
1,000	about 278
Very large	385

Under 100 questions cannot separate 97% from 93%. Use it for a smoke test, not a decision. 300 to 500 is the working range for tuning one collection. Go to 1,000 or more when the decision is expensive: a re-parse of a large corpus, or a production config change behind a customer-facing agent.

Structural changes need a holdout

Request parameters do not memorize questions, so tuning them on the full set is fine. The moment tuning touches the collection itself, it is not:

Chunk metadata added to make specific questions pass
boost rules keyed on the ground-truth chunk_ids
Relations created so a specific evidence chunk travels with a specific claim

These are per-item fixes and inflate any score measured on the same questions. Split the question set by document, not by question, tune on 70%, and report on the other 30%. With fewer than about 150 questions, use k-fold by document instead.

The structural levers, in the order the composite score’s diagnosis suggests them:

Processing type. Re-index with processing_type: "advanced" when recall@10 is weak on PDFs, scans, or DOCX with tables. Use captain_index_* for the source.
Chunk metadata. captain_set_chunk_metadata and captain_update_chunk_metadata attach fields that filter and boost can use (reference). No schema needed; sparse tags on a few chunks work.
Relations. captain_create_chunk_relation links a claim to its evidence table so include.related_chunks returns them together (reference). This is the lever for multi-hop questions, and where nDCG@10 starts to matter. See Advanced Search & Relations.

Keep the index honest

Scores describe the index they were measured on. If the files change and the index does not, last week’s numbers describe a collection that no longer exists. For any bucket-backed corpus, use Sync with deletion_policy: mirror so the collection tracks the bucket; the captain_create_*_sync and captain_reconcile_sync tools manage it. Note that with deletion_policy: archive, archived documents are hidden from search by default and their questions will fail; regenerate the question set after a policy change.

Record every run with enough context to compare it later: collection, environment, document and chunk counts, question set size, the config as JSON, recall@1/3/10, MRR, latency p50 and p95, failed count, timestamp. That is what the CLI writes to .captain/evals/runs/.

Promote the result

Tune in development, then move the collection with captain_copy_collection (reference). The copy branches the index rather than re-indexing, so the scores carry over. Apply the winning query configuration in the consumer that calls Captain; nothing is stored on the collection.

Tool map

Step	MCP tools
Map	`captain_list_collections`, `captain_list_documents`, `captain_list_chunks`, `captain_get_chunk`
Run	`captain_search_v3` with `semantic_ratio`, `rerank`, `exclude_chunk_types`, `filter`, `boost`, `include`, `limit`
Structural fixes	`captain_set_chunk_metadata`, `captain_update_chunk_metadata`, `captain_create_chunk_relation`, `captain_index_*`
Keep current	`captain_create_s3_sync` and the other providers, `captain_reconcile_sync`
Promote	`captain_copy_collection`, `captain_change_environment`