> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true

# Compare Query Configs

> Build a labeled test set from your own collection, score retrieval, and tune query settings from any MCP client.

## Agent Quick Reference: Compare query configs over MCP

* Loop: map the collection (`captain_list_documents`, `captain_list_chunks`), write one paraphrased question per sampled chunk, run each question through `captain_search_v3` under each candidate config with `limit: 10`, score recall\@1/3/10 and MRR at document level, compare configs as a paired test on the same questions.
* Ground truth is the chunk the question was written from: `{ chunkIds: [chunk_id], documentIds: [document_id] }`. No human grading.
* Question count: 300 to 500 for a decision; use the whole collection when it has fewer sampled chunks than that. Under 100 is a smoke test only.
* Candidate ladder: `{}` baseline, `{ rerank: true }`, `{ rerank: { enabled: true, candidate_limit: 50 } }`, `{ rerank: true, exclude_chunk_types: ["page_header", "page_footer", "footnote"] }`. Then `semantic_ratio` in 0.2 steps.
* A failed query (timeout, 5xx) is excluded and counted, never scored as a miss. Compare configs only over questions that succeeded in every run.
* Tune request parameters first (`semantic_ratio`, `rerank`, `exclude_chunk_types`, `filter`, `boost`, `limit`). Structural changes (`processing_type: "advanced"`, chunk metadata, relations) need a holdout split by document.
* Record per run: collection, environment, document and chunk counts, question set size, config, recall\@1/3/10, MRR, latency p50/p95, failed count.
* Promote a tuned collection with `captain_copy_collection`.

Use the MCP tools to build a test set from your collection, measure retrieval quality, and compare query configurations on the same questions. This guide walks through each step so the evaluation can be repeated as the collection or query settings change.

Start with the [MCP server](/guides/mcp) connected and a collection with documents in it. Everything below uses the MCP tools; the equivalent REST endpoints are linked where they matter.

Captain measures retrieval two ways. This guide covers the client-side loop: build the question set from the collection itself, run it through `captain_search_v3` under each candidate, and compare the candidates as a paired test. The [Evaluations](/guides/evaluations) guide covers the server-side route, where a case set of known expected documents is uploaded once and run as a paced job that keeps every answer. The case-set format, scoring rules, limits and billing live there.

## What you are measuring

A [v3 query](/reference/v3/query) runs two retrieval legs, keyword and semantic, blended by `semantic_ratio`, then optionally reranks the candidate pool. Every setting the evaluation tunes is a request parameter: `semantic_ratio`, `rerank` and its `candidate_limit`, `exclude_chunk_types`, `limit`, `filter`, and `boost`. Changing them never touches the index, so a candidate configuration costs one query per question to test.

Changes that mutate the collection (processing type, chunk metadata, relations) are the second, larger lever. They are covered at the end, with the extra discipline they need.

## The loop

### Map the collection

Call `captain_list_documents` for the collection, then `captain_list_chunks` for each document with `limit: 5`. Keep each chunk's `chunk_id`, `document_id`, and `text`.

This pool of sampled chunks is the ceiling on your question count: one question per chunk. Two sampling rules worth following:

* Take chunks from as many documents as possible before taking more chunks from one document. Questions from the same document are correlated and buy less information.
* Skip chunks whose `chunk_type` is `page_header`, `page_footer`, `footnote`, or `heading`, and chunks shorter than about 200 characters. A question written from a footer is a bad question.

### Write one question per chunk

For each sampled chunk, have a model write one question that the chunk answers, using this instruction:

```text
You write search evaluation questions. Given one chunk of text from a real
document, write ONE question a real user of this document collection would
plausibly ask, that this chunk answers. Do not reuse distinctive words or
phrases from the chunk; paraphrase the underlying need in different language,
the way someone who has not read this exact passage would phrase it. Reply
with ONLY the question, no preamble, no quotes.
```

The paraphrase rule is the whole trick. A question that copies the chunk's wording is a keyword test, not a retrieval test. Spot-check twenty questions against their chunks before trusting the set.

Store each question with its ground truth:

```json
{
  "id": "q-0042",
  "question": "What happens to a borrower's rate when the index resets mid-quarter?",
  "groundTruth": {
    "chunkIds": ["doc_9f3a...:17"],
    "documentIds": ["doc_9f3a..."]
  }
}
```

Because the question was generated from the chunk, relevance is known without human grading.

### Run a configuration

For every question, call `captain_search_v3` with the question text, the candidate configuration, and `limit: 10`. Record the ranked `document_id` and `chunk_id` of each result plus the latency.

Score at document level: the ground-truth document counts as a hit at its best rank in the top ten. Chunk-level matching is stricter and useful once the document-level score is high.

A question whose query fails (timeout, 5xx) is excluded from the score and counted separately. It is never scored as a miss.

### Score

| Metric     | What it answers                                                 | Weight in the composite |
| ---------- | --------------------------------------------------------------- | ----------------------- |
| recall\@10 | Was the document found at all                                   | 0.40                    |
| recall\@3  | Was it found early enough for an agent that reads the top three | 0.25                    |
| MRR        | When found, was it first                                        | 0.20                    |
| nDCG\@10   | Is the ordering right, only meaningful with multi-hop questions | 0.15                    |

With one relevant document per question, recall\@k is hit rate at k, and nDCG carries the same information as MRR, so leave it out until you add questions with more than one relevant chunk. F1 is deliberately absent: with one relevant document, precision\@10 cannot exceed 0.1, so F1 grades a flawless pipeline as failing.

Record latency p50 and p95 alongside the scores. Reranking costs roughly 200 ms; a deeper `candidate_limit` costs more.

### Try the candidate ladder

Run the same question set under each configuration, in the same session, against the same index:

| Candidate         | Config                                                                                  | Fixes                                        |
| ----------------- | --------------------------------------------------------------------------------------- | -------------------------------------------- |
| Baseline          | `{}`                                                                                    | Reference point                              |
| Rerank on         | `{ "rerank": true }`                                                                    | Found but not early: recall\@3, MRR          |
| Deeper pool       | `{ "rerank": { "enabled": true, "candidate_limit": 50 } }`                              | The right chunk was outside the default pool |
| Drop layout noise | `{ "rerank": true, "exclude_chunk_types": ["page_header", "page_footer", "footnote"] }` | Headers and footers outranking body text     |

Then move `semantic_ratio` in 0.2 steps from the default 0.5. Lower it for corpora full of exact strings (part numbers, error codes, tickers, gene symbols); raise it when callers phrase things differently from the documents.

### Pick a winner honestly

Two rules keep the comparison fair:

* **Compare over the intersection.** Recompute every candidate's score over the questions that succeeded in every run. A candidate that drops a hard question to a query failure looks better than one that answers it and scores a real miss.
* **A round with failures cannot win.** Fix the failures or exclude those questions everywhere before declaring a result.

Then ask whether the difference is real. The questions are the same across configs, so this is a paired comparison. Count the discordant questions: `b` where A hit and B missed, `c` where B hit and A missed. With 25 or more discordant pairs, `(|b - c| - 1)^2 / (b + c)` above 3.84 means significant at p = 0.05. Below 25, use an exact binomial test on `b` against `c`, or bootstrap the difference by resampling questions.

Rule of thumb: at 400 questions, two configs need to differ by about 3 points of recall\@10 before you should believe it; at 100 questions, about 6.

## How many questions

Choose a question count based on the number of sampled chunks available. For a proportion at 95% confidence and a 5-point margin, `n0 = 385`. With `N` sampled chunks available, `n = n0 / (1 + (n0 - 1) / N)`, and if that covers 80% or more of the pool, use the whole pool.

| Sampled chunks | Questions                          |
| -------------- | ---------------------------------- |
| 40             | 40 (whole pool, no sampling error) |
| 1,000          | about 278                          |
| Very large     | 385                                |

Under 100 questions cannot separate 97% from 93%. Use it for a smoke test, not a decision. 300 to 500 is the working range for tuning one collection. Go to 1,000 or more when the decision is expensive: a re-parse of a large corpus, or a production config change behind a customer-facing agent.

## Structural changes need a holdout

Request parameters do not memorize questions, so tuning them on the full set is fine. The moment tuning touches the collection itself, it is not:

* Chunk metadata added to make specific questions pass
* `boost` rules keyed on the ground-truth `chunk_ids`
* Relations created so a specific evidence chunk travels with a specific claim

These are per-item fixes and inflate any score measured on the same questions. Split the question set by **document**, not by question, tune on 70%, and report on the other 30%. With fewer than about 150 questions, use k-fold by document instead.

The structural levers, in the order the composite score's diagnosis suggests them:

* **Processing type.** Re-index with `processing_type: "advanced"` when recall\@10 is weak on PDFs, scans, or DOCX with tables. Use `captain_index_*` for the source.
* **Chunk metadata.** `captain_set_chunk_metadata` and `captain_update_chunk_metadata` attach fields that `filter` and `boost` can use ([reference](/reference/v3/documents/chunk/metadata)). No schema needed; sparse tags on a few chunks work.
* **Relations.** `captain_create_chunk_relation` links a claim to its evidence table so `include.related_chunks` returns them together ([reference](/reference/v3/documents/chunk/relations)). This is the lever for multi-hop questions, and where nDCG\@10 starts to matter. See [Advanced Search & Relations](/guides/advanced-querying).

## Keep the index honest

Scores describe the index they were measured on. If the files change and the index does not, last week's numbers describe a collection that no longer exists. For any bucket-backed corpus, use [Sync](/guides/sync/set-up) with `deletion_policy: mirror` so the collection tracks the bucket; the `captain_create_*_sync` and `captain_reconcile_sync` tools manage it. Note that with `deletion_policy: archive`, archived documents are hidden from search by default and their questions will fail; regenerate the question set after a policy change.

Record every run with enough context to compare it later: collection, environment, document and chunk counts, question set size, the config as JSON, recall\@1/3/10, MRR, latency p50 and p95, failed count, timestamp. Save the question set and run records as JSON files in your own evaluation workspace so each comparison can be reproduced.

## Promote the result

Tune in development, then move the collection with `captain_copy_collection` ([reference](/reference/collections/copy)). The copy branches the index rather than re-indexing, so the scores carry over. Apply the winning query configuration in the consumer that calls Captain; nothing is stored on the collection.

## Tool map

| Step             | MCP tools                                                                                                         |
| ---------------- | ----------------------------------------------------------------------------------------------------------------- |
| Map              | `captain_list_collections`, `captain_list_documents`, `captain_list_chunks`, `captain_get_chunk`                  |
| Run              | `captain_search_v3` with `semantic_ratio`, `rerank`, `exclude_chunk_types`, `filter`, `boost`, `include`, `limit` |
| Structural fixes | `captain_set_chunk_metadata`, `captain_update_chunk_metadata`, `captain_create_chunk_relation`, `captain_index_*` |
| Keep current     | `captain_create_s3_sync` and the other providers, `captain_reconcile_sync`                                        |
| Promote          | `captain_copy_collection`, `captain_change_environment`                                                           |