Advanced Search & Relations

Advanced search in v3 File Search is built around three concepts:

• Document metadata: stable file-level fields used for query filters and source context.
• Chunk custom metadata: custom annotations attached to a specific chunk.
• Chunk relations: typed links between chunks for graph-aware retrieval and review workflows.

Start with the File Search API guide for the basic v3 query flow. This guide covers the knobs that make retrieval more precise.

Document Metadata Filters

Use filter on v3 Query to restrict retrieval to documents that match file-level metadata.

1
{
2
  "query": "approved safety claims",
3
  "limit": 10,
4
  "filter": {
5
    "review_state": "approved",
6
    "year": { "$gte": 2024 },
7
    "$or": [
8
      { "department": "medical" },
9
      { "department": "regulatory" }
10
    ]
11
  }
12
}

Document metadata is best for stable fields such as access tier, source system, review status, file owner, product line, year, or jurisdiction.

Supported operators include bare equality, $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $and, and $or.

1
{ "userId": { "$eq": "user_123" } }

1
{ "metadata": { "$eq": { "userId": "user_123" } } }

Scope a Search to Specific Documents

To restrict a query to a known set of documents, filter on file_id with $in:

1
{
2
  "query": "renewal obligations",
3
  "limit": 10,
4
  "filter": {
5
    "file_id": { "$in": ["79ba3036381657b1c81bab0ceee6895c", "4b949882e07394f952967d1d9db00382"] }
6
  }
7
}

file_id values are the document_ids returned by List Documents and the indexing job manifest.

Chunk Types and Layout Exclusion

Every chunk of a parsed document carries a chunk_type: its layout role, classified from the parser’s block structure at indexing time. The closed set of values is body, table, heading, page_header, page_footer, footnote, and figure.

1
{
2
  "chunk_id": "chk_abc123_017",
3
  "text": "31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.",
4
  "chunk_type": "page_footer",
5
  "score": 0.81
6
}

chunk_type is null when a chunk has no label: content indexed before chunk types shipped, plain-text and media content, or regions the parser returned without block structure. The field appears once, at the top level of the result. It is never duplicated inside metadata.

Use exclude_chunk_types on v3 Query to keep page furniture out of retrieval. A compliance workload that does not want footers, footnotes, or section titles competing with body text sends:

1
{
2
  "query": "adverse event rates in the treatment arm",
3
  "limit": 10,
4
  "exclude_chunk_types": ["page_header", "page_footer", "footnote", "heading"]
5
}

Semantics to rely on:

• Exclusion happens at retrieval time, before reranking, and combines with filter when both are sent. You still receive up to limit results.
• Chunks with a null chunk_type always pass. Exclusion never removes unlabeled content, so older collections behave exactly as before.
• Unknown values return a 400 that lists the allowed set.
• Excluding body is allowed but adds an entry to warnings, since body is a document’s primary text.
• The response always echoes exclude_chunk_types, as an empty array when none were requested.

chunk_type is a reserved key: writing it through custom_metadata is rejected on every write surface.

Include Controls

Use include on v3 Query to request only the extra objects your application needs.

1
{
2
  "query": "contract renewal obligations",
3
  "limit": 10,
4
  "include": {
5
    "document": true,
6
    "metadata": true,
7
    "regions": true,
8
    "relations": true,
9
    "related_chunks": true,
10
    "archived": false
11
  }
12
}

Keep regions, relations, and related_chunks off unless the caller needs them. The base query response is easier for agents to consume when it returns only source text and core metadata.

archived belongs to this object because it answers the same question the rest of it answers: what comes back. Set it true to surface chunks that a sync archive deletion policy has retired. They stay out of results by default, so a normal query never mixes retired content into live answers.

Chunk Custom Metadata

Use chunk metadata when your application needs annotations below the document level, such as claim type, reviewer status, extraction confidence, entity IDs, or a workflow state for one chunk.

Metadata set through these endpoints is written to the search index, so the fields are usable in query filter expressions the same way as metadata set at upload time.

A small set of reserved internal field names (for example file_id, content, chunk_index) cannot be set as custom metadata; the endpoints reject them with a 400. Use your own field names such as userId, projectId, or claim_type.

The chunk metadata endpoints are:

• Get Chunk Metadata
• Replace Chunk Metadata
• Update Chunk Metadata
• Delete Chunk Metadata

1
import requests
2
3
BASE_URL = "https://api.captain.dev"
4
API_KEY = "your_api_key"
5
COLLECTION = "medical_claims"
6
CHUNK_ID = "chk_abc123_004"
7
8
response = requests.patch(
9
    f"{BASE_URL}/v3/collections/{COLLECTION}/chunks/{CHUNK_ID}/metadata",
10
    headers={
11
        "Authorization": f"Bearer {API_KEY}",
12
        "Content-Type": "application/json",
13
    },
14
    json={
15
        "metadata": {
16
            "claim_type": "safety",
17
            "review_state": "approved"
18
        }
19
    },
20
)

Query and document responses expose these annotations as custom_metadata on chunks. The metadata endpoint request and response body uses metadata because the URL is already scoped to one chunk.

Keyword and Semantic Balance

Captain searches two ways at once and merges the results. Keyword search (sparse, BM25) matches the words in the query. Semantic search (dense vector) matches its meaning. Keyword search finds ERR_4021 when you ask for ERR_4021; semantic search finds “the invoice was rejected” when you ask about failed payments.

semantic_ratio sets how much each side counts toward the final ranking:

1
{
2
  "query": "ERR_4021 retry backoff",
3
  "limit": 10,
4
  "semantic_ratio": 0.2
5
}

Lower it when your corpus is full of exact strings that must match literally: part numbers, error codes, SKUs, statute and case citations, gene symbols, ticker symbols. Raise it when callers ask questions in their own words and the corpus answers them in different words.

Between the endpoints both searches run, whatever you set. The ratio changes how loudly each side argues, not whether it gets to speak: at 0.2 the semantic side still retrieves and its rows still score, so a chunk only it found can still appear, just far lower.

The Endpoints Are Shortcuts

0.0 and 1.0 differ in kind, not degree: each turns the other retrieval method off rather than quieting it.

semantic_ratio: 0 is the fast path. Skipping vector search also skips embedding your query, which is much of a query’s cost, so it returns noticeably faster. Use it when your users type identifiers rather than questions: SKUs, error codes, case numbers, ticket IDs.

semantic_ratio: 1 skips the keyword search. Not a speed optimization, since the embedding still runs. Choose it for vector-only results, not for a faster query.

Two cases keep vector search running at 0, so they do not get the speedup:

• Queries using boost. Boost rules retrieve by vector, and a boost is your explicit instruction, independent of this setting.
• Collections holding images, video, or audio. Media is searched by vector only, so skipping it would return nothing from those files.

Metadata boosts are independent of semantic_ratio. A boost is an explicit instruction to favor certain chunks, so lowering semantic_ratio for an exact-term corpus does not weaken the boosts you set on it.

The response echoes the value applied, so you can confirm what a query actually ran with:

1
{
2
  "results": [ ... ],
3
  "semantic_ratio": 0.2
4
}

Tuning It

Start at the default and change it only when you can name the failure. If exact identifiers are ranking below prose that merely discusses them, lower semantic_ratio. If callers phrase things their own way and literal-match results crowd out the ones that actually answer the question, raise it.

Move in steps of 0.2 and compare against a fixed set of real queries with known right answers. semantic_ratio is a corpus-level property far more than a per-query one, so most applications find one value and keep it rather than varying it per request.

semantic_ratio shapes retrieval. When reranking is on the reranker still decides the final order of what retrieval surfaced, so semantic_ratio changes which chunks reach the reranker rather than how it sorts them.

Metadata Boosts

filter decides which chunks are eligible. A boost decides which eligible chunks win.

Search ranks chunks by matching words and meaning. Sometimes the correct chunk does not resemble the question at all:

Question: how long do customers have to return an item?

The chunk that answers it: “Merchandise may be presented for refund within 30 days of the invoice date.”

There are no shared words, and an embedding model has no way to know your organization treats “presented for refund” as the return policy, so this chunk ranks low or does not surface. A support agent would call it the answer.

A boost is for the cases where your application already knows which chunk is right and search does not:

• A person marked it. A reviewer runs a set of test questions, sees that the correct passage did not surface, and tags it: answers: ["returns-window"]. Boost that tag whenever that question comes up again.
• Your application derived it from the question. The question is about refunds, so your code adds a boost on policy_area: "returns" - a fact you can act on that the index could not match by wording alone.
• It was already cited in this conversation. A follow-up question should stay anchored to the passages the previous answer used, so boost those chunk_ids.

Boosts are opt-in per request. Send no boost field and retrieval behaves exactly as it does today.

1
{
2
  "query": "how long do customers have to return an item",
3
  "limit": 10,
4
  "boost": [
5
    { "field": "answers", "in": ["returns-window"], "weight": 3.0 }
6
  ]
7
}

boost takes a list of boost rules. A rule is one entry in that list: some metadata to look for, and how much to favour the chunks carrying it. Each rule does two things: it retrieves matching chunks through their own retrieval pass, and it multiplies their retrieval score by weight. The retrieval pass is what makes reviewer feedback work. A chunk that answers the question but shares no wording with it is not in the normal search results at all, so a re-score alone could never surface it.

Rule Shapes

A rule is exactly one of these three shapes. Send up to 10 of them in one request.

chunk_ids accepts the chunk_id from a previous response, or the stable document_id:chunk_index form. Prefer document_id:chunk_index for anything you store: it keeps resolving after the document is re-indexed.

Choosing a Weight

weight is a multiplier on the chunk’s retrieval score, from 0.2 to 5.0. It is not a percentage and not a rank.

A rule’s retrieval pass is what makes a chunk appear; the weight decides where it lands among the results. This is the part most people get backwards. Because a rule fetches the chunks it names, a selective rule surfaces its chunk even at a low weight, and raising the weight to 5.0 does not make a chunk “more likely” to be found. Reach for a higher weight when a boosted chunk is appearing but not high enough, not when it is missing entirely.

Start at 1.5 to 2.0 and raise it only if the ordering is wrong. A rule that matches one specific chunk needs far less weight than a rule that matches hundreds, since a broad rule has to lift its matches past each other as well as past the query’s own results.

When several rules match the same chunk their weights multiply, and the result is capped at 5.0. If a chunk still will not appear, a bigger weight is the wrong fix: check that the rule matched at all (matched in the response), then use reserve below.

Guaranteeing a Slot with reserve

Reranking is a second pass that re-scores retrieved chunks by how well their text answers the question. It runs after boosting, and it can undo a boost.

That is the whole tension. A reviewer-marked chunk does not resemble the question, which is exactly why it needed a boost. The reranker reads it, sees text that does not look like an answer to this question, scores it low, and drops it again. Your boost retrieved the chunk and the reranker discarded it.

reserve settles that conflict in your favour. reserve: 1 means “whatever the reranker decides, this rule’s best-matching chunk gets a place on the page.”

Same query and same boost, with limit: 1:

Soft is the default deliberately: a boost set too broadly should not be able to force an unhelpful chunk onto the first page. Leaving reserve at 0 keeps the reranker’s judgement as the backstop. Setting it is you overriding that judgement on purpose, for a fixed number of slots.

Set reserve on a rule when a chunk must appear:

1
{
2
  "query": "how long do customers have to return an item",
3
  "limit": 10,
4
  "rerank": true,
5
  "boost": [
6
    { "field": "answers", "in": ["returns-window"], "weight": 3.0, "reserve": 1 }
7
  ]
8
}

reserve: 1 guarantees that rule’s best-matching chunk a place on the page. The sum of reserve across all rules cannot exceed limit. Results placed this way are marked "reserved": true.

If you send a boost with reranking on and no reserve, the response includes a warning explaining that the reranker owns the final order. Setting rerank: false also lets boosts decide the order outright.

Which Fields You Can Boost

Boosts apply to your own metadata - the same custom_metadata keys that filter accepts, whether they were set at upload time or through the chunk metadata endpoints. If you can filter on a field today, you can boost on it today. Nothing needs to be re-indexed.

Captain’s own internal fields (file_id, job_id, chunk_index, organization_id, and similar) are rejected with a 400, as are field names beginning with $.

Your metadata does not have to be uniform. Chunks in one collection can carry completely different key sets, and a boost on a key that a chunk does not have simply never matches it - there is no schema to declare and nothing to keep in sync. Values can be strings, numbers, booleans, or lists; a list matches when any element matches. This means a sparse tag applied to five chunks in a corpus of fifty thousand works exactly as well as a field every chunk carries.

Seeing What a Boost Did

Every boosted response explains itself, so a reviewer can tell the difference between “the boost did not match” and “the boost matched but the reranker disagreed”.

Each rule is echoed back with counts:

1
{
2
  "boost": [
3
    {
4
      "field": "answers",
5
      "in": ["returns-window"],
6
      "weight": 3.0,
7
      "reserve": 1,
8
      "matched": 4,
9
      "in_pool": 4,
10
      "on_page": 1,
11
      "applied": true
12
    }
13
  ]
14
}

• matched - retrieved chunks the rule matched
• in_pool - how many reached the reranking stage
• on_page - how many are in the results you received
• applied - false when boosts were skipped for this request; a warning explains why

Each boosted result carries its own record, and reports metadata in match_sources:

1
{
2
  "chunk_id": "ffe1545103a8178a64da66f7f5280b03:0",
3
  "score": 0.2792,
4
  "match_sources": ["content_embedding", "metadata"],
5
  "boost": {
6
    "multiplier": 3.0,
7
    "clauses": [0],
8
    "pre_boost_rank": 2,
9
    "pre_boost_score": 0.0455,
10
    "reserved": false
11
  }
12
}

pre_boost_rank and pre_boost_score are where the chunk stood before the boost, which is what you want when tuning a weight.

Warnings cover the cases where a boost did less than you asked: a rule that matched nothing, a field this collection has no filterable schema for, and reranking without reserve.

Limits

Documents, Chunks, and Regions

Use Get Document, List Chunks, and Get Chunk when you need source inspection without running a new search.

Set include_regions=true when the UI or downstream workflow needs extracted layout regions. Regions are useful for PDF and document review flows, but should not be requested by default in simple agent search flows.

Chunk Relations

Relations link one chunk to another with a typed edge. Use them when your application needs source graph behavior, such as cited-by links, contradiction links, section-to-table links, or claim-to-evidence links.

The relation endpoints are:

• List Chunk Relations
• Create Chunk Relation
• Delete Chunk Relation

1
response = requests.post(
2
    f"{BASE_URL}/v3/collections/{COLLECTION}/chunks/{CHUNK_ID}/relations",
3
    headers={
4
        "Authorization": f"Bearer {API_KEY}",
5
        "Content-Type": "application/json",
6
    },
7
    json={
8
        "target_chunk_id": "chk_abc123_009",
9
        "relation_type": "supports",
10
        "metadata": {
11
            "reviewer": "clinical-review"
12
        }
13
    },
14
)

Relation-Aware Queries

Use include.relations, include.related_chunks, and relation_direction on v3 Query when a search result should carry graph context.

1
{
2
  "query": "claims supported by clinical evidence",
3
  "limit": 10,
4
  "include": {
5
    "relations": true,
6
    "related_chunks": true
7
  },
8
  "relation_direction": "outgoing"
9
}

Use relation-aware queries when the graph should affect what the caller can inspect after retrieval. Use plain queries when the caller only needs the best matching source chunks.