File Types

Text and Code

Captain indexes plain text, Markdown, JSON, YAML, logs and source code through one lane that reads every file as text. This page covers which files go through it, how they are chunked, the JSON parsing script, the raw text route, and the limits.

This is useful for

  • Runbooks, READMEs and wikis kept as Markdown, where the section under a heading is the answer
  • JSON exports from other systems, shaped into records with a short parsing script before indexing
  • Logs, configs and source files that should be findable by an exact string
  • Short text sent straight from your application through the raw text route

What goes through this lane

Every extension under Text and code on Supported File Types: .txt, .md, .json, .yaml, .yml, and 151 source, markup and config extensions. All are read as plain text. There is no syntax-aware parsing: a Python file and a text file are chunked the same way, and an HTML file is indexed with its tags in place.

Not indexed: compiled or binary variants (.pyc, .pyo, .class, .o, .so, .dll, .wasm, .jar, .svgz), .plist, and files that usually hold secrets (.env, .pem, .key, .crt). A file with a text extension whose first 8 KB contains a null byte is treated as binary and skipped with zero chunks; the job does not fail. Extensionless files from a connector are indexed when the storage reports a text content type.

How text is chunked

1

Sections

A file of up to about 40,000 tokens is one section. Longer files are split at Markdown headings and fenced code blocks when the file is Markdown, and at line boundaries otherwise. A single line longer than the budget (a minified bundle, a one-line JSON blob) is cut at whitespace first, then at , ; } ], and only then at a character.

2

Summaries and passages

Each section gets a generated summary (two or three sentences) and a set of tags, and is tiled into shorter passages that are embedded. A result contains the passage and the summary and tags of its section. Passages do not overlap.

3

No page ranges

page_start and page_end are null for text and code. Page ranges exist for documents. Encoding is UTF-8, with a fallback to Latin-1; a UTF-8 byte order mark is kept as a character.

4

JSON parsing scripts

A .json file indexed without a parsing_script is read as raw text, including the braces. With a parsing_script, a JavaScript function receives the parsed document and returns the text to index, so chunks fall on records rather than on syntax. The script runs in a sandbox with no network or filesystem access, must return a non-empty string, and has 60 seconds. The document must be a single JSON value of at most 250 MB. JSON Lines files are indexed as text. See JSON Parsers.

Raw text

POST /v2/collections/{name}/index/text indexes a string directly, up to 10 MB, with an optional filename, mask_pii and custom_metadata. The file name is forced to end in .txt. The document’s identity is upload://<filename>, so sending the same name again updates that document instead of creating a second one.

Limits and billing

Files in this lane are limited to 250 MB. When the storage can report a size up front, an oversized file is rejected before download; otherwise it is rejected after download. Text is billed at 0.5 credits per 3,000 characters, rounded up per file. See File Size Limits.

Examples

Goal: return the section that explains a procedure, from a docs/ tree indexed from a bucket.

request
{
  "query": "rotate the signing key",
  "limit": 3
}
result (one item)
{
  "text": "## Rotating the signing key\n\n1. Generate a new key pair with `make keys`...",
  "document": {
    "id": "…",
    "name": "docs/ops/auth.md"
  },
  "metadata": {
    "summary": "Steps to rotate the JWT signing key without downtime, including the dual-key window.",
    "file_tags": [
      "auth",
      "keys",
      "rotation",
      "ops"
    ]
  }
}

Markdown files are split at headings, so the result is the section under the matching heading. metadata.summary describes the section.

Goal: index a JSON export so each result is a product entry rather than a fragment of syntax.

parsing_script
export default function (doc) {
  return doc.products
    .map(p => `${p.sku} ${p.name}\n${p.description}\nCategory: ${p.category}`)
    .join("\n\n");
}
request
{
  "query": "stainless 40 mm ball valve",
  "limit": 5,
  "filter": {
    "catalogue": {
      "$eq": "2026"
    }
  }
}

The script defines what one record looks like as text, so the chunker cuts on records. The SKU stays in the text and matches an exact query.

Goal: find an error across a month of logs and narrow it by day. A 200 MB log is under the limit but is one document; split logs by day or by service before indexing and set the day as custom_metadata.

request
{
  "query": "connection pool exhausted",
  "limit": 10,
  "filter": {
    "day": {
      "$gte": "2026-09-15"
    }
  },
  "max_chunks_per_document": 3
}

Logs are cut at line boundaries, so a passage is a run of whole lines. Per-day files give the day filter something to match.

© 2026 Captain