> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true

# Index S3 Manifest

POST https://api.filesearch.dev/v2/collections/{collection_name}/index/s3/manifest
Content-Type: application/json

Index an explicit list of objects from an S3 bucket, given as a JSON Lines manifest in the same bucket. One bulk job; nothing is listed or scanned.

One object per line, UTF-8, uncompressed. `uri` is required, the rest are optional:

```json
{"uri": "s3://my-documents-bucket/inbox/2026-q4-msa.pdf", "source_identity": "contract-8841", "custom_metadata": {"department": "legal", "counterparty": "Northwind"}}
{"uri": "s3://my-documents-bucket/inbox/2026-q4-sow.pdf", "source_identity": "contract-8842", "size": 482113, "etag": "9b2cf535f27731c974343645a3985328"}
{"uri": "s3://my-documents-bucket/archive/2025-renewal.pdf", "source_identity": "contract-7310"}
{"uri": "s3://my-documents-bucket/inbox/handbook.pdf"}
```

### Physical location vs. logical identity

`uri` is **where the file currently lives**: this bucket, this key. It must be inside `bucket_name`, and it is the only field Captain reads bytes from.

`source_identity` is **what the document is**, in your own terms. Captain keeps one document per identity, so the identity is what decides whether a later index is an update or a new document:

* Give `s3://my-documents-bucket/inbox/2026-q4-msa.pdf` the identity `contract-8841`. Move the file to `s3://my-documents-bucket/archive/2026-q4-msa.pdf` and index it again with the same identity, and Captain updates that one document. Its chunks are replaced and the old ones are retired.
* Omit `source_identity` and the identity is the `uri`. The same file at a new key is a different identity, so you get a second document and both versions answer queries.

A repeat of an identity already in the collection is skipped while `skip_existing` is true (the default), and replaced when you send `overwrite_existing: true` or `skip_existing: false`. While it is skipping, Captain compares content. A repeat whose content is unchanged is skipped. A repeat whose content differs is an identity conflict: it is neither skipped nor indexed, and the job reports it under `identity_conflicts` so that you can send it again with `overwrite_existing: true`. When either version has no checksum, the repeat is skipped. Use your own record id, not a path, whenever content can move or be delivered twice.

### The other line fields

* `custom_metadata`: merged over the job-level `custom_metadata`, key by key, for this file only. A line's value wins on a shared key.
* `checksum` and `version_id`: recorded on the document. A `checksum` that is the object's ETag, written bare or as `etag:...`, is compared with the object before anything is read, and a line that does not match is reported as unreachable instead of being indexed. Write any other algorithm with its prefix, for example `sha256:...`; it is recorded but not verified. `version_id` pins the object version that is read.
* `size` and `etag`: when both are present, as in an S3 Inventory export, Captain does not read the object's metadata before indexing. That is what lets a large manifest start quickly.

### Limits and error handling

A manifest holds up to **1,000,000 lines** and **500 MB**, with each line at most **10 KB**. It must be plain UTF-8 `.jsonl` or `.ndjson`; a compressed body is refused. Per line, `custom_metadata` takes at most 50 keys, keys up to 64 characters and string values up to 1,024 characters. The job-level `custom_metadata` is capped at 64 KB serialized. It is attached to every file in the job, so metadata size times file count must stay under 2 GB: a few hundred bytes fits the full million lines, and the full 64 KB fits about 32,000 files. Larger jobs should carry less job-level metadata or be split.

A wrong `manifest_uri` is a 400 before the job is created, as is a manifest larger than the size cap. Request bodies are capped at 1 MB, which is why a large object list is a manifest in storage rather than an inline array.

Unusable lines (invalid JSON, not a JSON object, a `uri` outside the bucket, a duplicate `uri`, an unknown key) and unreachable objects (missing, access denied, unsupported file type) are counted on the job. `GET /v2/jobs/{job_id}` returns a `manifest` object with the totals and a sample of up to 20 of each, giving the line number or `uri` and a reason. Neither fails the job: a million-line manifest with a few thousand bad lines indexes the rest.

Headers:

* Authorization: Bearer \{api\_key} - Captain API key for authentication
* X-Organization-ID: Organization UUID

Returns:
\{ job\_id, status: "pending" }

Reference: https://docs.runcaptain.com/reference/indexing/s3/manifest

## Request

### Path parameters

- `collection_name` (string, required)

### Headers

- `authorization` (string, optional, nullable)

### Body (application/json)

This endpoint expects an IndexS3ManifestRequest.

- `bucket_name` (string, required)
- `manifest_uri` (string, required) — S3 URI of the manifest, inside bucket\_name (s3://bucket/path/manifest.jsonl). UTF-8 JSON Lines, uncompressed, one object per line: \{"uri": ...} plus optional source\_identity, custom\_metadata, checksum, version\_id, size and etag. Up to 1,000,000 lines, 500 MB total and 10 KB per line.
- `processing_type` (enum, required) — Document processing type. 'advanced' uses agentic OCR with AI-enhanced extraction for complex layouts, tables, figures, charts, and documents containing images. 'basic' provides reliable OCR optimized for general document indexing and high-volume processing. 'auto' inspects each PDF and uses advanced only when it has photos, drawings, tables or scanned pages (logos and signatures do not count), otherwise basic; each file bills at the tier it used.
  - Allowed values: `advanced`, `basic`, `auto`
- `auth` (S3AssumeRoleAuth, optional, nullable) — Cross-account role-assumption auth. When provided, Captain calls sts:AssumeRole on the supplied role_arn (with the supplied external_id) instead of using static IAM-user keys. Mutually exclusive with aws_access_key_id/aws_secret_access_key.
- `aws_access_key_id` (string, optional, nullable) — AWS access key ID with read access to the bucket. Use this for long-lived IAM-user credentials. Omit when using the role-based 'auth' block.
- `aws_secret_access_key` (string, optional, nullable) — AWS secret access key. Use this for long-lived IAM-user credentials. Omit when using the role-based 'auth' block.
- `bucket_region` (string, optional, default: us-east-1)
- `custom_metadata` (map from string to IndexS3ManifestRequestCustomMetadata, optional, nullable) — Custom metadata attached to every indexed chunk. A line's own custom_metadata is merged over it for that file. Keys are strings; values: str, int, float, bool, or List[str].
- `mask_pii` (boolean, optional, default: false) — When true, mask detected PII (emails, names, SSNs, etc.) in parsed text before embedding and storing.
- `max_files` (integer, optional, nullable) — Stop after this many manifest lines, in file order.
- `overwrite_existing` (boolean, optional, default: false) — When true, files that already exist in the collection will be deleted and re-indexed with the latest changes. Requires skip_existing=false. Setting both to true returns a 400 error.
- `parsing_script` (string, optional, nullable) — Relative path to a JS parsing script for JSON files (e.g. 'research/paper-parser'). When provided, .json files are processed through a sandboxed V8 isolate. Without this, .json files are indexed as raw text.
- `pii_engine` (enum, optional, default: kev) — Masking engine for this job. `kev` (default): the Kev classifier model on Captain's infrastructure; supports `pii_fields` and `pii_instructions`; no fallback, so a file kev cannot mask fails to index and the rest of the job continues. `jev`: TypeSafe's hosted Jev model, slightly more accurate, best-effort availability; supports custom fields and instructions and may list fallbacks in `pii_fallback`. `presidio`: the legacy pattern engine, built-in categories only (rejects `pii_fields` or `pii_instructions` with 422), never a fallback. Requires `mask_pii: true`. The earlier names `captain-jev` and `captain-presidio` keep working as before; new requests should use `kev`, `jev` or `presidio`.
  - Allowed values: `kev`, `jev`, `presidio`, `captain-jev`, `captain-presidio`
- `pii_fallback` (IndexS3ManifestRequestPiiFallback, optional) — Ordered fallback engines for `jev`, as `{"engines": [...], "retry_budget_seconds": n}`. Omitted or an empty `engines` list means no fallback. Only valid with `pii_engine: "jev"`; with `kev` or `presidio` a non-empty list answers 422. On the multipart file upload endpoint, send it as a JSON string form field. With the earlier engine name `captain-jev` it is `true` (default, fall back to `captain-presidio`) or `false`.
- `pii_fields` (list of PIIField, optional, nullable) — Additional PII categories to mask for this job, on top of the built-in set. Requires mask_pii to be true. Up to 20 fields. Each masked value is replaced with the field's tag in angle brackets and appears in the masking report under the field's name.
- `pii_instructions` (string, optional, nullable) — Plain-language guidance that steers what counts as personal data for this job, for the built-in categories and any pii_fields. For example: 'Names of hospital staff may stay; patient names, record numbers and bed assignments must be masked.' Requires mask_pii to be true. Up to 1,000 characters.
- `skip_existing` (boolean, optional, default: true) — When true, files already indexed in the collection are skipped and will not be re-indexed with incoming changes. When false, all incoming files are indexed regardless of whether they already exist.
- `transcription_language` (string, optional, nullable) — AWS Transcribe language code for the spoken audio (e.g. 'es-US', 'pt-BR'). Omit to auto-detect per file. Video and audio files only. Supported codes: https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html

## Response

### 200

Successful Response

- `job_id` (string, required)
- `status` (string, optional, default: pending)
- `custom_metadata` (map from string to any, optional, nullable) — The custom_metadata Captain accepted for this job, echoed back as validated. Null when none was supplied.

## Errors

### 400 Bad Request Error

manifest missing or unreadable, or overwrite_existing and skip_existing both true

- `error` (string, required) — Error kind. `HTTP_ERROR` for errors raised by the route.
- `message` (string, required) — What went wrong.
- `path` (string, required) — The request path.
- `status_code` (integer, optional) — The HTTP status code.

### 422 Unprocessable Entity Error

Validation Error

- `error` (string, required) — `VALIDATION_ERROR`.
- `message` (string, required)
- `details` (list of V2CollectionsCollectionNameIndexS3ManifestPostResponsesContentApplicationJsonSchemaDetailsItems, required) — One entry per invalid field, with `type`, `loc` and `msg`.
- `path` (string, required)

## Types

### S3AssumeRoleAuth

- `external_id` (string, required) — External ID Captain issued you when you enrolled the AWS integration. Prevents the confused-deputy problem — your role's trust policy must require this exact value via an sts:ExternalId condition.
- `role_arn` (string, required) — ARN of the IAM role in your AWS account for Captain to assume (e.g. 'arn:aws:iam::123456789012:role/CaptainS3ReadRole'). The role's trust policy must allow Captain's principal and require the external_id below.
- `type` ("assume_role", required) — Auth type discriminator. Currently only 'assume_role' is supported.

### IndexS3ManifestRequestCustomMetadata

- string
- integer
- double
- boolean
- list of string

### IndexS3ManifestRequestPiiFallback

Ordered fallback engines for `jev`, as `{"engines": [...], "retry_budget_seconds": n}`. Omitted or an empty `engines` list means no fallback. Only valid with `pii_engine: "jev"`; with `kev` or `presidio` a non-empty list answers 422. On the multipart file upload endpoint, send it as a JSON string form field. With the earlier engine name `captain-jev` it is `true` (default, fall back to `captain-presidio`) or `false`.

- PiiFallbackPolicy: PiiFallbackPolicy
- boolean

### PIIField

A customer-defined PII category to mask in addition to the built-in ones.

- `name` (string, required) — Tag for this category. Uppercase letters, digits and underscores, 2 to 40 characters, unique within the request and not one of the built-in tags. Masked values are replaced with the tag in angle brackets (a field named EMPLOYEE\_ID produces \<EMPLOYEE\_ID>) and the masking report lists them under this name.
- `description` (string, required) — What the category covers, in plain language, as you would explain it to a colleague. This is what decides whether a value is masked, so say what the values look like and where they appear.
- `examples` (list of string, optional, nullable) — Up to 10 sample values, used only to make the description concrete. They are not a pattern: values that do not look like the examples are still masked when the description covers them.

### V2CollectionsCollectionNameIndexS3ManifestPostResponsesContentApplicationJsonSchemaDetailsItems

### PiiFallbackPolicy

Fallback policy for `pii_engine: "jev"`. If every engine fails, the file fails to index and the rest of the job continues.

- `engines` (list of enum, optional, default: []) — Engines to try, in order, when `jev` fails with a rate limit, timeout, capacity limit, provider error or invalid answer. `jev-openrouter` and `jev-ai-gateway` are the same Jev model served through OpenRouter and Vercel AI Gateway. Custom fields and instructions apply on every engine. `presidio` is not allowed, and an engine may appear only once.
  - Allowed values: `jev-openrouter`, `jev-ai-gateway`, `kev`
- `retry_budget_seconds` (integer, optional, default: 60) — Extra retry and wait time per engine, for each piece of a file, before moving to the next engine. Time spent on successful answers does not count. 0 means one attempt per engine. Normal request timeouts still apply.

## Examples

**Request**

```json
{
  "bucket_name": "my-documents-bucket",
  "manifest_uri": "s3://my-documents-bucket/manifests/batch-2026-09.jsonl",
  "processing_type": "advanced",
  "auth": {
    "external_id": "captain-ext-8f2c1a",
    "role_arn": "arn:aws:iam::123456789012:role/captain-read",
    "type": "assume_role"
  },
  "bucket_region": "us-east-1",
  "custom_metadata": {
    "batch": "2026-09"
  }
}
```

**Response**

```json
{
  "job_id": "3f9c2b1e7d5a4c8b9e0f1a2b3c4d5e6f-1789700000",
  "status": "pending",
  "custom_metadata": {
    "batch": "2026-09"
  }
}
```

**SDK Code**

```python Python REST
import json
import boto3
import requests

BASE_URL = "https://api.filesearch.dev"
API_KEY = "your_api_key"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

lines = [
    {
        "uri": "s3://my-documents-bucket/inbox/2026-q4-msa.pdf",
        "source_identity": "contract-8841",
        "custom_metadata": {"department": "legal", "counterparty": "Northwind"},
    },
    {
        "uri": "s3://my-documents-bucket/inbox/2026-q4-sow.pdf",
        "source_identity": "contract-8842",
        "size": 482113,
        "etag": "9b2cf535f27731c974343645a3985328",
    },
    {
        "uri": "s3://my-documents-bucket/archive/2025-renewal.pdf",
        "source_identity": "contract-7310",
    },
    {
        "uri": "s3://my-documents-bucket/inbox/handbook.pdf",
    },
]
boto3.client("s3").put_object(
    Bucket="my-documents-bucket",
    Key="manifests/batch-2026-09.jsonl",
    Body="\n".join(json.dumps(line) for line in lines),
)

response = requests.post(
    f"{BASE_URL}/v2/collections/my_documents/index/s3/manifest",
    headers=headers,
    json={
        "bucket_name": "my-documents-bucket",
        "manifest_uri": "s3://my-documents-bucket/manifests/batch-2026-09.jsonl",
        "auth": {
            "type": "assume_role",
            "role_arn": "arn:aws:iam::123456789012:role/captain-read",
            "external_id": "captain-ext-8f2c1a"
        },
        "bucket_region": "us-east-1",
        "processing_type": "advanced",
        "custom_metadata": {
            "batch": "2026-09"
        }
    },
    timeout=60.0
)

if response.status_code in [200, 201]:
    data = response.json()
    print(f"Job {data['job_id']} is {data['status']}")
else:
    print(f"Error {response.status_code}: {response.text}")

```

```typescript TypeScript REST
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const BASE_URL = "https://api.filesearch.dev";
const API_KEY = "your_api_key";

const lines = [
    {
        uri: "s3://my-documents-bucket/inbox/2026-q4-msa.pdf",
        source_identity: "contract-8841",
        custom_metadata: { department: "legal", counterparty: "Northwind" },
    },
    {
        uri: "s3://my-documents-bucket/inbox/2026-q4-sow.pdf",
        source_identity: "contract-8842",
        size: 482113,
        etag: "9b2cf535f27731c974343645a3985328",
    },
    {
        uri: "s3://my-documents-bucket/archive/2025-renewal.pdf",
        source_identity: "contract-7310",
    },
    {
        uri: "s3://my-documents-bucket/inbox/handbook.pdf",
    },
];
await new S3Client({ region: "us-east-1" }).send(new PutObjectCommand({
    Bucket: "my-documents-bucket",
    Key: "manifests/batch-2026-09.jsonl",
    Body: lines.map((line) => JSON.stringify(line)).join("\n"),
}));

const response = await fetch(
    `${BASE_URL}/v2/collections/my_documents/index/s3/manifest`,
    {
        method: "POST",
        headers: {
            "Authorization": `Bearer ${API_KEY}`,
            "Content-Type": "application/json"
        },
        body: JSON.stringify({
            bucket_name: "my-documents-bucket",
            manifest_uri: "s3://my-documents-bucket/manifests/batch-2026-09.jsonl",
            auth: {
                type: "assume_role",
                role_arn: "arn:aws:iam::123456789012:role/captain-read",
                external_id: "captain-ext-8f2c1a"
            },
            bucket_region: "us-east-1",
            processing_type: "advanced",
            custom_metadata: {
                batch: "2026-09"
            }
        })
    }
);

if (response.ok) {
    const data = await response.json();
    console.log(`Job ${data.job_id} is ${data.status}`);
} else {
    console.error(`Error ${response.status}: ${await response.text()}`);
}

```