Index GCS Manifest | Captain Docs

Index an explicit list of objects from a Google Cloud Storage bucket, given as a JSON Lines manifest in the same bucket. One bulk job; nothing is listed or scanned.

One object per line, UTF-8, uncompressed. uri is required, the rest are optional:

{"uri": "gs://my-gcs-documents/inbox/2026-q4-msa.pdf", "source_identity": "contract-8841", "custom_metadata": {"department": "legal", "counterparty": "Northwind"}}
{"uri": "gs://my-gcs-documents/inbox/2026-q4-sow.pdf", "source_identity": "contract-8842", "size": 482113, "etag": "9b2cf535f27731c974343645a3985328"}
{"uri": "gs://my-gcs-documents/archive/2025-renewal.pdf", "source_identity": "contract-7310"}
{"uri": "gs://my-gcs-documents/inbox/handbook.pdf"}

Physical location vs. logical identity

uri is where the file currently lives: this bucket, this key. It must be inside bucket_name, and it is the only field Captain reads bytes from.

source_identity is what the document is, in your own terms. Captain keeps one document per identity, so the identity is what decides whether a later index is an update or a new document:

Give gs://my-gcs-documents/inbox/2026-q4-msa.pdf the identity contract-8841. Move the file to gs://my-gcs-documents/archive/2026-q4-msa.pdf and index it again with the same identity, and Captain updates that one document. Its chunks are replaced and the old ones are retired.
Omit source_identity and the identity is the uri. The same file at a new key is a different identity, so you get a second document and both versions answer queries.

A repeat of an identity already in the collection is skipped while skip_existing is true (the default), and replaced when you send overwrite_existing: true or skip_existing: false. While it is skipping, Captain compares content. A repeat whose content is unchanged is skipped. A repeat whose content differs is an identity conflict: it is neither skipped nor indexed, and the job reports it under identity_conflicts so that you can send it again with overwrite_existing: true. When either version has no checksum, the repeat is skipped. Use your own record id, not a path, whenever content can move or be delivered twice.

The other line fields

custom_metadata: merged over the job-level custom_metadata, key by key, for this file only. A line’s value wins on a shared key.
checksum and version_id: recorded on the document. A checksum that is the object’s ETag, written bare or as etag:..., is compared with the object before anything is read, and a line that does not match is reported as unreachable instead of being indexed. Write any other algorithm with its prefix, for example sha256:...; it is recorded but not verified. version_id pins the object generation that is read.
size and etag: when both are present, as in a Storage Insights export, Captain does not read the object’s metadata before indexing. That is what lets a large manifest start quickly.

Limits and error handling

A manifest holds up to 1,000,000 lines and 500 MB, with each line at most 10 KB. It must be plain UTF-8 .jsonl or .ndjson; a compressed body is refused. Per line, custom_metadata takes at most 50 keys, keys up to 64 characters and string values up to 1,024 characters. The job-level custom_metadata is capped at 64 KB serialized. It is attached to every file in the job, so metadata size times file count must stay under 2 GB: a few hundred bytes fits the full million lines, and the full 64 KB fits about 32,000 files. Larger jobs should carry less job-level metadata or be split.

A wrong manifest_uri is a 400 before the job is created, as is a manifest larger than the size cap. Request bodies are capped at 1 MB, which is why a large object list is a manifest in storage rather than an inline array.

Unusable lines (invalid JSON, not a JSON object, a uri outside the bucket, a duplicate uri, an unknown key) and unreachable objects (missing, access denied, unsupported file type) are counted on the job. GET /v2/jobs/{job_id} returns a manifest object with the totals and a sample of up to 20 of each, giving the line number or uri and a reason. Neither fails the job: a million-line manifest with a few thousand bad lines indexes the rest.

Headers:

Authorization: Bearer {api_key} - Captain API key for authentication
X-Organization-ID: Organization UUID

Returns: { job_id, status: “pending” }

Index an explicit list of objects from a Google Cloud Storage bucket, given as a JSON Lines manifest in the same bucket. One bulk job; nothing is listed or scanned. One object per line, UTF-8, uncompressed. `uri` is required, the rest are optional: ```json {"uri": "gs://my-gcs-documents/inbox/2026-q4-msa.pdf", "source_identity": "contract-8841", "custom_metadata": {"department": "legal", "counterparty": "Northwind"}} {"uri": "gs://my-gcs-documents/inbox/2026-q4-sow.pdf", "source_identity": "contract-8842", "size": 482113, "etag": "9b2cf535f27731c974343645a3985328"} {"uri": "gs://my-gcs-documents/archive/2025-renewal.pdf", "source_identity": "contract-7310"} {"uri": "gs://my-gcs-documents/inbox/handbook.pdf"} ``` ### Physical location vs. logical identity `uri` is **where the file currently lives**: this bucket, this key. It must be inside `bucket_name`, and it is the only field Captain reads bytes from. `source_identity` is **what the document is**, in your own terms. Captain keeps one document per identity, so the identity is what decides whether a later index is an update or a new document: - Give `gs://my-gcs-documents/inbox/2026-q4-msa.pdf` the identity `contract-8841`. Move the file to `gs://my-gcs-documents/archive/2026-q4-msa.pdf` and index it again with the same identity, and Captain updates that one document. Its chunks are replaced and the old ones are retired. - Omit `source_identity` and the identity is the `uri`. The same file at a new key is a different identity, so you get a second document and both versions answer queries. A repeat of an identity already in the collection is skipped while `skip_existing` is true (the default), and replaced when you send `overwrite_existing: true` or `skip_existing: false`. While it is skipping, Captain compares content. A repeat whose content is unchanged is skipped. A repeat whose content differs is an identity conflict: it is neither skipped nor indexed, and the job reports it under `identity_conflicts` so that you can send it again with `overwrite_existing: true`. When either version has no checksum, the repeat is skipped. Use your own record id, not a path, whenever content can move or be delivered twice. ### The other line fields - `custom_metadata`: merged over the job-level `custom_metadata`, key by key, for this file only. A line's value wins on a shared key. - `checksum` and `version_id`: recorded on the document. A `checksum` that is the object's ETag, written bare or as `etag:...`, is compared with the object before anything is read, and a line that does not match is reported as unreachable instead of being indexed. Write any other algorithm with its prefix, for example `sha256:...`; it is recorded but not verified. `version_id` pins the object generation that is read. - `size` and `etag`: when both are present, as in a Storage Insights export, Captain does not read the object's metadata before indexing. That is what lets a large manifest start quickly. ### Limits and error handling A manifest holds up to **1,000,000 lines** and **500 MB**, with each line at most **10 KB**. It must be plain UTF-8 `.jsonl` or `.ndjson`; a compressed body is refused. Per line, `custom_metadata` takes at most 50 keys, keys up to 64 characters and string values up to 1,024 characters. The job-level `custom_metadata` is capped at 64 KB serialized. It is attached to every file in the job, so metadata size times file count must stay under 2 GB: a few hundred bytes fits the full million lines, and the full 64 KB fits about 32,000 files. Larger jobs should carry less job-level metadata or be split. A wrong `manifest_uri` is a 400 before the job is created, as is a manifest larger than the size cap. Request bodies are capped at 1 MB, which is why a large object list is a manifest in storage rather than an inline array. Unusable lines (invalid JSON, not a JSON object, a `uri` outside the bucket, a duplicate `uri`, an unknown key) and unreachable objects (missing, access denied, unsupported file type) are counted on the job. `GET /v2/jobs/{job_id}` returns a `manifest` object with the totals and a sample of up to 20 of each, giving the line number or `uri` and a reason. Neither fails the job: a million-line manifest with a few thousand bad lines indexes the rest. Headers: - Authorization: Bearer {api_key} - Captain API key for authentication - X-Organization-ID: Organization UUID Returns: { job_id, status: "pending" }

Path parameters

collection_namestringRequired

Request

This endpoint expects an object.

bucket_namestringRequired

manifest_uristringRequired

GCS URI of the manifest, inside bucket_name (gs://bucket/path/manifest.jsonl). UTF-8 JSON Lines, uncompressed, one object per line: {“uri”: …} plus optional source_identity, custom_metadata, checksum, version_id, size and etag. Up to 1,000,000 lines, 500 MB total and 10 KB per line.

processing_typeenumRequired

Document processing type. ‘advanced’ uses agentic OCR with AI-enhanced extraction for complex layouts, tables, figures, charts, and documents containing images. ‘basic’ provides reliable OCR optimized for general document indexing and high-volume processing.

service_account_jsonstringRequired

custom_metadatamap from strings to strings or integers or doubles or booleans or lists of strings or nullOptional

Custom metadata attached to every indexed chunk. A line’s own custom_metadata is merged over it for that file. Keys are strings; values: str, int, float, bool, or List[str].

mask_piibooleanOptionalDefaults to false

When true, detected PII (emails, phone numbers, SSNs, credit cards, names, and locations) is masked in the parsed content before it is embedded and stored — replaced with entity tags like <PERSON> and <EMAIL_ADDRESS>. For images (including images embedded in PDFs), PII text visible in the image is also pixel-redacted. Opt-in; defaults to false, which leaves content unchanged.

max_filesinteger or nullOptional>=1

Stop after this many manifest lines, in file order.

overwrite_existingbooleanOptionalDefaults to false

When true, files that already exist in the collection are re-indexed and replaced with zero downtime: the new version is built alongside the live one and atomically swapped in when complete, so the previous version keeps serving search results throughout the rebuild. The document keeps the same document_id across overwrites, and its status reads ‘updating’ in the document listing while the rebuild runs. Requires skip_existing=false. Setting both to true returns a 400 error.

parsing_scriptstring or nullOptional

Relative path to a JS parsing script for JSON files (e.g. ‘research/paper-parser’). When provided, .json files are processed through a sandboxed V8 isolate. Without this, .json files are indexed as raw text.

pii_engineenumOptionalDefaults to captain-jev

Masking engine for this job. captain-jev (default): TypeSafe’s Jev classifier reads every token of the text in context, decides which values are personal data and which kind each one is; supports pii_fields and pii_instructions. captain-presidio (legacy): pattern recognizers and a named-entity model, runs entirely inside Captain, built-in categories only; rejects pii_fields and pii_instructions with 422. Requires mask_pii to be true.

pii_fallbackbooleanOptionalDefaults to true

What happens to a file when the Jev engine cannot be reached. true (default): the file is masked by captain-presidio instead, without custom fields, and the job reports engine captain-presidio with fallback true. false: the file fails to index rather than being stored with less than the requested masking. Only meaningful with pii_engine captain-jev.

pii_fieldslist of objects or nullOptional

Additional PII categories to mask for this job, on top of the built-in set. Requires mask_pii to be true. Up to 20 fields. Each masked value is replaced with the field’s tag in angle brackets and appears in the masking report under the field’s name.

pii_instructionsstring or nullOptional<=1000 characters

Plain-language guidance that steers what counts as personal data for this job, for the built-in categories and any pii_fields. For example: ‘Names of hospital staff may stay; patient names, record numbers and bed assignments must be masked.’ Requires mask_pii to be true. Up to 1,000 characters.

skip_existingbooleanOptionalDefaults to true

When true, files already indexed in the collection are skipped and will not be re-indexed with incoming changes. When false, all incoming files are indexed regardless of whether they already exist.

transcription_languagestring or nullOptional

AWS Transcribe language code for the spoken audio (e.g. ‘es-US’, ‘pt-BR’). Omit to auto-detect per file. Video and audio files only. Supported codes: https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html

Response

Successful Response

job_idstring

statusstringOptionalDefaults to pending

custom_metadatamap from strings to any or nullOptional

The custom_metadata Captain accepted for this job, echoed back as validated. Null when none was supplied.

Errors

400

Bad Request Error

422

Unprocessable Entity Error