Indexing

File Size Limits

Uploading and indexing limits

A file goes through two stages on the way in, and each one caps size differently. Captain first takes in the bytes, and how big they can be depends on the endpoint that sent them. It then parses, chunks, and embeds the file, where the ceiling comes from the file type instead. A file has to clear both.

Indexing through cloud storage skips the uploading step entirely. Grant Captain read access to a bucket or drive and it pulls the file straight from the source, so the upload limit never comes into play. Indexing works the same either way.

Step 1. Upload

Upload methodUpload limit
Dashboard upload (and any presigned S3 upload)500 MB per file
POST /v2/collections/{name}/index/file100 MB per file, 20 files per request
POST /v2/collections/{name}/index/text10 MB of text
Cloud storage connectorsNo upload step, so no upload limit

Connectors are available for:

  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
  • Cloudflare R2
  • Supabase Storage
  • Backblaze B2
  • Dropbox
  • Google Drive
  • SharePoint
  • OneDrive

The dashboard switches to a multipart upload on its own for anything over 4 MB.

Step 2. Indexing

These limits apply to every file, including the ones Captain reads from cloud storage.

JSON

.json files are limited to 250 MB, whether or not you pass a parsing_script.

If you pass a parsing_script, the file also has to fit in the sandbox that runs your script:

ConstraintLimit
JSON passed to the parsing script250 MB
Script execution time60 seconds
Returned stringMust fit in V8, roughly 512 MB

Scripts that build large intermediate objects can still run out of memory below that ceiling. See Parsers for the full parsing script contract.

Text, Markdown, CSV, YAML

.txt, .md, .csv, .tsv, .yaml, and .yml are limited to 250 MB per file, the same ceiling as JSON. Files above that are rejected before they are downloaded, so you get the error in seconds rather than after a long job.

Split larger files. Chunking a huge document into per-record or per-section files also produces better search results than one enormous document.

Everything else

PDF, Office documents, images, audio, and video have no indexing limit of their own, so the upload limit is the only ceiling they have to clear. Very large media files take longer to process because they are transcribed or analyzed frame by frame.

What happens when you go over

A file too big for the upload step comes back right away with a 400 naming the limit for the endpoint it hit. Here is a 180 MB PDF sent to POST /v2/collections/{name}/index/file, which tops out at 100 MB:

1{
2  "detail": "File 'annual-report.pdf' exceeds maximum size of 100MB"
3}

That same PDF uploads fine from the dashboard, which takes 500 MB.

A file too big for the indexing step behaves differently. Captain accepts the job, then fails it while reading the file, since that is the first point where the type limit gets checked. Cloud storage jobs always fail this way because there is no upload step to catch them earlier. The error lands on the job record, readable with GET /v2/jobs/{job_id}.

Working with larger files

The right fix depends on which limit is blocking the file.

A text-based file over 250 MB is hitting the indexing limit, and splitting it is the only way through. No upload route gets around that one. Per-record or per-section files also search better than one enormous document, so this is usually worth doing anyway.

A PDF, Office doc, image, audio, or video between 100 MB and 500 MB is only hitting the upload limit on POST /v2/collections/{name}/index/file. Send it from the dashboard instead, which takes 500 MB, and it indexes normally.

A non-text file over 500 MB is past every upload endpoint. Put it in a cloud storage bucket and index it from there. Captain reads it from the source, so the upload limit drops out and only the indexing limits are left.

© 2026 Captain