Documents
Captain indexes PDF and Word files as chunks that keep their layout role and page range. This page covers the two processing tiers, how a document is chunked, what a result contains, and the limits.
This is useful for
- Contracts and policies, where a clause needs a page number and a highlight box to cite
- Scanned invoices, forms and correspondence at volume, where OCR text has to be searchable
- Reports and decks, where headings, tables and charts should be returned as separate blocks
- Any file a reader will open at the right page rather than read in a snippet
Processing tiers
Every document job sets processing_type. Captain does not switch tiers on its own.
Word files are converted to PDF before either tier runs, so .docx and .doc are handled
exactly like PDFs from that point, page ranges included.
How a document is chunked
Layout roles
Each block of the page is labelled with a role: body, table, heading,
page_header, page_footer, footnote or figure. Blocks with different roles are
never packed into the same passage. This keeps running headers and footers out of body
text, so a query for “confidential” does not match the stamp at the top of every page.
Use exclude_chunk_types on a query to drop roles you do not want in results.
Sections and passages
The document is split into sections of up to 40,000 tokens. Each section gets a generated summary of two or three sentences and 8 to 15 tags. Each section is then tiled into passages of about 250 tokens, and the passages are what the search matches. A result contains the passage and its section’s summary, so a hit on one clause also says which part of the document it belongs to. Passages do not overlap.
Context across section boundaries
When a document spans several sections, the next section starts with a short summary of the previous one. A clause that crosses the boundary is readable from either side.
Tables and figures
Tables are stored as text inside the chunk (HTML tables in advanced), so cell values are
searchable. Figures are described in words. Chart values are stored only when the
extraction can be verified against the image; otherwise the figure is kept with
extraction: "unavailable" and no values.
Page ranges and regions
Every result carries location.page_start and location.page_end. Add
include.regions to get the blocks behind a result, each with page, a normalised
bbox (top, left, width, height from 0 to 1), type and confidence. To show a
page, call GET /v2/collections/{collection}/documents/{document_id}/pages/{page_number}.
Limits and failures
Documents of 75 pages or more are parsed in parallel, 75 pages at a time. There is no page cap. The only size ceiling is the one on the route the file arrived by (see File Size Limits). A document that cannot be parsed fails on its own: the error is recorded on that file in the job and the rest of the batch continues. Both tiers run OCR; language detection is automatic.
With mask_pii on, Captain masks the chunk text, the block regions and any figure crops. If
masking fails, the document is not indexed. The report at GET /v2/jobs/{job_id}/pii lists
what was found, with block-level boxes. See PII Masking.
Examples
Find a clause and highlight it in a viewer
Goal: return the termination clause from a set of contracts indexed with advanced,
with the box to draw on the page.
The clause is one body passage with its own page range. regions[0].bbox is the
rectangle to highlight.
Scanned invoices at volume
Goal: make invoice numbers and totals searchable across thousands of scans at the lowest
cost. Index with basic; OCR runs on the scanned regions at 1 credit per page. Move a
supplier to advanced when its invoices have handwritten annotations or dense line-item
tables.
The OCR text is chunked like any other text. supplier comes from the job’s
custom_metadata, which is copied onto every passage.
A board deck exported to PDF
Goal: find the revenue chart on slide 9 by what it shows, and keep slide numbers out of results.
The figure is described in words and labelled figure. Slide numbers are page_footer
blocks, so excluding that role removes them.
See Supported File Types for the extension table.