Records
A record is one thing people search for: a product, an article, a listing or a contract. Each record is a JSON object with an id and the fields search should know about.
Records reach search on their own. Added and replaced records become searchable within a few seconds. Updates and deletes take effect within about a second. There’s no build step to run.
Reading records takes a search key or a scoped key. Adding, updating and deleting records takes a write key, which Captain issues. Keep the write key on a server, and never ship it in a browser or an app.
Structure a record
Here is a product record with the kinds of fields most indexes use:
Every record needs an id, in the field named at onboarding, which is usually id. When you send a record whose id the index already holds, the new record replaces the old one.
Captain agrees each field’s type with you at onboarding. When a record has a field the schema doesn’t name, Captain types it from its values. Dates, geo points and lists of objects are recognised, and other strings become text or keyword fields. Index settings covers which fields are searched, faceted and sorted.
A nested object turns into dotted field names, so attributes.weight_g filters like any other field. A list gives a field several values, so color = "tan" matches the record above.
Values have to match their field’s type. A record with a value that doesn’t fit is rejected, and the reply names the record, the field and the problem.
Field types
Each field has one of these types. The type decides what a value looks like and what search can do with it.
- Text is a string of words, such as a title or a description. Search matches the words in it, and titles count for more than descriptions. Text fields can’t be filtered.
- Keyword is a string, or a list of strings, such as a brand or a category. Search can filter, facet and group on the exact value, and its words also match searches.
- Enum is one value from a set agreed at onboarding, such as a condition or a status. Search can filter and facet on it. A value outside the set is rejected.
- Identifier is a code, such as a SKU, a part number, a CUSIP, or a case or account number. A search for the exact code ranks that record first, and the field can be filtered exactly.
- Number is a number, or a number with a unit, such as
"5 lb"or"30 in". Search can filter, range, sort and facet on it. Values in other units of the same quantity are converted. - Date is an ISO 8601 date or datetime, or unix seconds. Search can filter, range and sort on it. Dates are stored in UTC.
- Geo is a location, written as
{"lat": …, "lon": …},[lat, lon]or"lat,lon". Search can filter by distance, as instore_location WITHIN 25 km OF (39.74, -104.99). - Bool is
trueorfalse. The strings"yes"and"no"also work. Search can filter on it. - URL is a link. It comes back with the record.
- Children is a list of objects, such as the clauses of a contract or the line items of an order. Their text is searched with the parent record,
children.fieldfilters match when any child matches, and a child can be cited as the passage that answered a search.
List Filters returns each field’s type and the filter operators it accepts.
Per-record access
On an index with per-record access control, each record carries an access list. The list names the users, groups or roles allowed to see the record. Its field is named at onboarding, and is acl by default.
A scoped key is minted for one end user. It returns only the records whose access list names that user, or one of that user’s groups. What a record without an access list means, whether visible to nobody or to everyone, is agreed at onboarding.
A write key can change access lists, which includes making a record visible to everyone. Check how the integration sets access fields before going live.
How changes reach search
Captain applies every change in the order it arrives. Each write gets a seq number that marks its place in that order. A new index starts at seq 1, including one created with the name of an index you deleted. Get Update Status reports the last seq that search has applied. When it reaches the seq of your write, the change is live.
Added and replaced records become searchable within a few seconds. Search keeps answering the whole time, from the records it already has.
Updates and deletes take effect within about a second. A user who loses access to a record stops seeing it as soon as that update is applied.
Suggestions and spelling correction learn new words, and forget removed ones, within a few minutes. Captain refreshes them in the background after changes settle, and search keeps serving while it does.
Writes always go to your tenant’s primary region, and reads are served from the primary. A tenant can have other regions provisioned, but they don’t serve reads until cross-region sync ships. This section will describe it then. Until then, regions_applied lists only the primary region and regions_pending is empty.
When added records wait for a build
Get Update Status also carries build, where the automatic build of added records stands. Its state is idle, waiting (with starts_in_s), building or failed. A failed build is tried again on its own, three times in all, 30 seconds and then two minutes apart. While a try is left, retry_in_s says when the next one starts, and message says why the last one failed. After the third, the next write starts another.
A records update or delete answers with job_id, the write job it belongs to. It also lists the ids it skipped because the index doesn’t have them, in not_found with their count in not_found_count. And it lists the ids that exist only in added records not built yet, in waiting_for_build with waiting_for_build_count. Those apply once that build is served.
A large first load can take a while to build. While the build runs, build in Get Update Status names its job in job_id and shows how far it has got in progress. Get Job with that id returns the same progress, and each step under stages. A write key reads the automatic builds of its own indexes. A build another key started answers 404 JOB_NOT_FOUND.
step is the step running now, named as in stages. It’s queued before the first step and done after the last. step_number counts from 1 up to steps. records_done is how many records the running step has done, out of records_total. A step that doesn’t count records reports every record done.
curl
Python
TypeScript
When your records will be searchable
While added records wait for a build or are being built, Get Update Status carries eta: an estimate of when they will be searchable. It is null when nothing is waiting.
Most adds to an index that already serves don’t wait for a build. They match keyword searches within about a second and are fully searchable a few seconds later. While they’re being made fully searchable, eta has mode fast_lane, searchable_in_s of 0 because keyword search finds them already, and two stages: keyword, which is done, and meaning, which is running. Once they’re fully searchable, eta is null, even though Captain later rebuilds the index’s ranking tables in the background.
A first build, a request over 1,000 records, clearing an index and a settings change still wait for a build, and eta estimates it as below.
Captain works the estimate out from the work ahead of your records and from how fast your own recent builds ran, stage by stage. It updates as the build moves, so ask again for a fresher answer.
The estimate is least accurate for the first build on a new tenant. No build has run there yet, so basis is default and the range is wide. It sharpens after the first build finishes. A build much larger than any before it is also less certain, so its range is wider.
When an index outgrows its tenant
There’s no record limit. Every write is accepted and stored durably before its reply. When the records outgrow what the tenant has room for, Captain resizes the tenant and then builds. Search keeps answering from the last build meanwhile. Get Update Status shows build.state as scaling with eta_s, the seconds until the new records are searchable, about, and message says the same in words.
Read your own writes
Every write that answers 202 also sends its seq in the X-Captain-Seq response header, the same number as the seq in the body.
Every answer about an index carries X-Captain-Applied-Seq: the last seq that search had applied when the request started. The results reflect at least that much. The header is missing on 401 and 403 answers, and for an index that doesn’t exist.
Every answer about an index also carries X-Captain-Index-Incarnation, and Get Update Status returns the same value as incarnation. It’s an opaque id that changes only when the index is deleted and created again, or restored from a backup. A re-created index starts again at seq 1, and a restored one continues from the backup’s seq. So compare seq numbers only when the incarnation matches.
To make sure a read sees a write you just made, send that write’s seq in the X-Captain-Min-Seq request header. It works on search, typeahead, browse, similar and Get Records. Captain answers from a copy of the index that has applied at least that seq, and never waits for one: when none has, you get the most current answer available. The x-captain-min-seq-met response header tells you which:
true: the answer includes your write.false: it doesn’t yet. Retry shortly.unknown: the answer carries noX-Captain-Applied-Seq, for example a401, a403or an index that doesn’t exist.
The header must be a whole number. Anything else gets 400 BAD_MIN_SEQ. An empty value counts as no header.
Every answer names the region that served it in the X-Captain-Cell response header, for example use1 for us-east-1. Writes always go to your primary region, and today reads do too. If Captain retried the request in a second region, the value ends in (second attempt). Browsers can read it, because it’s listed in Access-Control-Expose-Headers.
curl
Python
TypeScript
Get records
Get Records returns records by id. Repeat the ids parameter once for each record, up to 100 per request. Asking for a single id works the same way.
Ids that don’t exist come back under missing. So do ids the key isn’t allowed to see. The two cases look the same on purpose, so a scoped key can’t learn that a hidden record exists.
curl
Python
TypeScript
Add or replace records
Add Records takes a batch of records. A record with a new id is added, and a record with an id the index already holds replaces the old one. The records become searchable within a few seconds.
You can send the batch in three formats. A JSON array holds up to 1,000 records and 8 MiB, sent as application/json. NDJSON, with one record per line, holds up to 100,000 records and 64 MiB, sent as application/x-ndjson. CSV with a header row has the same limits as NDJSON, sent as text/csv. CSV cells are typed like JSON values, and an empty cell counts as a missing value.
curl
Python
TypeScript
The reply says what happened. accepted is the number of records taken, split into new and replaced. record_count is the number of records the index now holds. max_records is null: an index has no record limit, and Captain grows the tenant to fit what you add. The seq marks this change’s place in order.
By default, one bad record rejects the whole request, and the reply names every bad record. To add the good records anyway, send on_error=skip. The bad ones are then listed under rejected. Each bad record is named by its line or array position, with a code: BAD_JSON, BAD_ROW, MISSING_ID or NOT_AN_OBJECT.
When the same id appears twice in one request, the last copy wins and the reply adds a DUPLICATE_ID warning. Send on_duplicate=fail to reject the request instead.
For a large file sent in parts, set line_offset to the first line number of each part. Errors then report line numbers from the whole file.
Update fields
Update Records changes some fields of existing records and leaves the rest alone. Send each record’s id with only the fields to change, and set a field to null to clear it. A request can update up to 1,000 records. The changes take effect within about a second.
curl
Python
TypeScript
Updates to keyword, enum, identifier, number, date, geo and bool fields, and to access lists, take effect within about a second. An update that names a text field, such as a title or a description, is merged into the stored record, and the whole record is added again, as Add Records does. The fields you didn’t send are kept. Such records are listed in readded, and become searchable within a few seconds, like an added record. A request that names a text field re-adds every record it updates.
An id the index doesn’t have is skipped and listed in not_found, and the rest of the request still applies.
The reply counts the records updated and deleted, and gives the change’s seq. stale_features_until_build lists any features, such as typeahead or spelling, that still show the old values. They catch up within a few minutes, when Captain refreshes them in the background.
Delete records
Delete Record removes one record by its id. The record leaves search results, browse totals and Get Records within about a second.
curl
Python
TypeScript
To delete several records at once, send Update Records with a delete list of up to 1,000 ids.
curl
Python
TypeScript
A deleted record’s words leave suggestions and spelling within a few minutes, along with the background refresh. Deleted records never appear in results or in Get Records in the meantime.
An id the index doesn’t have is not an error. Delete Record answers 202 with deleted: 0 and the id in not_found, and Update Records lists such ids in not_found too, for updates and deletes alike. So a retried delete is always safe.
Delete records by filter
Delete Records By Filter removes every record a filter matches, in the same language search’s filter takes. Use it to drop discontinued products or a whole brand without listing their ids. Records added but not yet searchable are matched too.
curl
Python
TypeScript
The records leave search within about a second, like any delete. The reply’s job_id tracks it, and Get Update Status shows it applied once applied_seq reaches seq.
Send "dry_run": true to count the matches without deleting anything. The reply then has matched and deleted: 0.
Repeating the call is safe. The same filter then matches nothing, deletes nothing and answers 200 with matched: 0. A filter that names an unknown field, or uses an operator the field doesn’t take, answers 400 FILTER_INVALID and deletes nothing.
Empty an index
Delete All Records removes every record and keeps the index, with its fields, settings, synonyms and rules. Searches stop returning the records within about a second, including while the index’s first build is still running.
curl
Python
TypeScript
To swap in a fresh catalog, empty the index, then add the new records. Searches return nothing in between, so for a switch with no gap build the new catalog in a second index and point your app at it once it is ready.
Check an index
Get Index reports how many records the index holds and whether any changes are still being applied.
curl
Python
TypeScript
The status field is one of four values. empty means the index has no records yet. pending means changes were accepted and are still being applied. building means Captain is refreshing suggestions and spelling in the background, while search serves as usual. ready means every change is applied.
Manage indexes
A write key manages the indexes it can write: it lists them, deletes them, and reads and cancels their jobs. A key for every index manages them all.
List indexes
List Indexes returns each index the key can write, with the same fields Get Index returns.
curl
Python
TypeScript
Delete an index
Delete Index removes the index, its records, settings, synonyms and rules. It stops answering searches at once.
curl
Python
TypeScript
Deleting an index also changes the keys that named it:
- A key limited to this index alone is revoked. Its id is in
keys_revoked. It could reach nothing else, and revoking it means an index created later with the same name is never reachable with it. - A key that also covers other indexes loses only this one. Its id is in
keys_narrowed, and it keeps working on the rest. - A key for every index is never changed.
So deleting a staging index never touches the keys your production index uses, unless a key was limited to the staging index alone. Captain can issue a new key if you need one.
An index that was searchable answers 202 with a job that removes its stored data. An index that never had searchable records answers 200. A queued or running refresh of the index is cancelled.
Switch to a new index with an alias
An alias is a name that points at one index. Search through the alias, then rebuild into a new index and switch the alias to it in one call. Every request through the alias reaches the new index once the call answers, and your app’s config doesn’t change. Your tenant’s other regions get the switch with the rest of your writes, usually within a few seconds.
Every endpoint under /v1/indexes/{index}, the Algolia routes and multiple searches accept the alias in place of an index name. A key that names the alias keeps working after a switch, and reaches the index only through the alias.
An alias and an index can’t share a name. Deleting an index that an alias points to answers 409 INDEX_HAS_ALIAS, so an app searching through the alias never starts getting 404s. Point the alias elsewhere or delete it first. DELETE /v1/indexes/{alias} with a key that may delete answers 409 ALIAS_NOT_INDEX (any other key gets the same 401 or 403 as for a name that doesn’t exist), and Delete Alias (DELETE /v1/aliases/{alias}) removes only the alias. List Aliases (GET /v1/aliases) returns every alias and its index.
Copy Index without "records": false copies the records too, which then become searchable on their own. Copying creates an index, so it takes a write key for every index.
Read usage
Get Usage (GET /v1/usage) reports, per index and in total, how many records each index holds and serves, how many records and bytes were added, and how many searches and events it got. days sets the window: that many whole UTC days ending today, from start_date to end_date, 30 by default and at most 90. Today counts up to the moment of the call. Usage answers within about 5 seconds. On a very large search log, days not yet counted are left out, and the index and the reply say partial: true. Counting continues after the reply, so a later call is whole. Today’s numbers can lag by up to 5 minutes. A write key sees the indexes it writes.
searches counts a search once however many keystrokes led to it, and search_requests counts every request. accepted_records counts every record added since the index was created, replacements included.
List jobs
List Jobs returns the jobs the key can read, newest first: the jobs it started, and the automatic builds of its indexes. status is running (the default), queued, succeeded, failed or all. index limits the list to one index, and limit sets how many come back (50 by default, at most 500).
curl
Python
TypeScript
Cancel a job
Cancel Job stops a refresh that has not started yet, one with status queued. A write key cancels the jobs it started.
curl
Python
TypeScript
The job ends with status failed and cancelled: true, and the index keeps serving what it served before. The next records you add start the next refresh. A running refresh, a finished job, or a write job answers 409 JOB_NOT_CANCELLABLE. A write job’s records are already accepted, so undo them with another write instead.
Import from Algolia
An existing Algolia index can come into Captain through Import From Algolia. Send either an export of the Algolia index, with its records, settings, synonyms and rules, or read access to it. The import runs at once, with your write key.
Importing into an index that already has records deletes all of them first, then adds the Algolia records in their place. To keep an index’s records, import into a new index instead, and switch your app to it when it’s ready.
To give read access, send the Algolia application id, a key with read permissions only, and the index name. Captain refuses a key that can write to Algolia before it reads anything, with 400 ALGOLIA_KEY_CAN_WRITE. Algolia keys are never stored.
Send "object_search": true. The index then gets typed fields from the Algolia settings, the same kind of index Create Index makes, and Index Settings applies to it. Without it, the import uses an older mapping with no field settings.
curl
Python
TypeScript
Reading from Algolia answers 202 with an import job: Captain reads the index, adds the records with their settings, synonyms and rules, and starts the build (result.build_job). Send "build": false to skip the build. Get Job reports its progress, and result.counts how many settings, synonyms and rules were mapped, approximated or not mapped.
An export in the body is imported at once and answers 200. Its report lists each setting, synonym and rule as mapped, approximated or not_mapped, with the reason. Add "build": true to start the build too. Send "dry_run": true with an export to see that report without storing anything.
An index the import creates is keyed by objectID. Importing into a new index needs a write key for every index. A second import into the same index while one runs answers 409 IMPORT_RUNNING. Index Settings lists how each Algolia setting, synonym and rule carries over.
curl
Python
TypeScript
Retries
Send an Idempotency-Key header with every write. If a request fails in transit and you retry it with the same key within 24 hours, Captain returns the first reply instead of applying the change twice. That reply carries the header Idempotent-Replayed: true. Reusing a key for a different request answers 422.