Records
A record is one thing people search for: a product, an article, a listing or a contract. Each record is a JSON object with an id and the fields search should know about.
Records reach search on their own. Added and replaced records become searchable within a few seconds. Updates and deletes take effect within about a second. There’s no build step to run.
Reading records takes a search key or a scoped key. Adding, updating and deleting records takes a write key, which Captain issues. Keep the write key on a server, and never ship it in a browser or an app.
Structure a record
Here is a product record with the kinds of fields most indexes use:
Every record needs an id, in the field named at onboarding, which is usually id. When you send a record whose id the index already holds, the new record replaces the old one.
Captain agrees each field’s type with you at onboarding. When a record has a field the schema doesn’t name, Captain types it from its values. Dates, geo points and lists of objects are recognised, and other strings become text or keyword fields.
A nested object turns into dotted field names, so attributes.weight_g filters like any other field. A list gives a field several values, so color = "tan" matches the record above.
Values have to match their field’s type. A record with a value that doesn’t fit is rejected, and the reply names the record, the field and the problem.
Field types
Each field has one of these types. The type decides what a value looks like and what search can do with it.
- Text is a string of words, such as a title or a description. Search matches the words in it, and titles count for more than descriptions. Text fields can’t be filtered.
- Keyword is a string, or a list of strings, such as a brand or a category. Search can filter, facet and group on the exact value, and its words also match searches.
- Enum is one value from a set agreed at onboarding, such as a condition or a status. Search can filter and facet on it. A value outside the set is rejected.
- Identifier is a code, such as a SKU, a part number, a CUSIP, or a case or account number. A search for the exact code ranks that record first, and the field can be filtered exactly.
- Number is a number, or a number with a unit, such as
"5 lb"or"30 in". Search can filter, range, sort and facet on it. Values in other units of the same quantity are converted. - Date is an ISO 8601 date or datetime, or unix seconds. Search can filter, range and sort on it. Dates are stored in UTC.
- Geo is a location, written as
{"lat": …, "lon": …},[lat, lon]or"lat,lon". Search can filter by distance, as instore_location WITHIN 25 km OF (39.74, -104.99). - Bool is
trueorfalse. The strings"yes"and"no"also work. Search can filter on it. - URL is a link. It comes back with the record.
- Children is a list of objects, such as the clauses of a contract or the line items of an order. Their text is searched with the parent record,
children.fieldfilters match when any child matches, and a child can be cited as the passage that answered a search.
List Filters returns each field’s type and the filter operators it accepts.
Per-record access
On an index with per-record access control, each record carries an access list. The list names the users, groups or roles allowed to see the record. Its field is named at onboarding, and is acl by default.
A scoped key is minted for one end user. It returns only the records whose access list names that user, or one of that user’s groups. What a record without an access list means, whether visible to nobody or to everyone, is agreed at onboarding.
A write key can change access lists, which includes making a record visible to everyone. Check how the integration sets access fields before going live.
How changes reach search
Captain applies every change in the order it arrives. Each write gets a seq number that marks its place in that order. Get Update Status reports the last seq that search has applied. When it reaches the seq of your write, the change is live.
Added and replaced records become searchable within a few seconds. Search keeps answering the whole time, from the records it already has.
Updates and deletes take effect within about a second. A user who loses access to a record stops seeing it as soon as that update is applied.
Suggestions and spelling correction learn new words, and forget removed ones, within a few minutes. Captain refreshes them in the background after changes settle, and search keeps serving while it does.
When a tenant serves search from more than one region, a change applies first in the region that received it. The reply lists where the change is live in regions_applied, and where it’s still on its way in regions_pending. Every region catches up automatically.
curl
Python
TypeScript
Get records
Get Records returns records by id. Repeat the ids parameter once for each record, up to 100 per request. Asking for a single id works the same way.
Ids that don’t exist come back under missing. So do ids the key isn’t allowed to see. The two cases look the same on purpose, so a scoped key can’t learn that a hidden record exists.
curl
Python
TypeScript
Add or replace records
Add Records takes a batch of records. A record with a new id is added, and a record with an id the index already holds replaces the old one. The records become searchable within a few seconds.
You can send the batch in three formats. A JSON array holds up to 1,000 records and 8 MiB, sent as application/json. NDJSON, with one record per line, holds up to 100,000 records and 64 MiB, sent as application/x-ndjson. CSV with a header row has the same limits as NDJSON, sent as text/csv. CSV cells are typed like JSON values, and an empty cell counts as a missing value.
curl
Python
TypeScript
The reply says what happened. accepted is the number of records taken, split into new and replaced. record_count is the number of records the index now holds, and max_records is the most it can hold. The seq marks this change’s place in order.
By default, one bad record rejects the whole request, and the reply names every bad record. To add the good records anyway, send on_error=skip. The bad ones are then listed under rejected. Each bad record is named by its line or array position, with a code: BAD_JSON, BAD_ROW, MISSING_ID or NOT_AN_OBJECT.
When the same id appears twice in one request, the last copy wins and the reply adds a DUPLICATE_ID warning. Send on_duplicate=fail to reject the request instead.
For a large file sent in parts, set line_offset to the first line number of each part. Errors then report line numbers from the whole file.
Update fields
Update Records changes some fields of existing records and leaves the rest alone. Send each record’s id with only the fields to change, and set a field to null to clear it. A request can update up to 1,000 records. The changes take effect within about a second.
curl
Python
TypeScript
Updates work on every field type except text. To change a title or a description, send the whole record again through Add Records. An update to a text field answers 422 FIELD_NEEDS_REBUILD.
The reply counts the records updated and deleted, and gives the change’s seq. stale_features_until_build lists any features, such as typeahead or spelling, that still show the old values. They catch up within a few minutes, when Captain refreshes them in the background.
Delete records
Delete Record removes one record by its id. The record leaves search results, browse totals and Get Records within about a second.
curl
Python
TypeScript
To delete several records at once, send Update Records with a delete list of up to 1,000 ids.
curl
Python
TypeScript
A deleted record’s words leave suggestions and spelling within a few minutes, along with the background refresh. Deleted records never appear in results or in Get Records in the meantime.
Check an index
Get Index reports how many records the index holds and whether any changes are still being applied.
curl
Python
TypeScript
The status field is one of four values. empty means the index has no records yet. pending means changes were accepted and are still being applied. building means Captain is refreshing suggestions and spelling in the background, while search serves as usual. ready means every change is applied.
Import from Algolia
An existing Algolia index can come into Captain through Import From Algolia. Send either an export of the Algolia index, with its records, settings, synonyms and rules, or read access to it.
To give read access, send the Algolia application id, a key with read permissions only, and the index name. Captain refuses a key that can write to Algolia before it reads anything. Algolia keys are never stored.
curl
Python
TypeScript
The reply is an import job. Captain stores the import and reviews it with you, and then loads the records into the index. The job ends with the status landed_for_review once the import is stored. Get Job reports its progress.
curl
Python
TypeScript
Retries
Send an Idempotency-Key header with every write. If a request fails in transit and you retry it with the same key within 24 hours, Captain returns the first reply instead of applying the change twice. That reply carries the header Idempotent-Replayed: true. Reusing a key for a different request answers 422.