Set Up Sync

How Sync works

One setup, then hands-off. Point a sync at a bucket. Captain indexes the existing files, then tracks the source so your search results always reflect what is in storage.

Many sources, one collection. Set up a sync for each bucket you want included, and Captain keeps all of them current in the same collection. Each sync tracks its own source independently, so a change or deletion in one source only affects the files that came from it.

Two mechanisms keep a collection current:

• Scheduled reconciliation runs on the cadence you choose. Captain lists the source, compares it against what it has already indexed, and applies the difference: new files get indexed, changed files get re-indexed, and removed files follow your deletion policy.
• Real-time events are an optional add-on. Forward change notifications from your source and updates land within seconds. Delivery is best-effort, so reconciliation always stays on as the backstop.

Events are enrolled per sync with one call, POST /v2/syncs/{sync_id}/webhooks; each provider section below shows its exact wiring, and the shared settings (cadence, deletion policy, pause, on-demand reconcile) work identically everywhere.

Choose your provider

Six storage providers with one behavior, plus Google Drive from the Studio. Pick yours:

AWS S3

Google Cloud Storage

Azure Blob Storage

Cloudflare R2

Supabase Storage

Backblaze B2

Google Drive

Studio only

AWS S3

Authentication

An AWS S3 sync can authenticate two ways.

Role assumption (recommended). Create an IAM role in your own account that grants read access to the bucket, and Captain reads through that role using a Captain-issued external ID. No long-lived keys leave your account. The setup is the same as S3 indexing, so follow the S3 Cross-Account IAM guide, then pass an auth block of type assume_role with your role_arn and external_id.

Access key. Pass an auth block of type access_key with an access_key_id and secret_access_key. Captain stores the secret securely and never returns it. Use this when a role is not an option.

Create a sync

Creating a sync starts the first full index of the collection and returns the sync record right away. The indexing itself runs in the background.

1
import requests
2
3
BASE_URL = "https://api.captain.dev"
4
API_KEY = "cap_your_api_key"
5
6
resp = requests.post(
7
    f"{BASE_URL}/v2/collections/company-knowledge/sync/s3",
8
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
9
    json={
10
        "bucket": "my-company-docs",
11
        "prefix": "knowledge-base/",
12
        "region": "us-east-1",
13
        "auth": {
14
            "type": "assume_role",
15
            "role_arn": "arn:aws:iam::123456789012:role/captain-s3-read",
16
            "external_id": "captain-abc123",
17
        },
18
        "processing_type": "advanced",
19
        "deletion_policy": "mirror",
20
        "sync_interval_minutes": 15,
21
    },
22
)
23
print(resp.json())

Real-time events

For AWS S3, events run entirely inside AWS: your bucket publishes to an SNS topic in your account, and that topic delivers straight to a queue on Captain’s side. There’s no HTTP endpoint to host and no signature handshake to configure.

One-click setup (Launch Stack)

A CloudFormation quick-create link stands up the whole wiring below in one pass, in your own AWS account: the SNS topic, the bucket’s event notification, and the enrollment with Captain.

The template is open source: github.com/runcaptain/captain-sync-templates. Launch it in the same region as your bucket (edit the region= segment of the link to match) and have your sync id and Captain API key ready; the stack prompts for them as parameters.

Manual setup

1. Create an SNS topic in the bucket’s region.
2. On the bucket, add an event notification for s3:ObjectCreated:* and s3:ObjectRemoved:* (scoped to your sync prefix if you set one) with the topic as its destination.
3. Enroll the sync with your topic ARN. Captain binds the topic to this sync and returns the queue to subscribe it to:

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={"sns_topic_arn": "arn:aws:sns:us-east-1:123456789012:my-bucket-events"},
5
)
6
print(resp.json()["queue_arn"])
7
print(resp.json()["instructions"])

4. Subscribe the returned queue_arn to your topic (protocol sqs). The subscription confirms automatically; there’s nothing to click through. Leave raw message delivery off (the default): Captain matches your events to this sync by the topic ARN carried in the SNS envelope, so raw delivery strips the routing information and your events are ignored.

Only events published through the topic you enrolled reach your sync. Events sit alongside scheduled reconciliation rather than replacing it. Delivery is at-least-once and can drop or reorder messages, so keep a cadence set even with events enabled.

Google Cloud Storage

Authentication

A GCS sync authenticates with a service account key. Create a service account with read access to the bucket (roles/storage.objectViewer), download its JSON key, and pass the whole key as service_account_json when you create the sync. Captain stores the key securely and never returns it.

Create a sync

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/gcs",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "service_account_json": open("service-account.json").read(),
8
        "processing_type": "advanced",
9
        "deletion_policy": "mirror",
10
        "sync_interval_minutes": 15,
11
    },
12
)
13
print(resp.json())

GCS buckets are global, so there is no region or endpoint_url to pass.

Real-time events

GCS publishes bucket changes to Pub/Sub, and Pub/Sub pushes them straight to Captain. Enroll first: Captain returns an ingest_url, an HTTPS endpoint hosted by Captain that listens for this sync’s notifications.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

Then wire the bucket to it:

$
# 1. A topic for the bucket's change notifications
>
gcloud pubsub topics create my-captain-events
>
>
# 2. Publish object creations and deletions to it (scoped to your prefix if set)
>
gsutil notification create -t my-captain-events -f json \
>
    -e OBJECT_FINALIZE -e OBJECT_DELETE gs://my-bucket
>
>
# 3. Push the notifications to your ingest_url
>
gcloud pubsub subscriptions create my-captain-events-sub \
>
    --topic my-captain-events --push-endpoint="<ingest_url>" --ack-deadline=30

OBJECT_FINALIZE fires for new and overwritten files and triggers a re-index; OBJECT_DELETE follows your deletion policy. Metadata-only changes are ignored. Keep the ingest_url private, because it authenticates the event stream for your sync.

Azure Blob Storage

Authentication

An Azure Blob sync authenticates with the storage account’s name and access key (key1 or key2 from the account’s Access keys blade). Pass them as account_name and account_key when creating the sync. Captain stores the key securely and never returns it. When the key is rotated on the Azure side, hand Captain the new one with PATCH /v2/syncs/{sync_id} and an account_key field: the new key is validated against the container before it replaces the stored one.

Create a sync

The container name is passed as container and echoed back as bucket in sync responses.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/azure",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "container": "policy-documents",
6
        "account_name": "contosodocs",
7
        "account_key": "<storage account key>",
8
        "prefix": "docs/",
9
        "processing_type": "advanced",
10
        "deletion_policy": "mirror",
11
        "sync_interval_minutes": 60,
12
    },
13
)
14
print(resp.json())

Azure storage accounts are addressed by name, so there is no region or endpoint_url to pass. sync_interval_minutes defaults to 60 for Azure so reconciliation backstops any missed event delivery.

Real-time events

Azure publishes blob changes through Event Grid, which delivers them straight to Captain over HTTPS. Enroll first: Captain returns an ingest_url, an HTTPS endpoint hosted by Captain that listens for this sync’s notifications.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

One-click setup (Deploy to Azure)

A Deploy to Azure template stands up the whole wiring in one pass, in your own Azure subscription: the Event Grid system topic on the storage account, the event subscription pointed at your ingest_url, and a self-verifying enrollment check. The deployment only goes green if Event Grid proves it can deliver to Captain and Captain confirms the webhook is enrolled on your sync.

The template is open source: github.com/runcaptain/captain-sync-templates. Deploy it into the resource group that holds the storage account, in the same region, with your sync id, Captain API key, and ingest_url ready; the template prompts for them as parameters.

Manual setup

One az CLI command creates the same subscription:

$
az eventgrid event-subscription create \
>
    --name captain-sync \
>
    --source-resource-id $(az storage account show -n contosodocs -g <resource-group> --query id -o tsv) \
>
    --endpoint "<ingest_url>" --endpoint-type webhook \
>
    --included-event-types Microsoft.Storage.BlobCreated Microsoft.Storage.BlobDeleted \
>
    --subject-begins-with "/blobServices/default/containers/policy-documents/blobs/docs/" \
>
    --event-delivery-schema eventgridschema

Azure validates the endpoint during the create (a 30-second handshake with Captain), so a failed create means the ingest_url is wrong or stale. BlobCreated fires for new and overwritten blobs and triggers a re-index; BlobDeleted follows your deletion policy. Renames on hierarchical-namespace (ADLS Gen2) accounts converge on the next reconcile. Keep the ingest_url private, because it authenticates the event stream for your sync; if it leaks, rotate it with {"rotate_secret": true} on the webhooks call, then update the event subscription’s endpoint to the new URL (which re-runs the handshake).

Cloudflare R2

Authentication

R2 speaks the S3 API with its own keys: create an R2 API token (Access Key ID + Secret Access Key). R2 derives its endpoint from your account ID and jurisdiction, so pass account_id rather than a full URL.

Create a sync

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/r2",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "account_id": "your-account-id",
8
        "access_key_id": "your-r2-key-id",
9
        "secret_access_key": "your-r2-secret",
10
        "jurisdiction": "default",  # "default", "eu", or "fedramp"
11
        "processing_type": "advanced",
12
        "sync_interval_minutes": 15,
13
    },
14
)

Real-time events

Enroll first to get your ingest_url:

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

Cloudflare only delivers R2 bucket notifications to a Queue inside your own account, and only a Worker in your account can read that queue; nothing outside Cloudflare can subscribe to it. So the event path needs one small piece on your side: bucket notifications land in your queue, and a forwarder Worker passes each one to your ingest_url, where Captain applies the change. Queues require the Workers Paid plan.

$
npx wrangler queues create captain-sync-events
$
npx wrangler r2 bucket notification create my-bucket \
>
    --event-type object-create --event-type object-delete \
>
    --prefix "docs/" --queue captain-sync-events

Then deploy the forwarder. wrangler.toml:

1
name = "captain-r2-forwarder"
2
main = "worker.js"
3
compatibility_date = "2025-01-01"
4
5
[[queues.consumers]]
6
queue = "captain-sync-events"
7
max_batch_size = 10
8
max_batch_timeout = 5
9
10
[vars]
11
INGEST_URL = "<ingest_url>"

worker.js:

1
export default {
2
  async queue(batch, env) {
3
    for (const msg of batch.messages) {
4
      try {
5
        const res = await fetch(env.INGEST_URL, {
6
          method: "POST",
7
          headers: { "Content-Type": "application/json" },
8
          body: JSON.stringify(msg.body),
9
        });
10
        if (res.status >= 500) msg.retry();
11
        else msg.ack();
12
      } catch (e) {
13
        msg.retry();
14
      }
15
    }
16
  },
17
};

Run npx wrangler deploy and changes in the bucket start flowing to your collection.

Supabase Storage

Authentication

Supabase Storage speaks the S3 API through a per-project endpoint. Get the S3 endpoint, an Access Key ID, and a Secret Access Key under Storage → S3 Connection, then pass the endpoint as endpoint_url.

Create a sync

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/supabase",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "endpoint_url": "https://PROJECT_REF.storage.supabase.co/storage/v1/s3",
8
        "access_key_id": "your-key-id",
9
        "secret_access_key": "your-secret",
10
        "processing_type": "advanced",
11
        "sync_interval_minutes": 15,
12
    },
13
)

Real-time events

Enroll first to get your ingest_url:

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

Supabase file changes appear as rows in the storage.objects table, so events come from a Database Webhook rather than the Storage UI. In the dashboard, open Integrations, install Database Webhooks if the project doesn’t have it yet, and create a webhook with:

• Table: objects in schema storage. Not buckets, which only fires when a bucket itself is created or deleted.
• Events: INSERT, UPDATE, and DELETE.
• Type: HTTP Request, method POST, URL: your ingest_url.

Each file change now posts directly to the Captain-hosted ingest_url, and Captain applies it to the collection. The webhook fires for every bucket in the project; Captain ignores events from buckets other than the one your sync covers.

Backblaze B2

Authentication

B2 speaks the S3 API with application keys. Create an Application Key (keyID + applicationKey) under Account → Application Keys, using a bucket-scoped key rather than the master key. The endpoint and region appear on the bucket’s details page.

Create a sync

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/backblaze",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "endpoint_url": "https://s3.us-west-004.backblazeb2.com",
8
        "region": "us-west-004",
9
        "access_key_id": "your-key-id",
10
        "secret_access_key": "your-app-key",
11
        "processing_type": "advanced",
12
        "sync_interval_minutes": 15,
13
    },
14
)

Real-time events

Enroll first to get your ingest_url:

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

Enable Event Notifications for your Backblaze account first (in the console); the API rejects rules until the feature is on. Then create an event notification rule on the bucket, in the console or via b2_set_bucket_notification_rules:

• Event types: b2:ObjectCreated:* and b2:ObjectDeleted:*.
• Prefix: your sync prefix, if set.
• Target: webhook, with your ingest_url as the URL.

B2 sends a test event when the rule is saved; Captain accepts it, so validation passes without side effects.

Google Drive

Sign in and import

1. In the Studio, open Home, choose the collection and environment you want the files in, and pick Google Drive as the source.
2. Sign in with Google. Captain asks for read-only access to Drive. The connection is durable: you will not be asked again for this Google account unless you revoke access or sign out of Google Drive in the Studio.
3. Pick a folder or files in the Google picker. Picking a folder indexes everything in it, including subfolders; picking individual files indexes those files and treats their folder as the sync scope.
4. The import runs as an indexing job. Native Google Docs, Slides, and Sheets are exported at fetch time (Docs and Slides as PDF, Sheets as a spreadsheet) so they parse like any other document.

Keep it in sync

When the import finishes, the Studio asks “Keep this folder in sync?”

Enable it and the folder is watched from then on. On every check Captain asks Drive what changed since the last one and applies only the difference:

• New files in the folder (or any subfolder) are indexed.
• Modified files are re-indexed in place. The updated content replaces the earlier version of the same document, so a file edited ten times is still one document in the collection.
• Deleted or trashed files follow the deletion policy.
• Files that did not change are not touched, so a check on a quiet folder costs nothing.

A file that fails to fetch on one check (a transient Drive error, for example) is retried on later checks; the Sync Manager shows how many retries are pending.

Manage the sync

Open Sync Manager in the Studio. Google Drive syncs sit next to your bucket syncs with the same columns: source, collection, status, cadence, last sync, and files updated. From the row menu you can:

• Sync now: run a check immediately instead of waiting for the next tick.
• Edit: change the cadence or the deletion policy.
• Pause and Resume: stop checking without losing the configuration.
• Delete: stop the sync and forget its change tracking. Already indexed documents stay in the collection.

Each row carries an activity band showing the running or latest indexing job. Expand it to see recent jobs with their file counts and status, or open one in Activity for per-file detail. These jobs are ordinary indexing jobs, so they are also visible through GET /v2/jobs.

If Google access lapses

If the Google account’s access is revoked, or the sign-in expires, the sync pauses and its status reads Reconnect required. Nothing already indexed is affected. Click Reconnect on the Google Drive integration card (or on the row in Sync Manager), sign in again, and checking resumes from where it left off.

Sync settings on every provider

Everything below behaves identically on all six storage providers; the sections above cover only what differs (credentials, the create call, event wiring). Google Drive syncs use the same cadence, deletion policy, pause, resume, and delete semantics, but as Sync Manager controls in the Studio rather than API calls; Sync now is the on-demand reconcile.

Cadence

sync_interval_minutes sets how often scheduled reconciliation runs.

Set sync_interval_minutes to an explicit null for a manual sync with no scheduled reconciliation. Leaving the field out means the provider’s default: manual on every provider except Azure Blob, which defaults to a 60-minute cadence so reconciliation backstops missed event deliveries. Real-time events and on-demand reconcile work either way. When changes need to land faster than every five minutes, use real-time events rather than a shorter cadence. The cadence is a backstop interval, not a latency guarantee.

Change the cadence at any time:

1
requests.patch(
2
    f"{BASE_URL}/v2/syncs/{sync_id}",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={"sync_interval_minutes": 60},
5
)

Change detection

Reconciliation decides what changed by comparing each object’s ETag, a content hash, against the value Captain recorded when it last indexed the object. It doesn’t compare file size, so an in-place overwrite that keeps the same byte count still gets caught and re-indexed. Objects whose content hasn’t changed are skipped, so each pass only touches what actually changed.

Deletion propagation

When an object is no longer in the bucket, deletion_policy decides what happens to its indexed copy:

Removed objects are matched by their exact bucket and key, so deletion targets the right document. archive is a one-way transition for a removed object: once its source object is gone from the bucket, the archived copy stays archived. Re-uploading the same key later starts a new live document at that key and does not un-archive the prior copy, so a query with include: { archived: true } can return both: the current live version and the earlier archived one.

Scoping and parsing

prefix scopes a sync to part of the bucket (an empty prefix syncs everything), include_patterns and exclude_patterns are glob filters over object keys, and processing_type picks the parsing tier (advanced for full document understanding, basic for faster and cheaper processing). All four are accepted on every provider’s create call.

Reconcile on demand

Run a reconciliation right away instead of waiting for the next scheduled tick. It returns the changes it found and applied.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/reconcile",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json())
6
# {"sync_id": "...", "job_id": "...", "added": 0, "modified": 1,
7
#  "removed": 0, "unchanged": 0, "deleted_documents": 0}

Pause, resume, and delete

Pause a sync to stop syncing without losing its configuration: send a PATCH to /v2/syncs/{sync_id} with {"status": "paused"}, and resume with {"status": "active"}.

Delete a sync with DELETE /v2/syncs/{sync_id}. This is a soft delete: syncing stops, the record is marked inactive, and its per-object sync state is cleared. The history is kept.

How it fits together

A common production setup: create the sync with a cadence such as 15 and deletion_policy: mirror, enroll the event webhook for faster updates, and leave scheduled reconciliation on as the backstop. Reconciliation catches any events that were dropped and brings the collection back in line with the bucket.

Endpoint reference

Full request and response schemas are in the Sync section of the API Reference. At a glance: