Set Up Sync

Set up Sync with your cloud storage

Pick a provider to jump to its setup. The same sync behavior (cadence, change detection, deletion propagation, and real-time events) applies to every provider.

AWS S3

Google Cloud Storage

Cloudflare R2

Supabase Storage

Backblaze B2

How Sync works

One setup, then hands-off. Point a sync at a bucket. Captain indexes the existing files, then tracks the source so your search results always reflect what is in storage.

Many sources, one collection. Set up a sync for each bucket you want included, and Captain keeps all of them current in the same collection. Each sync tracks its own source independently, so a change or deletion in one source only affects the files that came from it.

Two mechanisms keep a collection current:

• Scheduled reconciliation runs on the cadence you choose. Captain lists the source, compares it against what it has already indexed, and applies the difference: new files get indexed, changed files get re-indexed, and removed files follow your deletion policy.
• Real-time events are an optional add-on. Forward change notifications from your source and updates land within seconds. Delivery is best-effort, so reconciliation always stays on as the backstop.

You turn events on per sync with a single call, POST /v2/syncs/{sync_id}/webhooks. What it returns depends on the provider. For AWS S3, events travel AWS-natively: you pass your SNS topic’s ARN and the response contains a queue_arn to subscribe the topic to, with no URL involved. For every other provider (GCS, R2, Supabase, Backblaze), the response contains an ingest_url: an HTTPS endpoint hosted by Captain, unique to your sync, that you point the store’s change notifications at. Keep it private, because the secret embedded in it is what authenticates your event stream. Each provider section below shows the exact wiring.

AWS S3

Authentication

An AWS S3 sync can authenticate two ways.

Role assumption (recommended). Create an IAM role in your own account that grants read access to the bucket, and Captain reads through that role using a Captain-issued external ID. No long-lived keys leave your account. The setup is the same as S3 indexing, so follow the S3 Cross-Account IAM guide, then pass an auth block of type assume_role with your role_arn and external_id.

Access key. Pass an auth block of type access_key with an access_key_id and secret_access_key. Captain stores the secret securely and never returns it. Use this when a role is not an option. It is also how the S3-compatible stores below authenticate, since they do not support IAM roles. For step-by-step credential setup with screenshots, see Connect Cloud Storage.

Create a sync

Creating a sync starts the first full index of the collection and returns the sync record right away. The indexing itself runs in the background.

1
import requests
2
3
BASE_URL = "https://api.runcaptain.com"
4
API_KEY = "cap_your_api_key"
5
6
resp = requests.post(
7
    f"{BASE_URL}/v2/collections/company-knowledge/sync/s3",
8
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
9
    json={
10
        "bucket": "my-company-docs",
11
        "prefix": "knowledge-base/",
12
        "region": "us-east-1",
13
        "auth": {
14
            "type": "assume_role",
15
            "role_arn": "arn:aws:iam::123456789012:role/captain-s3-read",
16
            "external_id": "captain-abc123",
17
        },
18
        "processing_type": "advanced",
19
        "deletion_policy": "mirror",
20
        "sync_interval_minutes": 15,
21
    },
22
)
23
print(resp.json())

A few fields worth calling out: prefix scopes the sync to part of the bucket (an empty prefix syncs everything), include_patterns and exclude_patterns are glob filters over object keys, and processing_type picks the parsing tier (advanced or basic). deletion_policy and sync_interval_minutes are covered below.

Cadence

sync_interval_minutes sets how often scheduled reconciliation runs.

Set sync_interval_minutes to null (or leave it out) for a manual sync with no scheduled reconciliation. Real-time events and on-demand reconcile still work. When changes need to land faster than every five minutes, use real-time events rather than a shorter cadence. The cadence is a backstop interval, not a latency guarantee.

Change the cadence at any time:

1
requests.patch(
2
    f"{BASE_URL}/v2/syncs/{sync_id}",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={"sync_interval_minutes": 60},
5
)

Change detection

Reconciliation decides what changed by comparing each object’s ETag, a content hash, against the value Captain recorded when it last indexed the object. It does not compare file size, so an in-place overwrite that keeps the same byte count is still caught and re-indexed. Objects whose content has not changed are skipped, so each pass only does work for what actually changed.

Deletion propagation

When an object is no longer in the bucket, deletion_policy decides what happens to its indexed copy:

Removed objects are matched by their exact bucket and key, so deletion targets the right document.

Real-time events

For faster propagation, wire your bucket’s change notifications to Captain. For AWS S3 this runs entirely inside AWS: your bucket publishes to an SNS topic in your account, and that topic delivers straight to a queue on Captain’s side. There is no HTTP endpoint to host and no signature handshake to configure.

1. Create an SNS topic in the bucket’s region.
2. On the bucket, add an event notification for s3:ObjectCreated:* and s3:ObjectRemoved:* (scoped to your sync prefix if you set one) with the topic as its destination.
3. Enroll the sync with your topic ARN. Captain binds the topic to this sync and returns the queue to subscribe it to:

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={"sns_topic_arn": "arn:aws:sns:us-east-1:123456789012:my-bucket-events"},
5
)
6
print(resp.json()["queue_arn"])
7
print(resp.json()["instructions"])

4. Subscribe the returned queue_arn to your topic (protocol sqs). The subscription confirms automatically; there is nothing to click through. Leave raw message delivery off (the default) — Captain matches your events to this sync by the topic ARN carried in the SNS envelope, so raw delivery strips the routing information and your events are ignored.

Only events published through the topic you enrolled reach your sync. Events sit alongside scheduled reconciliation rather than replacing it. Delivery is at-least-once and can drop or reorder messages, so keep a cadence set even with events enabled.

Reconcile on demand

Run a reconciliation right away instead of waiting for the next scheduled tick. It returns the changes it found and applied.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/reconcile",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json())
6
# {"sync_id": "...", "job_id": "...", "added": 0, "modified": 1,
7
#  "removed": 0, "unchanged": 0, "deleted_documents": 0}

Pause, resume, and delete

Pause a sync to stop syncing without losing its configuration: send a PATCH to /v2/syncs/{sync_id} with {"status": "paused"}, and resume with {"status": "active"}.

Delete a sync with DELETE /v2/syncs/{sync_id}. This is a soft delete: syncing stops, the record is marked inactive, and its per-object sync state is cleared. The history is kept.

Google Cloud Storage

Authentication

A GCS sync authenticates with a service account key. Create a service account with read access to the bucket (roles/storage.objectViewer), download its JSON key, and pass the whole key as service_account_json when you create the sync. Captain stores the key securely and never returns it. For a step-by-step walkthrough with screenshots, from creating the service account to downloading the key, see Connect Cloud Storage.

Create a sync

Creating a sync starts the first full index of the collection and returns the sync record right away, exactly as with S3.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/gcs",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "service_account_json": open("service-account.json").read(),
8
        "processing_type": "advanced",
9
        "deletion_policy": "mirror",
10
        "sync_interval_minutes": 15,
11
    },
12
)
13
print(resp.json())

GCS buckets are global, so there is no region or endpoint_url to pass. The other fields (prefix, include_patterns, exclude_patterns, processing_type, deletion_policy, sync_interval_minutes) work the same as in the S3 section, and so do cadence, change detection, deletion propagation, on-demand reconcile, and pause and delete.

Real-time events

GCS publishes bucket changes to Pub/Sub, and Pub/Sub pushes them straight to Captain. Enroll first: Captain returns an ingest_url, an HTTPS endpoint hosted by Captain that listens for this sync’s notifications.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

Then wire the bucket to it:

$
# 1. A topic for the bucket's change notifications
>
gcloud pubsub topics create my-captain-events
>
>
# 2. Publish object creations and deletions to it (scoped to your prefix if set)
>
gsutil notification create -t my-captain-events -f json \
>
    -e OBJECT_FINALIZE -e OBJECT_DELETE gs://my-bucket
>
>
# 3. Push the notifications to your ingest_url
>
gcloud pubsub subscriptions create my-captain-events-sub \
>
    --topic my-captain-events --push-endpoint="<ingest_url>" --ack-deadline=30

OBJECT_FINALIZE fires for new and overwritten files and triggers a re-index; OBJECT_DELETE follows your deletion policy. Metadata-only changes are ignored. Keep the ingest_url private, because it authenticates the event stream for your sync. As with every provider, delivery is best-effort, so leave a reconciliation cadence set as the backstop.

S3-Compatible

Sync also works with Cloudflare R2, Supabase Storage, and Backblaze B2 over their S3-compatible interfaces. Each has its own create endpoint that takes the fields native to that store. These stores do not support IAM roles, so they authenticate with an access key. Once a sync exists it is managed with the same endpoints, whichever store it came from.

Real-time events work here too. Enroll with an empty body and Captain returns an ingest_url: an HTTPS endpoint hosted by Captain, unique to this sync, that listens for your store’s change notifications. You configure the store to send its notifications there (the steps differ per store, shown in each tab), and Captain applies each change to the collection as it arrives. Keep the URL private, because the secret embedded in it is what authenticates the event stream for your sync.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["ingest_url"])
6
print(resp.json()["instructions"])

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/r2",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "account_id": "your-account-id",
8
        "access_key_id": "your-r2-key-id",
9
        "secret_access_key": "your-r2-secret",
10
        "jurisdiction": "default",  # "default", "eu", or "fedramp"
11
        "processing_type": "advanced",
12
        "sync_interval_minutes": 15,
13
    },
14
)

$
npx wrangler queues create captain-sync-events
$
npx wrangler r2 bucket notification create my-bucket \
>
    --event-type object-create --event-type object-delete \
>
    --prefix "docs/" --queue captain-sync-events

1
name = "captain-r2-forwarder"
2
main = "worker.js"
3
compatibility_date = "2025-01-01"
4
5
[[queues.consumers]]
6
queue = "captain-sync-events"
7
max_batch_size = 10
8
max_batch_timeout = 5
9
10
[vars]
11
INGEST_URL = "<ingest_url>"

1
export default {
2
  async queue(batch, env) {
3
    for (const msg of batch.messages) {
4
      try {
5
        const res = await fetch(env.INGEST_URL, {
6
          method: "POST",
7
          headers: { "Content-Type": "application/json" },
8
          body: JSON.stringify(msg.body),
9
        });
10
        if (res.status >= 500) msg.retry();
11
        else msg.ack();
12
      } catch (e) {
13
        msg.retry();
14
      }
15
    }
16
  },
17
};

Cadence, deletion propagation, and on-demand reconcile behave the same on every provider. Real-time events behave the same once wired, but each store connects its notifications differently, as shown in the tabs above.

How it fits together

A common production setup: create the sync with a cadence such as 15 and deletion_policy: mirror, enroll the event webhook for faster updates, and leave scheduled reconciliation on as the backstop. Reconciliation catches any events that were dropped and brings the collection back in line with the bucket.

Endpoint reference

Full request and response schemas are in the Sync section of the API Reference. At a glance: