Back to changelog

Every File in a Bucket Is Its Own Document

When a bucket is indexed, each file is now one document, identified by its bucket and its full path. A file's name no longer decides which document it is.

This applies to Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, Supabase Storage and Backblaze B2.

Same name, different folders. Files such as a/report.txt and b/report.txt are always separate documents, each with its own content. Before, two such files indexed in one job could both end up holding one file's content, and the other file's text could not be found by search.

Names that start with a timestamp. Files whose names start with a 13-digit number are no longer merged. Before, ts/1700000000000-export.txt, ts/1700000000001-export.txt and ts/export.txt were treated as one document.

Re-syncing a bucket. Indexing an unchanged bucket again no longer reports unchanged files as identity conflicts. A new file with a timestamp at the start of its name is now indexed on a re-sync with default settings. Before, it was reported as a conflict and left out.

Document IDs stay the same. Every existing document keeps its document_id. Nothing needs to be indexed again to keep an ID.

Do I need to change anything?

  • If search results were complete and correct for your bucket: no.
  • If content was missing or duplicated for files that share a name: index those files again with overwrite_existing set to true and skip_existing set to false. Each file then holds its own content.
  • If files with timestamp names were merged into one document: they become separate documents the next time they are indexed. Indexing the folder again with overwrite_existing set to true and skip_existing set to false does this at once. The document that existed before keeps its ID, and each of the others is listed under the ID it already had.
  • Files sent through Index File: nothing changes. Uploading the same filename again still updates the same document.

Not fixed yet. When the same object path exists in two different buckets, and both buckets are indexed into one collection, the two files still collide: the one indexed last replaces the other. Until this is fixed, index buckets that share paths into separate collections. JSON files on Google Cloud Storage or Azure processed with a parsing script are still matched by name.

Same day: Search Results Carry Document Metadata, Metadata per File in One Upload, Uploaded Files Keep Their Names, Word Files Return Plain Text