Skip to content

Commit a96d971

Browse files
committed
[IMP] fs_attachment_s3_migration: checksum-based deduplication
Before this change, each attachment was processed individually causing duplicate S3 uploads for files with identical content. The migration domain was incomplete (missing type, checksum, db_datas filters), and writing to the datas field risked data loss if source files were unreadable. After this change, attachments are grouped by checksum within each batch - uploading each unique file once and updating all related attachment records to point to the same S3 path. The domain now properly excludes URL attachments, DB-stored files, and already migrated files. Record updates use _force_write_store_fname instead of datas writes, eliminating data loss risk. Added configurable force-DB rules respect and updated error logging. Task: 5048
1 parent ec02e7a commit a96d971

File tree

6 files changed

+438
-158
lines changed

6 files changed

+438
-158
lines changed

fs_attachment_s3_migration/data/queue_job_channel_data.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
To increase parallelism, configure in queue.job.channel settings or
1010
via server environment configuration:
1111
12-
Check https://github.com/OCA/queue/tree/18.0/queue_job documentation for more details.
12+
Check https://github.com/OCA/queue/tree/16.0/queue_job documentation for more details.
1313
1414
-->
1515
<record id="queue_channel_s3_migration" model="queue.job.channel">

fs_attachment_s3_migration/models/fs_storage.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,13 @@ class FsStorage(models.Model):
1919
help="queue_job channel to use for migration jobs.",
2020
)
2121

22+
migration_use_storage_force_db_rules = fields.Boolean(
23+
string="Use Storage Force-DB Rules",
24+
default=True,
25+
help="If checked, respect force_db_for_default_attachment_rules during "
26+
"migration. Small images and assets will be skipped.",
27+
)
28+
2229
def action_open_migration_wizard(self):
2330
"""Open the S3 migration wizard for this storage."""
2431
self.ensure_one()

0 commit comments

Comments
 (0)