Code

Implement custom storage for orgs#2093

Code

Open

tw4l wants to merge 69 commits intomainwebrecorder/browsertrix:mainfrom issue-578-custom-storagewebrecorder/browsertrix:issue-578-custom-storageCopy head branch name to clipboard

+944-94

Commits

Commits on Feb 24, 2025

• Add back custom storage endpoints

  tw4l

  committed
  6253106

  Copy full SHA for 6253106

• Flush out tests for setting custom storage

  tw4l

  committed
  4b88b32

  Copy full SHA for 4b88b32

• Fix test issue with bucket not existing for now

  tw4l

  committed
  0a90002

  Copy full SHA for 0a90002

• Add additional tests

  tw4l

  committed
  5b2a31c

  Copy full SHA for 5b2a31c

• Fix custom storage so it works as expected

  Show description for ac11a24
  tw4l

  committed
  ac11a24

  Copy full SHA for ac11a24

• Actually unset custom replica storage before deleting

  tw4l

  committed
  7b2ac32

  Copy full SHA for 7b2ac32

• Add TODO where custom storage deletion is failing

  tw4l

  committed
  26d5cc4

  Copy full SHA for 26d5cc4

• Fix check for whether storage label is in use

  tw4l

  committed
  3c2dbf8

  Copy full SHA for 3c2dbf8

• Remove todo on endpoint that's fine

  tw4l

  committed
  6d75066

  Copy full SHA for 6d75066

• Add todos re: tasks necessary to change storage

  tw4l

  committed
  8eb4d99

  Copy full SHA for 8eb4d99

• Check that no crawls are running before updating storage

  tw4l

  committed
  89f1064

  Copy full SHA for 89f1064

• Start adding post-storage update logic

  tw4l

  committed
  f9c81e5

  Copy full SHA for f9c81e5

• WIP: Add background job to copy old s3 bucket to new

  tw4l

  committed
  371cf27

  Copy full SHA for 371cf27

• WIP: Start adding logic to handle replica location updates

  tw4l

  committed
  b8609cd

  Copy full SHA for b8609cd

• Add additional note

  tw4l

  committed
  9b6c3ae

  Copy full SHA for 9b6c3ae

• Fix argument

  tw4l

  committed
  587952c

  Copy full SHA for 587952c

• Fix another argument

  tw4l

  committed
  ede158e

  Copy full SHA for ede158e

• Fixups

  tw4l

  committed
  5627130

  Copy full SHA for 5627130

• Fix linting

  tw4l

  committed
  7eb0f95

  Copy full SHA for 7eb0f95

• More linting fixes

  tw4l

  committed
  dc87351

  Copy full SHA for dc87351

• Refactor, seperate storage and replicas updates

  tw4l

  committed
  f1045f6

  Copy full SHA for f1045f6

• More refactoring

  tw4l

  committed
  2a746b0

  Copy full SHA for 2a746b0

• Make post-update task methods private

  tw4l

  committed
  03060dd

  Copy full SHA for 03060dd

• Check if any bg jobs running before changing storage

  tw4l

  committed
  dd58cdb

  Copy full SHA for dd58cdb

• Check bg job finished as well

  tw4l

  committed
  8ef2b2d

  Copy full SHA for 8ef2b2d

• Fixups

  tw4l

  committed
  8c12e26

  Copy full SHA for 8c12e26

• Storage update improvements

  tw4l

  committed
  92d08b2

  Copy full SHA for 92d08b2

• Fixup

  tw4l

  committed
  824a439

  Copy full SHA for 824a439

• Remove TODO

  tw4l

  committed
  ce01439

  Copy full SHA for ce01439

• Remove another todo

  tw4l

  committed
  3949a86

  Copy full SHA for 3949a86

• More fixups

  tw4l

  committed
  9e0c8df

  Copy full SHA for 9e0c8df

• Add provider to s3storage for rclone

  tw4l

  committed
  e8fa3bb

  Copy full SHA for e8fa3bb

• Fix typo

  tw4l

  committed
  ba9007f

  Copy full SHA for ba9007f

• Make API endpoints that change storage superuser-only for now

  tw4l

  committed
  4ba7e56

  Copy full SHA for 4ba7e56

• Add typing for init_storages_api, import Callable

  tw4l

  committed
  6e9a0c6

  Copy full SHA for 6e9a0c6

• Add missing User import

  tw4l

  committed
  85d34d1

  Copy full SHA for 85d34d1

• Fix StorageOps in operator main

  tw4l

  committed
  7ea417a

  Copy full SHA for 7ea417a

• Always use oid prefix in s3 storage

  Show description for d41d5c8
  tw4l

  committed
  d41d5c8

  Copy full SHA for d41d5c8

• Post-rebase fixups and remove create bucket fallback

  Show description for 87ee14a
  tw4l

  committed
  87ee14a

  Copy full SHA for 87ee14a

• Create extra test buckets in CI

  tw4l

  committed
  cf8a0d5

  Copy full SHA for cf8a0d5

• Add test for non-verified custom storage

  tw4l

  committed
  0c85a63

  Copy full SHA for 0c85a63

• Refactor to move updates to FastAPI background tasks

  tw4l

  committed
  6f7b09a

  Copy full SHA for 6f7b09a

• Include default replicas in /storage response if no org replicas

  tw4l

  committed
  336fac8

  Copy full SHA for 336fac8

• Fix unsetting of presigned URLs

  tw4l

  committed
  9879119

  Copy full SHA for 9879119

• Add --progress flag to rclone copy command

  tw4l

  committed
  f2fcba6

  Copy full SHA for f2fcba6

• Increase ttl seconds after finished for testing on dev

  tw4l

  committed
  6351ebd

  Copy full SHA for 6351ebd

• Ensure there are no double slashes between bucket name and oid

  tw4l

  committed
  e17a750

  Copy full SHA for e17a750

• Increase memory limit/request for copy job to 500Mi

  tw4l

  committed
  e3b53e6

  Copy full SHA for e3b53e6

• Reduce copy job ttlSecondsAfterFinished to 60

  tw4l

  committed
  a3a905a

  Copy full SHA for a3a905a

• Add storage tag to API endpoints

  tw4l

  committed
  fa9315f

  Copy full SHA for fa9315f

• Add flags to rclone to reduce memory usage, set limit to 350Mi

  tw4l

  committed
  706c31f

  Copy full SHA for 706c31f

• Fix positional operator in storage ref update

  tw4l

  committed
  aab19f9

  Copy full SHA for aab19f9

• One more positional operator fix

  tw4l

  committed
  dcfb1b7

  Copy full SHA for dcfb1b7

• Update docstrings and comments

  tw4l

  committed
  e9c2aa5

  Copy full SHA for e9c2aa5

• Make all-storages response valid JSON with response model

  tw4l

  committed
  35827e6

  Copy full SHA for 35827e6

• Add admin docs for storage

  tw4l

  committed
  3597d01

  Copy full SHA for 3597d01

• Fix API endpoint path in docs example

  tw4l

  committed
  e361526

  Copy full SHA for e361526

• Docs typo fix

  tw4l

  committed
  29fbe6f

  Copy full SHA for 29fbe6f

• Add provider field note

  tw4l

  committed
  adf5fc3

  Copy full SHA for adf5fc3

• Docs language cleanup

  tw4l

  committed
  6e0b547

  Copy full SHA for 6e0b547

• Check /all-storages in backend tests

  tw4l

  committed
  b4b1443

  Copy full SHA for b4b1443

• Add API endpoint for background job progress

  tw4l

  committed
  63ca2b9

  Copy full SHA for 63ca2b9

• Fix linting

  tw4l

  committed
  de52804

  Copy full SHA for de52804

• Format post-rebase with Black

  tw4l

  committed
  0e2debb

  Copy full SHA for 0e2debb

• Format with Black

  tw4l

  committed
  731526a

  Copy full SHA for 731526a

• Fix linting

  tw4l

  committed
  92ecc3f

  Copy full SHA for 92ecc3f

• Fix linking

  tw4l

  committed
  ff37866

  Copy full SHA for ff37866

• Cast job to CopyBucketJob

  tw4l

  committed
  bb2291d

  Copy full SHA for bb2291d

• Fix type errors from rebase

  tw4l

  committed
  9c3b053

  Copy full SHA for 9c3b053