-
Notifications
You must be signed in to change notification settings - Fork 5
How backups work
The whole backup picture in one place: what is protected, by what, how often, and — where it matters — why it was built that way rather than some other way.
The companion page, Off-site backups, is the procedure for the copies a person takes by hand. This page is the system: the automated mechanisms, the policy behind them, and how to check any of it is actually working.
Every figure here was measured on 2 September 2026 unless dated otherwise.
Sizes come from CloudWatch BucketSizeBytes and VolumeBytesUsed, from
du -sb, and from s3api head-object. None of them come from collStats,
whose free-list figures on this cluster froze byte-identical across a 4 GiB
write and should not be quoted.
GridFS is not a separate system. fs.files and fs.chunks are ordinary
collections living inside the same DocumentDB database as projects. The file
bytes are rows in a database. That is why:
- the cluster's billed volume is measured in hundreds of gigabytes;
- deleting a project is a database write, not a file operation;
- one cluster snapshot captures the catalogue and the payload in one act. There is no separate file backup to schedule, and there never was.
S3 sits outside that and holds something different in kind. amprepo-private
is a cache of prepared project downloads: on a download the application asks
S3 for the object and, on a miss, pulls the tarball out of GridFS, re-uploads
it, and serves that. So losing an object in it costs one slow download, not
data. It does not need a backup of its own.
The exception, and the only piece of site data that is not in the cluster at
all, is caper.sqlite3 — Django's own database, holding the user accounts,
their email and OAuth links, and the CMS pages. It has its own mechanism below.
| Store | Holds | Size | Covered by |
|---|---|---|---|
DocumentDB caper
|
Production projects, samples, and every GridFS file behind them | 237.65 GiB | Cluster snapshots |
DocumentDB caper-dev
|
The dev site, same shape, same cluster | 147.48 GiB | Cluster snapshots |
caper.sqlite3 |
364 accounts, 339 email links, 148 OAuth links, the CMS pages | 29.61 MB | Nightly upload to S3 |
S3 amprepo-private
|
Prepared project downloads served to users | 210.64 GiB | Nothing, deliberately — it is a cache |
S3 refs.ampliconrepository.org
|
AmpliconArchitect reference builds | 26.30 GiB | Nothing yet — see gaps |
S3 amprepobucket
|
Static assets, rebuildable from the repository | 151.5 MiB | Not needed |
caper and caper-dev are two databases on one cluster,
ampliconubuntu. Dev therefore shares prod's storage bill, its snapshot scope
and its I/O. Anything that connects should assert which database it is talking
to rather than assume.
Daily, AWS-managed, retention 35 days, taken in the 00:00–00:30 UTC window. This is the mechanism that covers both databases and all 362.50 GiB of GridFS payload. Nothing runs on our side; it is a cluster setting.
aws docdb describe-db-clusters \
--db-cluster-identifier ampliconubuntu \
--query 'DBClusters[0].[BackupRetentionPeriod,PreferredBackupWindow,EarliestRestorableTime]'
Retention is not the same as reach. It was raised from 1 day to 35 on
24 Aug 2026, and the window gains a day per day rather than jumping. Read
EarliestRestorableTime to see how far back point-in-time recovery actually
goes; it reaches the full 35 days around 27 Sep 2026. Until then, take a manual
snapshot before anything destructive.
Identical in kind to the automated ones with one property that matters: they never expire. That is the whole reason to take one before a migration, a bulk delete, or an engine upgrade.
aws docdb create-db-cluster-snapshot \
--db-cluster-identifier ampliconubuntu \
--db-cluster-snapshot-identifier manual-<what>-<yyyy-mm-dd>
They are also the thing that quietly accumulates. Delete them when the reason they were taken has passed — but check first whether point-in-time recovery still reaches back past the snapshot, because that is what makes it redundant.
backup_sqlite.py, from the crontab on both hosts at 00:05:
5 0 * * * docker exec -w /srv amplicon-<env> bash -c 'set -a; source /srv/caper/config.sh; set +a; \
/opt/venv/bin/python /srv/backup_sqlite.py --db /srv/caper/caper.sqlite3 --execute' \
>> ~/backup_sqlite.log 2>&1
What it does, and why each part is there:
- Copies the live database with
sqlite3.Connection.backup()rather thancp. Copying an open SQLite file withcpcan capture a torn write; the backup API cannot. - Drops every session row from its own copy. The live file is ~176 MiB and is essentially all sessions — the site mints one per anonymous request and the traffic is nearly all crawlers. What is actually irreplaceable is about a megabyte.
- Gzips and uploads to
s3://amprepo-backups/<env>/sqlite/, storage class Glacier Instant Retrieval. -
Skips the upload when a content hash says nothing changed. The instance
role can
HeadObjectits own prefix and read back the storedcontent-sha256, so an unchanged night costs a hash, not an upload.
There is deliberately no retention tier. Glacier IR bills a 90-day minimum per object, so deleting early saves nothing and could only lose something.
Check it ran:
aws s3 ls s3://amprepo-backups/prod/sqlite/ --recursive | tail -5
tail -20 ~/backup_sqlite.log # on the host
Three download controls that build and stream the accounts database, a metadata
dump of every collection except fs.chunks, and a manifest of every project
URL. They call backup_sqlite.py and dump_metadata.write_dump() rather than
reimplementing them, so there is no second definition of "what belongs in a
backup" to drift.
Each download writes a record — who took it, when, and the collection totals at that moment — and the page reports the difference against the totals now, because a copy whose age nobody knows is not a backup. The comparison is by count, so it catches anything created or removed but not an edit that leaves the counts equal; the page says so rather than implying a guarantee it cannot make.
The full procedure, including how to pull current project versions through the API, is on Off-site backups.
Restoring a snapshot has been rehearsed, on 31 Aug 2026, and it passed: both databases came back at identical sizes, every collection count matched, and three GridFS files over 1 MiB hashed identical to live. It took 1.5 hours and about $0.30. Counting proves the catalogue restored; hashing proves the payload did.
aws docdb restore-db-cluster-from-snapshot \
--db-cluster-identifier <scratch-name> \
--snapshot-identifier <snapshot> \
--engine docdb
# then create an instance in it — the cluster alone serves nothing
Three things worth knowing before you sit and watch it:
-
Expect about 90 minutes of total silence. No CloudWatch metric of any kind
is published while a cluster is
creating. An absence of metrics is not a failure. -
The cluster reports
availablebefore its instance does. Wait for the instance, not the cluster. - Delete the scratch cluster afterwards, instance first. It bills like a real one.
Re-run this after any migration and at least annually. A backup nobody has restored is a belief, not a backup.
| Decision | Made | Reasoning |
|---|---|---|
| Soft-deleted projects are retained forever | 2 Sep 2026 | Nothing expires them on a timer. The queue is worked by hand when someone decides to. A 90-day auto-purge was considered and dropped. |
| Superseded project versions are never deleted to reclaim space | 2 Sep 2026 | Old versions back published results and are needed for reproducibility. 64% of production is superseded versions; that is a fact about the site, not a target. |
No versioning on refs.ampliconrepository.org
|
2 Sep 2026 | The bucket publishes an md5sum beside each tarball, and that is the integrity check. Versioning would double the cost of a store that changes once a year. |
| No automated pull from a lab node | 31 Aug 2026 | The off-AWS copy is occasional and operator-run, not a service with a cron and a key on a machine outside the account. |
| Off-AWS copy leaves through an admin page, not a job | 2 Sep 2026 | The copy that survives an account-level event is the one on somebody's own disk, and no server can put it there. |
| Off-AWS cadence: every ~6 months | 31 Aug 2026 | The site does not change often. The page reports drift since the last download, so the cadence self-corrects. |
| No off-account snapshot copy | Aug 2026 | Instance loss is the likely failure and is covered. Account loss is the institution's to mitigate. |
| No weekly GridFS sync tier | Aug 2026 | It would duplicate the daily snapshots, worse. Deleted from the plan. |
-
The DocumentDB payload cannot leave AWS. There is no
start-export-taskin the DocumentDB API; a snapshot can be copied to another region and nowhere else. So the catalogue goes off AWS at a few tens of megabytes and the 362.50 GiB does not. That is a stated position, not an oversight — a rebuilt site would be re-uploaded projects behind their original URLs, which is what the URL manifest exists to make possible. -
amprepo-backupsis itself unversioned. The nightly job holds bothPutObjectandDeleteObjecton its own prefix, so a bug in the uploader or a mistypedaws s3 rmis not recoverable. One command fixes it, and it matters more here than on the bucket it protects.
No longer on this list: the reference builds. refs.ampliconrepository.org
holds 26.30 GiB in ten tarballs, and every analysis this site has served was
produced against them. A verified copy was taken off AWS on 2 Sep 2026 —
download_refs.py, all ten files downloaded and verified against the md5sums
published beside them, 28,234,799,306 bytes in about twelve minutes. Re-run it
with --verify-only after any change to the reference set; it hashes what is
already there and downloads nothing.
A separate question that gets asked repeatedly, so it has its own script rather than living in somebody's memory:
python check_volume_reclamation.py --profile <your-aws-profile>
Between 31 August and 2 September 2026, 257.80 GiB was deleted from the
cluster. CloudWatch's VolumeBytesUsed — the number AWS bills on — did not
move. The script reports whether it has since.
The trap it exists to avoid. VolumeBytesUsed dips below its plateau
several times a day on its own, for an hour or two at a time, and has gone as
low as 697.34 GiB. Every one of those dips has returned to the same plateau. A
dip is not reclamation, however deep it goes and however long it lasts — one was
read as the start of reclamation once and was wrong.
So the script reads the maximum over a trailing eight hours, not the current value. The longest excursion seen is a little over three hours, so an eight-hour window always contains plateau time and its maximum is the plateau even when the read happens mid-dip. Only a fall in that number means storage came back.
Two things it will not do, deliberately:
- It does not report
collStats.unusedStorageSize. That figure froze byte-identical across a 4 GiB write on this cluster and was the basis for a "~43 GB is already free" claim that was not true. - It does not treat the latest datapoint as the answer. That value is printed, labelled as possibly mid-dip, and excluded from the verdict.
As of 2 September 2026 the verdict is no reclamation. That is a fact about the storage engine, not a fault, and nothing on our side is waiting on it. If it ever changes, re-baseline the script so the next reader compares against the new plateau rather than the old one.
# 1. Is the cluster still snapshotting, and how far back does recovery reach?
aws docdb describe-db-clusters --db-cluster-identifier ampliconubuntu \
--query 'DBClusters[0].[BackupRetentionPeriod,EarliestRestorableTime]'
# 2. Which snapshots exist, and is the manual one still needed?
aws docdb describe-db-cluster-snapshots \
--query 'DBClusterSnapshots[].[DBClusterSnapshotIdentifier,SnapshotType,SnapshotCreateTime,Status]' \
--output text
# 3. Did last night's sqlite backup land?
aws s3 ls s3://amprepo-backups/prod/sqlite/ --recursive | tail -3
# 4. How stale is the copy that is off AWS?
# Admin > Download Backups shows the date and the drift since.
If (4) says the totals have moved, take the three downloads again. That is the only step in this system that needs a person, and it is the one that matters most.