feat(scripts): reproducible demo seed via snapshot dump/load#604
Merged
Conversation
Adds a self-contained pipeline that captures the current hand-crafted
demo state (78 users, 288 posts, 6 active + 48 past mentorships, 45
ratings, 100 messages, 389 binary uploads) into a portable snapshot
and replays it on a fresh DB without any HTTP/API calls.
- scripts/seed.sh: single bash entry point (setup/dump/load/status).
Bootstraps a local .venv on first run, pre-flights docker, and
dispatches to seed_snapshot.py.
- scripts/seed_snapshot.py: psycopg COPY + docker-cp based dump/load.
Writes seed-owned rows in FK order, restores backend uploads tree.
Idempotent — DELETE-then-COPY semantics on load.
- scripts/seed_snapshot_data/: committed snapshot artifact (~109 MB).
Lets a fresh checkout reach demo state in ~1.5 seconds.
- scripts/seed_demo_*.py: the full Phase A-V builder that produced
the snapshot, kept around for future regenerations. Phase A now
auto-applies coordinates + backfills bio/goals/availability from
the markdown corpus so the snapshot is reproducible end-to-end.
- docker-compose.yml: three new env defaults required by the seed
flow (APP_RATELIMIT_ENABLED, FOLLOW_GRAPH_SYNC_ENABLED, FOLLOW_RANKER).
- .gitignore: ignore .venv/, .env.*, seed-pipeline runtime artifacts.
Fresh-machine flow:
docker compose up -d
./scripts/seed.sh load
Add a docker-compose service that loads the snapshot into Postgres and
into the shared uploads volume when the stack is brought up with
`--profile demo`. Without the flag the service is dormant.
- docker-compose.yml: new `seed` service, profile-gated. Uses
python:3.12-slim, depends on backend health, mounts the scripts/
tree read-only plus the uploads_data volume, runs seed_snapshot.py
in `--uploads-mode local` so it writes directly to the volume
(no docker daemon access inside compose).
- scripts/seed_snapshot.py: support running inside compose. DSN now
auto-detects POSTGRES_HOST (so the same script works from the host
and from inside the `seed` service). New `--uploads-mode local`
flag short-circuits the `docker cp` path and copies files with
shutil instead.
- README.md: document the `--profile demo` flow plus the host-side
`./scripts/seed.sh load` wrapper.
Fresh-machine deploy flow now compresses to:
docker compose --profile demo up -d
…and the stack comes up already seeded with the hand-crafted demo
state. Without the profile, behaviour is unchanged: clean DB.
Without these flags the `--profile demo` stack boots, but the UI
quietly falls back to the legacy ranker, blank mentor explanations,
and a 401 on any registration attempt. Make the demo defaults explicit
and explain each knob.
- .env.example: pre-set every flag the demo dataset wants
(APP_RATELIMIT_ENABLED=false, APP_EMAIL_ENABLED=false,
APP_SPAM_ENABLED=false; MENTOR_ADVANCED_RANKER=true,
MENTOR_EXPLANATION_ENABLED=true, MENTOR_EXPLANATION_TIMEOUT_MS=15000;
FEED_ADVANCED_RANKER=true; FOLLOW_RANKER=advanced). The
docker-compose `:-` fallbacks still ship prod-safe defaults, so
forgetting to copy .env.example yields a prod-shaped boot.
- README.md: new "Environment variable reference" section with three
tables — seed-friendliness, advanced ranker stack, and keys to set
manually (chiefly OPENAI_API_KEY for LLM explanations + semantic
affinity).
Re-runnable demo flow now matches the snapshot's intent end-to-end:
cp .env.example .env
docker compose --profile demo up -d
Two friction points uncovered while dry-running the README on a fresh
checkout:
1. Quick start and the seed section listed two different commands
(`docker compose up --build` and `docker compose --profile demo up -d`).
Collapse them into one canonical fresh-clone command:
docker compose --profile demo up --build -d
`--profile demo` enables the one-shot seed service; `--build` covers
the first-run image build (backend, frontend, custom Neo4j-GDS).
Omit the profile to get a clean DB instead.
2. `docker compose down` without `--profile demo` leaves the seed
container orphaned with a stale network reference; the next `up`
then fails with "network not found". Document explicitly that the
teardown command must include the profile too, and surface the
`run --rm seed` recipe for re-seeding against a running stack.
Also collapse the duplicated "Seed local data" block into a single
"Seeding options" section that points back at the Quick start for the
primary command and only lists the host-side wrapper / legacy seeder
as alternates.
The production droplet runs the same docker-compose.yml as local dev (bundled Postgres + Neo4j, prod-grade .env), so the demo profile works identically there. Wire it into the existing deploy workflow: - .github/workflows/deploy.yml: add `--profile demo` to the compose invocation so the one-shot `seed` service runs after the backend reports healthy. Also pin command_timeout: 30m explicitly (it had rotted out of this branch but is needed for cold rebuilds), and add `set -e` so the workflow fails loudly if any step breaks. - README.md: rewrite the Production deployment section to describe the actual deploy path (GitHub Actions → SSH → docker compose --profile demo up --build -d) and list the env flags the droplet needs to surface the full advanced-ranker UI. The standalone docker-compose.prod.yml flow stays documented as the hardened alternative. The seed is idempotent and scoped to %@seed.test rows. Real users registered through the UI on the live site are never touched.
The orphan-cleanup queries in AttachmentRepository previously checked
only `messages` and `feed_post_attachments`, but four tables hold a
foreign key to `attachments(id)`:
- messages.attachment_id (V15, ON DELETE SET NULL)
- feed_post_attachments.attachment_id (V41, NO ACTION)
- task_assignment_attachments.attachment_id (V23, ON DELETE CASCADE)
- task_submission_attachments.attachment_id (V23, ON DELETE CASCADE)
Because the task-attachment junctions cascade on attachment delete,
the daily 03:15 UTC orphan sweep would silently drop task assignment
PDFs and task submission files older than the 24h retention window:
1. `findOrphansOlderThan` returns the attachment row as a candidate
(passes the messages/feed_post_attachments NOT EXISTS check).
2. `deleteIfStillOrphan` deletes the attachment row.
3. The CASCADE on `task_*_attachments.attachment_id` drops the
junction row, removing the task's file reference entirely.
4. Result: the file vanishes from the task detail view on both
web and mobile, with no audit trail.
Add `NOT EXISTS` clauses for both task junctions to both queries, and
expand the javadoc to enumerate the canonical list of FK sources so
future schema additions stay in sync.
No migration is needed (the queries are in code, not in
flyway_schema_history). Existing orphan rows in production that have
already been swept are unrecoverable, but the seed snapshot
re-uploads its task PDFs on every deploy so the demo dataset
self-heals.
LifelongWay
approved these changes
May 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a self-contained pipeline that captures the current hand-crafted
demo state (78 users, 288 posts, 6 active + 48 past mentorships,
45 ratings, 100 messages, 389 binary uploads) into a portable
snapshot and replays it on a fresh DB without any HTTP/API calls.
After this lands, a teammate can reach the full demo state with two
commands on a fresh checkout:
Load time on a working stack: ~1.5 seconds. No OpenAI / Unsplash /
Resend / thispersondoesnotexist dependencies at load time — every binary
the demo shows is committed under
scripts/seed_snapshot_data/uploads/.What's new
scripts/seed.sh— single bash entry point. Subcommands:setup,dump [DIR],load [DIR],status. Bootstraps a local.venvon first run, pre-flights Docker, and dispatches to thePython tooling.
scripts/seed_snapshot.py— psycopgCOPY+docker cpbaseddump/load. Writes seed-owned rows in FK order, restores the
backend uploads tree. Idempotent: DELETE-then-COPY on load, so
a re-load over an existing state cleanly converges.
scripts/seed_snapshot_data/— committed snapshot artifact(~109 MB). 27 CSVs in
data/, 151 photos + 238 attachments inuploads/, and amanifest.jsonwith row counts + id-sequencehigh-water marks.
scripts/seed_demo_*.py— the full Phase A-V builder thatproduced the snapshot. Phase A now auto-applies city → lat/lon
coordinates and backfills
mentors.bio,mentors.mentoring_goals,mentees.background_info,mentees.goalsfrom the markdowncorpus, plus tops up any mentor with 0 availability slots. That
makes the snapshot regenerable end-to-end without manual fixups.
docker-compose.yml— three new env defaults required by theseed flow:
APP_RATELIMIT_ENABLED,FOLLOW_GRAPH_SYNC_ENABLED,FOLLOW_RANKER. All default to safe production values; the seedscripts flip them per env when needed.
.gitignore—.venv/,.env.*(with example overrides),seed-pipeline runtime artifacts under
scripts/.seed_*.jsonandscripts/.seed_*_tmp/.What's in the demo state
Files in the snapshot directory
How load works
DELETE FROM <table> WHERE <seed-filter>for every dumped table,child-first, leaves bootstrap users (
admin@group7.com) untouched.COPY <table> (...) FROM STDIN WITH (FORMAT csv, HEADER false)for every CSV, parent-first.
setval(pg_get_serial_sequence(...), MAX(id))on every relevantsequence so subsequent UI registrations don't collide.
docker cp scripts/seed_snapshot_data/uploads/{photos,attachments} <backend>:/app/uploads/to restore the binary files. Existingfiles in the container are left in place (orphans are harmless;
nothing in the DB references them).
Test plan
docker compose up -dbringsthe stack up cleanly.
./scripts/seed.sh setupcreates.venv/and installsscripts/requirements-seed.txt../scripts/seed.sh statusshows seed counts (0 if first run)../scripts/seed.sh loadfinishes in under 5 seconds andreports the row totals listed in the table above.
defne.korkmaz.74@seed.test/Seed1234!on theweb and on the mobile app — both should show the demo persona,
her past mentorships, AI match recommendations, and her profile
photo.
yildiz.demir.78@seed.test/Seed1234!and confirmthe active mentorship with Rıza Yavuz shows tasks (with PDF
attachments), milestones, meeting notes, and a message thread.
./scripts/seed.sh dumpre-captures current state to thesame snapshot directory in under 15 seconds, leaving the manifest
with up-to-date counts.
Notes for reviewers
(no git LFS). The trade-off is that a fresh checkout has the full
demo immediately, with no external service round-trips. Open to
switching to LFS if size becomes a problem.
scripts/seed_demo_*.py) is what producedthe snapshot. It's kept around so future demo iterations can
regenerate the snapshot from updated markdown content — but the
builder is not required for
./scripts/seed.sh load, whichonly needs the snapshot directory.
APP_RATELIMIT_ENABLED=trueis the safe production default, sothe docker-compose change does not loosen any prod-relevant
setting. Local dev can flip it via
.envwhen needed.