Skip to content

feat(scripts): reproducible demo seed via snapshot dump/load#604

Merged
LifelongWay merged 6 commits into
devfrom
feat/seed-snapshot-pipeline
May 13, 2026
Merged

feat(scripts): reproducible demo seed via snapshot dump/load#604
LifelongWay merged 6 commits into
devfrom
feat/seed-snapshot-pipeline

Conversation

@mehmetborasarioglu

Copy link
Copy Markdown
Contributor

Summary

Adds a self-contained pipeline that captures the current hand-crafted
demo state (78 users, 288 posts, 6 active + 48 past mentorships,
45 ratings, 100 messages, 389 binary uploads) into a portable
snapshot and replays it on a fresh DB without any HTTP/API calls.

After this lands, a teammate can reach the full demo state with two
commands on a fresh checkout:

docker compose up -d
./scripts/seed.sh load

Load time on a working stack: ~1.5 seconds. No OpenAI / Unsplash /
Resend / thispersondoesnotexist dependencies at load time — every binary
the demo shows is committed under scripts/seed_snapshot_data/uploads/.

What's new

  • scripts/seed.sh — single bash entry point. Subcommands:
    setup, dump [DIR], load [DIR], status. Bootstraps a local
    .venv on first run, pre-flights Docker, and dispatches to the
    Python tooling.
  • scripts/seed_snapshot.py — psycopg COPY + docker cp based
    dump/load. Writes seed-owned rows in FK order, restores the
    backend uploads tree. Idempotent: DELETE-then-COPY on load, so
    a re-load over an existing state cleanly converges.
  • scripts/seed_snapshot_data/ — committed snapshot artifact
    (~109 MB). 27 CSVs in data/, 151 photos + 238 attachments in
    uploads/, and a manifest.json with row counts + id-sequence
    high-water marks.
  • scripts/seed_demo_*.py — the full Phase A-V builder that
    produced the snapshot. Phase A now auto-applies city → lat/lon
    coordinates and backfills mentors.bio, mentors.mentoring_goals,
    mentees.background_info, mentees.goals from the markdown
    corpus, plus tops up any mentor with 0 availability slots. That
    makes the snapshot regenerable end-to-end without manual fixups.
  • docker-compose.yml — three new env defaults required by the
    seed flow: APP_RATELIMIT_ENABLED, FOLLOW_GRAPH_SYNC_ENABLED,
    FOLLOW_RANKER. All default to safe production values; the seed
    scripts flip them per env when needed.
  • .gitignore.venv/, .env.* (with example overrides),
    seed-pipeline runtime artifacts under scripts/.seed_*.json and
    scripts/.seed_*_tmp/.

What's in the demo state

Entity Count
Users 78 (1 admin + 20 mentors + 57 mentees)
Profile photos 78 (1 per user)
Feed posts 288, 71 with attached images
Comments / likes 655 / 852
Comment likes 1 639
Follow edges 377
Mentorships (active) 6
Mentorships (past) 48
Mentor ratings 45 (mean ~3.9 / 5)
Messages 100
Tasks 34 (7 with attached PDF files)
Milestones 28
Meetings 51 (mix of completed + scheduled)
Mentor-pair conversations 3

Files in the snapshot directory

scripts/seed_snapshot_data/
├── data/                  # 27 CSVs, FK-ordered
├── uploads/
│   ├── photos/            # 151 profile photos
│   └── attachments/       # 238 post images + task PDFs
└── manifest.json

How load works

  1. Pre-flight checks: docker daemon up, db + backend containers running.
  2. DELETE FROM <table> WHERE <seed-filter> for every dumped table,
    child-first, leaves bootstrap users (admin@group7.com) untouched.
  3. COPY <table> (...) FROM STDIN WITH (FORMAT csv, HEADER false)
    for every CSV, parent-first.
  4. setval(pg_get_serial_sequence(...), MAX(id)) on every relevant
    sequence so subsequent UI registrations don't collide.
  5. docker cp scripts/seed_snapshot_data/uploads/{photos,attachments} <backend>:/app/uploads/ to restore the binary files. Existing
    files in the container are left in place (orphans are harmless;
    nothing in the DB references them).

Test plan

  • On a checkout of this branch, docker compose up -d brings
    the stack up cleanly.
  • ./scripts/seed.sh setup creates .venv/ and installs
    scripts/requirements-seed.txt.
  • ./scripts/seed.sh status shows seed counts (0 if first run).
  • ./scripts/seed.sh load finishes in under 5 seconds and
    reports the row totals listed in the table above.
  • Login as defne.korkmaz.74@seed.test / Seed1234! on the
    web and on the mobile app — both should show the demo persona,
    her past mentorships, AI match recommendations, and her profile
    photo.
  • Login as yildiz.demir.78@seed.test / Seed1234! and confirm
    the active mentorship with Rıza Yavuz shows tasks (with PDF
    attachments), milestones, meeting notes, and a message thread.
  • ./scripts/seed.sh dump re-captures current state to the
    same snapshot directory in under 15 seconds, leaving the manifest
    with up-to-date counts.

Notes for reviewers

  • The snapshot directory is ~109 MB and is committed directly
    (no git LFS). The trade-off is that a fresh checkout has the full
    demo immediately, with no external service round-trips. Open to
    switching to LFS if size becomes a problem.
  • The Phase A-V builder (scripts/seed_demo_*.py) is what produced
    the snapshot. It's kept around so future demo iterations can
    regenerate the snapshot from updated markdown content — but the
    builder is not required for ./scripts/seed.sh load, which
    only needs the snapshot directory.
  • APP_RATELIMIT_ENABLED=true is the safe production default, so
    the docker-compose change does not loosen any prod-relevant
    setting. Local dev can flip it via .env when needed.

Adds a self-contained pipeline that captures the current hand-crafted
demo state (78 users, 288 posts, 6 active + 48 past mentorships, 45
ratings, 100 messages, 389 binary uploads) into a portable snapshot
and replays it on a fresh DB without any HTTP/API calls.

- scripts/seed.sh: single bash entry point (setup/dump/load/status).
  Bootstraps a local .venv on first run, pre-flights docker, and
  dispatches to seed_snapshot.py.
- scripts/seed_snapshot.py: psycopg COPY + docker-cp based dump/load.
  Writes seed-owned rows in FK order, restores backend uploads tree.
  Idempotent — DELETE-then-COPY semantics on load.
- scripts/seed_snapshot_data/: committed snapshot artifact (~109 MB).
  Lets a fresh checkout reach demo state in ~1.5 seconds.
- scripts/seed_demo_*.py: the full Phase A-V builder that produced
  the snapshot, kept around for future regenerations. Phase A now
  auto-applies coordinates + backfills bio/goals/availability from
  the markdown corpus so the snapshot is reproducible end-to-end.
- docker-compose.yml: three new env defaults required by the seed
  flow (APP_RATELIMIT_ENABLED, FOLLOW_GRAPH_SYNC_ENABLED, FOLLOW_RANKER).
- .gitignore: ignore .venv/, .env.*, seed-pipeline runtime artifacts.

Fresh-machine flow:

    docker compose up -d
    ./scripts/seed.sh load
Add a docker-compose service that loads the snapshot into Postgres and
into the shared uploads volume when the stack is brought up with
`--profile demo`. Without the flag the service is dormant.

- docker-compose.yml: new `seed` service, profile-gated. Uses
  python:3.12-slim, depends on backend health, mounts the scripts/
  tree read-only plus the uploads_data volume, runs seed_snapshot.py
  in `--uploads-mode local` so it writes directly to the volume
  (no docker daemon access inside compose).
- scripts/seed_snapshot.py: support running inside compose. DSN now
  auto-detects POSTGRES_HOST (so the same script works from the host
  and from inside the `seed` service). New `--uploads-mode local`
  flag short-circuits the `docker cp` path and copies files with
  shutil instead.
- README.md: document the `--profile demo` flow plus the host-side
  `./scripts/seed.sh load` wrapper.

Fresh-machine deploy flow now compresses to:

    docker compose --profile demo up -d

…and the stack comes up already seeded with the hand-crafted demo
state. Without the profile, behaviour is unchanged: clean DB.
Without these flags the `--profile demo` stack boots, but the UI
quietly falls back to the legacy ranker, blank mentor explanations,
and a 401 on any registration attempt. Make the demo defaults explicit
and explain each knob.

- .env.example: pre-set every flag the demo dataset wants
  (APP_RATELIMIT_ENABLED=false, APP_EMAIL_ENABLED=false,
  APP_SPAM_ENABLED=false; MENTOR_ADVANCED_RANKER=true,
  MENTOR_EXPLANATION_ENABLED=true, MENTOR_EXPLANATION_TIMEOUT_MS=15000;
  FEED_ADVANCED_RANKER=true; FOLLOW_RANKER=advanced). The
  docker-compose `:-` fallbacks still ship prod-safe defaults, so
  forgetting to copy .env.example yields a prod-shaped boot.
- README.md: new "Environment variable reference" section with three
  tables — seed-friendliness, advanced ranker stack, and keys to set
  manually (chiefly OPENAI_API_KEY for LLM explanations + semantic
  affinity).

Re-runnable demo flow now matches the snapshot's intent end-to-end:

    cp .env.example .env
    docker compose --profile demo up -d
Two friction points uncovered while dry-running the README on a fresh
checkout:

1. Quick start and the seed section listed two different commands
   (`docker compose up --build` and `docker compose --profile demo up -d`).
   Collapse them into one canonical fresh-clone command:

       docker compose --profile demo up --build -d

   `--profile demo` enables the one-shot seed service; `--build` covers
   the first-run image build (backend, frontend, custom Neo4j-GDS).
   Omit the profile to get a clean DB instead.

2. `docker compose down` without `--profile demo` leaves the seed
   container orphaned with a stale network reference; the next `up`
   then fails with "network not found". Document explicitly that the
   teardown command must include the profile too, and surface the
   `run --rm seed` recipe for re-seeding against a running stack.

Also collapse the duplicated "Seed local data" block into a single
"Seeding options" section that points back at the Quick start for the
primary command and only lists the host-side wrapper / legacy seeder
as alternates.
The production droplet runs the same docker-compose.yml as local dev
(bundled Postgres + Neo4j, prod-grade .env), so the demo profile works
identically there. Wire it into the existing deploy workflow:

- .github/workflows/deploy.yml: add `--profile demo` to the compose
  invocation so the one-shot `seed` service runs after the backend
  reports healthy. Also pin command_timeout: 30m explicitly (it had
  rotted out of this branch but is needed for cold rebuilds), and
  add `set -e` so the workflow fails loudly if any step breaks.
- README.md: rewrite the Production deployment section to describe
  the actual deploy path (GitHub Actions → SSH → docker compose
  --profile demo up --build -d) and list the env flags the droplet
  needs to surface the full advanced-ranker UI. The standalone
  docker-compose.prod.yml flow stays documented as the hardened
  alternative.

The seed is idempotent and scoped to %@seed.test rows. Real users
registered through the UI on the live site are never touched.
The orphan-cleanup queries in AttachmentRepository previously checked
only `messages` and `feed_post_attachments`, but four tables hold a
foreign key to `attachments(id)`:

  - messages.attachment_id                  (V15, ON DELETE SET NULL)
  - feed_post_attachments.attachment_id     (V41, NO ACTION)
  - task_assignment_attachments.attachment_id  (V23, ON DELETE CASCADE)
  - task_submission_attachments.attachment_id  (V23, ON DELETE CASCADE)

Because the task-attachment junctions cascade on attachment delete,
the daily 03:15 UTC orphan sweep would silently drop task assignment
PDFs and task submission files older than the 24h retention window:

  1. `findOrphansOlderThan` returns the attachment row as a candidate
     (passes the messages/feed_post_attachments NOT EXISTS check).
  2. `deleteIfStillOrphan` deletes the attachment row.
  3. The CASCADE on `task_*_attachments.attachment_id` drops the
     junction row, removing the task's file reference entirely.
  4. Result: the file vanishes from the task detail view on both
     web and mobile, with no audit trail.

Add `NOT EXISTS` clauses for both task junctions to both queries, and
expand the javadoc to enumerate the canonical list of FK sources so
future schema additions stay in sync.

No migration is needed (the queries are in code, not in
flyway_schema_history). Existing orphan rows in production that have
already been swept are unrecoverable, but the seed snapshot
re-uploads its task PDFs on every deploy so the demo dataset
self-heals.
@LifelongWay LifelongWay self-requested a review May 13, 2026 10:20
@LifelongWay LifelongWay merged commit 64d8e62 into dev May 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants