Add storage sync, tessellation modes, and stuck-job watchdog

[davecoulter]

Summary

• Decouples ingest pipelines from raw file transfer via an mc-based staging→canonical storage sync (Celery task + Beat cron), with LocalStorageService.list_prefix fixed for true string-prefix matching and dot-file exclusion.
• Adds configurable tile-tessellation region sources (cone, project_footprint, bounding_box) and an overlap_only keep-policy, replacing the old pre-computed moc_to_tile input with a region_source-discriminated request schema.
• Adds a stuck-job reconciliation watchdog (JobConfiguration dispatch linkage, per-job staleness override, reconcile_stuck_jobs service + API/CLI/Beat cron) that auto-fails jobs left IN_PROCESS by an uncatchable worker crash or OOM kill.
• Adds a per-task Celery time-limit contract (TimeLimitedTask/DiffpypeTask, __init_subclass__-enforced so no concrete task can skip declaring its own soft_time_limit_seconds/tracked_entity_model) and a begin_tracked_job stale-redelivery guard, closing a real race condition between the watchdog and Redis's own visibility_timeout broker redelivery.
• Migrates Celery Beat to a dedicated, database-backed scheduler (sqlalchemy-celery-beat, in its own celery_schema Postgres schema) with a dedicated beat container and SQLAdmin views for runtime schedule editing without restarts.

CLI Verification

All steps completed prior to PR creation via genTests.

• Branch confirmed: feature/operational-services-watchdog
• Images rebuilt: docker compose build api worker_light worker_heavy beat
• Containers recreated: docker compose up -d api worker_light worker_heavy beat
• Migrations applied: 0013 (JobConfiguration.task_name), 0014 (Beat scheduler tables in celery_schema) — both applied and verified live
• Test suite passed: 309 passed, 99.59% coverage (docker compose exec api uv run pytest --cov=src --cov-fail-under=90 -q)
• Sphinx build passed (local)
• docs/diagrams/infrastructure_topology.md updated (new beat container + staging sync data flow) and visually reviewed

Application QA

All steps completed and verified prior to PR creation via genTests.

• Staging→canonical sync (local + MinIO/S3 paths) — files mirrored correctly, dot-files/partial-prefixes excluded
• Per-task time-limit contract enforced at worker startup — confirmed a misconfigured task fails at class-definition time, not per-call; real worker traceback confirmed current code (not stale image) after a rebuild
• Stale-redelivery guard (begin_tracked_job) — re-dispatching an already-COMPLETE mosaic logged tracked_job_stale_redelivery_skipped and left DB state untouched
• Tessellation cone/project_footprint/bounding_box modes — each returned correct tile grids against real project calibration data; overlap_only true/false produced correctly differing tile counts
• Tessellation 422 validation — missing conditional params for each region_source correctly rejected
• Tessellation CLI parity (tessellate-tiles) — exact match against the equivalent API call
• JobConfiguration dispatch linkage — new mosaic dispatch correctly populated job_configuration_id/task_name
• reconcile_stuck_jobs — verified via API, CLI, and Beat cron dispatch, including a per-job staleness_timeout_seconds override read from job_kwargs
• Beat schedule seeding (diffpype-manage seed-db) — seeded both cron schedules correctly
• Dynamic Beat schedule edit via SQLAdmin, no restart required — confirmed after fixing a PeriodicTaskAdmin form crash found live during this step
• jobs/reconcile error response — malformed threshold_seconds correctly returned 422

🤖 Generated with Claude Code

[github-actions]

Test Results

309 tests  +67   309 ✅ +67   44s ⏱️ +12s
  1 suites ± 0     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit ee1bd8a. ± Comparison against base commit 70433ef.