Skip to content

Add project-scoped calibrations, native footprints, and S3 storage - #32

Merged
davecoulter merged 2 commits into
mainfrom
feature/schema-storage-spatial-types
Jul 28, 2026
Merged

davecoulter merged 2 commits into
mainfrom
feature/schema-storage-spatial-types

Conversation

@davecoulter

Copy link
Copy Markdown
Owner

Summary

  • Scope Level2Calibration to Project (composite unique on level2_image_id+project_id, replacing the old strict 1:1) so the same raw image can carry a distinct calibration per project; enforce Level2Image.base_filename uniqueness.
  • Replace the Text moc_str footprint columns with a native int8multirange footprint column (src/db/spatial_types.MOCType), resolving GitHub issue Migrate Tile.healpix_index to a native range type #26 without depending on healpix-alchemy — confirmed unmaintained since 2021 and unimportable against modern mocpy. GiST-indexed; a live && overlap query proves the indexed spatial-match capability the migration exists for.
  • Add S3StorageService (boto3) + a local MinIO service in docker-compose.yml for FITS storage, swappable to real S3 via env vars.
  • Add a real (unmocked) MinIO integration test and the matching CI wiring (.github/workflows/ci.yml) — CI previously had no object storage at all.
  • Housekeeping: drop the broken healpix-alchemy dependency; correct now-stale "HealpixAlchemy" references across docs and governance files (CLAUDE.md, README.md, architecture docs, the infra topology diagram).

CLI Verification

All steps completed prior to PR creation via genTests.

  • Branch confirmed: feature/schema-storage-spatial-types
  • Images rebuilt: docker compose build api worker_light worker_heavy
  • Containers recreated: docker compose up -d api worker_light worker_heavy
  • Migration 0008 applied (alembic upgrade head → 0008 (head); round-tripped via downgrade -1/upgrade head to prove the corrected migration, including the GiST indexes, applies cleanly — alembic check reports no drift)
  • Test suite passed in-container: docker compose exec api uv run pytest --cov=src --cov-fail-under=90 -q → 136 passed, 1 legitimate host-only skip, 98.94% coverage
  • Sphinx build passed (local, -W)
  • docs/diagrams/infrastructure_topology.md updated (new minio node + edges, de-stale'd db-node label) and visually reviewed in rendered HTML

Application QA

All steps completed and verified prior to PR creation via genTests.

  • base_filename uniqueness — duplicate insert correctly rejected with IntegrityError
  • Project-scoped calibration — same image + two different projects both succeeded; same image + same project correctly rejected
  • Footprint fidelity + native overlap query — stored/reloaded MOC matched on sky_fraction and empty symmetric difference; a live && query returned only the spatially-overlapping calibration
  • MinIO console reachable — bucket browsable at localhost:9001, confirmed visually
  • S3StorageService live round-trip over the real container network — uploaded/downloaded content matched exactly
  • Infra failure injection — upload against a stopped MinIO failed fast and cleanly (EndpointConnectionError, no hang); succeeded again immediately after restart

🤖 Generated with Claude Code

Doc 27: calibrations can now vary per project (not just per image),
footprints move from a dead-dependency Text column to a native
queryable PostgreSQL range type, and FITS storage moves to
S3-compatible object storage (MinIO locally). Establishes the
schema/storage foundation the ingest service (doc 28) depends on.
@github-actions

github-actions Bot commented Jul 28, 2026 •

Copy link
Copy Markdown

Test Results

138 tests  +10   138 ✅ +10   9s ⏱️ -1s
  1 suites ± 0     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 9c33efd. ± Comparison against base commit 7cabc26.

This pull request removes 2 and adds 12 tests. Note that renamed tests count towards both.
src.db.tests.test_integration ‑ test_level2_image_and_calibration_one_to_one
src.db.tests.test_integration ‑ test_level2_image_calibration_is_one_per_image
src.db.tests.test_integration ‑ test_base_filename_is_unique
src.db.tests.test_integration ‑ test_footprint_moc_roundtrip_fidelity
src.db.tests.test_integration ‑ test_footprint_none_and_empty_and_update_paths
src.db.tests.test_integration ‑ test_footprint_overlap_query_returns_only_spatial_matches
src.db.tests.test_integration ‑ test_level2_calibration_roundtrip_and_back_reference
src.db.tests.test_integration ‑ test_same_image_different_projects_allows_two_calibrations
src.db.tests.test_integration ‑ test_same_image_same_project_calibration_is_rejected
src.services.tests.test_storage_service ‑ test_download_file_delegates_to_client_with_bucket
src.services.tests.test_storage_service ‑ test_init_builds_client_from_settings
src.services.tests.test_storage_service ‑ test_upload_file_delegates_to_client_with_bucket
…

♻️ This comment has been updated with latest results.

@davecoulter
davecoulter merged commit ddf8466 into main Jul 28, 2026
3 checks passed
@davecoulter
davecoulter deleted the feature/schema-storage-spatial-types branch July 28, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant