Repository navigation
Populate domain graph: ingest, tiles, epochs, mosaics - #36
Merged
Merged
Conversation
Populates the doc 26 domain schema end-to-end: real FITS ingest, spatial tiling, MJD-based epoch clustering, and mosaic job dispatch, so the future Aladin UI and JWST pipeline docs have real, tested services and data to build on instead of empty tables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ingest_service,tile_service,epoch_service,mosaic_service) plusproject_service, each with a matching FastAPI route anddiffpype-manageCLI command (API/CLI parity throughout), and apopulate-demo-projectCLI coordinator that drives all four end-to-end for dev/demo use.Project.slug+IngestBatch(0009), andLevel3Mosaic.ra/declbarycenter columns closing GitHub issue Add computed barycenter coords to Level3Mosaic #27 (0010).run_ingest_batchstreams one file at a time (download -> parse -> delete, incremental progress) rather than materializing a whole batch first; bothrun_ingest_batchandrun_mosaic_drizzleroute toheavy_memory. Both dispatch-time service functions now mark their job rowFAILED(instead of leaving it orphanedPENDING) if the Celery broker is unreachable at dispatch.NO_COLOR-aware) and a curated, salient column set formosaic-status(no raw MOC dump, no bare FK ids);MosaicStatus's API response drops the same footprint payload for the same reason.tile_service.tile_with_most_calibrationsreplaces a naive "first tile in the list" selection with one chosen by actual data overlap.S3StorageServiceno longer forces empty/local-only credential kwargs onto boto3, so it works against both MinIO (static keys) and real AWS (default credential chain) unchanged.CLI Verification
All steps completed prior to PR creation via
genTests.feature/domain-graph-populationdocker compose build api worker_light worker_heavydocker compose up -dalembic upgrade head, verified via downgrade/upgrade round-trip)uv run pytest --cov=src --cov-fail-under=90 -q— 227 passed, 94.4% coverage-W, zero warnings)pre-commit run --all-files— ruff, ruff-format, mypy all cleandocs/diagrams/infrastructure_topology.md— N/A, no service/topology change (local storage bind mount is a volume addition to existing services, not a new service or inter-service connection)Application QA
All steps completed and verified prior to PR creation via
genTests, against real ingested JWST FITS data (not synthetic fixtures).docker compose kill worker_heavyinjection) — confirmed row correctly surfaces as stuckin_process, no silent auto-completion on worker restartpending; now markedfailed)populate-demo-projectfull coordinator run — confirmed end-to-end (ingest -> tiles -> epochs -> mosaic allcomplete) after fixing a naive tile-selection bugNO_COLORoverride respected🤖 Generated with Claude Code