Skip to content

Migrate to psycopg3, add point-HEALPix indexing, remove DummyImage - #38

Merged
davecoulter merged 1 commit into
mainfrom
feature/psycopg3-healpix-dummy-cleanup
Jul 29, 2026
Merged

davecoulter merged 1 commit into
mainfrom
feature/psycopg3-healpix-dummy-cleanup

Conversation

@davecoulter

Copy link
Copy Markdown
Owner

Summary

  • Migrated the database driver from psycopg2 to psycopg3, enabling native
    range/multirange binding and eliminating MOCType's text-literal
    serialization workaround (resolves Evaluate migrating the DB driver from psycopg2 to psycopg3 #30).
  • Added a native depth-29 point-HEALPix range type (PointHEALPixType) to
    Tile, Level2Image, and Level3Mosaic, enabling an indexed
    footprint-contains-point spatial query; added IngestBatch.job_configuration_id
    ahead of doc 30's watchdog work (resolves Migrate Tile.healpix_index to a native range type #26).
  • Fully decommissioned the Stage-0 DummyImage scaffolding (model, service
    functions, task, CLI commands, routes, admin view, seed data); simplified
    DiffpypeTask.on_failure to be entity-agnostic now that nothing but
    DummyImage ever used its status-write (resolves Decommission Stage-0 DummyImage scaffolding #33).

CLI Verification

All steps completed prior to PR creation via genTests.

  • Branch confirmed: feature/psycopg3-healpix-dummy-cleanup
  • Images rebuilt: docker compose build api worker_light worker_heavy
  • Containers recreated: docker compose up -d api worker_light worker_heavy
  • Migrations applied: 0011 (point-HEALPix columns/indexes, IngestBatch FK), 0012
    (drop dummy_images) — verified via alembic current and a full downgrade/upgrade
    round-trip through both migrations
  • Test suite passed: docker compose exec api uv run pytest --cov=src --cov-fail-under=90 -q
    (242 passed, 99.4% coverage)
  • Sphinx build passed (local)
  • Pre-commit passed (ruff, ruff-format, mypy)
  • docs/diagrams/infrastructure_topology.md — N/A, no docker-compose service or
    container-layer changes

Application QA

All steps completed and verified prior to PR creation via genTests.

  • Live psycopg3 write/read smoke test — created a real Project via CLI inside the
    container; the INSERT+SELECT round-tripped cleanly through the new driver.
  • Ingest populates Level2Image.healpix_index — ingested 717 real FITS files;
    every resulting row had healpix_index populated, none NULL.
  • Tile creation populates Tile.healpix_index — tessellated and created 54 real
    tiles over live ingested data; every tile had healpix_index populated, and
    tile↔calibration associations formed correctly against real overlapping footprints.
  • Mosaic creation populates Level3Mosaic.healpix_index — clustered a real epoch
    and created a mosaic from real constituent calibrations; ra/decl/healpix_index
    all populated as expected for the has-constituents case.
  • Live footprint @> healpix_index containment query — ran the real GiST-indexed
    containment query against live tile/image data; EXPLAIN confirmed it uses the
    index, not a sequential scan.
  • Schema check — confirmed all three new GiST indexes and the
    IngestBatch.job_configuration_id FK exist exactly as specified.
  • DummyImage decommission — confirmed the dummy_images table, /jobs/dummy
    routes, and run-dummy/get-dummy CLI commands are all fully removed (404s and
    argparse errors as expected).
  • on_failure's simplified behavior — dispatched a real Celery failure through
    execute_cli_tool; confirmed logging + dead-letter dispatch fire correctly
    end-to-end with no DB-related crash, verified in both worker logs and Flower.

🤖 Generated with Claude Code

Doc 29 completes the Stage 1 database foundation: psycopg3 replaces
psycopg2, enabling native range/multirange binding and eliminating
MOCType's text-literal serialization workaround. Tile, Level2Image,
and Level3Mosaic gain an indexed point-in-footprint containment
query via the new PointHEALPixType. The Stage-0 DummyImage
scaffolding is fully decommissioned now that IngestBatch/Level3Mosaic
are the real async-tracking pattern.
@github-actions

Copy link
Copy Markdown

Test Results

242 tests   - 26   242 ✅  - 26   34s ⏱️ ±0s
  1 suites ± 0     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 8307c00. ± Comparison against base commit 37da9b2.

This pull request removes 42 and adds 16 tests. Note that renamed tests count towards both.
src.api.tests.test_cli ‑ test_cmd_get_dummy_closes_session
src.api.tests.test_cli ‑ test_cmd_get_dummy_prints_elapsed_run_time
src.api.tests.test_cli ‑ test_cmd_get_dummy_prints_error_for_missing_image
src.api.tests.test_cli ‑ test_cmd_get_dummy_prints_table_for_found_image
src.api.tests.test_cli ‑ test_cmd_run_dummy_calls_dispatch_with_config_and_closes_session
src.api.tests.test_cli ‑ test_cmd_run_dummy_logs_job_id_to_stdout
src.api.tests.test_cli ‑ test_elapsed_label_none_when_no_timestamps
src.api.tests.test_cli ‑ test_elapsed_label_queue_time_when_pending
src.api.tests.test_cli ‑ test_elapsed_label_run_time_when_finished
src.api.tests.test_cli ‑ test_elapsed_label_run_time_when_still_running
…
src.api.tests.test_main ‑ test_old_unversioned_projects_path_returns_404
src.db.tests.test_integration ‑ test_footprint_contains_healpix_index_point_query
src.db.tests.test_integration ‑ test_ingest_batch_job_configuration_nullable
src.db.tests.test_integration ‑ test_ingest_batch_job_configuration_relationship
src.db.tests.test_integration ‑ test_ingest_batch_status_roundtrip
src.db.tests.test_integration ‑ test_sysadmin_seeding_creates_sysadmin_user
src.db.tests.test_spatial_types ‑ test_moc_type_binds_a_native_multirange_and_reads_back_a_moc
src.db.tests.test_spatial_types ‑ test_moc_type_none_passes_through_both_directions
src.db.tests.test_spatial_types ‑ test_point_healpix_bind_and_result_round_trip
src.db.tests.test_spatial_types ‑ test_point_healpix_bind_encodes_tuple_as_single_cell_range
…

@davecoulter
davecoulter merged commit 70433ef into main Jul 29, 2026
3 checks passed
@davecoulter
davecoulter deleted the feature/psycopg3-healpix-dummy-cleanup branch July 29, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant