Move common resample code to stcal #8986

mcara · 2024-11-27T17:00:02Z

This PR add the common resample code used by both JWST and Roman pipelines to stcal. For now only imaging mode was switched to the new code. This PR supersedes #8695

This work is related to https://jira.stsci.edu/browse/AL-835

The code in this PR requires the code from spacetelescope/stcal#320

Tasks

news fragment change types...

changes/<PR#>.general.rst: infrastructure or miscellaneous change
changes/<PR#>.docs.rst
changes/<PR#>.stpipe.rst
changes/<PR#>.datamodels.rst
changes/<PR#>.scripts.rst
changes/<PR#>.fits_generator.rst
changes/<PR#>.set_telescope_pointing.rst
changes/<PR#>.pipeline.rst

stage 1

changes/<PR#>.group_scale.rst
changes/<PR#>.dq_init.rst
changes/<PR#>.emicorr.rst
changes/<PR#>.saturation.rst
changes/<PR#>.ipc.rst
changes/<PR#>.firstframe.rst
changes/<PR#>.lastframe.rst
changes/<PR#>.reset.rst
changes/<PR#>.superbias.rst
changes/<PR#>.refpix.rst
changes/<PR#>.linearity.rst
changes/<PR#>.rscd.rst
changes/<PR#>.persistence.rst
changes/<PR#>.dark_current.rst
changes/<PR#>.charge_migration.rst
changes/<PR#>.jump.rst
changes/<PR#>.clean_flicker_noise.rst
changes/<PR#>.ramp_fitting.rst
changes/<PR#>.gain_scale.rst

stage 2

changes/<PR#>.assign_wcs.rst
changes/<PR#>.badpix_selfcal.rst
changes/<PR#>.msaflagopen.rst
changes/<PR#>.nsclean.rst
changes/<PR#>.imprint.rst
changes/<PR#>.background.rst
changes/<PR#>.extract_2d.rst
changes/<PR#>.master_background.rst
changes/<PR#>.wavecorr.rst
changes/<PR#>.srctype.rst
changes/<PR#>.straylight.rst
changes/<PR#>.wfss_contam.rst
changes/<PR#>.flatfield.rst
changes/<PR#>.fringe.rst
changes/<PR#>.pathloss.rst
changes/<PR#>.barshadow.rst
changes/<PR#>.photom.rst
changes/<PR#>.pixel_replace.rst
changes/<PR#>.resample_spec.rst
changes/<PR#>.residual_fringe.rst
changes/<PR#>.cube_build.rst
changes/<PR#>.extract_1d.rst
changes/<PR#>.resample.rst

stage 3

changes/<PR#>.assign_mtwcs.rst
changes/<PR#>.mrs_imatch.rst
changes/<PR#>.tweakreg.rst
changes/<PR#>.skymatch.rst
changes/<PR#>.exp_to_source.rst
changes/<PR#>.outlier_detection.rst
changes/<PR#>.tso_photometry.rst
changes/<PR#>.stack_refs.rst
changes/<PR#>.align_refs.rst
changes/<PR#>.klip.rst
changes/<PR#>.spectral_leak.rst
changes/<PR#>.source_catalog.rst
changes/<PR#>.combine_1d.rst
changes/<PR#>.ami.rst

other

changes/<PR#>.wfs_combine.rst
changes/<PR#>.white_light.rst
changes/<PR#>.cube_skymatch.rst
changes/<PR#>.engdb_tools.rst
changes/<PR#>.guider_cds.rst

codecov · 2024-11-27T17:44:54Z

Codecov Report

Attention: Patch coverage is 92.75000% with 29 lines in your changes missing coverage. Please review.

Project coverage is 73.57%. Comparing base (a86a74e) to head (747e319).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/resample/resample.py	94.04%	10 Missing ⚠️
jwst/resample/resample_spec_step.py	68.00%	8 Missing ⚠️
jwst/resample/resample_utils.py	88.73%	8 Missing ⚠️
jwst/resample/resample_step.py	80.00%	2 Missing ⚠️
jwst/resample/resample_spec.py	98.92%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8986      +/-   ##
==========================================
- Coverage   73.77%   73.57%   -0.21%     
==========================================
  Files         372      372              
  Lines       37273    37108     -165     
==========================================
- Hits        27498    27302     -196     
- Misses       9775     9806      +31

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

braingram

Thanks for putting this together.

I'd like to see the CI run and regression tests for this PR. I added one comment about psutil which should fix one of the issues preventing the CI from running.

Would you update the pyproject.toml to point to the source branch of spacetelescope/stcal#320? That should (hopefully) allow the CI to run so we can see the coverage and test results with this PR.

Also would you queue up a regtest run?

jwst/resample/resample.py

mcara · 2024-11-28T16:35:12Z

Regression tests pass: https://github.com/spacetelescope/RegressionTests/actions/runs/12059744878

jwst/resample/resample.py

jwst/resample/resample_utils.py

mcara · 2024-12-04T14:41:59Z

Initial regression tests that were using the new code only for imaging are: https://github.com/spacetelescope/RegressionTests/actions/runs/12146883313

After switching spectral code to use the new class, here are the results of regression tests: https://github.com/spacetelescope/RegressionTests/actions/runs/12155412170

I most failures that I see are unrelated to my code, I think (such as test_miri_image_detector1[ramp], test_miri_image_detector1[rate], ...). The spectral test failures have new header cards and these could be OKified. Spectral tests are easily identifiable by Extra keyword 'NDRIZ' - these are spectral tests with extra meta info (related to resample). Imaging tests should be OK.

emolter

This is looking great Mihai! Thanks for all the hard work. Most of these comments are minor.

It still feels a bit sad to have to convert the datamodels into dictionaries to pass into stcal, but I can't really think of a better way to do it while maintaining the observatory-agnostic nature of stcal. So I guess I'm happy with it. Maybe it's a bit easier on the romancal side when working with asdf files as compared with fits?

jwst/outlier_detection/utils.py

jwst/resample/resample.py

jwst/resample/resample_spec.py

emolter · 2024-12-04T22:52:00Z

jwst/resample/resample_spec.py

+        # delete these to reduce the number of failures in the regression tests
+        # TODO: this is helpful info that should be stored in products.
+        # del model.meta.resample.pixel_scale_ratio
+        # del model.meta.resample.pixfrac
+        # del model.meta.resample.weight_type
+        # del model.meta.resample.pointings
+        # del model.meta.cal_step.resample


Can you elaborate on why these cause failures? should these be reinstated before merge?

It seems like currently for spectral data none of the above fields related to resample info are being set. So, if I turn this ON right now, you will see multiple failures in the regression tests. I think these should be turned on before merging, if @melanieclarke is not against storing this info for spectral data.

I am all for storing this info for spectral data! I thought these were added, though, in the ResampleSpecStep? There may be some weirdness around what happens with MultiSlitModels.

jwst/resample/resample_spec.py

jwst/resample/resample_step.py

perrygreenfield

LGTM

jwst/resample/resample.py

braingram · 2025-02-05T19:56:13Z

jwst/resample/resample.py

+            weight_type=weight_type,
+            good_bits=good_bits,
+            output_wcs=output_wcs,
+            output_model=None,


Is this ever not None?

Not at this moment but if Roman will want to be able to "resume" adding new images to an existing resampled image - this is how it would be done.

My vote is to not add that complexity to these PRs.

Roman decided against this mode of operation.
What is needed for Roman is to be able to pass an output WCS object to resample - essentially something like the "custom WCS" but as an object, not file read from disk.

We definitely need to be able to specify the output WCS in normal operations in Roman.

Re adding to an existing stack, I don't think we've really figured out how we want to handle very deep stacks for Roman (GBTDS, HLTDS, general astrophysics). This kind of resume operation would be a good approach, but it's unclear to me when to turn it on and how to change outlier detection when it gets turned on, and it's easy to imagine weird observation patterns that could lead the outlier rejection to be compromised. The alternative we've talked about for deep stacks is "stack of stack" approaches which have appeal but will only scale so far with our current single intermediate "pass" layer.

So on the Roman side I would have described it more as not being part of our nominal baseline and being uncertain than as a clear decision against, FWIW. But I realize that's vague and I don't know what level of complexity supporting it adds to the code!

My preference would be to keep this code. However, given strong feelings for not supporting the ability to continue adding new models to an existing resampled image and given high uncertainty that this will ever be needed I have removed all the code from this and stcal (747e319) PR that supports this.

braingram

I'd like to understand why in_memory is no longer being applied in some conditions with the current PR.

jwst/resample/resample_step.py

…eprecated funct

mcara · 2025-02-07T05:42:50Z

Latest regression tests: https://github.com/spacetelescope/RegressionTests/actions/runs/13193971330

mcara added the resample label Nov 27, 2024

mcara requested review from braingram, perrygreenfield, nden and emolter November 27, 2024 17:00

mcara self-assigned this Nov 27, 2024

mcara requested a review from a team as a code owner November 27, 2024 17:00

github-actions bot added the testing label Nov 27, 2024

mcara force-pushed the new-resample-cls2 branch from adc3ca4 to c7286ee Compare November 27, 2024 17:21

braingram reviewed Nov 27, 2024

View reviewed changes

jwst/resample/resample.py Outdated Show resolved Hide resolved

github-actions bot added the installation label Nov 27, 2024

emolter reviewed Nov 29, 2024

View reviewed changes

jwst/resample/resample.py Outdated Show resolved Hide resolved

emolter reviewed Nov 29, 2024

View reviewed changes

jwst/resample/resample_utils.py Show resolved Hide resolved

github-actions bot added outlier_detection documentation labels Dec 3, 2024

mcara force-pushed the new-resample-cls2 branch from 3df4da3 to 3553412 Compare December 3, 2024 17:27

mcara marked this pull request as draft December 3, 2024 18:28

mcara force-pushed the new-resample-cls2 branch from 1db5ba1 to 679aa51 Compare December 3, 2024 18:29

mcara marked this pull request as ready for review December 4, 2024 14:42

emolter reviewed Dec 4, 2024

View reviewed changes

melanieclarke added this to the Build 11.3 milestone Dec 12, 2024

mcara force-pushed the new-resample-cls2 branch from bb0839e to aff1ded Compare December 15, 2024 11:24

perrygreenfield approved these changes Dec 16, 2024

View reviewed changes

jwst/resample/resample.py Outdated Show resolved Hide resolved

mcara force-pushed the new-resample-cls2 branch from a4301e9 to b37078e Compare January 28, 2025 05:20

mcara requested a review from a team as a code owner January 28, 2025 05:20

mcara force-pushed the new-resample-cls2 branch 2 times, most recently from fa6feda to b13bd51 Compare January 28, 2025 07:50

braingram reviewed Feb 5, 2025

View reviewed changes

mcara force-pushed the new-resample-cls2 branch from 417bc89 to 2c53ccc Compare February 5, 2025 20:04

braingram requested changes Feb 6, 2025

View reviewed changes

jwst/resample/resample_step.py Outdated Show resolved Hide resolved

mcara force-pushed the new-resample-cls2 branch from 025406c to ee16194 Compare February 6, 2025 21:20

mcara added 21 commits February 6, 2025 16:55

Move common resample code to stcal

02441fe

fix style

9cc6d26

add temporary dependency on stcal my fork

2cdb4f9

remove available memory check

e145265

add changelog

9ea4722

update spec code to use new classes

72b7822

fix test failures and pep8

08a3183

fix diff in s_region for spec

baa3905

Address reviewer comments

f592dc4

Address reviewer comments - 2

1edd94b

function name change

8f9d384

finish changes to parameter name: wht_type->weight_type

81acd54

Address reviewer comments

33b3e50

Deprecate build_driz_weight -> stcal; move load_custom_wcs -> utils

91382be

remove drizzle as dependency

bcf2053

Remove reproject and reformat deprecation code

86af1a9

Fix deprecation warnings and docstrings. Add drizzle dependency for d…

57d31f5

…eprecated funct

add comment to pyproject

91f89dd

Address reviewers comments

48c2ac3

Address reviewers comments

ee3657c

clarify finalized; add locked flag

727ec4b

mcara force-pushed the new-resample-cls2 branch from ee16194 to 727ec4b Compare February 6, 2025 21:55

mcara added 3 commits February 6, 2025 23:00

Limit support for output_wcs only to dict

85239f9

Correct comment about in_memory

73ccb29

Remove code related to resuming resampling onto an existing output model

747e319

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move common resample code to stcal #8986

Move common resample code to stcal #8986

mcara commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

braingram left a comment

mcara commented Nov 28, 2024

mcara commented Dec 4, 2024 •

edited

Loading

emolter left a comment

emolter Dec 4, 2024

mcara Dec 12, 2024

melanieclarke Dec 12, 2024

perrygreenfield left a comment

braingram Feb 5, 2025

mcara Feb 5, 2025

braingram Feb 5, 2025

nden Feb 5, 2025

schlafly Feb 6, 2025

mcara Feb 7, 2025 •

edited

Loading

braingram left a comment

mcara commented Feb 7, 2025

Move common resample code to stcal #8986

Are you sure you want to change the base?

Move common resample code to stcal #8986

Conversation

mcara commented Nov 27, 2024 • edited Loading

Tasks

stage 1

stage 2

stage 3

other

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

braingram left a comment

Choose a reason for hiding this comment

mcara commented Nov 28, 2024

mcara commented Dec 4, 2024 • edited Loading

emolter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

perrygreenfield left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcara Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

braingram left a comment

Choose a reason for hiding this comment

mcara commented Feb 7, 2025

mcara commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

mcara commented Dec 4, 2024 •

edited

Loading

mcara Feb 7, 2025 •

edited

Loading