task: Add more SPICE frames for MAG L2 processing and fix issue with SPICE epochs missing by mfacchinelli · Pull Request #2378 · IMAP-Science-Operations-Center/imap_processing

mfacchinelli · 2025-11-04T16:30:14Z

Change Summary

Overview

This change:

Adds transformation of MAG L2 data to 3 new frames (GSE, GSM, RTN)
Fixes a few issues with processing MAG L2 data due to:
a. DSRF transformation not being defined during periods of thruster firing (transformation matrix is now all NaNs)
b. Truncates L2 data to the specified day at the beginning of L2 processing
c. Truncation also crops epoch_et

Note

The current implementation will require a lot of memory for processing Burst mode in 128 cadence (as the data for 1 entire day will be held in memory for 5 different frames).

New Dependencies

N/A

New Files

N/A

Deleted Files

N/A

Updated Files

imap_mag_global_cdf_attrs.yaml
- Adds definition for GSM reference frame
mag_l2.py:
- Adds conversion to all supported reference frames
mag_l2_data.py:
- Adds enum for GSM reference frame
- Also crops epoch_et when truncating
geometry.py:
- If SPICE tranformation fails becuase of SpiceNOFRAMECONNECT, i.e., no SPICE data is available for a specific timestamp (this happens every day between 10 and 10:30 AM for the despun reference frame, due to thruster firing), a NaN matrix is returned, instead of an error being thrown
test_mag_l2.py:
- Modifies existing test to expect 5 datasets for each frame
- Adds a test to make sure NaN matrices are supported
test_geometry.py:
- Adds a test to make sure get_rotation_matrix returns NaN matrix when despun frame not defined for input et

Testing

Testing was added to cover all cases.

…certain epoch, instead NaN matrix is returned

maxinelasp · 2025-11-04T17:03:07Z

Relates to: #2297, #2295

tmplummer

I know this is still just a draft, but I thought that I would leave some comments anyway.

tmplummer · 2025-11-04T20:56:34Z

+    ):  # numpydoc ignore=GL08
+        try:
+            return spiceypy.pxform(*arg, **kwargs)
+        except spiceypy.utils.exceptions.SpiceNOFRAMECONNECT:


This implementation concerns me. I want this function to fail loudly by default rather than just logging an exception level message. I would be more comfortable with adding a keyword argument called something like allow_spice_noframeconnect that defaults to False.

try: return spiceypy.pxform(*arg, **kwargs) except spiceypy.utils.exceptions.SpiceNOFRAMECONNECT as e: if not allow_spice_noframeconnect: raise(e) logger.debug(f"Returning NaN rotation matrix due to spiceypy error: {e}") return np.full((3, 3), np.nan)

Makes sense, done!

tmplummer · 2025-11-04T21:01:23Z

+@pytest.mark.external_kernel
+def test_get_rotation_matrix_no_transformation_defined_for_et(furnish_kernels):
+    """Test error handling in get_rotation_matrix()."""
+    kernels = [
+        "naif0012.tls",
+        "imap_100.tf",
+        "imap_sclk_0000.tsc",
+        "imap_science_100.tf",
+        "sim_1yr_imap_attitude.bc",
+        "sim_1yr_imap_pointing_frame.bc",
+        "de440s.bsp",
+    ]
+    with furnish_kernels(kernels):
+        # Midnight is not defined in pointing frame
+        et = spiceypy.utc2et("2029-01-01T00:00:00.000")
+        rotation = get_rotation_matrix(et, SpiceFrame.IMAP_MAG_O, SpiceFrame.IMAP_DPS)
+        assert np.isnan(rotation).all()


It would be good to also check that the vectorized form works:

# one hour after midnight should have coverage ets = np.array([et, et + 3600]) rotations = get_rotation_matrix(ets, SpiceFrame.IMAP_MAG_O, SpiceFrame.IMAP_DPS) assert rotations.shape == (2, 3, 3) assert np.isnan(rotations[0]).all() assert np.isfinite(rotations[1]).all()

I've added this test, and a test for the default allow_spice_noframeconnect value, where we throw an error.

tmplummer · 2025-11-04T21:11:13Z

+        try:
+            l2_data.rotate_frame(frame)
+            frames.append(l2_data.generate_dataset(attributes, day))
+        except Exception:


Broad exceptions like this are generally bad and in this case all details about what caused the failure to rotate the data to the desired frame is lost.

Are there specific reasons you expect this to fail and can you handle those specific exceptions?

I've removed this as this change addresses the failures we have seen in the past.

Btw, the logger.exception method does log the exception information automatically (including stacktrace), without having to pass the exception directly:
https://docs.python.org/3/library/logging.html#logging.Logger.exception

The same is true when passing exc_info to other logging methods, like I've done for the debug call.

…nect`

mfacchinelli · 2025-11-05T10:52:42Z

I know this is still just a draft, but I thought that I would leave some comments anyway.

Thanks Tim, I've implemented the feedback. I'll get it tested in the dev environment with the files that have failed in the past.

tmplummer · 2025-11-05T17:12:16Z

This all looks reasonable to me. @mfacchinelli, will you please convert this to a regular PR, as opposed to a Draft, and add @maxinelasp and @greglucas as additional reviewers?

mfacchinelli · 2025-11-05T17:14:28Z

This all looks reasonable to me. @mfacchinelli, will you please convert this to a regular PR, as opposed to a Draft, and add @maxinelasp and @greglucas as additional reviewers?

I can't add reviewers, but the tag should also work!

maxinelasp

This looks good to me! I am adding the SPICE kernels as required for L2 that I missed (ephemeris kernels). Let me know when you're done testing and ready to merge it. Thanks for the fix!

maxinelasp · 2025-11-06T22:00:49Z

    DSRF = SpiceFrame.IMAP_DPS
    SRF = SpiceFrame.IMAP_SPACECRAFT
    GSE = SpiceFrame.IMAP_GSE
+    GSM = SpiceFrame.IMAP_GSM


Do we need to add GSM for the L1D processing as well?

No, GSM is not used for L1d

mfacchinelli · 2025-11-07T09:47:53Z

This looks good to me! I am adding the SPICE kernels as required for L2 that I missed (ephemeris kernels). Let me know when you're done testing and ready to merge it. Thanks for the fix!

Thanks @maxinelasp. I've tested it locally, and it generated all the expected L2 files. Mhairi had a look at them and she was happy.

For me this is ready to merge.

mfacchinelli added 4 commits November 4, 2025 15:37

fix: getting rotation matrix not fail if no spice is available for a …

4ab8931

…certain epoch, instead NaN matrix is returned

fix: also truncate epoch_et

79d6bbc

task: add support for all L2 frames

5d20e15

test: add tests for new L2 functionality

4b65b28

mfacchinelli changed the title ~~task: Add more SPICE frames for MAG L2 processing and fix issues with SPICE epochs missing~~ task: Add more SPICE frames for MAG L2 processing and fix issue with SPICE epochs missing Nov 4, 2025

test: add coverage for frame transformation error logging

72f385b

tmplummer reviewed Nov 4, 2025

View reviewed changes

mfacchinelli added 2 commits November 5, 2025 10:23

task: apply Tim's feedback

7b7e282

test(fix): L1d frame transformation also uses `allow_spice_noframecon…

93b9066

…nect`

alastairtree mentioned this pull request Nov 5, 2025

BUG - MAG L2 file generation fails with spice error #2309

Closed

mfacchinelli added 3 commits November 5, 2025 11:28

fix: pass exception directly

933a8a2

test: make sure matrices are not NaN

f1229f4

fix: add Burst GSM reference frame

8fa75c3

mfacchinelli marked this pull request as ready for review November 5, 2025 17:13

tmplummer requested review from greglucas and maxinelasp November 5, 2025 22:12

maxinelasp approved these changes Nov 6, 2025

View reviewed changes

mfacchinelli mentioned this pull request Nov 7, 2025

L2 data can be generated by the SDC ImperialCollegeLondon/imap-pipeline-core#178

Closed

6 tasks

maxinelasp merged commit 786936a into IMAP-Science-Operations-Center:dev Nov 7, 2025
14 checks passed

This was referenced Nov 17, 2025

MAG: L2 produced in all reference frames #2295

Closed

Mag L1D Repair attributes #2441

Merged

Conversation

mfacchinelli commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Overview

New Dependencies

New Files

Deleted Files

Updated Files

Testing

Uh oh!

maxinelasp commented Nov 4, 2025

Uh oh!

tmplummer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfacchinelli commented Nov 5, 2025

Uh oh!

tmplummer commented Nov 5, 2025

Uh oh!

mfacchinelli commented Nov 5, 2025

Uh oh!

maxinelasp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfacchinelli commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mfacchinelli commented Nov 4, 2025 •

edited

Loading

mfacchinelli commented Nov 7, 2025 •

edited

Loading