2935 hi l1c backgrounds utilities by subagonsouth · Pull Request #2936 · IMAP-Science-Operations-Center/imap_processing

subagonsouth · 2026-04-08T19:38:03Z

Summary

Adds background configuration infrastructure for HI L1C processing. Refactors configuration accessor pattern to support both calibration products and backgrounds with shared validation logic.

Key Changes

`imap_processing/hi/utils.py`

Refactored configuration accessors: Created _BaseConfigAccessor base class to eliminate code duplication between CalibrationProductConfig and new BackgroundConfig
Added BackgroundConfig accessor: Pandas DataFrame accessor for background configuration
Added background filtering utilities: iter_background_events_by_config() generator and _filter_events_by_config_row() helper
Bug fixes:
- Fixed validation logic: if index_name in df.index → if index_name not in df.index.names
- Fixed pandas FutureWarning in EsaEnergyStepLookupTable by using explicit dtypes

`imap_processing/tests/hi/test_utils.py`

Added TestBackgroundConfig class with 7 tests covering CSV loading, validation, rate/uncertainty calculations
Fixed TestCalibrationProductConfig.test_wrong_columns to properly test column validation

`imap_processing/tests/hi/conftest.py`

Added hi_test_background_config_path fixture

Related Issues

Closes: #2935

Add background counts binning helper to hi utils

Copilot

Pull request overview

This PR adds initial infrastructure for HI L1C “background” configuration handling by introducing a shared DataFrame accessor base class, a new BackgroundConfig accessor, and utilities to filter direct-event datasets by background config selections.

Changes:

Refactors existing HI calibration-product DataFrame accessor logic into a shared _BaseConfigAccessor and updates CalibrationProductConfig to use it.
Introduces BackgroundConfig DataFrame accessor with CSV loading + validation and adds background event filtering generator/helpers.
Updates/extends HI unit tests and fixtures to cover the new background config accessor and improves column-validation testing.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
`imap_processing/hi/utils.py`	Adds `_BaseConfigAccessor`, introduces `BackgroundConfig`, and adds background event filtering utilities; also adjusts lookup table DataFrame initialization dtypes.
`imap_processing/tests/hi/test_utils.py`	Adds tests for `BackgroundConfig` CSV loading/validation and fixes `CalibrationProductConfig` wrong-column test setup.
`imap_processing/tests/hi/conftest.py`	Adds a fixture for a backgrounds config CSV path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

imap_processing/tests/hi/conftest.py

imap_processing/hi/utils.py

Copilot · 2026-04-08T19:42:30Z

imap_processing/hi/utils.py

    @classmethod
    def from_csv(cls, path: str | Path) -> pd.DataFrame:
        """
-        Read configuration CSV file into a pandas.DataFrame.
+        Read calibration product configuration CSV file into a pandas.DataFrame.

        Parameters
        ----------
        path : str or pathlib.Path
-            Location of the Calibration Product configuration CSV file.
+            Location of the calibration product configuration CSV file.



Type hint for from_csv is path: str | Path, but callers/tests pass file-like objects (e.g., io.StringIO) which pd.read_csv supports. Consider widening the annotation (and docstring wording) to include file-like inputs (e.g., str | Path | IO[str]) to reflect the actual supported API.

Copilot · 2026-04-08T19:42:30Z

imap_processing/hi/utils.py

+    @classmethod
+    def from_csv(cls, path: str | Path) -> pd.DataFrame:
        """
-        Get the calibration product numbers from the current configuration.
+        Read background configuration CSV file into a pandas.DataFrame.
+
+        Parameters
+        ----------
+        path : str or pathlib.Path
+            Location of the background configuration CSV file.



Type hint for BackgroundConfig.from_csv is path: str | Path, but tests pass file-like objects (e.g., io.StringIO) which pd.read_csv supports. Consider widening the annotation (and docstring wording) to include file-like inputs so the public API matches real usage.

laspsandoval

Just a minor suggestion.

laspsandoval · 2026-04-09T19:52:01Z

imap_processing/hi/utils.py

+        "coincidence_type_list",
+        *[
+            f"tof_{det_pair}_{limit}"
+            for det_pair in tof_detector_pairs


for det_pair in _BaseConfigAccessor.tof_detector_pairs

Yeah, good catch. That was a bit sloppy :)

laspsandoval · 2026-04-09T19:52:22Z

imap_processing/hi/utils.py

+        "calibration_prod",
+        "esa_energy_step",
+    )
+    tof_detector_pairs = ("ab", "ac1", "bc1", "c1c2")


Remove the above line.

laspsandoval · 2026-04-09T19:53:58Z

imap_processing/hi/utils.py

+        "coincidence_type_list",
+        *[
+            f"tof_{det_pair}_{limit}"
+            for det_pair in CalibrationProductConfig.tof_detector_pairs


for det_pair in _BaseConfigAccessor.tof_detector_pairs

laspsandoval · 2026-04-09T19:54:12Z

imap_processing/hi/utils.py

+        "calibration_prod",
+        "background_index",
+    )
+    tof_detector_pairs = CalibrationProductConfig.tof_detector_pairs


Remove above.

subagonsouth added 3 commits April 8, 2026 13:23

Add background config to hi utils

57627b7

Add background counts binning helper to hi utils

Add fixture to conftest

1325ea5

Remove unused accessor functions

0802773

subagonsouth requested a review from Copilot April 8, 2026 19:38

subagonsouth self-assigned this Apr 8, 2026

subagonsouth added this to IMAP Apr 8, 2026

Copilot started reviewing on behalf of subagonsouth April 8, 2026 19:38 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

subagonsouth added 2 commits April 8, 2026 14:02

Add background config test file

2991c39

Copilot feedback changes

e565f67

subagonsouth requested review from laspsandoval, sdhoyt and vmartinez-cu April 8, 2026 20:16

subagonsouth added 3 commits April 8, 2026 15:25

Fix documentation build

becbf1b

Fix typo in csv comment section

49db01a

Add proper unit tests for utils iter functions

4b42c01

laspsandoval approved these changes Apr 9, 2026

View reviewed changes

Clean up Config definitions

77fbaf3

subagonsouth merged commit a5dec5a into IMAP-Science-Operations-Center:dev Apr 9, 2026
14 checks passed

subagonsouth deleted the 2935-hi-l1c-backgrounds---utilities branch April 9, 2026 21:01

github-project-automation bot moved this to Done in IMAP Apr 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2935 hi l1c backgrounds utilities#2936

2935 hi l1c backgrounds utilities#2936
subagonsouth merged 9 commits intoIMAP-Science-Operations-Center:devfrom
subagonsouth:2935-hi-l1c-backgrounds---utilities

subagonsouth commented Apr 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

laspsandoval left a comment

Uh oh!

laspsandoval Apr 9, 2026

Uh oh!

subagonsouth Apr 9, 2026

Uh oh!

laspsandoval Apr 9, 2026

Uh oh!

laspsandoval Apr 9, 2026

Uh oh!

laspsandoval Apr 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

subagonsouth commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

imap_processing/hi/utils.py

imap_processing/tests/hi/test_utils.py

imap_processing/tests/hi/conftest.py

Related Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

laspsandoval left a comment

Choose a reason for hiding this comment

Uh oh!

laspsandoval Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

subagonsouth Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

laspsandoval Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

laspsandoval Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

laspsandoval Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

subagonsouth commented Apr 8, 2026 •

edited

Loading

`imap_processing/hi/utils.py`

`imap_processing/tests/hi/test_utils.py`

`imap_processing/tests/hi/conftest.py`