Skip to content

Refactor MESSAGEix-Materials build#475

Open
macflo8 wants to merge 40 commits intomainfrom
enh/materials-clean-up
Open

Refactor MESSAGEix-Materials build#475
macflo8 wants to merge 40 commits intomainfrom
enh/materials-clean-up

Conversation

@macflo8
Copy link
Contributor

@macflo8 macflo8 commented Feb 10, 2026

This PR refactors the model building code of MESSAGEix-Materials and updates some parameter sets. It does not alter the model structure of MESSAGEix-Materials.

The following changes have been made:

  • Remove deprecated functions and CLI commands
  • Add type hints and docstrings to all functions
  • Drop .model.material.util pycountry functions in favour of .model.util.pycountry
  • Reduce set of parameters with year_act & year_vtg dimension by entries where: $y^A - y^V > \text{technical\_lifetime}$
  • Add separate input files for emission factors and relations
  • Convert .xlsx input files to .csv
  • Move all "other industry" parametrisation functions to .model.material.data_other_industry
  • Refactor share_constraints.py

Effective outcome:

  • Refactoring reduced build run time by ~35% (~6.5 vs ~4 mins) without power sector build and 75% with power sector build (~16 vs ~4 mins)
  • Parameter data set row count of aluminum, cement and steel models reduced by ~40%

How to review

  1. Read the diff and note that the CI checks all pass.
  2. Look at the materials doc in the ReadTheDocs preview build of the documentation —in particular, the entry in “What's New” and linked entries.
  3. Ensure that changes/additions are self-documenting, i.e. that another
    developer (someone like the reviewer) will be able to understand what the code
    does in the future.

PR checklist

  • Continuous integration checks all ✅
  • Add or expand tests; coverage checks both ✅
  • Add, expand, or update documentation.
  • Update doc/whatsnew.

@macflo8 macflo8 self-assigned this Feb 10, 2026
@macflo8 macflo8 added material MESSAGEix-Materials variant enh New features or functionality labels Feb 10, 2026
@macflo8 macflo8 force-pushed the enh/materials-clean-up branch from 75a4f2a to f83d20d Compare February 10, 2026 18:52
@macflo8 macflo8 force-pushed the enh/materials-clean-up branch from f83d20d to 6b9418b Compare February 10, 2026 20:05
macflo8 added a commit that referenced this pull request Feb 11, 2026
macflo8 added a commit that referenced this pull request Feb 11, 2026
@macflo8 macflo8 force-pushed the enh/materials-clean-up branch from 293cf41 to 64fca17 Compare February 11, 2026 12:34
@codecov
Copy link

codecov bot commented Feb 11, 2026

Codecov Report

❌ Patch coverage is 77.08333% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.2%. Comparing base (54fe267) to head (a12464b).

Files with missing lines Patch % Lines
...ge_ix_models/model/material/data_other_industry.py 17.3% 38 Missing ⚠️
message_ix_models/model/material/data_generic.py 30.7% 27 Missing ⚠️
...sage_ix_models/model/material/data_power_sector.py 67.9% 17 Missing ⚠️
...sage_ix_models/model/material/share_constraints.py 72.5% 14 Missing ⚠️
message_ix_models/model/material/build.py 50.0% 6 Missing ⚠️
message_ix_models/model/material/data_steel.py 94.2% 4 Missing ⚠️
message_ix_models/model/material/data_aluminum.py 97.9% 1 Missing ⚠️
message_ix_models/model/material/data_ammonia.py 50.0% 1 Missing ⚠️
message_ix_models/model/material/data_methanol.py 0.0% 1 Missing ⚠️
message_ix_models/model/material/data_petro.py 98.7% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main    #475     +/-   ##
=======================================
+ Coverage   69.5%   73.2%   +3.6%     
=======================================
  Files        300     301      +1     
  Lines      24598   24254    -344     
=======================================
+ Hits       17104   17755    +651     
+ Misses      7494    6499    -995     
Files with missing lines Coverage Δ
message_ix_models/model/material/cli.py 23.5% <100.0%> (-0.5%) ⬇️
message_ix_models/model/material/data_cement.py 95.2% <100.0%> (-0.1%) ⬇️
message_ix_models/model/material/data_util.py 82.5% <100.0%> (+41.0%) ⬆️
...ssage_ix_models/model/material/demand/math_util.py 83.3% <ø> (ø)
message_ix_models/model/material/util.py 80.9% <100.0%> (+6.1%) ⬆️
...ssage_ix_models/tests/model/material/test_build.py 91.3% <100.0%> (ø)
..._models/tests/model/material/test_data_aluminum.py 100.0% <100.0%> (ø)
...ix_models/tests/model/material/test_data_cement.py 100.0% <ø> (ø)
..._ix_models/tests/model/material/test_data_petro.py 100.0% <ø> (ø)
...els/tests/model/material/test_data_power_sector.py 100.0% <100.0%> (ø)
... and 12 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Add ``bound_activity_up`` for furnace_elec_cement to timeseries_R12 in
generic_furnace_boiler_techno_economic.xlsx
- Remove in-code generation of same parameter data
- Generate cement CCS emission relation in .material.data_cement
- Remove from .material.build
@macflo8 macflo8 force-pushed the enh/materials-clean-up branch from 4c4ec18 to a12464b Compare February 18, 2026 13:43
@macflo8 macflo8 marked this pull request as ready for review February 18, 2026 13:44
@khaeru
Copy link
Member

khaeru commented Feb 18, 2026

To make things slightly easier for reviewers:

@macflo8
Copy link
Contributor Author

macflo8 commented Feb 19, 2026

Thanks, @khaeru for adding the links and instructions!
Just for context:
I have started to restructure the existing documentation into several pages in this PR. However the documentation still needs to be updated and extended. I would like to keep the scope of this PR to code refactoring and rather open a separate "Improve material docs" PR next.

@yiyi1991
Copy link
Contributor

Super helpful clean-ups!

(The CommShareConfig class should be even outside the material module and be utilized more often from now on. Another similar ModeShareConfig will be nice to have too. )

@khaeru
Copy link
Member

khaeru commented Feb 26, 2026

I have started to restructure the existing documentation into several pages in this PR. However the documentation still needs to be updated and extended. I would like to keep the scope of this PR to code refactoring and rather open a separate "Improve material docs" PR next.

OK, understood. I usually start from the What's New and linked docs page as a starting-point to understand changes made in a PR, before going on to the code. But since this is the plan, I will just look at the code/data files directly (hopefully today), and expect they will be documented in another PR.

@macflo8
Copy link
Contributor Author

macflo8 commented Feb 26, 2026

I documented the changes of this PR in the docs here:
https://iiasa-energy-program-message-ix--475.com.readthedocs.build/projects/models/en/475/material/v1.2.1.html

I tried to cover most changes of this PR there. The follow-up documentation PR will rather tackle general documentation, as I realized that previous PRs did not comprehensively update/extend the material docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enh New features or functionality material MESSAGEix-Materials variant

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants