Skip to content

Add RollingDataset to close past forcing data - #1052

Open
evetion wants to merge 1 commit into
mainfrom
feat/rolling-time
Open

Add RollingDataset to close past forcing data#1052
evetion wants to merge 1 commit into
mainfrom
feat/rolling-time

Conversation

@evetion

@evetion evetion commented Aug 21, 2026

Copy link
Copy Markdown
Member

Issue addressed

Fixes #1037

Explanation

Wflow uses deferopen=false in opening multifile forcing data, so every monthly file remains open. Each accessed forcing variable in each file gets a 64 MB HDF5 chunk cache.
For 10.5 years, 126 monthly files * 3 variables * 64 MB that results in roughly 24 GiB.

This adds a RollingDataset instead of the MFD NCDataset, that only holds one forcing file open at a time.

Checklist

  • Updated tests or added new tests
  • Branch is up to date with main
  • Tests & prek hooks pass
  • Updated documentation if needed
  • Updated changelog.qmd if needed
  • A review by Copilot was done to ensure the requirements in AGENTS.md are satisfied

Additional Notes (optional)

Changing deferopen=true makes the model slower, as it would reopen the file on each var access. Ideally we upstream this to NCDatasets, I will make an issue.

Partially AI generated with GPT-5.6 Sol.

@evetion

evetion commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

Happy to have someone from the core team take over this PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate memory usage (leak?)

1 participant