Deflate by huard · Pull Request #87 · CSHS-CWRA/RavenHydroFramework

huard · 2026-04-02T22:08:06Z

Pull Request Checklist:

This PR addresses an already opened issue (for bug fixes / features)
- This PR fixes Compress (deflate) netCDF outputs #86
Tests for the changes have been added (for bug fixes / features)
- (If applicable) Documentation has been added / updated (for bug fixes / features)

What kind of change does this PR introduce?

Compresses NetCDF standard and custom outputs using the deflate filter.
Chunks NetCDF standard and custom outputs, optimizing for time series analysis (one chunk covers the entire time series).

Compression is particularly useful with simulations that do not include q_obs. Without compression, we still need to store the full array of NaNs.

Does this PR introduce a breaking change?

Should not, but I'm going to run more tests.

Note that this PR is based off v4.12 because the main does not compile on my end, nor on the CI. This PR should not be merged before main is merged into this branch.

for more information, see https://pre-commit.ci

…to deflate

Copilot

Pull request overview

This PR targets issue #86 by adding NetCDF-4 deflate compression and explicit chunking to reduce standard output file sizes and improve time-series access patterns.

Changes:

Add zlib/deflate compression and chunking to standard NetCDF outputs (notably 2D time×basin variables) and to custom NetCDF outputs.
Change the standard-output NetCDF time dimension from NC_UNLIMITED to a fixed length derived from Options.duration/Options.timestep.
Introduce NetCDF compression/chunking constants in RavenInclude.h and some related formatting-only cleanups.

Reviewed changes

Copilot reviewed 5 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/StandardOutput.cpp	Adds deflate + chunking and makes standard-output `time` dimension fixed; updates NetCDFAddMetadata2D to apply compression/chunking.
src/CustomOutput.cpp	Adds deflate + chunking to custom NetCDF variables and introduces `ComputeNumTimeSteps()` to size `time`.
src/CustomOutput.h	Declares `ComputeNumTimeSteps()`.
src/RavenInclude.h	Adds `NETCDF_DEFLATE_LEVEL` and `NETCDF_CHUNKSIZE_MB` constants.
src/Makefile	Forces NetCDF compilation/linking on by default.
src/Reservoir.cpp / src/Reservoir.h	Whitespace/formatting-only changes.
src/ParsePropertyFile.cpp	Whitespace-only changes.
src/ChannelXSect.cpp	Comment whitespace-only change.
src/Assimilate.cpp	Whitespace-only change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…t, storage, etc)

…ework into deflate" This reverts commit 6b0fca9, reversing changes made to 10ad207.

…ng. Add compression to other outputs (storage, rainfall)

huard · 2026-04-09T20:08:34Z

Tested against the RavenPy test suite and no error. Bonus, the outputs from the test take half the disk space.

analytophile · 2026-04-10T16:54:34Z

@huard showing off my github ignorance today - I thought I had merged the main into deflate but I still see recent changes in main not included in the 'Files changed'. It seems like it only merged the conflicts I approved but not the unconflicted changes?? Can you tell me how to do this? There was a separate button for the other pull requests that made this easy, but not here...?

analytophile

not sure why my merge with main didn't take, but it is still holding onto these features of the previous branch as if they were additions in deflate

analytophile · 2026-04-10T17:08:24Z

should revert to main

analytophile · 2026-04-10T17:08:41Z

revert to main

analytophile · 2026-04-10T17:10:09Z

should revert to main

analytophile · 2026-04-10T17:10:26Z

revert to main

analytophile · 2026-04-10T17:11:17Z

revert to main

analytophile · 2026-04-10T17:13:58Z

revert to main

analytophile · 2026-04-10T17:14:19Z

revert to main

analytophile · 2026-04-10T17:14:37Z

      for (int p = 0; p < _nSubBasins; p++) {
        if((_pSubBasins[p]->IsGauged()) && (_pSubBasins[p]->IsEnabled())){
          if (Options.assim_method==DA_ECCC){ASSIM<<","<<_aDAQadjust[p]<<" ";}
+          else                              {ASSIM<<","<<_aDAscale  [p]<<" ";}


analytophile · 2026-04-10T17:15:21Z

+
+  if(!overriding)
+  {
+    for(int n=0;n<_nQlatHist;n++) {


analytophile · 2026-04-10T17:15:43Z

  void            SetGauged                (const bool isgauged);
  void            Disable                  ();
  void            Enable                   ();
+  double          ScaleAllFlows            (const double &scale_factor, const bool scale_last, const double &tstep, const double &t);


huard · 2026-04-10T18:00:25Z

What happened is that I merged main, then reverted that merge because I couldn't compiile. I'll try to make a clean branch and open a new PR.

huard and others added 8 commits April 2, 2026 14:35

Added deflate instructions and global NETCDF_DEFLATE_LEVEL option

0479810

set compression for standard outputs

3fd23eb

parameterized chunksize_mb

7f18bde

Merge branch 'main' into deflate

62241cf

[pre-commit.ci] auto fixes from pre-commit.com hooks

1abceab

for more information, see https://pre-commit.ci

add function to count the number of custom output time steps

e6dd682

Add chunking to CustomOutput

10ad207

Merge branch 'deflate' of github.com:CSHS-CWRA/RavenHydroFramework in…

6b0fca9

…to deflate

huard requested review from analytophile and Copilot April 7, 2026 22:16

Copilot started reviewing on behalf of huard April 7, 2026 22:17 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread src/StandardOutput.cpp Outdated

Comment thread src/CustomOutput.cpp Outdated

Comment thread src/CustomOutput.cpp Outdated

Comment thread src/CustomOutput.cpp Outdated

Comment thread src/Makefile Outdated

Comment thread src/StandardOutput.cpp

huard added 7 commits April 8, 2026 09:05

try to extend compression logic to other output variables (constituen…

f2b6316

…t, storage, etc)

Revert "Merge branch 'deflate' of github.com:CSHS-CWRA/RavenHydroFram…

f63733c

…ework into deflate" This reverts commit 6b0fca9, reversing changes made to 10ad207.

add n_out_time in Options struct

f72a9e0

Set time dimension back to illimited. No need to define it for chunki…

c1d0585

…ng. Add compression to other outputs (storage, rainfall)

oups

9e49de0

clean-up

54667b6

reset the makefile

1fd66f1

Merge branch 'main' into deflate

c005ee9

analytophile approved these changes Apr 10, 2026

View reviewed changes

Merge branch 'main' into deflate

bb37df4

analytophile requested changes Apr 10, 2026

View reviewed changes

huard mentioned this pull request Apr 10, 2026

Add filters to compress (deflate) and chunk standard and custom netCDF outputs #94

Merged

3 tasks

huard closed this Apr 10, 2026

Conversation

huard commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist:

What kind of change does this PR introduce?

Does this PR introduce a breaking change?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huard commented Apr 9, 2026

Uh oh!

analytophile commented Apr 10, 2026

Uh oh!

analytophile left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huard commented Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

huard commented Apr 2, 2026 •

edited

Loading