Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add converting batches to NetCDF and saving them out #238

Merged
merged 29 commits into from
Nov 16, 2023
Merged

Conversation

jacobbieker
Copy link
Member

@jacobbieker jacobbieker commented Nov 10, 2023

Pull Request

Description

This PR adds support for converting multiple input dataarrays/datasets into a single Dataset to save it out as NetCDF. This is to help with #231 and make batches more interpretable on disk .

Fixes #237

How Has This Been Tested?

Unit tests

  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

@jacobbieker jacobbieker self-assigned this Nov 10, 2023
Copy link

codecov bot commented Nov 10, 2023

Codecov Report

Attention: 47 lines in your changes are missing coverage. Please review.

Comparison is base (2d67722) 78.60% compared to head (62ea906) 79.05%.
Report is 7 commits behind head on main.

Files Patch % Lines
ocf_datapipes/training/common.py 75.83% 36 Missing ⚠️
ocf_datapipes/training/windnet.py 92.80% 10 Missing ⚠️
ocf_datapipes/utils/utils.py 97.14% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #238      +/-   ##
==========================================
+ Coverage   78.60%   79.05%   +0.45%     
==========================================
  Files         129      130       +1     
  Lines        5693     5868     +175     
==========================================
+ Hits         4475     4639     +164     
- Misses       1218     1229      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jacobbieker jacobbieker marked this pull request as ready for review November 15, 2023 10:28
@jacobbieker jacobbieker requested a review from dfulu November 15, 2023 13:49
@jacobbieker
Copy link
Member Author

This has the start of windnet forecasting, but mostly is just adding how to combine and uncombine the output from PVNet datapipe to save to NetCDF, and load it from that.

Copy link
Member

@dfulu dfulu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@jacobbieker jacobbieker force-pushed the jacob/netcdf branch 2 times, most recently from 549be2f to 21765b7 Compare November 16, 2023 16:25
@jacobbieker jacobbieker merged commit 77be003 into main Nov 16, 2023
@jacobbieker jacobbieker deleted the jacob/netcdf branch November 16, 2023 19:01
@jacobbieker jacobbieker mentioned this pull request Nov 20, 2023
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Save out batches to NetCDF
2 participants