Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Caching to Disk in Datapipe #182

Closed
wants to merge 5 commits into from
Closed

Conversation

jacobbieker
Copy link
Member

Pull Request

Description

This gives the option to cache to disk the outputs of some datapipes, and gives examples in the training pipeline on how to do so. This should help with compute intensive data pipelines.

Fixes #

How Has This Been Tested?

Unit tests and running it.

  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

@jacobbieker jacobbieker added the enhancement New feature or request label Apr 3, 2023
@jacobbieker jacobbieker self-assigned this Apr 3, 2023
@codecov
Copy link

codecov bot commented Apr 3, 2023

Codecov Report

Merging #182 (57ba8db) into main (86fa3de) will decrease coverage by 0.06%.
The diff coverage is 8.33%.

❗ Current head 57ba8db differs from pull request most recent head 4a21574. Consider uploading reports for the commit 4a21574 to get more accurate results

@@            Coverage Diff             @@
##             main     #182      +/-   ##
==========================================
- Coverage   77.79%   77.73%   -0.06%     
==========================================
  Files         124      124              
  Lines        5097     5106       +9     
==========================================
+ Hits         3965     3969       +4     
- Misses       1132     1137       +5     
Impacted Files Coverage Δ
ocf_datapipes/training/metnet_pv_site.py 22.98% <8.33%> (-1.38%) ⬇️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@jacobbieker
Copy link
Member Author

This has already partly been done in #238

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant