Grab bag of outstanding issues

[omad]

Work Generation

• Incremental. Need a way to compare what's on S3 to what's not.
  • Our current unit of work for COG conversion is a NetCDF file. Either stacked or unstacked. It will be awkward to compare Stacked NetCDF files to data existing on S3 since they represent different one dataset vs many..

COG Converter

• Configurable COG parameters when generating overviews.
  • Resampling method for overlays for different products
  • Number of overview levels
  • Compression/chunk size (maybe, deflate/512 is good, but...)
• Is it faster/easier/more configurable to use rio cogeo than raw GDAL.
• Review/test the parameters used

Uploader

• Specify bucket instead of having COG-Conversion define it.
• Give uploader an option to move files to a COMPLETE directory instead of deleting them. Will let us test upload to a dev bucket, and then run again against the prod bucket.
• SPEED How fast can we upload in a single thread, do we need parallel upload processes?
• MAYBE Ability to watch multiple directories?

[ashoka1234]

For the COG-conversion

• Validation of COG datasets - On which side do we do this?

[omad]

A very quick test of rio cogeo indicates no significant performance difference.

[omad]

• Easier submission to large parallel PBS jobs
• Useful progress logging when run in PBS or as a background process
  • We currently use tqdm for an interactive progress bar, it might be possible to use it for background progress logs
• Finer grained progress when converting stacked NetCDF files
• Progress and speed metrics from the Uploader

[ashoka1234]

For the COG-conversion

• Further configurability of upload directory structure, for example some products wants flat directory structure in AWS when only yearly time makes sense, i.e. without months and days
• What do we do when time indicated in the file name rather than the timestamp of dataset makes sense

[omad]

• Make the src_template more flexible and simplify class COGProductConfiguration by using either parse or accepting regexes.
• Look into using MPIPoolExecutor for distributing work inside PBS jobs

[emmaai]

For the COG-conversion

* [x]  Validation of `COG` datasets - On which side do we do this?

On NCI, done!