add visulization script

[peterdudfield]

Pull Request

Description

Add a script to add visualization
It loads a batch and makes a markdown report

This does not include

• does not plot x and y of nwp data, or satellite data

Could do a lot of refactoring, into smaller functions, so code is not repeated

#231

How Has This Been Tested?

Tested it locally

• Yes

Checklist:

• My code follows OCF's coding style guidelines
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked my code and corrected any misspellings

report

Batch visualization

We are looking at example 3

Wind

GSP

(and other examples)

gsp

shape value.shape=torch.Size([8, 21, 1])
Max 0.58
Min 0.00

gsp_id

shape value.shape=torch.Size([8, 1])
Max 284.00
Min 2.00

gsp_time_utc

shape value.shape=torch.Size([8, 21])
Max 1673744400.00
Min 1657915200.00

gsp_x_osgb

shape value.shape=torch.Size([8, 1])
Max 609538.81
Min 172042.17

gsp_y_osgb

shape value.shape=torch.Size([8, 1])
Max 732058.75
Min 180926.89

gsp_t0_idx

4

gsp_effective_capacity_mwp

shape value.shape=torch.Size([8, 1])
Max 330.09
Min 1.13

gsp_nominal_capacity_mwp

shape value.shape=torch.Size([8, 1])
Max 351.39
Min 1.19

gsp_solar_azimuth

shape value.shape=torch.Size([8, 21])
Max 4.36
Min -4.23

gsp_solar_elevation

shape value.shape=torch.Size([8, 21])
Max 1.21
Min -7.31

NWP

Provider ukv

(plus other examples)

nwp

Shape value.shape=torch.Size([8, 11, 10, 24, 24])
Max 21.84
Min -4.25

nwp_target_time_utc

Shape=(11,)
Max 2022-08-02 08:00:00
Min 2022-08-01 22:00:00

nwp_channel_names

Channel	Max	Min	Mean	Std
t	2.88	-2.01	0.73	1.43
dswrf	2.11	-0.58	-0.32	0.57
dlwrf	1.83	-1.59	0.23	1.02
hcc	1.86	-0.76	0.34	0.93
mcc	1.41	-0.98	-0.19	0.84
lcc	1.27	-1.27	-0.52	0.99
sde	0.05	-0.03	-0.03	0.01
r	1.46	-3.50	-0.49	1.37
si10	1.82	-1.42	-0.61	0.70
prate	4.38	-0.16	0.15	0.85
Shape=(10,)

nwp_step

Shape value.shape=torch.Size([8, 11])
Max 14.00
Min 1.00

nwp_t0_idx

2

nwp_init_time_utc

Shape=(11,)
Max 2022-08-01 21:00:00
Min 2022-08-01 21:00:00

Provider ecmwf

(plus other examples)

nwp

Shape value.shape=torch.Size([8, 11, 8, 12, 12])
Max 13.22
Min -2.61

nwp_target_time_utc

Shape=(11,)
Max 2022-08-02 08:00:00
Min 2022-08-01 22:00:00

nwp_channel_names

Channel	Max	Min	Mean	Std
t2m	3.38	-2.27	0.87	1.60
hcc	1.43	-0.94	0.48	0.94
mcc	1.76	-0.86	0.08	0.96
lcc	1.45	-1.18	-0.16	1.10
tcc	0.79	-1.75	0.22	0.80
sde	5.63	-0.09	0.24	1.05
u10	1.76	-0.92	0.05	0.62
v10	1.08	-0.86	0.09	0.48
Shape=(8,)

nwp_step

Shape value.shape=torch.Size([8, 11])
Max 25.00
Min 7.00

nwp_t0_idx

2

nwp_init_time_utc

Shape=(11,)
Max 2022-08-01 12:00:00
Min 2022-08-01 12:00:00

[Sukh-P]

@peterdudfield sorry for the slow response, I think this is useful for quick sanity checks on the batch data so thank you for making it! I guess we could extend it to solar later by adding in a parameter for what the target is and and if block, might also be good to add support for netcdf files/batches as well as .pt files since we some batches created in that format?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 148 lines in your changes missing coverage. Please review.

Project coverage is 78.36%. Comparing base (ab5b649) to head (6bab08b).
Report is 163 commits behind head on main.

Files with missing lines	Patch %	Lines
ocf_datapipes/batch/visualise.py	0.00%	148 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #339      +/-   ##
==========================================
+ Coverage   75.24%   78.36%   +3.12%     
==========================================
  Files         128      126       -2     
  Lines        6208     5962     -246     
==========================================
+ Hits         4671     4672       +1     
+ Misses       1537     1290     -247     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AUdaltsova]

could it be good to add it to the batch init so you can import it with from ocf_datapipes.batch import visualise_batch?

[peterdudfield]

could it be good to add it to the batch init so you can import it with from ocf_datapipes.batch import visualise_batch?

sounds a good idea. Can you add that?

[AUdaltsova]

also sorry to be like this but i think both file and function should be called either with visualiSe or with visualiZe

[peterdudfield]

also sorry to be like this but i think both file and function should be called either with visualiSe or with visualiZe

yea, could you change it to with s?

[AUdaltsova]

@peterdudfield I'm happy to do both but beforeI do: there is actually a folder for visualisation in dpipes that appears to be empty: https://github.com/openclimatefix/ocf_datapipes/tree/main/ocf_datapipes/visualization

Do we scrap that or do we move this script there? (I would probably be for moving the script there, batch folder feels like more of a place for handling during batch creation, not for inference)

[peterdudfield]

@peterdudfield I'm happy to do both but beforeI do: there is actually a folder for visualisation in dpipes that appears to be empty: https://github.com/openclimatefix/ocf_datapipes/tree/main/ocf_datapipes/visualization

Do we scrap that or do we move this script there? (I would probably be for moving the script there, batch folder feels like more of a place for handling during batch creation, not for inference)

yea perhaps move this function to that folder

[AUdaltsova]

I've updated some stuff but can someone with access to tensor batches run that before w merge please (this does not work with .nc batches) ? want to be sure I didn't break anything :)

[peterdudfield]

How come it doesn't work with .nc batches? Do they have to be then converted to a tensor and then it might work?

[AUdaltsova]

How come it doesn't work with .nc batches? Do they have to be then converted to a tensor and then it might work?

It expects the data to have BatchKeys which .nc files don't have, they get transformed into a batch after opening; I'll look into adding a function that does that here so it can handle .nc too :)

[peterdudfield]

How come it doesn't work with .nc batches? Do they have to be then converted to a tensor and then it might work?

It expects the data to have BatchKeys which .nc files don't have, they get transformed into a batch after opening; I'll look into adding a function that does that here so it can handle .nc too :)

Yea, we should think what will likely to do in of-data-sample, perhaps we wont save .nv files anymore, im not sure what the current thinking is