WB2 Latitude Flipping

[stefanwayon]

Hello, I’ve been using the data_process/convert_makani_output_to_wb2.py script to convert roll-outs output from inference to Zarr, and I am unsure if the output latitudes are correct. I can see there is some flipping involved, with a comment about colatitudes. However, if I plot the data in the Zarr file, it appears flipped, which to me suggests the original latitudes (coming from ARCO ERA5) didn’t need flipping.

Am I missing something in the WB2 conventions?

Below a script to visualise a field in the output Zarr.

import cartopy.crs as ccrs
import cartopy
import matplotlib.pyplot as plt

def _plot(da):
    fig = plt.figure(figsize=(10, 10))
    ax = plt.axes(projection=ccrs.Robinson())
    ax.coastlines(resolution="10m")
    plot = da.plot(
        cmap=plt.cm.coolwarm, transform=ccrs.PlateCarree(), cbar_kwargs={"shrink": 0.6}
    )

ds_global = xr.open_zarr("/raid/stefan/fcn3/runs/fcn3_ensemble/eval_rollouts/scores/rollouts_global.zarr")
_plot(ds_global.isel(time=0, number=0, prediction_timedelta=19)["2m_temperature"])

[azrael417]

Hello Stefan, I think you are correct. I am preparing an MR which streamlines the computation of lat and Lon coordinates in makani.

Note that the lat lon values on weather bench can be confusing though, for example for the pangu model data hosted there, the order is reversed to what it should be. Their Era5 copies though seem to have the right order.

[stefanwayon]

Just had a look at WB2 HRES t0 data, and you are right about the order being reversed vs. ERA5. However, both the coordinates and the data are reversed, so they both should be flipped when going from makani to WB2 (or neither and rely on Zarr/xarray to sort coordinates when needed).

I’ll try to add a --flip-lat option to the script on my next pass over the code to toggle behaviour, since it seems either behaviour could be useful.

[azrael417]

I am not sure we need that, since if the zarr data is correct it should be read correctly, right? I have an MR #62 which, among many other things, removes the flipping.

[stefanwayon]

Ah, I now see why the WB2 data ordering might be an issue: if trying to convert WB2-style data to makani, the coordinates are actually ignored:

makani/data_process/convert_wb2_to_makani_input.py

Lines 195 to 196 in 885c45a

	wb2_sel = wb2_data[scwb2]
	data = wb2_sel[wb2_sel["time"].isin(timebatch)].values

Will add a fix to that script to match the coordinate order to the one in the metadata file (which I assume matters for the network).

[azrael417]

Yes, I think I did that because for example the Pangu-generated data have a wrong ordering in the wb2 data file, so I wasn't sure if there are more files like that. If you add awareness of Lon lat, would you mind adding a flag to ignore this awareness? Maybe issuing a warning if that is the case, so that the user is really aware of what he is doing?

So I think the best behavior would be to respect it, but allow for a user override. So basically the default case is to assume that the wb2 file has the right order, and then match that to what the metafile expects. And then otherwise allow to flip it or no. So maybe instead of a binary flag add a mode with 3 options:

auto/match: respect ordering from wb2 file and match to metafile (maybe requiring flipping)
force-flip: flip input data
force-noflip: read as-is, do not flip, assume correct ordering

Does this make sense?