Many IrisLoadWarning in logs
#2718
Comments
|
Probably a side effect or related in one way or another to #2716 🍺 |
|
No, looks like this is actually a new feature of Iris 3.12.0: https://scitools-iris.readthedocs.io/en/stable/generated/api/iris.loading.html#iris.loading.LoadProblems One can print How do we want to deal with this? I can think of several options:
@ESMValGroup/technical-lead-development-team |
|
Debug log is already hard to parse as is, I'd say we leave a message at the end of each preproc block with the iris load warning (only ONCE per block) in the debug log and then route the more verbose warnings in either cmor log or a new iris_log. Probably cmor since as you said it's most likely related. |
|
It would be good to distuinguish between two kinds of load problems:
|
|
The simplest way to deal with category 1. would be to apply the fixes using ncdata before loading into iris, so the warning never appears. That does not work for legacy code, so we may want to use something like |
Yeah, sounds like a good approach for the future (would need something like #2579 before though). Unfortunately I don't see an easy way of determining which problem is fixed and which not for our present fixes. For example, this is the content of In practice, we would need to check the content of For data where we expect lots of problems (native model output), we could simple ignore the warning within the config file. For the other cases we could show the problems in the debug or CMOR log? |
|
Actually, I think a possible solution could also be to reroute ALL iris warnings into a dedicated I don't think anyone takes those serious... We could do it the same way we do it for the CMOR log: If there have been problems, issue a single warning at the end of our log saying that there have been problems. |
|
If you're happy about the way the ICON data is loaded, it should be safe to add the warning from the top post to list of ignored warnings for the ICON project in config-developer.yml and that should solve most of the problem. If there are other warnings related to loading data outside of the ICON project, we could move that setting from config-developer.yml to the extra facets facility to ignore warnings in a more fine grained way of ignoring load warnings for CMIP6 and CMIP7 data as a short term solution, until we have moved things to fixing before loading? In most cases, it should be fairly straightforward to see from the fix if a problem has been fixed or not. E.g. the example with the invalid missing or fill value above would be a bit of code using masked_equal. |
|
This is probably the safest approach. One problem I see with this is that it requires a lot of work: you need to find out where the warning is raised, which dataset raises it, if it's safe to ignore, etc. Here is an example log of the run that I used to create the multi-model plot of our new paper: From the more than 5000 lines in the log, more than 4000 are Iris warnings. They are most likely raised in the area_statistics code, but one would need to check that. I think it would be much more user-friendly to show those warnings in a separate file. |
|
Hiding these warnings in a separate log file where nobody will look at them defeats the purpose of having them at all. Can't we just silence the irrelevant warnings from area_statistics? Another option could be to issue each warning only once? The 1582 warnings in that log file are reduced to only 12 unique warnings. |
|
I agree that it's certainly not optimal, but having 1500 warnings in a user's log also kind of defeats the purpose of them. No one will read them. The warnings from area_statistics is just one example - there are many more preprocessors that emit them. Thus, I think that hiding them in a separate file and then trying to address them is a good compromise (in terms of user-friendliness vs. being careful). I can start on another PR that ignores the area_statistics warnings once we reached a conclusion here. |
After updating to the latest Iris version, logs are full of
IrisLoadWarning: Not all file objects were parsed correctly. See iris.loading.LOAD_PROBLEMS for details.I got this when running a recipe that uses lots of native data (ICON), there are probably less warnings for CMIP models.
The text was updated successfully, but these errors were encountered: