Is there a way to enrich the "State" with the background error estimated in the EnsVar ?

[andreapiacentini]

Dear Friends,
BSC got into a problem which can be explained by having an observation located in a point where the ensemble has almost 0 spread, but close enough to the front of a perturbed area, and this results in an abnormally high increment located on the front (so that, by the very weak correlation, there is a reasonable increment at the observation location).
It looks to me that this case fits perfectly in the Quality Control comparing ombg to threshold * Background Error at obs location.
The latter is a member of the ObsDiag group in the IODA formalism and I understand (by reading the .h files) that it can be interpolated at obs locations by using a specific Composite Observation Operator but it feeds on a State component which has a derived name: for the simulated variable var it needs var_background_error.
I feel like in a 4DEnsVar it should be easy to derive this information from the ensemble.
Any thought about this approach ? Knowing anybody going that way ?
Thanks
Andrea

[andreapiacentini]

Anyway, after a pretty boring session of ens_mean_variance followed by some NCO magics for enriching the state by hand, it turns out that the BackgroundErrorVertInterp lacks the TLAD part and the 4DEnsVar is not able to enroll it in the LinearObsOperator factory. It also lacks most of the nice parameters defining the vertical interpolation options (and the opportunity to replace the model vertical coordinate by a constant conventional coordinate) that the VertInterp operator has. This nice application would therefore require some work on UFO !

[jeromebarre]

Not sure if you find out that the problem was in a badly conditioned set of ensembvle trajectories for the 4EnVar to work as intended or is the problem in UFO. This isn't clear to me from what you describe above.
I guess you refer to https://github.com/JCSDA-internal/ufo/tree/develop/src/ufo/operators/backgrounderrorvertinterp
I would not use this... and I am not sure why in the 4DEnVar context you need the TL/AD of this. The obs part of jacobian is done with the actual observation operator TL/AD in your case https://github.com/JCSDA-internal/ufo/blob/develop/src/ufo/operators/vertinterp/ObsVertInterpTLAD.cc

[andreapiacentini]

Of course, the original problem is in the bad ensemble, but pragmatically, that can happen.
The bad ensemble translates into some region where the ensemble spread is dangerously small. Since it translates into a wrong approximation of the background error, it being too small, I'd like - at least as a temporary patch - to flag out observations from locations where the assimilation can't get any reasonable increment.
The https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/stable/inside/jedi-components/ufo/qcfilters/GenericQC.html#background-check-filter:~:text=for%20more%20detail.-,The%20third%20filter,-compares%20the%20departure QC filter is meant to do so, but I hardly see how to make it work.
The unit tests all relies on a precomputed IODA file containing the background error at the observation locations e.g. https://github.com/JCSDA/ufo/blob/master/test/testinput/unit_tests/filters/qc_backgroundcheck.yaml#L168
But if you start reading the headers of the filters, you realize that they can come from the interpolation of a specific GeoVal via the backgrounderrorvertinterp operator.
Accordingly to the instructions it has to be in a Composite operator together with its companion VertInterp, yet when I tried it off (adding the ensemble stddev by hand to the states), the 4DEnsVar complained about the lack of TLAD.
Is it clearer now ?
I don't even know if it is the path to follow.
Cheers