F10.7 driver issue

[twfang]

There were significant jumps in the WRS runs shown in the neutral density Testbed plots. The issue is associated with the way how we deal with jumps in F10.7 values. Keeping the records of these email exchanges.

[twfang]

From Adam:
Checking the F10.7 from the drivers during the WFS cycles on 10/14:

compared to what the WRS saw:

Thinking on it, I'm almost positive it's the F10.7 jump limit that was triggered. I bet the WRS parser isn't looking backward and so it doesn't see the jump, or something of that nature.

[twfang]

From Adam:

2024-10-06 20:00:00 265.0
2024-10-07 20:00:00 277.0
2024-10-08 20:00:00 225.0
2024-10-09 20:00:00 220.0
2024-10-10 20:00:00 216.0
2024-10-11 20:00:00 214.0
2024-10-12 20:00:00 214.0
2024-10-13 20:00:00 195.0
2024-10-14 20:00:00 200.0
2024-10-15 20:00:00 200.0
2024-10-16 20:00:00 200.0
2024-10-17 20:00:00 205.0

So the 277 -> 225 gets rejected for being an absolute jump of >35... the 225 is deleted, then 220, 216, etc. for the same reason, and we're left with a static value of 277 throughout the forecast period. The WRS parser initializes the same way, which gives that initial 277 value for five minutes I think, and then subsequent updates didn't grab the 6th and 7th as a part of writing new data to the driver file, so that jump limit isn't enforced.

[twfang]

From Adam,

The issue isn't with the data. The data matches GFZ. The issue is with the F10 jump limit of 35 that we vetted with the most recent round of upgrades.

[twfang]

Apologies if this is overly verbose, but I am opting to dump absolutely everything I know on this topic into this email.

I understand the forecast office to always use the observed 20UT value (put into the database at 21UT and available to us in an xml file on WCOSS2 just after 21:15UT) unless they need to do a flare adjustment.

There's not much value in focusing on the specific dates; the reason why values from a week ago are included before interpolation is just a matter of operational redundancy. The script determines which dates it is trying to find values for, and then parses the F10 and Kp values into a lookup table for every entry in every xml file available in that span, overwriting the values from the older files with values from newer files. So for 10/14 00Z cycles, it has to search in order 10/12, 10/13, and 10/14, because e.g. MSIS in IPE needs Kpa values from up to 36h prior to initialization. An xml file issued on 10/12 would include values back to 10/9, since the files include 3 days prior, current day, and 3 days after. There's an additional two days buffer built in to protect against operational issues we've experienced in the past, or edge cases on timing at the start of a UT day.

The difference between WFS and WRS resulting in that constant jumping behavior (WFS running with artificially high F10, WRS initializes from that runs forward with correct F10, density drops over time, then it reinitializes high six hours later from a bunk-F10 WFS run) would only express itself when the initially high jump limited value drops from the WRS backward search. The WRS process is to write the first IPE timestep's worth of the driver file normally (thus five minutes of that initial incorrect value) and then allow a wrapper script to append to the driver files (and write lockfiles that allow the nowcast to progress). When the wrapper does the backward search described above, it's no longer initializing from 36h back, so it stops seeing that older value above the jump limit ~36h sooner.

As a result, there was approximately a five day stretch where both WFS and WRS F10 were static (and wrong) at 277, a ~36h stretch where WFS was wrong and WRS was right, and then things returned to normal. Even if we were more strict about which dates the script put into its internal database prior to interpolating and writing the values out, this issue would've only expressed and resolved itself sooner.

I made the choice to preferentially delete the second value in the pair rather than the higher value for two reasons. We would have to make multiple (n-1, I think) passes at the jump limit algorithm if the higher value is to get wiped rather than a single pass. I'm also not entirely sure how the interpolation/extrapolation algorithm would work if there's no prior value to interpolate from. If there's no axis-wise right value for interpolation, it just keeps the prior value. If the left value is missing, I think the script would fail, and it'd default to the minimum value of 75, but I'm not entirely sure of that.

It's unfortunate, I guess, that the code did exactly what it was intended to do. If I remember correctly, we tested this against an entire solar cycle of F10 values and identified no issues resembling this. I can look into a different approach that preferentially removes the high value to see how that would work through the same verification period, and am open to other ideas as well.

Adam