From Visuals to Hypotheses: Visualization as Reasoning in AI-Led Weather Forecast Evaluation

[Hackshaven]

1. Introduction
   • Motivation: Weather models (HRRR, RRFS) produce massive high-frequency datasets.
   • Challenge: Traditional verification relies on predefined statistical metrics (bias, RMSE, skill scores). These miss unexpected structures in model output.
   • Key Idea: Visualization is not just communication but reasoning — capable of generating new hypotheses about model behavior.
   • Contribution: Present Zyra as an agentic framework that explores model output visually, proposes hypotheses, and supports formal verification workflows.
   • Important Note: This work represents a hypothesis and conceptual demonstration, not completed verification; the goal is to augment and guide existing verification methods.

⸻

2. Related Work
   • Verification methods in NWP (object-based verification, FSS, traditional skill scores).
   • Visualization in weather forecasting (human forecasters use radar loops, spaghetti plots, etc. for intuitive reasoning).
   • AI for weather forecasting (ML surrogates, evaluation frameworks).
   • Gap: No existing work treats visualization as an agent-driven reasoning step.

⸻

3. Methodology: Zyra as a Visual Reasoning Agent

3.1. Pipeline Overview
• Zyra pipeline: data acquisition → preprocessing → visualization → hypothesis proposal → validation → loop closure.
• Distinction: Zyra does not just plot; it chooses visual encodings and proposes scientific hypotheses.

3.2. Data Source
• HRRR (hourly 3 km forecasts).
• RRFS (ensemble, FV3-based next-gen system).
• Variables: reflectivity, CAPE, shear, winds, precipitation.

3.3. Visualization Strategies
• Composite reflectivity loops (storm initiation timing).
• Vertical cross-sections (storm structure / asymmetry).
• Ensemble spaghetti plots (spread / propagation bias).

3.4. Hypothesis Generation
• Zyra flags anomalies from visual inspection:
• Example: storms initiating earlier/later than CAPE would suggest.
• Example: ensemble spread skewed toward one propagation direction.

3.5. Hypothesis Validation (Conceptual)
• Zyra’s proposals are not verification results themselves.
• Instead, they guide what statistical/physics checks to run (e.g., compare to observed radar, precipitation).
• Positioning: supports, not replaces, verification.

⸻

4. Case Study (Hypothetical Demonstration)
   • Illustrative workflows (small subset of HRRR/RRFS fields).
   • Show Zyra-generated visualizations with annotations.
   • Example Hypotheses (framed as “candidates for future verification”):
   1. Asymmetry in convective initiation timing relative to CAPE.
   2. Ensemble spread bias in storm propagation directions.
      • Figures:
      • Figure 1: Workflow diagram of Zyra pipeline.
      • Figure 2: Reflectivity loop with Zyra annotation (“early storm initiation”).
      • Figure 3: Ensemble spaghetti plot highlighting asymmetry in spread.
      • Figure 4: Conceptual loop closure (visual → hypothesis → statistical test).

⸻

5. Discussion
   • Novelty: Zyra proposes hypotheses serendipitously from visual patterns, unlike traditional metrics which must be predefined.
   • Positioning: This work is a hypothesis — not yet completed verification, but a conceptual framework to support NOAA’s verification toolkit.
   • Value: Helps human forecasters and scientists discover non-obvious behaviors in models, accelerating verification design.
   • Limitations:
   • Risk of spurious correlations.
   • Needs careful statistical follow-up.
   • Requires strong provenance logging to ensure reproducibility.

⸻

6. Responsible AI Statement

Needs work.

⸻

7. Reproducibility Statement

Needs work.

⸻

8. Conclusion
   • Reassert core claim: Visualization can act as a reasoning agent in NWP model evaluation.
   • Zyra demonstrates potential to:
   • Propose hypotheses not encoded in verification metrics.
   • Support reproducibility through pipeline architecture.
   • Enhance trust and interpretability in AI-led science.
   • Next steps:
   • Integrate Zyra outputs with object-based and neighborhood verification methods.
   • Expand to probabilistic evaluation of RRFS ensembles.

⸻

📊 Figures & Illustrations to Include
1. Pipeline Diagram (Zyra as agent in loop).
2. Annotated Reflectivity Loop (highlight early storm initiation anomaly).
3. Ensemble Spaghetti Plot (show bias in storm tracks).
4. Loop Closure Diagram (visual → hypothesis → statistical validation → updated visualization).

⸻

✅ This structure:
• Aligns with Agents4Science requirements (novel AI-led hypothesis generation).
• Keeps us safe by saying this is a hypothesis, not verification results.
• Anchors in NOAA GSL mission (verification and operational model evaluation).

[Hackshaven]

This idea is still very open-ended — we’d love your input!
• The Agents4Science 2025 call for papers is here:
https://agents4science.stanford.edu/call-for-papers.html
• The Zyra repository is here:
http://github.com/NOAA-GSL/zyra/
• For background on Zyra’s philosophy and architecture, see the Zyra Wiki.

Key questions we’d like feedback on:
1. Which HRRR/RRFS verification challenges would benefit most from a “visual reasoning” agent?
2. What datasets or case studies should we prioritize to make this compelling?
3. What forms of collaboration (statistical testing agents, visualization methods, domain expertise) would strengthen this submission?

If you’re interested in shaping this, please reply here — even small contributions (ideas, datasets, test cases) could make a big difference.