Why are there multiple FDS version runs in Verification and Validation guides?

[Er9y714]

As reading an article today I have learned (never noticed before) that there are results from multiple FDS versions in Validation and Verification Guides of FDS . For example in Validation Guide of v6.11.0 the majority of the results are from FDS v6.10.1. Whereas, 6.11.0 is majority in Verification guide but also consists of 6.10.1 and 6.9.1.

As far I as remember, all those V&V simulations are run and checked for each major release. Including earlier versions to a certain release document looks counter intuitive.

I was wondering if there is any reason having a mixture of versions in the guides?

Thank you,

[drjfloyd]

The validation cases take a very long time to run. If you look at the git hashes in the Validation Guide you will see that they vary and that they do not equal the git hash of the release version of 6.10.1. These represent various states of the FDS source since the release of FDS 6.10.1. For example, 6.10.1 was released April 2025 with the git hash of 12efa16, and the git hash for the plots for the NIST FSE 2008 validation cases is 20dc4c7 which is the state of the source in Dec 2025.

There are a handful of verification cases which are not part of the continuous integration testing done each night as they take too long to run to do overnight. These are periodically cycled through with the Validation Cases.

[Er9y714]

I see, thank you for the clarification.

[bwklein]

I hadn't really looked into it before, but I assumed that prior to a release, that all V&V cases were run and reviewed for discrepancies from the prior release. I know it cannot be done for the nightly tests, but a comprehensive run of the entire suite prior to a release seems like a reasonable QA check. Something I have mentioned in the past was to have a table of prior release metrics for the suite, to quickly check for significant changes from the last version. These metrics would be case/quantity specific, but would help track changes over time to the results of FDS, knowing if the results are getting better or worse as time goes on. At a minimum, it could be the table of uncertainties and biases in the Validation guide.

[drjfloyd]

With the exception of a very small number of cases, every Verification case is run every night and we do not do a release unless all of those nightly cases pass. Every case has one or more metrics being examined. There are 1000s of verificaiton comparisions. The few cases that are not part of the nightly build are primarily fundamental flow phenomena that are regions of the code that are not frequently changed. Even those cases were run between 6.10.1 and 6.11.0.

All of the validation cases were run between 6.10.1 and 6.11.0. Running the entire suite takes weeks if the full NIST cluster is devoted to nothing but running validation cases. This is the same cluster used for development. This is not a commercial operation where there is incomming revenue to spin up cloud computing at will for running a massive compute load. A balance needs to be struck between being able to continue doing work and support releases.

[bwklein]

I appreciate that most of the Verification cases are run nightly, and that a full V&V run was completed between 6.10.1 and 6.11.0. I take issue with the claim that commercial operations have more resources to waste on frivolous or unnecessary compute services, I know we don't. Also, this isn't a attack on the dev team, my comment was to express that the V&V process wasn't aligned with expectations (mine and others) for a release package. I fully understand how time and resource consuming this activity can be.

How does any single developer or the project team in general have confidence that everything is still fine for every test case, with multiple developers changing different parts of the code, unless you run the tests? How does the team go about selecting the correct subset of cases to run if the entire suite isn't processed? The fact that the suite was run recently demonstrates that a balance can be found, and the activity is worthwhile.

It seems from the response that two things are true.

1. Verification Cases are the primary set of test cases for the functionality of FDS. Which makes sense as they are designed to exercise specific features and capabilities of the model.
2. The team is confident that a good process in place by which the best subset of Validation Cases are selected and run prior to a release as a secondary check on how code changes might impact results compared to experimental data.

According to the Configuration Management plan, 'For major new releases of the model, a complete suite of test cases should be compared to those from previous versions of the model. ' As there hasn't been a major release of FDS in over 12.5 years, you are already doing more than stated in the SQAP for minor version 6.11.0. 😄

I don't think in general the V&V document matters, nobody is reading it for fun on the weekends. Until something about the model is called into question, and then the relevant parts of the document become a big deal to those who are excited about something. The fallback is always to just take the cases you think are relevant, and run them in whatever release you have questions about, which makes out of date information in the last version of the V&V Guide even more irrelevant in that moment.

[obscureed]

@bwklein -- I am not sure from your comments whether you think that the current V&V testing is too onerous, or lags behind versions too much, or is just right. In places, you seem to say that tests should only be run when problems are found (in a released version?), but elsewhere you note that it is difficult to predict the effects of multiple changes.

I'm just pitching in to say that the validation of FDS is one of its major strengths. To be of value, it has to be documented and up-to-date, as it currently is.

I agree with you that it would be useful to have more detail on the changes from version to version. The Release Notes give a high-level summary, and we can in theory trawl through every edit on github, but a middle-level list would be useful.

[bwklein]

@obscureed I know first hand the value of the FDS V&V process and the documentation as I was the one who set up the initial automated system for processing and plotting the output from FDS for the documents. To clarify my points above. There was a specific issue where some questions arose about some phenomena in FDS and I discovered through that inquiry that the latest release of the Validation document did not have updated plots generated from the results of the latest release of FDS. I assumed that prior to a release (Major and Minor versions) that the full set of Validation and Verification cases where run and processed for that version of the simulator. That was not the case, and the plots in the docs might not be generated from simulations run with the latest release.

Jason clarified that there is an internal process where the developers choose which Validation cases might be more or less impacted by changes to the code since the last time the case was run, and they update the plots and results analysis for those cases that are more likely to be impacted. I know that the time and effort to run and process all of the output is significant, but I also know that it would look better for the document released with a version of the program, to fully reflect the output of that version. It would have helped clarify some things for the above mentioned inquiry.

The FDS development project is the gold standard in my mind for how to document and support life safety simulation tools. Many try to replicate it in their own work. As the gold standard, people come to expect great things and can be let down when their expectations are not met.

I summarized my thoughts above, but in essence, I think that the development team is doing a great job and are thoughtfully managing their time and effort commitments. This keeps the project moving forward and in the right direction and I am honored to have played some small role in the effort over the years.

[rmcdermo]

FYI, yesterday we pushed up a set of baseline validation statistics using current values (gotta start somewhere).

See PR # 16586

Now the nightly run of FDS_validation_script.py, which runs scripts/validation_git_stats.py, tests the current model sigma and bias against the baseline and reports deviations larger than 10%. A change in the wrong direction creates an Error, a change for the better creates a Warning, hence a note to ourselves that we may want to tighten the baseline (replace the baseline with the improved stats). Over time, this should push us the right direction. A discussion of this process was also added to the configuration manual under "Firebot: Automated Software Quality Testing" >> "Stage 7: Generation of verification and validation plots and regression statistics".