Benchmark procedure for LTC task

[lan496]

(This is a continuation from the discussion in my Linkedin's post)

Matbench-discovery is now working on integrating the lattice thermal conductivity (LTC) task by k_SRME. This new benchmark definitely adds a new dimension to evaluating uNNPs. However, I've recently noticed that many uNNPs have a non-negligible dependency on LTC predictions from displacement distances (see details in our group's blog post). Therefore, I believe that testing with a single displacement distance is insufficient for properly evaluating uNNPs by their metrics.

@janosh thinks testing more than one displacement distance can be informative, similar to the approach used in the geo-opt task (comment). I also think it would be beneficial to test with several displacement distances.

If we use multiple displacement distances, how should we select them? Or, from a broader perspective, how can we quantify the smoothness of the uNNP PES based on this benchmark?

[MSimoncelli]

Thanks @lan496 for this very good question!

I agree that computing k_SRME for many different finite displacements may provide additional information, however there are a few things that one should keep in mind.

Computing k_SRME for many different displacements would also require having reference DFT data computed for many different displacements.

In DFT, the finite displacements have to be chosen within a physically and numerically sensible range of 0.005 Å - 0.05 Å (this range ensures that energy perturbations are numerically large enough to be above the DFT accuracy, and physically small enough to be treated as perturbations). In DFT, varying the finite displacement within this range produces, almost without exception, unimportant differences (~<3%) on vibrational and thermal properties within Born-Oppenheimer DFT. These 3% variations are much smaller than those probed by k_SRME between DFT and MLP.

Since a good MLP is expected to reproduce the DFT PES, for a fixed finite displacement a good MLP is expected to reproduce the reference DFT value computed with the same displacement. In other words, having a low k_SRME is a necessary condition for a MLP to be accurate.
Importantly, the weak dependence of DFT vibrational and thermal properties on the finite displacement is material-dependent, therefore a fMLP that accurately reproduces DFT data for vibrational and thermal properties of many materials has to display a weak or negligible dependence on the finite displacement, like DFT.

Whenever a MLP predicts vibrational properties with a dependence on the displacement larger than that observed in reference DFT data, it means that the MLP fails to reproduce the smoothness of the DFT PES.
MLP with PES having smoothness different from DFT may result in:

• failures to properly predict crystal symmetries (since unphysical local minima emerge), which are critical for crystallographers who synthesize materials;
• nonsensical molecular dynamics, since inaccurate derivatives of the PES lead to inaccurate forces. There are examples of this on the MLP arena website.

[chiang-yuan]

As one can see, displacements of 0.01 A and 0.1 A really make strike difference and go into different regimes (positive or negative PES curvature) for many structures, and one do not measure the same vibration anymore

[JaGeo]

@chiang-yuan thanks for the example! I think it is very important when designing a benchmark what someone interested in phonons and their interactions would look for. At least for me, the good description of the phonon modes would be a part of it. I don't care much about the MLIP architecture as long as I can use the MLIP instead of expensive DFT calculations to investigate phonons.

[timduignan]

Very interesting result thanks @chiang-yuan, is the data for those curves available by any chance?

[chiang-yuan]

@timduignan - We will put it on repo once the preprint is released. Thanks for your interest!

[rfhari]

Thanks for the engaging discussion - it was interesting to read!

Just my unsolicited two cents: I understand that phonon properties, (particularly the acoustic branches), computed with a well-trained MLIP characteristically not show significant variations within a physically reasonable range of perturbation as the system remains around equilibrium. This, of course, can be somewhat subjective depending on the material’s anharmonicity.

That said, I was just wondering if Temperature Dependent Phonons (via TDEP or ALAMODE) could be an alternative approach or metric in MLIP Area? Since the displacements and forces used to compute FC2 and FC3 are sampled from actual MD trajectories, they inherently can capture larger deviations from equilibrium at finite temperatures. While this may possibly introduce some classical effects depending on the system & temperature, the obtained force constants could be helpful in validating the MLIP, as they are derived directly from MD snapshots.