[BUG] `dp eval-desc` give mismatch descriptors on certain System in certain GPU node

[QuantumMisaka]

Bug summary

On certain system, dp eval-desc will generate mismatch number of descriptor frames against System (number of frames is larger, where descriptors have 241 frames and the dpdata.System only have 175 frames)

DeePMD-kit Version

3.1.2 (modified as #5177)

Backend and its version

Pytorch

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

descritor_mismatch_example.tar.gz

• ./data contain target dpdata.System
• ./desc contain mismatch descriptor

Steps to Reproduce

dp --pt eval-desc -s [data] -m DPA-3.1-3M.pt --head Hybrid_Perovskite

Further Information, Files, and Links

No response

[dosubot]

This mismatch is a known issue, especially for multi-frame systems in the deepmd/npy/mixed format. It usually happens when .npy files in your system directory have inconsistent frame or atom counts, or if atomic and global tensor files are mixed up. All frames must have the same number of atoms and consistent atom types to avoid this problem.

Try these steps: check your input files for consistency, run dp eval-desc with fewer frames using the -n parameter, or set the environment variable DP_INFER_BATCH_SIZE to a lower value. If the mismatch persists, process each frame or system individually and concatenate the results to sidestep batch processing issues. Also, make sure your model file is compatible with the DeePMD-kit version you’re using—older frozen models can cause problems with descriptor evaluation. More details and workarounds are discussed here.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[njzjz]

The shapes in your example look good

>>> import numpy as np
>>> np.load("data/Fe0O0C6H5/set.000/coord.npy").shape
(644, 33)
>>> np.load("data/Fe0O0C6H5/set.000/box.npy").shape
(644, 9)
>>> np.load("desc/Fe0O0C6H5.npy").shape
(644, 11, 128)

[QuantumMisaka]

@njzjz Wait a minute, I upload mistake files

[QuantumMisaka]

In [4]: np.load("data/Fe-C-ActMat2024/C27Fe96/set.000/coord.npy").shape
Out[4]: (175, 369)

In [5]: np.load("desc_cli/Fe-C-ActMat2024/C27Fe96.npy").shape
Out[5]: (241, 123, 128)

descritor_mismatch_example.tar.gz

[njzjz]

I failed to reproduce the error.

(base) jzzeng@node3:~/tmp/descritor_mismatch_example$ srun --gres=gpu:1 dp eval-desc -m DPA3.pth -s data/ -o test1
[2026-01-29 17:44:33,052] DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-01-29 17:44:36,102] DEEPMD INFO    # -------output of dp eval_desc-------
[2026-01-29 17:44:36,102] DEEPMD INFO    # processing system : data/C27Fe96
[2026-01-29 17:44:36,163] DEEPMD INFO    # evaluating descriptors for 175 frames
[2026-01-29 17:44:44,680] DEEPMD INFO    Adjust batch size from 1024 to 2048
[2026-01-29 17:44:56,268] DEEPMD INFO    Adjust batch size from 2048 to 4096
[2026-01-29 17:45:29,060] DEEPMD INFO    Adjust batch size from 4096 to 8192
[2026-01-29 17:45:44,786] DEEPMD INFO    Adjust batch size from 8192 to 4096
[2026-01-29 17:46:23,447] DEEPMD INFO    # descriptors saved to test1/C27Fe96.npy
[2026-01-29 17:46:23,447] DEEPMD INFO    # descriptor shape: (175, 123, 128)
[2026-01-29 17:46:23,447] DEEPMD INFO    # -----------------------------------
[2026-01-29 17:46:23,447] DEEPMD INFO    # eval_desc completed successfully
(base) jzzeng@node3:~/tmp/descritor_mismatch_example$ python
Python 3.13.9 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 19:16:10) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.load("test1/C27Fe96.npy").shape
(175, 123, 128)
>>>

[QuantumMisaka]

I can't reproduce the error either in deepmd-kit 3.1.2 in CPU.

(/opt/apps/conda_env/deepmd-kit-3.1.2) liuzhaoqing@login-01:~/WORK/FT2DP-DPEVA/dpeva/test/mismatch_repro/descritor_mismatch_example/data$ dp --pt eval-desc -m ../../../DPA-3.1-3M.pt -s C27Fe96 --head OC22
[2026-01-29 21:08:34,115] DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-01-29 21:08:36,841] DEEPMD WARNING You can use the environment variable DP_INFER_BATCH_SIZE tocontrol the inference batch size (nframes * natoms). The default value is 1024.
[2026-01-29 21:08:38,439] DEEPMD WARNING You can use the environment variable DP_INFER_BATCH_SIZE tocontrol the inference batch size (nframes * natoms). The default value is 1024.
[2026-01-29 21:08:38,442] DEEPMD INFO    # -------output of dp eval_desc------- 
[2026-01-29 21:08:38,442] DEEPMD INFO    # processing system : C27Fe96
[2026-01-29 21:08:38,553] DEEPMD INFO    # evaluating descriptors for 175 frames
[2026-01-29 21:12:54,348] DEEPMD INFO    # descriptors saved to desc/C27Fe96.npy
[2026-01-29 21:12:54,348] DEEPMD INFO    # descriptor shape: (175, 123, 128)
[2026-01-29 21:12:54,348] DEEPMD INFO    # ----------------------------------- 
[2026-01-29 21:12:54,348] DEEPMD INFO    # eval_desc completed successfully

(/opt/apps/conda_env/deepmd-kit-3.1.2) liuzhaoqing@login-01:~/WORK/FT2DP-DPEVA/dpeva/test/mismatch_repro/descritor_mismatch_example/data$ dp --pt eval-desc -m ../../../DPA-3.1-3M.pt -s C27Fe96 --head Hybrid_Perovskite
[2026-01-29 21:13:31,285] DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-01-29 21:13:34,048] DEEPMD WARNING You can use the environment variable DP_INFER_BATCH_SIZE tocontrol the inference batch size (nframes * natoms). The default value is 1024.
[2026-01-29 21:13:35,843] DEEPMD WARNING You can use the environment variable DP_INFER_BATCH_SIZE tocontrol the inference batch size (nframes * natoms). The default value is 1024.
[2026-01-29 21:13:35,846] DEEPMD INFO    # -------output of dp eval_desc------- 
[2026-01-29 21:13:35,846] DEEPMD INFO    # processing system : C27Fe96
[2026-01-29 21:13:35,951] DEEPMD INFO    # evaluating descriptors for 175 frames
[2026-01-29 21:17:32,644] DEEPMD INFO    # descriptors saved to desc/C27Fe96.npy
[2026-01-29 21:17:32,644] DEEPMD INFO    # descriptor shape: (175, 123, 128)
[2026-01-29 21:17:32,644] DEEPMD INFO    # ----------------------------------- 
[2026-01-29 21:17:32,644] DEEPMD INFO    # eval_desc completed successfully

However I can reproduce this error in GPU

~/WORK/FT2DP-DPEVA/dpeva/test/mismatch_repro/descritor_mismatch_example/data$ srun --gres=gpu:1 --partition=4V100 --qos=rush-1o2gpu  dp --pt eval-desc -m ../../../DPA-3.1-3M.pt -s C27Fe96 --head Hybrid_Perovskite
[2026-01-29 21:18:54,330] DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-01-29 21:18:56,547] DEEPMD INFO    # -------output of dp eval_desc------- 
[2026-01-29 21:18:56,548] DEEPMD INFO    # processing system : C27Fe96
[2026-01-29 21:18:56,569] DEEPMD INFO    # evaluating descriptors for 175 frames
[2026-01-29 21:18:58,378] DEEPMD INFO    Adjust batch size from 1024 to 2048
[2026-01-29 21:19:00,596] DEEPMD INFO    Adjust batch size from 2048 to 4096
[2026-01-29 21:19:07,477] DEEPMD INFO    Adjust batch size from 4096 to 8192
[2026-01-29 21:19:12,309] DEEPMD INFO    Adjust batch size from 8192 to 4096
[2026-01-29 21:19:18,594] DEEPMD INFO    # descriptors saved to desc/C27Fe96.npy
[2026-01-29 21:19:18,594] DEEPMD INFO    # descriptor shape: (241, 123, 128)
[2026-01-29 21:19:18,594] DEEPMD INFO    # ----------------------------------- 
[2026-01-29 21:19:18,594] DEEPMD INFO    # eval_desc completed successfully

Much strange. The GPU resource is 4V100 in S-A-I

[QuantumMisaka]

In 4V100PX

(/opt/apps/conda_env/deepmd-kit-3.1.2) liuzhaoqing@login-01:~/WORK/FT2DP-DPEVA/dpeva/test/mismatch_repro/descritor_mismatch_example/data$ srun --gres=gpu:1 --partition=4V100PX --qos=rush-1o2gpu  dp --pt eval-desc -m ../../../DPA-3.1-3M.pt -s C27Fe96 --head Hybrid_Perovskite -o desc_test_4V100PX
[2026-01-29 21:30:47,901] DEEPMD WARNING To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-01-29 21:30:50,149] DEEPMD INFO    # -------output of dp eval_desc------- 
[2026-01-29 21:30:50,150] DEEPMD INFO    # processing system : C27Fe96
[2026-01-29 21:30:50,171] DEEPMD INFO    # evaluating descriptors for 175 frames
[2026-01-29 21:30:51,989] DEEPMD INFO    Adjust batch size from 1024 to 2048
[2026-01-29 21:30:54,221] DEEPMD INFO    Adjust batch size from 2048 to 4096
[2026-01-29 21:31:00,704] DEEPMD INFO    Adjust batch size from 4096 to 2048
[2026-01-29 21:31:10,250] DEEPMD INFO    # descriptors saved to desc_test_4V100PX/C27Fe96.npy
[2026-01-29 21:31:10,250] DEEPMD INFO    # descriptor shape: (208, 123, 128)
[2026-01-29 21:31:10,250] DEEPMD INFO    # ----------------------------------- 
[2026-01-29 21:31:10,250] DEEPMD INFO    # eval_desc completed successfully

@Entropy-Enthalpy、@Stardust0831 You should see this

[QuantumMisaka]

Report by Gemini:

Diagnostic Report: Descriptor Mismatch Analysis

Issue Summary A frame count discrepancy was identified between the descriptor file ( desc/C27Fe96.npy , 241 frames ) and the reference structure file ( coord.npy , 175 frames ).

Data Integrity Analysis

• Validity: All 241 frames in the mismatched descriptor are valid . Every frame finds an exact match (distance ≈ 0) within the reference dataset.
• No Corruption: The mismatch is not caused by data corruption or incorrect calculation parameters, but rather by duplication and disordering of valid data.
  Detailed Anomaly Pattern The mismatch is characterized by block-wise duplication and a specific frame loss:

1. Redundant Writes (Total +67 frames):
   • Frame 0 : Written 3 times (+2).
   • Frames 57-92 (36 frames): Written 2 times (+36).
   • Frames 95-122 (28 frames): Written 2 times (+28).
   • Frame 157 : Written 2 times (+1).
2. Data Loss (-1 frame):
   • Frame 94 : Completely missing from the output.
     Root Cause Diagnosis The pattern of block-wise duplication combined with specific frame loss strongly suggests a Race Condition or Concurrency Bug in the I/O pipeline.

• Mechanism: Multiple workers or processes likely attempted to append to the same file without proper locking or atomic operations.
• Result: Overlapping write operations caused certain data blocks to be flushed twice (duplication) while overwriting or skipping the boundary frame (Frame 94).

[njzjz]

This may be related to the handling of the automatic batch size. But I don't know why I cannot reproduce the error.

[njzjz]

But I don't know why I cannot reproduce the error.

Maybe the model on my machine gets OOM before the variable is saved.

[Entropy-Enthalpy]

I suspect this is a problem specific to the distributed parallel file system of HPC (such as Lustre/BeeGFS/GPFS).

[iProzd]

Maybe we need to: first do autobatchsize --> clean up the desc results after OOM occured in autobatchsize --> do the evaluation from begining again with appropriate batchsize.