Memory Overflow During Band Structure Prediction Fine-tuning with Si 3x3x3 Supercell

[3532564593]

Dear HamGNN Development Team,

I'm currently working on fine-tuning the HamGNN model for band structure prediction using silicon 3×3×3 supercells. Despite implementing several memory optimization strategies, I'm experiencing GPU memory overflow during the fine-tuning phase.

Here's my current setup and what I've tried:

Task: Fine-tuning for band structure prediction

System: Si 3×3×3 supercells

Batch size: Reduced to 1

Memory status: ~27.9GB/32GB used before fine-tuning

The memory overflow occurs specifically during the fine-tuning stage, even with these optimizations in place.

I would appreciate your insights on:

What could be causing this memory overflow despite the minimal batch size?

Are there specific memory optimization techniques you recommend for fine-tuning with large supercells?

What GPU models and memory configurations does your team typically use for training with 3×3×3 supercells?

Thank you for your time and assistance. I'm happy to provide any additional details about my setup if needed.

Best regards

[QuantumLab-ZY]

Hi @3532564593,

Could you please share the error you obtained after the first-round training on the Hamiltonian matrix?
If the error is already sufficiently low, it might not be necessary to proceed with the second-round band structure training.

Best regards,
Yang Zhong

[3532564593]

Dear Professor Zhong,

Thank you very much for your prompt and helpful response.

As you suggested, I have attached the error plot obtained after the first-round training on the Hamiltonian matrix. The error values appear to be quite low, but I would like to confirm whether it is acceptable to proceed directly to the band structure calculation without a second round of training.

Additionally, I have a follow-up question regarding computational resources. Currently, for a 3×3×3 Si supercell, the training process requires approximately 30 GB of GPU memory. If I intend to study larger systems in the future (e.g., doped structures), would it be feasible to apply perturbations directly to the primitive cell instead of constructing a supercell? Could this approach help mitigate potential GPU memory issues?

Any insights or suggestions you might have would be greatly appreciated.

Thank you once again for your time and support.

Best regards,
Jiaxin Han

[QuantumLab-ZY]

This level of precision is sufficient to obtain a reliable band structure. You can have a test using this model first.

For defect systems, constructing a supercell is generally necessary. The size of the supercell to be constructed depends on your GPU's memory.