Speed up training and inference: AMP, compile, fused Adam, batching - #166
Open
jcwang587 wants to merge 1 commit into
Open
Speed up training and inference: AMP, compile, fused Adam, batching#166jcwang587 wants to merge 1 commit into
jcwang587 wants to merge 1 commit into
Conversation
Training (cgcnn-tr / cgcnn-ft), on CUDA devices: - bf16 torch.autocast around the forward pass and loss - model.compile() (in-place, keeps state_dict keys clean) - fused Adam Data movement: - non_blocking=True on all host-to-device transfers so pin_memory actually overlaps copies with compute - persistent_workers when DataLoader workers > 0, keeping worker processes and their per-process LRU caches alive across epochs Inference: - cgcnn_pred batches inference (new batch_size parameter, default 256) instead of a hardcoded batch size of 1 - cgcnn_descriptor handles batches correctly; each crystal's descriptor is now a 1D (h_fea_len,) array instead of (1, h_fea_len) Cleanups: cgcnn_ft no longer builds a throwaway CIFData just to read feature dimensions, layer resets use .to(args.device) instead of conditional .cuda(), and a redundant .to(device) on the Boltzmann bias tensor is gone. Tutorial notebooks are re-executed against the new behavior and p02's torch.load cell modernized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Training (
cgcnn-tr/cgcnn-ft), on CUDA devices:torch.autocastaround forward pass and lossmodel.compile()(in-place, keepsstate_dictkeys clean)Data movement:
non_blocking=Trueon all host-to-device transfers sopin_memoryactually overlaps copies with computepersistent_workerswhen DataLoader workers > 0, keeping workers and their per-process LRU caches alive across epochsInference:
cgcnn_predbatches inference (newbatch_sizeparameter, default 256) instead of hardcoded batch size 1cgcnn_descriptorhandles batches correctly; each crystal's descriptor is now a 1D(h_fea_len,)array instead of(1, h_fea_len)-np.vstack/np.arrayconsumers are unaffectedCleanups:
cgcnn_ftno longer builds a throwawayCIFDatajust to read feature dimensions; layer resets use.to(args.device); redundant.to(device)on the Boltzmann bias removed. Tutorial notebooks re-executed against the new behavior and p02'storch.loadcell modernized.Verification
cgcnn-tr,cgcnn-ft,cgcnn-pr, and thecgcnn_predAPI (including the bundled pretrained model) all pass on this branch.Part 4/4, stacked on #165 (
chore/safe-checkpoint-loading).🤖 Generated with Claude Code