torch.OutOfMemoryError: CUDA out of memory with Runpod A100

[nanaj96]

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 42.00 MiB. GPU 0 has a total capacity of 44.52 GiB of which 2.00 MiB is free. Process 280468 has 5.57 GiB memory in use. Process 280474 has 5.35 GiB memory in use. Process 280470 has 5.53 GiB memory in use. Process 280467 has 5.57 GiB memory in use. Process 280473 has 5.75 GiB memory in use. Process 280472 has 5.49 GiB memory in use. Process 280469 has 5.65 GiB memory in use. Process 280471 has 5.57 GiB memory in use. Of the allocated memory 4.92 GiB is allocated by PyTorch, and 134.41 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)