Exe2: Different performance for GPU and CPU

[shani1610]

Hello,
Thank you for this great platform!
I'm working on the classification exercise,
I started by writing the code for CPU and once I got the desired loss and accuracy, I changed the device to GPU.
I get different performance even though I didn't change anything else but the device, does is makes sense? where should I look for the cause of this difference?
Could it be from the round function in:
y_pred = torch.round(torch.sigmoid(y_logits)) ?

the performance is shown in the plot, the upper is for GPU and the bottom is the CPU.

Thank you!

[0xm4ze]

Yes, it’s quite possible that changing the device from CPU to GPU could affect the performance of your model, even if you haven’t changed any other part of your code. Here are a few things to check:

1. Random Seed Initialization

Ensure that you have set the same random seed for both CPU and GPU computations. Different devices might use different implementations of operations, which can cause slight variations in results.

import torch
import numpy as np

seed = 42
torch.manual_seed(seed)
np.random.seed(seed)

2. Batch Size and Memory

When using a GPU, you might be using a different batch size compared to when using the CPU. Ensure that you’re using the same batch size for both CPU and GPU computations. If the GPU runs out of memory, it might be using a smaller batch size or may even cause errors or unexpected behaviors.

3. Deterministic Operations

Some operations in PyTorch are non-deterministic on GPUs (e.g., certain CUDA operations). You can set deterministic behavior with the following code:

torch.use_deterministic_algorithms(True)

This might help in making sure that your results are consistent.

4. Numerical Precision

GPUs often use single precision (float32) by default, which might lead to small numerical differences compared to double precision (float64) used on CPUs. Ensure that you are aware of these differences and consider normalizing or scaling your data to mitigate their effects.

5. Code and Library Versions

Ensure that the versions of PyTorch and other libraries are the same across your CPU and GPU setups. Different versions might have different implementations or optimizations that could affect performance.

6. Round Function Impact

The torch.round() function applied after torch.sigmoid() may indeed cause some variation, especially if your logits are very close to 0 or 1. This can be more noticeable on GPUs due to differences in precision and numerical stability. Consider using a different thresholding method or evaluate the impact of rounding by checking the distribution of your logits.

7. Hardware Differences

Different GPUs and CPUs might have different performance characteristics, which can influence the training and inference speed. Consider profiling the execution time on both devices to see if there are any bottlenecks or inefficiencies.

In summary, while it's normal to see some variations in performance when switching between CPU and GPU, checking the above factors should help you identify and mitigate any issues.