Training & Loss Curves

[nkarast]

Hi all,

Many thanks for the time and effort put into making these videos and the material available for the community! 🥇

I was following along the implementation and training of a CNN on the MNIST and Pizza/Steak/Sushi datasets tutorial and I noticed that during training, for few epochs I kept getting higher accuracy on the test compared to the metric on the training set. Then I was following the exercise solution video and I noticed that Daniel is getting a similar output.

Epoch: 0 | Loss: 0.324 | Test Loss: 0.085
Epoch: 1 | Loss: 0.077 | Test Loss: 0.055
Epoch: 2 | Loss: 0.061 | Test Loss: 0.103
Epoch: 3 | Loss: 0.053 | Test Loss: 0.044
Epoch: 4 | Loss: 0.045 | Test Loss: 0.036

Plotting the loss curves, we get test loss below the train loss. Now, from intuition I would expect that the test loss is (generally) always worse (higher) than the training loss and that as epochs go on, the ideal model (as described in the overfitting video) would have the test loss go down and approach the training loss curve.
We also see a similar behaviour with the pre-trained EfficientNet.

I was wondering whether my understanding of the loss curves is wrong. Could that be an artefact of the way the train/test sequence is set up?
From your experience, are there some specific cases where this behaviour can be alarming for a model (e.g. a silent bug) or is it generally not relevant which curve is higher as long as they move at the same direction?

Many thanks for your insights!

[Pouyaexe]

This behavior is common in many instances, specially when working on real-world problems. There could be lots of reasons behind it like using the L1 or L2 regressions. Maybe the testing samples had much simpler images to predict on, for example. I suggest you to read this article:
https://towardsdatascience.com/what-your-validation-loss-is-lower-than-your-training-loss-this-is-why-5e92e0b1747e
and this posts too:
https://stats.stackexchange.com/questions/187335/validation-error-less-than-training-error