Training classification models on Test set

[WissamElJ]

Hi!
First of all, thank you so much for the detailed curriculum that you guys made for PyTorch!

I have a question regarding the .train() stage and .eval() stage.
In the training stage, we are outputting the predictions on the test_dataloader.
Then in inference mode, we are outputting also the predictions on the test_dataloader.
Doesn't that give us the same result at the end?

Did you skip the validation set for simplicity (which in this case, replaces the test data in the training stage)

[LuluW8071]

In training stage, preds for train dataloader is outputted while on inference stage its prediction on test dataloader .
Its not same.

During model.train(), all the weights at each layers including dropout, batchnorm, layernorm are initialized and gives raw logits on train dataloader which is then transformed into probabilities and hence we get predictions

During model.eval(), weights of train mode are saved but dropout and normalization layers are excluded(turned off), and then generate predictions

[WissamElJ]

Are the dropout, batchnorm and layernorm have a standard placing in the neural network? I can see that in the PyTorch docs that we can initialize them.
Should we pay attention to them as we are building more complex neural networks? or stick to the way Pytorch uses them?

[LuluW8071]

Yes, these normalization layers have their own standard placing and advantages as to

• capture features faster by covariance shifting,
• add stability,
• deals with possibilities of overfit/underfit,
• dropout features, and many more.

These are still used in SOTA models. Even LLMs have their own normalization called RMSNorm.

You can read batch normalization paper here for starters.

[WissamElJ]

Thank you so much!