Stochastic depth

[albertz]

We should have this implemented.
The name varies a bit in related literature and other implementations. Sometimes it is called stochastic layers, sometimes drop path, layer drop, even drop connect although that usually means sth different (dropout on weights).

This was introduced in this paper: Deep Networks with Stochastic Depth

This is commonly used for Transformers. For example:

• Very Deep Self-Attention Networks for End-to-End Speech Recognition
• Reducing Transformer Depth on Demand with Structured Dropout
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

External example implementations:

• quanpn90 / NMTGMinor (Very Deep Self-Attention Networks for End-to-End Speech Recognition). Jointly applied to attention and FF layer.
• torchvision (PR)
• rwightman / pytorch-image-models: Swin Transformer, DropPath
• Fairseq layerdrop

This needs to depend (conditional logic, #24) on the train flag (#18).