Weight norm

[albertz]

Implement weight norm.
The implementation can look very similar to PyTorch.

Probably similar would be weight dropout (#100) or other transformations or reparameterizations of weights.

The more generic issue is #59.

[albertz]

Note that weight norm only makes sense on model parameters but not on auxiliary parameters (used for e.g. collecting running statistics or so). For that reason, the nn.Parameter auxiliary flag was introduced.

[albertz]

While implementing weight norm, I stumbled upon the problem of weight decay on its parameterization (#241). Specifically, we would not want weight decay on g (or do we?).

In Lingvo, they solved this by parameterizing it as 1 + g instead (here). Do we also want this? Or do it just like layer norm?

[albertz]

We now have an initial implementation, see nn.weight_norm.

I ignore some of the raised aspects here, namely:

• Weight decay on it. Let's continue the discussion in Weight decay API maybe unintuitive #241.
• How it works well together when we use other backends (e.g. PyTorch). But this is more for Support different backends for nn returnn#1264.