You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Maybe we can reimplement dropout here, or make this optionally using stateless random ops on RETURNN side.
RandIntLayer on RETURNN side should also have an option for using the stateless random op.
For param init, we probably need all the other common random ops anyway, like random_uniform, random_normal, random_truncated_normal. And then we directly should use the stateless variants.
The text was updated successfully, but these errors were encountered:
Should dropout still use the functional API or become a Module? Should we always make the state var for RandomLayer (nn.random) explicit or is this maybe something where we allow an exception?
This makes all logic regarding randomness much more explicit.
The current sources of randomness are:
dropout
rand_int
Maybe we can reimplement
dropout
here, or make this optionally using stateless random ops on RETURNN side.RandIntLayer
on RETURNN side should also have an option for using the stateless random op.For param init, we probably need all the other common random ops anyway, like random_uniform, random_normal, random_truncated_normal. And then we directly should use the stateless variants.
The text was updated successfully, but these errors were encountered: