Skip to content

Use stateless random ops #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
albertz opened this issue Jan 19, 2022 · 3 comments
Closed

Use stateless random ops #95

albertz opened this issue Jan 19, 2022 · 3 comments
Milestone

Comments

@albertz
Copy link
Member

albertz commented Jan 19, 2022

This makes all logic regarding randomness much more explicit.

The current sources of randomness are:

Maybe we can reimplement dropout here, or make this optionally using stateless random ops on RETURNN side.

RandIntLayer on RETURNN side should also have an option for using the stateless random op.

For param init, we probably need all the other common random ops anyway, like random_uniform, random_normal, random_truncated_normal. And then we directly should use the stateless variants.

@albertz
Copy link
Member Author

albertz commented Feb 1, 2022

We now have the generic RandomLayer (via rwth-i6/returnn#911) which supports stateless random ops, or rather explicit handling of state.

@albertz
Copy link
Member Author

albertz commented Feb 1, 2022

Should dropout still use the functional API or become a Module? Should we always make the state var for RandomLayer (nn.random) explicit or is this maybe something where we allow an exception?

Currently I tend to keep dropout as a function...

@albertz
Copy link
Member Author

albertz commented Feb 1, 2022

I think nothing really needs to be done here for now.

@albertz albertz closed this as completed Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant