Allow non-deterministic layers in DMoE

[learning-at-home]

Allow non-deterministic layers in DMoE

[mryab]

Because different random states may be used for different batch elements at backward, tracking them all (e.g. changing CUDA random state for each example in the batch) is a huge performance hit. If you want complete determinism, a somewhat better option is to generate all random variables on the client and send them as inputs to the expert.

hivemind.utils.custom_layers.DeterministicDropout is an example of such a layer for the simplest case of dropout. You can see an example of its usage in tests/test_mode.py:test_determinism

[mryab]

Solved by #31.