Testing?

[lmnt-com]

Testing?

[sharvil]

There are essentially 4 approaches that I'm taking.

If there are known good implementations in core PyTorch or TensorFlow, I check the output of my implementation against those. I do this for LSTM and GRU.

If there aren't such implementations, I write the Python equivalent forward pass, let autodiff give me gradients for the backward pass, and check my C++ forward/backward pass outputs against those.

To make sure PyTorch and TensorFlow are in sync, I compute forward/backward outputs from TensorFlow, copy weights to the PyTorch implementation and check those outputs against what I got from the TF implementation.

Finally, I do gradient checking locally during development to make sure I've derived the right equations and that my implementation matches the derivation. I haven't checked this code in yet.

You're right in saying that I don't have checked-in tests for LayerNorm or Zoneout, though I've verified them locally. The trouble with Zoneout in particular is that it's stochastic so black-box testing won't work.

As for trustworthiness, I encourage you to try Haste for yourself. It's usually pretty easy to see if an implementation is good or not. Neural nets are unforgiving when it comes to bad gradients.

[sharvil]

BTW, if you think there are other approaches we could be taking to test Haste, I'd love to hear them! Code contributions would be even better. :)

[PetrochukM]

Great. Those testing strategies are good :) I think it'd be helpful if your tests were checked in, documented, and reproducible. It's hard to trust, otherwise.

For us to get state-of-the-art performance, we need to be able to triple check via tests!

I'd be happy to make code contributions if I was still using Haste. Sorry!