Reproducibility

[lmnt-com]

Reproducibility

[sharvil]

There isn't any checked in code yet that demonstrates that cuDNN and Haste compute the same function but we've locally verified the computation for the forward and backward passes of the LSTM and GRU layers. We also perform gradient checking to make sure the backward pass is correct.

I should also mention that I pushed a change earlier today that converts Haste LSTM/GRU layer weights to native PyTorch layer weights (see #3). It should be fairly straightforward to write a small program that verifies behavior if you need to.

[sharvil]

Here's some sample code that shows the native PyTorch LSTM and Haste LSTM computing (approximately) the same values:

import torch
import haste_pytorch as haste

batch_size = 256
time_steps = 250
input_size = 128
hidden_size = 256

x = torch.rand(time_steps, batch_size, input_size).cuda()
x.requires_grad_(True)

lstm = torch.nn.LSTM(input_size, hidden_size, 1).cuda()
y, _ = lstm(x)
grad = torch.rand_like(y)
y.backward(grad)
g1 = x.grad.data.clone()
x.grad.data.zero_()

hlstm = haste.LSTM(input_size, hidden_size)
hlstm.from_native_weights(
    lstm.weight_ih_l0,
    lstm.weight_hh_l0,
    lstm.bias_ih_l0,
    lstm.bias_hh_l0)
y, _ = hlstm(x)
y.backward(grad)
g2 = x.grad.data.clone()
x.grad.data.zero_()

print(torch.max(torch.abs(g2-g1)))

Let me know if you still have questions about cuDNN vs Haste behavior.