Bidirectional masking?

[lmnt-com]

Bidirectional masking?

[PetrochukM]

Sorry and thank you. We stopped using Haste because in our case it was slower than PyTorch and it didn't provide a significant enough performance increase for us to justify the additional overhead.

---

PyTorch API does support bidirectional RNNs, and we use them.

In order to support reverse_sequence, we used roll and flip. We implemented our own roll function, see here:

def roll(tensor, shift, dim=-1):
    """ Shift a tensor along the specified dimension.
    TODO: Create a `Roll` module so that `indices` are not recomputed each time.
    Args:
        tensor (torch.Tensor [*, dim, *]): The tensor to shift.
        shift (torch.Tensor [*]): The number of elements to shift `dim`. This tensor must have one
            less dimensions than `tensor`.
        dim (int): The dimension to shift.
    Returns:
        tensor (torch.Tensor [*, dim, *]): The tensor that was shifted.
    """
    shift = shift.unsqueeze(dim)
    assert shift.dim() == tensor.dim(
    ), 'The `shift` tensor must be the same size as `tensor` without the `dim` dimension.'
    indices = torch.arange(0, tensor.shape[dim], device=tensor.device)
    dim = tensor.dim() + dim if dim < 0 else dim

    # EXAMPLE:
    # indicies.shape == (3,)
    # tensor.shape == (1, 2, 3, 4, 5)
    # indices_shape == [1, 1, 3, 1, 1]
    indices_shape = [1] * dim + [-1] + [1] * (tensor.dim() - dim - 1)
    indices = indices.view(*tuple(indices_shape)).expand(*tensor.shape)

    indices = (indices - shift) % tensor.shape[dim]
    return torch.gather(tensor, dim, indices)

PyTorch does not provide a lengths parameter; therefore, we needed to implement masking on our own in order to use the PyTorch LSTM. Thanks for providing that mechanism!

We were using LSTMCell with unrolled sequences; therefore, we needed to convert our lengths to binary masks for each call.

[PetrochukM]

Furthermore, can this be used like an LSTMCell (https://pytorch.org/docs/stable/nn.html#torch.nn.LSTMCell) with the lengths parameter being used more like a binary mask?

[sharvil]

The PyTorch API doesn't provide bidirectional RNNs out of the box (the TensorFlow API does). Unfortunately, PyTorch doesn't have a reverse_sequence function so you'll either need to write your own or use the implementation here: pytorch/pytorch#1794.

Suppose you have reverse_sequence. Then the bidirectional RNN would look something like this:

reversed_inputs = reverse_sequence(inputs, lengths)
rnn_fwd = LSTM(...)
rnn_bwd = LSTM(...)

y_fwd, _ = rnn_fwd(inputs, lengths)
y_bwd, _ = rnn_bwd(reversed_inputs, lengths)
y_bwd = reverse_sequence(y_bwd, lengths)
y = concatenate([y_fwd, y_bwd], dim=-1)

I'm not sure what you mean by the lengths parameter being used like a binary mask. AFAICT, the standard PyTorch LSTMCell doesn't accept a lengths parameter. If you have a one-hot vector indicating end-of-sequence, you can just just do argmax(one_hot) + 1 to convert to lengths.

[sharvil]

Closing due to inactivity.