ndrplz / convlstm_pytorch Goto Github PK
View Code? Open in Web Editor NEWImplementation of Convolutional LSTM in PyTorch.
License: MIT License
Implementation of Convolutional LSTM in PyTorch.
License: MIT License
Hey,
I am trying to integrate your convLSTM cell into an existing model I have.
I did this in the following way:
def forward(self,x):
out = my_model(x)
out = out.unsqueeze(1) #make sure dimensions fit
out, self.hidden = self.convlstm(out, self.hidden)
return out[-1].squeeze(1)
self.hidden is none in the first run, but not none in the second, leading to:
# Implement stateful ConvLSTM
if hidden_state is not None:
raise NotImplementedError()
which is in your ConvLSTM module (line 141, 142)
would this be implemented just by:
# Implement stateful ConvLSTM
if hidden_state is not None:
hidden_state=hidden_state`
or am I misunderstanding something?
What is supposed to happen here?
Thanks for any help :-D
As stated in the title, doesn't it support bi-directional multi-layer LSTM?
I have a question regarding your implementation:
As I understood the original convolutional lstm formulation is as follows:
But in your implementation, u used only one convolution layer. I don't understand how these 2 correspond with each other. because in the formulation, c is only used in the Hadamard product and not in convolutions, but here c and h are both used in convolutions.
in fact, all weights are shared for all 4 formulas, although there are 11 weights in the original formula.
sorry, i didn't really understand conLSTM
when i use keras layers ConvLSTM2D(filters = 128, kernel_size=(3, 3), padding='same', return_sequences = False, go_backwards = True,kernel_initializer = 'he_normal' ),such as input shape is (batch,2,h,w,channel),2 i guess is time,and the output is (batch,h,w,128)
but in your code ,i didn't get same shape, can you help me to get the shape like keras convLSTM,thx
First off thank you for the implementation. Are Hadamard products in the paper missing? Please see the image below and paper that I am referring to for clarification.
I am not sure how to handle the situation of various length of input sample. I think padding is not okay for my problem. Is there any better way to solve it?
model = ConvLSTM(input_dim=channels,
hidden_dim=[64, 64, 128],
kernel_size=(3, 3),
num_layers=3,
batch_first=True[missing comma here]
bias=True,
return_all_layers=False)
how can we implement options for valid padding and using custom strides, where are changes required?
why out_channels=4 * self.hidden_dim?
Sorry but I've added a dropout layer behind the convolution, and the memory of the GPUs explode. How could I solve this?
Hello I am trying to reproduce the results of the paper in moving Mnist.
I developed my own implementation but it didnt converge. Now I will try with your proposed implementation.
I was wondering have you managed to use it in any dataset link Moving Mnist?
Does it converge?
By the way, I am not 100% sure how you implement the input to state transitions and final state to output transition. Do you do it via 2D(in the timestep) or 3D convs (in all the sequence)
Any ideas?
N.A.
First thanks for your implementing the conv-lstm which is useful in the video predict, I like to use pytorch, and find your implement, have you succeed training on the toy Moving MNIST dataset?
This code snippet taken and changed slightly from the docstring fails for me:
import torch
from CONVLSTM_Implementation import ConvLSTM
x = torch.rand((32, 10, 64, 128, 128))
convlstm = ConvLSTM(64, 16, (2, 2), 1, True, True, False)
_, last_states = convlstm(x)
with the following Error:
c_next = f * c_cur + i * g
RuntimeError: The size of tensor a (129) must match the size of tensor b (128) at non-singleton dimension 3
But I don't really know why. I guess I will use a different kernel_size for now.
Hey I was not sure how the logic in line 49 works
cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)
Since each of the 4 gates has operations of weights with inputs, how is the order of split determined? Why not something like cc_g, cc_f, cc_i, cc_go = torch.split(combined_conv, self.hidden_dim, dim=1)
? I am a bit confused how the LSTM equations in https://pytorch.org/docs/stable/nn.html#torch.nn.LSTMCell are implemented here.
Thanks in advance.
when i build my model with convlstm in the function def forward(self, x):
summary[m_key]["input_shape"] = list(input[0].size())
in torchsummary.py
the input is empty tuple, so the program error with '{IndexError}tuple index out of range'
Hello there, really interested in your work but i am still understanding the implementation. Can you explain you have used stacked version of x_(t) and h_(t-1) for convolution according to this:
but in implementation you used h_cur and c_cur rather then h_(t-1)?
Will really appreciate your quick answer.
Hello!
Is there any examples how to run training on 3 channels frames sequence?
When added the code below, I got an error:" RuntimeError: expected stride to be a single integer value or a list of 3 values to match the convolution dimensions, but got stride=[1, 1] ", does anyone tell me why this happen?
def main():
model = ConvLSTM(input_dim=3,
hidden_dim=[64, 64, 128, 128],
kernel_size=(3, 3, 3),
num_layers=4,
batch_first=True,
bias=True,
return_all_layers=False
)
print(model)
x = torch.randn((32, 10, 64, 128, 128))
now_states, last_states = model(x)
if name == "main":
main()
I noticed in the code that the input sequence updates the same cell iteratively before the hidden state is passed to the next layer. My understanding of LSTMs is that there can be multiple cells per layer? These cells then act as a sliding window over time, where cell 0 takes input from t0, cell 1 takes input from t1 and hidden state from cell 0, and so on. Can I clarify about this?
Sorry, my English is not very good. Why split_size_or_sections is self.hidden_dim? There are only 4 variables to receive the result, I think self.hidden_dim should be changed to 4
44 cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)
I saw that in the code, hidden_state is not implemented:
def forward(self, input_tensor, hidden_state=None):
"""
Parameters
----------
input_tensor: todo
5-D Tensor either of shape (t, b, c, h, w) or (b, t, c, h, w)
hidden_state: todo
None. todo implement stateful
meanwhile, hidden_dim is given.
What is the difference between those two variables?
If you run the example toy data in the script you will get the above error in pytorch 0.40. I am not sure if the eailer version will cause this issue. How can you make sure this is working?
Hi,
I just wanted to know if this implementation would work well in a DataParrallel model and if there was some adaptation to do ?
Thanks
I am trying to train a simple model:
model = seq2seq(input_dim=1,
hidden_dim=[64],
num_layers=1,
kernel_size=(3, 3),
batch_first=True,
bias=True,
return_all_layers=False).to(device)
but the computational graph is broken somewhere and I can't seem to find the place.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.