zhangpur/sr-lstm

States Refinement LSTM

sr-lstm's Issues

Question about data prerpocessing

Hi! I am really impressed by your work on sr-lstm and I want to ask a few questions about it to better understand your work. When I am dealing with the crowd dataset, I cropped 20 time-step window and it is usually the case that some pedestrian won't show up for all 20 time steps. How did you tackle this problem? When you used origin-shift technique to preprocess the dataset, did you shift the coordinates of all pedestrians at the last observed time step to zero? If that is the case, shouldn't that affect the spatial relationship between neighbors?

Thank you in advance and look forward to your reply!

Question about data preprocessing

Hi!

First of all thanks for your interesting work :)

I'm looking into your code (specifically, the data pre-processing stages) because I saw it has been cited/used in other trajectory prediction works and I'm interested too in giving it a try. While scrolling through the different pre-processing steps I found something odd when extracting the datasets sampling indexes and I don't know if it's some kind of bug or if the behaviour is instead intended. More precisely, I'm talking about the get_data_index() function inside utils.py.

SR-LSTM/utils.py

Lines 148 to 162 in 22de027

	def get_data_index(self,data_dict,setname,ifshuffle=True):
	'''
	Get the dataset sampling index.
	'''
	set_id = []
	frame_id_in_set = []
	total_frame = 0
	for seti,dict in enumerate(data_dict):
	frames=sorted(dict)
	maxframe=max(frames)-self.args.seq_length
	frames = [x for x in frames if not x>maxframe]
	total_frame+=len(frames)
	set_id.extend(list(seti for i in range(len(frames))))
	frame_id_in_set.extend(list(frames[i] for i in range(len(frames))))
	all_frame_id_list = list(i for i in range(total_frame))

In the first two lines inside the for loop, you're extracting all the frame IDs inside a given .txt file and you're subtracting the desired sequence length (as this task requires, 20 timesteps) from the last (greatest) ID: if I understand correctly what you're trying to do, you want to get rid of the last n frame IDs (n=sequence length) inside the frames list, in order to not exceed the last available frame when you'll afterwards extract the real 20-steps trajectories from the raw data.

If this is the case, I think the code is not working as expected, since

SR-LSTM/utils.py

Line 157 in 22de027

maxframe=max(frames)-self.args.seq_length

does not eliminate the last n IDs but only the last 2, because the IDs in frames are separated by a step of 10 (0, 10, 20, 30, ...) and not by a step of 1 (0, 1, 2, 3, ...) as expected.

If it's not the case, I didn't fully understand those lines.

Thanks in advance for your time :)
Alex

Hello! Excuse me

Hello!I want to ask what is the meaning of SKIP =[6,10,10,10,10,10,10] in the UTIls code of SR-LSTM?Thank you very much!

About train bug

when I use the train.py, have a bug like this

In Processor.py line 136, train_loss, _ =self.train_epoch(epoch), can not use the loss.backward()

Visualization of trajectories

Since there are some questions about visualization, I've listed the solutions below.

1. Use the ORIGINAL data (trajectories in meters, which I've updated named with 'true_pos_origin.csv') and download videos from the ETH/UCY website.

2. Value of the homography:
   `Homo_set= [np.array([[2.8128700e-02, 2.0091900e-03, -4.6693600e+00],
   [8.0625700e-04, 2.5195500e-02, -5.0608800e+00],
   [3.4555400e-04 , 9.2512200e-05 , 4.6255300e-01]]),

   np.array([[1.1048200e-02, 6.6958900e-04, -3.3295300e+00],
   [-1.5966000e-03 , 1.1632400e-02 , -5.3951400e+00],
   [1.1190700e-04 , 1.3617400e-05 , 5.4276600e-01]]),

   np.array( [[-2.5956517e-02, -5.1572804e-18, 7.8388681e+00],
   [-1.0953874e-03 , 2.1664330e-02 , -1.0032272e+01],
   [1.9540125e-20, 4.2171410e-19 , 1.0000000e+00]]),

   np.array([[-2.5956517e-02, -5.1572804e-18, 7.8388681e+00],
   [-1.0953874e-03, 2.1664330e-02, -1.0032272e+01],
   [1.9540125e-20, 4.2171410e-19, 1.0000000e+00]]),

   np.array([[ -2.3002776e-02, 5.3741914e-04, 8.6657256e+00],
   [ -5.2753792e-04, 1.9565153e-02, -6.0889188e+00],
   [ 0.0000000e+00 , -0.0000000e+00 , 1.0000000e+00]])]`

corresponding to 5 sequences (homography of other sequences is not found):
data_dirs_v = ['./data/eth/univ', './data/eth/hotel', './data/ucy/zara/zara01', './data/ucy/zara/zara02', 'data/ucy/univ/students003']

3. Use this to get trajectories from meter to pixel.
   Homo = np.linalg.inv(Homo_set[setid]) pred_pixel = np.dot(Homo, pred_meter.transpose())

Then you can draw trajectories in pixels along with the video frames. Hope to help.

Rotate shift

Hello, great work there.
I don't really understand what does the shift rotate function do to the batchs, why did you use it and why did you use the batch, batch_norm (batch - shift value) and the shift value ?
Thanks a lot

When running train.py using default

Hello.
When I run train.py, it occurs that:

Exception has occurred: AttributeError
module 'models' has no attribute 'SRLSTM'
File "/mnt/e/SR-LSTM-master/utils.py", line 543, in import_class
mod = getattr(mod, comp)
File "/mnt/e/SR-LSTM-master/Processor.py", line 17, in init
model=import_class(args.model)
File "/mnt/e/SR-LSTM-master/train.py", line 210, in
processor = Processor(args)

Can you help me ?
Thanks.

The inputs in training and validating isn't the same

Thank you for sharing the code!

There are some questions I don't understand. Could you help me with that ? Thank you very much!
To predict the future trajectory of the human, I have read some paper, such as social-lstm. I want to run it and test the code, so I configured the environment as yours and run the program "SR-LSTM" successfully.

But when I train the network(SR-LSTM or vanilla-LSTM), the following results are obtained, like this.

----epoch 62, train_loss=0.00379, valid_error=0.569, valid_final=1.349,test_error=0.000,valid_final=0.000

The train_loss is small, but valid_error is large which no longer declines. Is it overfitting?

I'd read the code and I found that the input in training is not the same as that in validating. When training, the network gets the data from time 0 to time n-1, and the output is from time 1 to time n, as shown in fig. 1.

When validating, the network gets the data from time 0 to time obs(observation), and the output is from time obs+1 to time n, as shown in fig. 2.

Maybe, this is why the valid_error is large but the train_loss is small. (I'm not sure because I'm new here, why not train as validating, only get the data from time 0 to time obs, and predict the data from time obs+1 to time n)

About predict process

In this paper, the authors declear that 8 ground truth positions are serve as observation, and predict the trajectories of following 12 time steps. However, when I take a close view on the function "forward", it seems to rely on 19 frames inputs. What is that meaning? @zhangpur
The related codes are as follows:
'''
for framenum in range(self.args.seq_length-1):
if framenum >= self.args.obs_length and iftest:

            node_index = seq_list[self.args.obs_length - 1] > 0
            nodes_current = outputs[framenum - 1, node_index].clone()

            nodes_abs=shift_value[framenum,node_index]+nodes_current     ##here the shift_value using the last 12 frames data

            nodes_abs=nodes_abs.repeat(nodes_abs.shape[0], 1, 1)
            corr_index=nodes_abs.transpose(0,1)-nodes_abs
        else:
            node_index=seq_list[framenum]>0
            nodes_current = nodes_norm[framenum,node_index]
            corr = nodes_abs[framenum, node_index].repeat(nodes_current.shape[0], 1, 1)
            nei_index = nei_list[framenum, node_index]
            nei_index = nei_index[:, node_index]
            # relative coords
            corr_index = corr.transpose(0,1)-corr
            nei_num_index=nei_num[framenum,node_index]

'''

I would appreciate it if you could give more details about that question.