Questions

[sberryman]

Questions

[Svito-zar]

I think, your idea is great. And I designed my system keeping this kind of applications in mind. I am curious how will it work. Please, let me know.

About the questions:

1. I can share a pretrain model later on. Maybe in a week on two.

2. I also expect activity-specific LSTM to yield better results. As long as the classification is correct. The risk here is that if classification is wrong it might have a very bad performance.
   I wanted to have my model general so that I can cope with any action.

3. Sequences were shuffled during the training, just as any other training batches for a gradient descent optimization of a neural network. It helps training to converge faster and to the better solution. You can try to comment out shuffling and observe how the performance changes.

Hope it helps.

[sberryman]

Thanks for the thorough reply.

1. That would be excellent, I can't imagine AlphaPose will be done for another 10-20 days anyways. I only have 3 GPUs and two are working on a SfM task right now.
2. Good point on misclassification, would your expectation be a single LSTM would do a better job overall? Assuming the system will never be 100% with the classification step.
3. I'm quite new to this if you can't tell, learning by trial and error while reading all these amazing research papers.
4. What about smoothing marker detection/reconstruction in a video sequence? In a few of the video sequences I have generated based on OpenPose results, I've noticed the detected joints jitter quite a bit. If the 6 million poses are not enough I can run inference on 3 more days of video and get another 6+ million poses. In fact I have 11 more days of video, so I expect at least 28 million poses in the entire dataset.

FYI, I'm working on opening up the video dataset I have acquired to the public/researches. Legal review for privacy reasons is under way and expected to be done in a few weeks.

Thanks again!

[Svito-zar]

Follow up:

2. I would expect that the results would depend on the class. For the classes, where classification is very accurate the accuracy of reconstruction can be higher for a more specific model (just for one class) than for more general one (for all the classes). While for those classes where classification accuracy is poor -
   the class-specific model might perform worst. I think you do not want this kind of artifact, so I would recommend using general model. But I think it would be interesting to compare them.

3. I think LSTM can be used for the smoothing as well because it has recurrent connections: between previous and current time-frames. So you could achieve smoothing together with missing markers reconstruction. I have played with that a bit and have got some amazing results, can send you a video, if you like.

Taras

[sberryman]

2. Makes sense, it would be very interesting to compare a class specific model to a generic model. Based on the idea of asking myself 'where would the left knee be in T+1 or T+5 of X person?' I would presume knowing the activity would improve my estimates quite a bit. I would guess the improvement would be more pronounced for predicting timesteps further than the next frame simply because humans can't move a whole lot in 33ms.

3. Yes, I would absolutely like to see the video. I can also share a little video I have created with the inference I have performed thus far. I can't share mine publicly yet, so I'll have to privately send it to you.

[Svito-zar]

3. Yes, let's exchange videos. My email is [email protected]