svito-zar/nn-for-missing-marker-reconstruction

This is the official implementation of the paper "A Neural Network Approach to Missing Marker Reconstruction in Human Motion Capture"

nn-for-missing-marker-reconstruction's Issues

Questions

First, thanks for the great paper and posting the code. Do you have any plans to share a pre-trained model?

I have a video dataset of pedestrians on a very busy boardwalk which includes walking, running, rollerblading, rollerskating, skateboarding, biking, electric scooters, etc. I performed inference using Mask RCNN, OpenPose and am now in the process of AlphaPose+PoseTrack. OpenPose detected over 6 million poses covering several thousand different identities. They are observed approaching one camera and usually departing on another. So almost every identity consists of a frontal and back view offset by 10-20 degrees.

Neither AlphaPose or OpenPose will infer 100% of the joints for every frame of video. I assumed based on the quantity of poses I would be able to use a LSTM to recover missing joints. My assumption is that classifying the activity (running, walking, etc.) and using a different LSTM based on the activity classification would yield better accuracy for missing joints.

What was the purpose of shuffling the sequences during training?

Sorry, lots of questions!

preprocessed dataset links not working

preprocessed dataset and test sequence links given in Data Preparation section are not working.

Separate test from train

I extracted the test part of train.py, created a test.py, loaded the trained model in test.py, and tested the sequence.
But the values I get when I test the same data are different.
`# TEST it
from FlatAE import FlatAutoEncoder

import time

from utils.data import *
from utils.flags import FLAGS
from train import test
from tensorflow.core.protobuf import saver_pb2

class DataInfo(object):
"""Information about the datasets
Will be passed to the FlatAe for creating corresponding variables in the graph
"""
def init(self, data_sigma, train_shape, eval_shape, max_val):
"""DataInfo initializer
Args:
data_sigma: variance in the dataset
train_shape: dimensionality of the train dataset
eval_shape: dimensionality of the evaluation dataset
"""
self._data_sigma = data_sigma
self._train_shape = train_shape
self._eval_shape = eval_shape
self._max_val = max_val

with tf.Graph().as_default():
dropout = FLAGS.dropout # keep probability value
variance = FLAGS.variance_of_noise
#learning_rate = FLAGS.learning_rate
batch_size = FLAGS.batch_size
num_hidden = FLAGS.num_hidden_layers
tf.set_random_seed(FLAGS.seed)
ae_hidden_shapes = [FLAGS.network_width for j in range(num_hidden)]

config = tf.ConfigProto(allow_soft_placement=True)
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
# Read the data
data, max_val, mean_pose = read_datasets_from_binary()
data_info = DataInfo(data.train.sigma, data.train._sequences.shape,
                             data.test._sequences.shape, max_val)
# Pad max values and the mean pose, if needed
if FLAGS.amount_of_frames_as_input > 1:
    max_val = np.tile(max_val, FLAGS.amount_of_frames_as_input)
    mean_pose = np.tile(mean_pose, FLAGS.amount_of_frames_as_input)

ae_shape = [FLAGS.frame_size * FLAGS.amount_of_frames_as_input] + ae_hidden_shapes + [
            FLAGS.frame_size * FLAGS.amount_of_frames_as_input]
# create model
ae = FlatAutoEncoder(ae_shape, sess, batch_size, variance, data_info)
sess.run(tf.local_variables_initializer())

with tf.variable_scope("test"):
    sess.run(tf.global_variables_initializer())
    saver = tf.train.Saver(write_version=saver_pb2.SaverDef.V2)
    chkpt_file = FLAGS.chkpt_dir + '/chkpt-' + str(FLAGS.chkpt_num)
    saver.restore(sess, chkpt_file)
    print("Model restored from the file " + str(chkpt_file) + '.')
    # print("---------TESTING------------")
    rmse = test(ae, FLAGS.data_dir + '/../test_seq/basketball.binary', max_val, mean_pose, False)
    print("\nbasketball: ", rmse)
    rmse = test(ae, FLAGS.data_dir + '/../test_seq/basketball.binary', max_val, mean_pose, False)
    print("\nbasketball: ", rmse))`

This is the code for test,missing markers are fixed,I got two different rmse.The same model, the same input and the same missing marks, I feel the same rmse value should appear,but no.

Is it that I call the model to make the test wrong or something else?I hope I can get help solving this problem.