melonwan/densereg

3D hand pose estimation via dense regression

Dense 3D Regression for Hand Pose Estimation

This respository contains tensorflow implementation of the paper. It is developped and tested on Debian GNU/Linux 8 64-bit.

Requirements:

• python 2.7
• tensorflow == 1.3
• tfplot (for visualization on tf summary files)
• matplotlib >= 2.0.2
• numpy
• opencv >= 2.4 (optional, for cpu visualization)

Data Preparations:

Download the datasets, create soft links for them to exp/data and run python data/${dataset}.py to create the TFRecord files. Details are here.

Usage:

Both training and testing functions are provided by model/hourglass_um_crop_tiny.py. Here is an example:

python model/hourglass_um_crop_tiny.py --dataset 'icvl' --batch_size 40 --num_stack 2 --fea_num 128 --debug_level 2 --is_train True

where the hyper parameter configuration is explained in the source python files.

Results:

We provide the estimation results by the proposed method for ICVL, NYU, MSRA15. They are xyz coordinates in mm, the 2D projection method is in the function xyz2uvd from here. Check here for comparison to other methods. Thanks @xinghaochen for providing the comparison.

Pretrained Models:

Run the script to download and install the corresponding trained model of datasets. $ROOT denote the root path of this project.

cd $ROOT
./exp/scripts/fetch_icvl_models.sh
./exp/scripts/fetch_msra_models.sh
./exp/scripts/fetch_nyu_models.sh

To perform testing, simply run

python model/hourglass_um_crop_tiny.py --dataset 'icvl' --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False
python model/hourglass_um_crop_tiny.py --dataset 'nyu' --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False
python model/hourglass_um_crop_tiny.py --dataset 'msra' --pid 0 --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False

in which msra dataset should use pid to indicate which person to test on. In the testing function, the third augument is used to indicate which model with corresponding training step will be restored. We use step of -1 to indicate our pre-trained model.

densereg's Issues

AttributeError: 'FigureCanvasBase' object has no attribute 'tostring_argb'

hello!Thank you for your generous sharing!There is some wrong when I train the NYU dataset.How to solve it?
AttributeError: 'FigureCanvasBase' object has no attribute 'tostring_argb'

     [[node pt_summary_1/Plot_1 (defined at D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\ops.py:114) ]]

Caused by op 'pt_summary_1/Plot_1', defined at:
File "hourglass_um_crop_tiny.py", line 918, in
run_train(dataset, val_dataset)
File "hourglass_um_crop_tiny.py", line 882, in run_train
train(model, restore_step)
File "..\model\train_single_gpu.py", line 59, in train
loss = model.loss(*batches)
File "hourglass_um_crop_tiny.py", line 419, in loss
tf.reshape(hm_uvd_pts, (3,-1,3))))
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\wrapper.py", line 87, in wrapped_fn
**merge_kwargs(kwargs, kwargs_call))
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\ops.py", line 167, in plot_many
im = plot(plot_func, arg, name=('Plot%d' % k), **kwargs)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\ops.py", line 114, in plot
name=name)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\ops\script_ops.py", line 468, in py_func
func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\ops\script_ops.py", line 282, in _internal_py_func
input=inp, token=token, Tout=Tout, name=name)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\ops\gen_script_ops.py", line 154, in py_func
"PyFunc", input=input, token=token, Tout=Tout, name=name)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

UnknownError (see above for traceback): AttributeError: 'FigureCanvasBase' object has no attribute 'tostring_argb'
Traceback (most recent call last):

File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\ops\script_ops.py", line 207, in call
ret = func(*args)

File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\ops.py", line 101, in _render_image
image = figure.to_array(fig)

File "D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\figure.py", line 90, in to_array
img = np.frombuffer(fig.canvas.tostring_argb(), dtype=np.uint8)

AttributeError: 'FigureCanvasBase' object has no attribute 'tostring_argb'

     [[node pt_summary_1/Plot_1 (defined at D:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tfplot\ops.py:114) ]]

NYU test results not good as it should be

After download nyu dataset and your pretrained model, I run the command python model/hourglass_um_crop_tiny.py --dataset 'nyu' --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 with is_train equals False in the code. But the results is not good as it should be:

[2018-04-19 16:06:11.933652]: 2727/393226 computed, with 0.10s
finish test
10mm percentage: 0.012239
20mm percentage: 0.057683
30mm percentage: 0.072225
40mm percentage: 0.095613

This is what i see in tensorboard:

Ground Truth:

Val:

And this is the generating result files:
testing-2018-04-19_15_59_22.194820-result.txt
testing-2018-04-19_15_59_22.194820-result_error.txt

What's the problem? Am I wrong with the code?

My environment:

• Ubuntu 16.04
• GTX titan xp 12GB
• cuda9.0 + cudnn7
• python 2.7
• tensorflow 1.7
• tfplot 0.2.0
• matplotlib 2.2.2
• opencv 2.4.11

AttributeError: dataset (ICVL-Hand)

When I run python data/icvl.py:
it reports the error:

Traceback (most recent call last):
File "data/icvl.py", line 271, in
run_preprocess()
File "data/icvl.py", line 215, in run_preprocess
preprocess_op=dataset.preprocess_op(128, 128))
File "/fast/junyan/HandPose/denseReg/data/dataset_base.py", line 199, in get_batch_op
result = preprocess_op(dm, pose, self.cfg)
File "data/icvl.py", line 154, in preprocess_op
dm, pose, cfg = crop_from_xyz_pose(dm, pose, cfg, input_width, input_height)
File "/fast/junyan/HandPose/denseReg/data/preprocess.py", line 64, in crop_from_xyz_pose
if FLAGS.dataset == 'icvl':
File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/flags.py", line 85, in getattr
return wrapped.getattr(name)
File "/root/anaconda3/lib/python3.6/site-packages/absl/flags/_flagvalues.py", line 472, in getattr
raise AttributeError(name)
AttributeError: dataset

I create the soft link in the exp/data/icvl.
The icvl data directory hierarchy looks like this:
**>Training

Depth(With many images in different directories）
labels.txt

Testing

Depth(With many images in different directories）
labels.txt**

I am not sure how can I resolve this problem.

data.preprocess.pose_sync not found.

I am wondering why data.preprocess.pose_sync not found in the data/preprocess.py file ?

Hourglass network has problem

1. hourglass module should have symmetric structure which means it has three residual modules in at the most inner layer.

    if n > 1:
        lower2 = _hourglass(lower1, n-1)
    else:
        lower2 = lower1

I think lower2 should also be add to residual module:

    if n > 1:
        lower2 = _hourglass(lower1, n-1)
    else:
        lower2 = lower1
        lower2 = _residual(lower2)

2. In the residual module, why half_num_in instead of half_num_out

Real-time performance isn't so good

Hi, I have tried the inference code in NYU dataset. However, I can't achieve "real-time" performance as mentioned in your paper.
For batch size=1: frame rate is 12
For batch size=3: frame rate is 17
Both are not good for real time(less than 24).
I wonder why the inference speed isn't good. Did you do infernece in TWO Titans?
Thank you very much.

Network architecture different between paper and code

Hi， thank you very much for sharing your code. I am little confused that about the network depicted in the paper and code.

                um_in = tf.concat([hg_outs, hm_out, hm3_out], axis=-1)
                um_in = _residual(_residual(um_in, 256))

                um_in_mask = tf.concat([hg_outs, hm_out, hm3_out], axis=-1)
                mask = tf.tile(tf.less(tiny_dm, -0.9), (1,1,1,um_in_mask.get_shape()[-1].value))
                um_in_mask = tf.where(mask, tf.zeros_like(um_in_mask), um_in_mask)
                um_in_mask = _residual(_residual(um_in_mask, 256))

There is two consecutive residual model flowing um_in and um_in_mask, respectively. However, I cannot find in the figure 1 of the paper.

Thank you!

train_multi_gpu

Hi, thank you very for your help. Have you ever run the project on multiple GPUs to speed up the trainding process? Thank you!

Estimate 3D hand pose in ASL Finger Spelling Dataset

Can I estimate 3D hand pose (uvd) in ASL Finger Spelling Dataset using the pretrained models?
Note that the resolution of the given depth images is different and there is no camera configuration.
Thanks in advance~

gpu_config.py

Hi,
When run the script, the file info.txt can not been found. Can you give some more information on that?
Thanks!

gpu_lock_path = '/tmp/lock-gpu*/info.txt'

Pre-trained model download link doesn't work anymore

Hi,

The download link in the script seems not working ant more, when I run the script, I got the following errors:
downloading the pretrained model(566M)... --2021-03-01 15:21:48-- https://polybox.ethz.ch/index.php/s/B2W1ngyUAitsv2e/download Resolving polybox.ethz.ch (polybox.ethz.ch)... 129.132.71.243 Connecting to polybox.ethz.ch (polybox.ethz.ch)|129.132.71.243|:443... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2021-03-01 15:21:49 ERROR 503: Service Unavailable.

Is it possible to update the link?

Thanks

'crop_from_uvd_pose' not found

When using

python model/hourglass_um_crop_tiny.py --dataset 'icvl' --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False

it seems to be an import error

from data.preprocess import crop_from_xyz_pose, crop_from_uvd_pose, crop_from_bbx, center_of_mass
ImportError: cannot import name crop_from_uvd_pose

And there is no crop_from_uvd_pose in data.preprocess indeed. It is in file 'data/icvl.py'.
How to solve it

NYU data preparation slow

Hi:
thanks for sharing your work!
I try to run the code following your instructions. and when i run the python script ./data/nyu.py.it takes 30 hours and still runing. Could you tell me is this normal or I did something wrong?

thank you.

Does it support 2D RGB image?

Can I test the pretrained model with webcam?
or should I prepare depth cam too?

AssertionError: Model diverged with loss = NaN

python model/hourglass_um_crop_tiny.py --dataset 'msra' --pid 0 --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False

finally into the long long training loop
Traceback (most recent call last):
  File "model/hourglass_um_crop_tiny.py", line 932, in <module>
    run_train(dataset, val_dataset)
  File "model/hourglass_um_crop_tiny.py", line 889, in run_train
    train(model, restore_step)
  File "/home/denseReg/model/train_single_gpu.py", line 149, in train
    assert not np.isnan(loss_value), 'Model diverged with loss = NaN'
AssertionError: Model diverged with loss = NaN

Why there is nyu_bbx.pkl?

I found you use nyu_bbx when testing on NYU dataset, which means you use 'ground truth' hand location data when running test. I think it should be predict by network itself instead of given by you.

So how did you get nyu_bbx.pkl in your experiment? Is it computed from ground truth xyz labels or predicted by another method which can compute bbx of hand location?

ERROR:tensorflow:Exception in QueueRunner: ./exp/data/msra15/tf/P0-9-of-100; No such file or directory

msra.py

if __name__ == '__main__':
    saveTFRecord()
    # run_check_record()
    # run_preprocess()

it means that i must uncomment the last two command of msra.py ?
But when i uncomment the last two command of msra.py, it always lead to error.
Can you help give me a little hint about how i can solve the problem ?

Can be this method used to estimate 3D human pose?

Can be this method used to estimate 3D human pose? please. @melonwan

msra.py run_preprocess() used the pose_sync which doesn't exist

hms, dcs = data.preprocess.pose_sync(norm_poses, normed_dms, cfgs, coms, 128, 128)

Error while testing/training

Hello,
While i try to run the model in test mode using the pre-trained model i always get an error at some point during testing (with num_shards=16 for test, this error happens after treating the first 400 points). The i receive this error:

flat indices[34, :] = [32, 26, 6] does not index into param (shape: [32,32,14]).
[[Node: map_3/while/GatherNd = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](map_3/while/TensorArrayReadV3_3, map_3/while/stack_1)]]

I tried to debug this with not much success, would you happen to know what is the problem?

Training data for NYU dataset

I find that you use all three views of depth images for training in the \denseReg-master\data\nyu.py .
def loadAnnotation(self, is_trun=False):
'''is_trun:
True: to load 14 joints from self.keep_list
False: to load all joints
'''
t1 = time.time()
path = os.path.join(self.src_dir, 'joint_data.mat')
mat = sio.loadmat(path)
camera_num = 1 if self.subset=='testing' else 3
joints = [mat['joint_xyz'][idx] for idx in range(camera_num)]
names = [['depth_{}_{:07d}.png'.format(camera_idx+1, idx+1) for idx in range(len(joints[camera_idx]))] for camera_idx in range(camera_num)]

But for fair comparison only view 1 images should be used for training.

Success rate on MSRA dataset

Hi, thanks to your great work!
If it's to your convenience, could you share the experimental data of ploting success rate on MSRA dataset? Namely, the fig. 6 on your paper. Because we are doing the research on depth hand pose, need to make a comparison. I would much appreciate it if you can share it with me!

direct regression and heatmaps

Hi,

1. The first method is coordinate regression method, whether the label is pixel coordinate or world coordinate for the loss calculation.

2. What is the next two line code meaning? Could you give more information in details? Thank you!

norm_xyz_pose = tf.divide(xyz_pose - tf.tile(com,[jnt_num]), POSE_NORM_RATIO) gt_oms = tf.reshape(gt_normed_poses, (-1,1,1,3*self._jnt_num)) - xyzs

error while training

hi,
I am now setup the DenseReg environment in local server, but encounter issue when executing the training command below,

python model/hourglass_um_crop_tiny.py --dataset 'nyu' --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train True

It seems the code doesn't find some file, but finally it enters the training log, can you help me to clarify what cause the issue? from my analysis, it seems
there is error happening in the below function .

135 #TODO: change to tf.train.SummaryWriter()
136 summary_writer = tf.summary.FileWriter(
137 model.summary_dir,
138 graph=sess.graph)

ERROR:tensorflow:Exception in QueueRunner: ./exp/data/nyu/tf_train/training-47-of-300; No such file or directory
[[Node: batch_processing/ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](batch_processing/TFRecordReaderV2, batch_processing/input_producer)]]
Exception in thread QueueRunnerThread-batch_processing/random_shuffle_queue-batch_processing/random_shuffle_queue_enqueue:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run
enqueue_callable()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1205, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
NotFoundError: ./exp/data/nyu/tf_train/training-47-of-300; No such file or directory
[[Node: batch_processing/ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](batch_processing/TFRecordReaderV2, batch_processing/input_producer)]]

ERROR:tensorflow:Exception in QueueRunner: ./exp/data/nyu/tf_train/training-4-of-300; No such file or directory
[[Node: batch_processing/ReaderReadV2_1 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](batch_processing/TFRecordReaderV2_1, batch_processing/input_producer)]]
Exception in thread QueueRunnerThread-batch_processing/random_shuffle_queue-batch_processing/random_shuffle_queue_enqueue_1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run
enqueue_callable()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1205, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
NotFoundError: ./exp/data/nyu/tf_train/training-4-of-300; No such file or directory
[[Node: batch_processing/ReaderReadV2_1 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](batch_processing/TFRecordReaderV2_1, batch_processing/input_producer)]]

finally into the long long training loop