PoseFix makes pose result of any methods better from a single '.json' file!
We achieved top performance by refining the state-of-the-art (HRNet, CVPR 2019). You can always make your results better!
This repo is official TensorFlow implementation of PoseFix: Model-agnostic General Human Pose Refinement Network (CVPR 2019) for model-agnostic human pose refinement from a single RGB image. What this repo provides:
- TensorFlow implementation of PoseFix: Model-agnostic General Human Pose Refinement Network.
- Flexible and simple code.
- Compatibility for most of the publicly available 2D multi-person pose estimation datasets including MPII, PoseTrack 2018, and MS COCO 2017.
- Human pose estimation visualization code (modified from Detectron).
This code is tested under Ubuntu 16.04, CUDA 9.0, cuDNN 7.1 environment with two NVIDIA 1080Ti GPUs.
Python 3.6.5 version with Anaconda 3 is used for development.
The ${POSE_ROOT} is described as below.
${POSE_ROOT}
|-- data
|-- lib
|-- main
|-- tool
`-- output
datacontains data loading codes and soft links to images and annotations directories.libcontains kernel codes for 2d multi-person pose estimation system.maincontains high-level codes for training or testing the network.toolcontains dataset converter.posetrack2coco_output.pyconvertsposetrackoutput files tococoformat.outputcontains log, trained models, visualized outputs, and test result.
You need to follow directory structure of the data as below.
${POSE_ROOT}
|-- data
|-- |-- MPII
| `-- |-- input_pose
| | |-- name_of_input_pose.json
| | |-- test_on_trainset
| | | | -- result.json
| |-- annotations
| | |-- train.json
| | `-- test.json
| `-- images
| |-- 000001163.jpg
| |-- 000003072.jpg
|-- |-- PoseTrack
| `-- |-- input_pose
| | |-- name_of_input_pose.json
| | |-- test_on_trainset
| | | | -- result.json
| |-- annotations
| | |-- train2018.json
| | |-- val2018.json
| | `-- test2018.json
| |-- original_annotations
| | |-- train/
| | |-- val/
| | `-- test/
| `-- images
| |-- train/
| |-- val/
| `-- test/
|-- |-- COCO
| `-- |-- input_pose
| | |-- name_of_input_pose.json
| | |-- test_on_trainset
| | | | -- result.json
| |-- annotations
| | |-- person_keypoints_train2017.json
| | |-- person_keypoints_val2017.json
| | `-- image_info_test-dev2017.json
| `-- images
| |-- train2017/
| |-- val2017/
| `-- test2017/
`-- |-- imagenet_weights
| |-- resnet_v1_50.ckpt
| |-- resnet_v1_101.ckpt
| `-- resnet_v1_152.ckpt
- In the
toolof TF-SimpleHumanPose, runpython mpii2coco.pyto convert MPII annotation files to MS COCO format (MPII/annotations). - In the
toolof TF-SimpleHumanPose, runpython posetrack2coco.pyto convert PoseTrack annotation files to MS COCO format (PoseTrack/annotations). - Download imagenet pre-trained resnet models from tf-slim and place it in the
data/imagenet_weights. - Except for
annotationsof the MPII and PoseTrack, all other directories are original version of downloaded ones. - If you want to add your own dataset, you have to convert it to MS COCO format.
- You can change default directory structure of
databy modifyingdataset.pyof each dataset folder.
You need to follow the directory structure of the output folder as below.
${POSE_ROOT}
|-- output
|-- |-- log
|-- |-- model_dump
|-- |-- result
`-- |-- vis
- Creating
outputfolder as soft link form is recommended instead of folder form because it would take large storage capacity. logfolder contains training log file.model_dumpfolder contains saved checkpoints for each epoch.resultfolder contains final estimation files generated in the testing stage.visfolder contains visualized results.- You can change default directory structure of
outputby modifyingmain/config.py.
- Run
pip install -r requirement.txtto install required modules. - Run
cd ${POSE_ROOT}/libandmaketo build NMS modules. - In the
main/config.py, you can change settings of the model including dataset to use, network backbone, and input size and so on.
input_pose/test_on_trainset/result.json should be prepared before training. This is test result on the training set with the groundtruth bbox and used when synthesizing input pose of not annotated keypoints in the training stage. Testing result of TF-SimpleHumanPose is used.
In the main folder, run
python train.py --gpu 0-1to train the network on the GPU 0,1.
If you want to continue experiment, run
python train.py --gpu 0-1 --continue--gpu 0,1 can be used instead of --gpu 0-1.
input_pose/name_of_input_pose.json is pose estimation result of any other method. You have to rename the it and also input_pose_path of the data/$DATASET/dataset.py. The input_pose/name_of_input_pose.json should be follow MS COCO format. To test on the PoseTrack dataset, run tool/posetrack2coco_output.py before testing to convert PoseTrack output files to COCO format.
Place trained model at the output/model_dump/$DATASET/ and pose estimation result of any other method (name_of_input_pose.json) to data/$DATASET/input_pose/.
In the main folder, run
python test.py --gpu 0-1 --test_epoch 140to test the network on the GPU 0,1 with 140th epoch trained model. --gpu 0,1 can be used instead of --gpu 0-1.
Here I report the performance of the PoseFix. Also, you can download pre-trained models of the PoseFix in here and test_on_trainset/result.json in here.
As this repo outputs compatible output files for MS COCO and PoseTrack, you can directly use cocoapi or poseval to evaluate result on the MS COCO or PoseTrack dataset. You have to convert the produced mat file to MPII mat format to evaluate on MPII dataset following this.
We additionally applied our PoseFix on HRNet (Ke etal. CVPR2019), and achieved the top performance.
| Method | AP | Ap .5 | AP .75 | AP (M) | AP (L) | AR | AR .5 | AR .75 | AR (M) | AR (L) |
|---|---|---|---|---|---|---|---|---|---|---|
| pose_hrnet_w48 | 76.3 | 90.8 | 82.9 | 72.3 | 83.4 | 81.2 | 94.2 | 87.1 | 76.7 | 87.6 |
| PoseFix + HRNet | 77.3 | 90.9 | 83.5 | 73.5 | 84.4 | 82.0 | 94.3 | 87.5 | 77.7 | 88.3 |
| Method | AP | Ap .5 | AP .75 | AP (M) | AP (L) | AR | AR .5 | AR .75 | AR (M) | AR (L) |
|---|---|---|---|---|---|---|---|---|---|---|
| pose_hrnet_w48 | 75.5 | 92.5 | 83.3 | 71.9 | 81.5 | 80.5 | 95.7 | 87.4 | 76.3 | 86.3 |
| PoseFix + HRNet | 76.7 | 92.6 | 84.1 | 73.1 | 82.6 | 81.5 | 95.8 | 88.1 | 77.5 | 87.2 |
- You have to set
dataset,backboneandinput_shapeto those of the model inconfig.py.
- You have to set
dataset,backboneandinput_shapeto those of the model inconfig.py.
-
Those who are suffer from out of bound index issue, please refer this issue. According to TF docs,
tf.scatter_ndwill ignore out of bound indices in GPU mode. However,BruceLeeeeehad a issue with that and fixed by clipping coordinates. -
For those who suffer from
FileNotFoundError: [Errno 2] No such file or directory: 'tmp_result_0.pkl'in testing stage, please prepare input pose properly. The pkl files are generated and deleted automatically in testing stage, so you don't have to prepare them. Most of this error comes from inproper human detection file.
This repo is largely modified from TensorFlow repo of CPN and PyTorch repo of Simple.
@InProceedings{Moon_2019_CVPR_PoseFix,
author = {Moon, Gyeongsik and Chang, Juyong and Lee, Kyoung Mu},
title = {PoseFix: Model-agnostic General Human Pose Refinement Network},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2019}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent