jufi2112/rgbd-kinect-pose

Demo for "Real-time RGBD-based Extended Body Pose Estimation" paper

Forked from the original repository. Some additions were made, especially to documentation and dockerfiles, in order to allow for an unproblematic setup for reproduction.

Real-time RGBD-based Extended Body Pose Estimation

This repository is a real-time demo for our paper that was published at WACV 2021 conference

The output of our module is in SMPL-X parametric body mesh model:

• RNN estimates body pose from joints detected by Azure Kinect Body Tracking API
• For face (expression and jaw) and hand pose we crop from rgb image:
  • for hand model we use minimal-hand
  • our face NN takes media-pipe keypoints as input

Combined system runs at 30 fps on a 2080ti GPU and 8 core @ 4GHz CPU.

How to use

Build

• Prereqs: your nvidia driver should support cuda 10.2, Windows or Mac are not supported.
• Docker setup:
  • Install docker engine
  • Install nvidia-docker
  • Set nvidia your default runtime for docker
  • Make docker run without sudo: create docker group and add current user to it:

    sudo groupadd docker
    sudo usermod -aG docker $USER
    

  • reboot
• Create docker image:
  • Build on your own: run 2 cmds
  • Or download docker image from currently not available
• Attach your Azure Kinect camera
• From the Microsoft Azure Kinect SDK repository, you have to setup udev rules on your host machine. See the instruction provided at the bottom of this file
• Check your Azure Kinect camera is working inside Docker container:
  • Enter Docker container: ./run_local.sh -c [10, 11] from docker dir
  • Then run python -m pyk4a.viewer --vis_color --no_bt --no_depth inside docker container

Download data

• Download our data archive smplx_kinect_demo_data.tar.gz
• Unzip: mkdir /your/unpacked/dir, tar -zxf smplx_kinect_demo_data.tar.gz -C /your/unpacked/dir
• Download models for hand, see link in "Download models from here" line in our fork, put to /your/unpacked/dir/minimal_hand/model
• To download SMPL-X parametric body model go to this project website, register, go to the downloads section, download SMPL-X v1.1 model, put to /your/unpacked/dir/pykinect/body_models/smplx
• /your/unpacked/dir should look like this
• Set data_dirpath and output_dirpath variables in config file:
  • data_dirpath is a path to /your/unpacked/dir
  • output_dirpath is used to check timings or to store result images
  • ensure these paths are visible inside docker container, set VOLUMES variable here

Run

• Run demo: in src dir run ./run_server.sh -c [10, 11], the latter will enter docker container and will use config file where shape of the person is loaded from an external file: in our work we did not focus on person's shape estimation

What else

Apart from our main body pose estimation contribution you can find this repository useful for:

• minimal_pytorch_rasterizer python package: CUDA non-differentiable mesh rasterization library for pytorch tensors with python bindings
• pyk4a python package: real-time streaming from Azure Kinect camera, this package also works in our provided docker environment
• multiprocessing_pipeline python package: set-up pipeline graph of python blocks running in parallel, see usage in server.py

Citation

If you find the project helpful, please consider citing us:

@inproceedings{bashirov2021real,
  title={Real-Time RGBD-Based Extended Body Pose Estimation},
  author={Bashirov, Renat and Ianina, Anastasia and Iskakov, Karim and Kononenko, Yevgeniy and Strizhkova, Valeriya and Lempitsky, Victor and Vakhitov, Alexander},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={2807--2816},
  year={2021}
}

Non-commercial use only