khrylx/egopose

[ICCV 2019] Official PyTorch Implementation of "Ego-Pose Estimation and Forecasting as Real-Time PD Control". ICCV 2019.

EgoPose

This repo contains the official implementation of our paper:

Ego-Pose Estimation and Forecasting as Real-Time PD Control
Ye Yuan, Kris Kitani
ICCV 2019
[website] [paper] [video]

Installation

Dataset

• Download the dataset from Google Drive in the form of a single zip or split zips (or BaiduYun link, password: ynui) and place the unzipped dataset folder inside the repo as "EgoPose/datasets". Please see the README.txt inside the folder for details about the dataset.

Environment

• Supported OS: MacOS, Linux
• Packages:
  • Python >= 3.6
  • PyTorch >= 0.4 (https://pytorch.org/)
  • gym (https://github.com/openai/gym)
  • mujoco-py (https://github.com/openai/mujoco-py) (MuJoCo is now free thanks to DeepMind! 🎉🎉🎉)
  • OpenCV: conda install -c menpo opencv
  • Tensorflow, OpenGL, yaml: conda install tensorflow pyopengl pyyaml
• Additional setup:
  • For linux, the following environment variable needs to be set to greatly improve multi-threaded sampling performance:
    export OMP_NUM_THREADS=1
• Note: All scripts should be run from the root of this repo.

Pretrained Models

• Download our pretrained models from this link (or BaiduYun link, password: kieq) and place the unzipped results folder inside the repo as "EgoPose/results".

Quick Demo

Ego-Pose Estimation

• To visualize the results for MoCap data:
  python ego_pose/eval_pose.py --egomimic-cfg subject_03 --statereg-cfg subject_03 --mode vis
  Here we use the config file for subject_03. Note that in the visualization, the red humanoid represents the GT.

• To visualize the results for in-the-wild data:
  python ego_pose/eval_pose_wild.py --egomimic-cfg cross_01 --statereg-cfg cross_01 --data wild_01 --mode vis
  Here we use the config file for cross-subject model (cross_01) and test it on in-the-wild data (wild_01).

• Keyboard shortcuts for the visualizer: keymap.md

Ego-Pose Forecasting

• To visualize the results for MoCap data:
  python ego_pose/eval_forecast.py --egoforecast-cfg subject_03 --mode vis

• To visualize the results for in-the-wild data:
  python ego_pose/eval_forecast_wild.py --egoforecast-cfg cross_01 --data wild_01 --mode vis

Training and Testing

• If you are interested in training and testing with our code, please see train_and_test.md.

Citation

If you find our work useful in your research, please cite our paper Ego-Pose Estimation and Forecasting as Real-Time PD Control:

@inproceedings{yuan2019ego,
    title={Ego-Pose Estimation and Forecasting as Real-Time PD Control},
    author={Yuan, Ye and Kitani, Kris},
    booktitle={Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
    year={2019},
    pages={10082--10092}
}

License

The software in this repo is freely available for free non-commercial use. Please see the license for further details.

egopose's Issues

Question about visualization

Hello, the above figure is taken from your paper. It seems the whole sequence is recorded within one figure. How do you achieve this? Does the current repo contain relevant visualization code? Thank you very much!

About data preprocessing.

I notice that gen_expert.py directly imports motion data from trajectory (*.p) files. What if I generate motion data from *.bvh files? Could you please provide the preprocessing script corresponding to *.bvh file? Thank you very much!

Question about Training Ego-Pose Estimation

Hi, I’ve tried Training Ego-Pose Estimation.
I executed the command:

python ego_pose / ego_mimic.py --cfg subject_02

However, the following error occurred:

File "mujoco_py / mjbatchrenderer.pyx", line 2, in mujoco_py.cymj
TypeError: self.ptr cannot be converted to a Python object for pickling

I tried using dill instead of pickle, but it didn't work.
How can I solve it? Thank you.

Alternative dataset link?

Hi, I have tried many times downloading the dataset but all failed. Could you please provide other links for this dataset (e.g., BaiduYun), or split the large one into multiple smaller parts? Thank you very much!

Question on wpos to qpos conversion

Hi,

Thank you very much for sharing this work, the result is very impressive to me.
May I ask a question on pose conversion?
I notice that the qpos is convert from amc file by RFC/convert_cmu_mocap.py, and I want to convert the wpos 3D pose to qpos , which is the inverse of function get_ee_pos (full body version).
My plan is to convert wpos to amc format, but I am still a bit confusion on this step.
May I ask if you have solution or suggestion for this problem?

Btw,
When I run the code ego_mimic_eval.py, I met the problem below:
$ python ego_pose/ego_mimic_eval.py --cfg subject_02 --iter 3000
Traceback (most recent call last):
File "ego_pose/ego_mimic_eval.py", line 41, in <module>
env = HumanoidEnv(cfg)
File "~xxx/EgoPose/ego_pose/envs/humanoid_v1.py", line 21, in __init__
self.video_dir = cfg.video_dir
AttributeError: 'Config' object has no attribute 'video_dir'

So I add it in EgoPose/ego_pose/utils/egomimic_config.py
self.video_dir = '%s/video' % self.cfg_dir
os.makedirs(self.video_dir, exist_ok=True)
I think the same issue occurs in ego_forecast_eval.py due to cleaning process.
Hope it helps :)

Thank you~