Giter Site home page Giter Site logo

eth-ait / dip18 Goto Github PK

View Code? Open in Web Editor NEW
135.0 13.0 47.0 14.35 MB

Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time

License: GNU General Public License v3.0

C# 23.89% Python 76.08% Shell 0.02%

dip18's Introduction

Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time

Code

This repository contains the code published alongside with our SIGGRAPH Asia paper. It is organised as follows: train_and_eval contains code to train and evaluate the neural networks proposed in the paper. live_demo contains Unity and Python scripts to use the models for real-time inference. data_synthesis contains a script to produce synthetic IMU measurements from SMPL sequences. Please refer to the READMEs in the respective subfolder for more details.

Data

To download the data please visit the project page. From this page you can also download the SMPL reference parameters for the TotalCapture dataset. To preprocess TotalCapture data, please refer to read_TC_data.py.

Visualization

Apart from the live demo, this repository does not offer any other visualization tools. However, the data can easily be visualized with the aitviewer. The examples provided by aitviewer contain two scripts that load data associated with DIP:

DIP-IMU rendering with aitviewer

Contact Information

For questions or problems please file an issue or contact [email protected] or [email protected].

Citation

If you use this code or data for your own work, please cite:

@article{DIP:SIGGRAPHAsia:2018,
	title = {Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time},
    	author = {Huang, Yinghao and Kaufmann, Manuel and Aksan, Emre and Black, Michael J. and Hilliges, Otmar and Pons-Moll, Gerard},
    	journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)},
    	volume = {37},
    	pages = {185:1-185:15},
    	publisher = {ACM},
    	month = nov,
    	year = {2018},
    	note = {First two authors contributed equally},
    	month_numeric = {11}
}

dip18's People

Contributors

kaufmanu avatar yinghaohuang91 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dip18's Issues

How can we get the 'gt' from our own IMU datas?

- data['gt']: A numpy-array of shape (seq_length, 72). Corresponds to the SMPL poses computed with SIP using all 17 sensors. The SMPL pose parameters are stored in the angle-axis format (3 joint angles per 24 joints = 72).
is mentioned in
overview_of_published_data.txt.

Now I read the data (accs, rots, etc) through my IMUs, but what should I do to get the pose?
I also didn't find the code corresponding to the paper(SIP):
Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs

Clarification on Sensor Calibration and Acceleration

Hello,

I am currently working on replicating the sensor calibration procedure described in your paper, but I had no luck so far applying the methods to my own sensors. I have two main points I would like clarification on:

  1. Sensor Alignment with SMPL Body Frame:

The paper mentions placing the head sensor onto the head such that the sensor axes align with the SMPL body frame. However, from the images provided, it appears that the head sensor is oriented with the Z-axis pointing up or down, whereas the SMPL body frame uses Y-up.
Here is the image from the repository showing sensor placement and pose:
Sensor Placement Image
And the image I found depicting the Xsens axes:
xsens.
My guess is that the image might be a generic visual aid or possibly there's a difference in sensor models.

  1. Acceleration Calibration Procedure:

While not explicitly detailed in the paper, I assume that the steps used for orientation calibration might be applicable to acceleration data as well. My approach involves:

  • Starting with the acceleration data in the inertial frame and removing the gravitational component (Z-axis in my setup).
  • Rotating the de-gravitized data by the inverse orientation of the head sensor.
  • Applying the per-bone offset.

Could you confirm if this approach is valid for calibrating accelerations?

Thank you for your assistance and for the insights provided in your paper.

CUDA issue

root@95d47060d3c9:/share/home/brcao/Repos/dip18/train_and_eval# python run_training.py --save_dir ./models --system local --data_file v9 --json ./models/tf-1527876409-imu_v9-birnn-fc1_512-lstm2_512-idrop2-relu-norm_ori_acc_smpl-auxloss_acc/config.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_training.py", line 18, in <module>
    import tensorflow as tf
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 72, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Any help would be appreciated.

The meaning of training data entries

Hi there,

I am looking at the training file: ./Synthetic_NPZ/training.npz from SyntheticNPZ.zip and I see that the dictionary it contains had the following entries:

  • data_id with shape (32092,) : n/a
  • file_id with shape (32092,) : names of files
  • orientation with shape (32092,) : entries with the shape (300, 45)
  • acceleration with shape (32092,) : entries with the shape (300, 15)
  • smpl_pose with shape (32092,) : entries with the shape (300, 135)
  • preprocessing [1] with shape (32092,)
  • statistics [2] with shape (32092,)

My questions:

  1. When and how did you compute the "preprocessing" entries [1] and the "statistics" entries [2]?
  2. How are the "preprocessing" [1] and "statistics" entries [2] used and which parts of the training / live demo depend on them?

Many thanks in advance!

How can I get the position of each joint?

In data folder DIP_IMU, the ground truth is provided in the form of joint angle data['gt']. Is there any way to translate the joint angle to the 3D position of each joint?

OS, cuDNN, Anaconda versions

Hi there,

could you please specify more precisely the versions of the OS, cuDNN and Anaconda which you tested the code with?
Thank you in advance!

evaluation result

Hi, I am currently using your DIP evaluation code and models to test the system on the DIP dataset. However, I have noticed discrepancies in the evaluation results compared to your paper.

Using the provided command, model, and "imu_own_test.npz" data, the results do not align with the reported outcomes in your research.

Could you please help me understand the potential reasons behind this difference in evaluation results?
image

Thank you for your assistance.

Data synthesis is too slow

Data synthesis takes a long time to finish. I try to process AMASS dataset, it costs several days to finish. Even using multiprocessing, it needs days to finish.
global_rigid_transformation takes too much time. Is there any tricks to accelerate data synthesis?
Thanks in advance.

Synthetic data generation

Hi there,

do you intend to release the code which you implemented for synthetic data generation from the AMASS dataset?

Thanks a lot!

Generate synthetic IMU data and FK

Hi,

Thanks so much for such excellent work. I wonder whether you have released the relevant code to generate synthetic IMU data from the AMASS dataset? Or could you tell us the implementation method to get synthetic IMU data using forward kinematics?

Thanks a lot!

IMU order

I'm looking at the data of DIP_IMU_and_Others/DIP_IMU/*.pkl.
I am suspicious of swapping the knees and elbows.
Could anyone make sure of the IMU order?

Visualizing data

Hi there,

I am trying to visualize the IMU data contained in DIPIMUandOthers [1] and I suspect that the current version of the live_demo/inference_server.py [2] and the format of the data are incompatible. The *.pkl files contain a dictionary which the code tries to parse into an ndarray.
Could you please comment on this?
Many thanks in advance!

[1] https://psfiles.is.tuebingen.mpg.de/downloads/dip/DIPIMUandOthers_zip
[2] https://github.com/eth-ait/dip18/blob/master/live_demo/inference_server.py

Pose Dimension in Synthetic60FPS

I noticed that the dimension of pose parameter in Synthetic60FPS is 135, which according to your overview_of_published_data.txt corresponds to the SMPL pose target stored in axis-angle format.
image
But SMPL only has 72-D pose parameters, while SMPL+H has 156-D pose parameters. Both of them are different from your published data, so I get confused here. Could you please tell me how to extract SMPL pose parameters from the 135-D vectors?

How to calibrate the imu data of totalcapture?

Hi, thanks for your excellent work.
Now I have the raw totalcapture datasets in https://cvssp.org/data/totalcapture/ and the smpl gt data in https://dip.is.tuebingen.mpg.de/ . The TotalCapture_Real.zip only contains calibrated 6 imu data. I want to use the other imu for my work. I have tried the method in "Real-Time full-body motion capture from video and IMUs". The formula is
$R_{b_{i}}^{g}=R_{i g} \cdot R_{i} \cdot R_{i b}^{-1}$
where R_ig and R_ib are in *_calib_imu_ref.txt and *_calib_imu_bone.txt respectively. But the result is wrong in visualization.
I also tried the calibration procedure mentioned in dip paper, and the result is wrong. Maybe I missed something important.
Would you help me provide the critical code for calibrating the imu ori and acc ?
Thank you very much!

The TotalCapture SMPL poses

Hi!
Thank you very much for the DIP dataset. Can you please provide the the TotalCapture SMPL poses (contain acc , ori and poses).I have got the Totalcapture raw dataset, but processing it is a big challenge.
Thanks!

TotalCapture SMPL models

Hi! Thank you all for your great work! I was wondering if the SMPL models generated from the TotalCapture dataset are available anywhere? The paper only mentioned the AMASS and DIP-IMU datasets being made available. The research I am conducting would greatly benefit from having access to the generated models from TotalCapture. Please let me know if it is possible for me to receive a copy of these models. Thanks!

Inquiry about the Discrepancy in Sequences between DIP_IMU.zip and DIP_IMU_nn.zip

HI ,I am writing to inquire about a discrepancy I noticed in the DIP_IMU dataset. In the DIP_IMU.zip file, it appears that there are a total of 19 sequences for subjects 9 and 10. However, when I checked the imu_own_test.npz file in the DIP_IMU_nn.zip, there are only 18 sequences available for these subjects.Could you kindly explain the reason for the difference in the number of sequences between the two versions of the dataset?

Playground and TotalCapture_dataprocess

Dear DIP Authors.

Thank you very much for your work. I have obtained the raw data of TotalCapture and I would like to ask if I can get from you the script you used to process the TotalCapture dataset?

I found a lot of datasets about Playground, I would like to ask which dataset you used? If so, can I also refer to your scripts for the Playground dataset?

Locations of IMUs in DIP-IMU dataset

Dear DIP-authors,

Please excuse my frequent posting here. With this one, at least I hope to provide value to future research by adding to the usability of your open-sourced dataset DIP-IMU!

For my use case of DIP-IMU, I need to know the exact locations of the 17 IMUs on the body. All the information I can find on this is in the overview_of_published_data.txt which comes inside the dataset DIP_IMU_and_Others:

The order of IMUs is: [head, spine2, belly, lchest, rchest, lshoulder, rshoulder, lelbow, relbow, lhip, rhip, lknee, rknee, lwrist, lwrist, lankle, rankle].

Sadly, this does not correspond to the standard Xsens MVN configuration. I guess you didn't care about tracking feet and hands but instead used those sensors to get more data from the torso.

It would be of most value to me if you could help me figure out where exactly you placed the sensors on the subjects. Maybe you still have a photo from the day of the recordings?

If not, I hope you could maybe memorize it? To help with this, I made the following sketch with my best guess of the sensor placement. The red captions relate to the unknown sensors. Please correct where I was wrong.

imu_order

Finally, I'd like to create some value for future research on this. Thus, I will post the indices of the corresponding SMPL and SMPL-X vertices here once I get a good understanding of where exactly they are located.

Many thanks for your time and best wishes,
Nicola

Question: Super Noisy Acceleration Data?

Hi!

Thank you so much for creating this really useful IMU dataset. I encountered a problem when I was playing around with it and was wondering if you could kindly help me with it:

When I load the raw IMU data, select a random sensor, and visualize its readings, the IMU orientations look pretty much reasonable and consistent, but the acceleration visualizations (direction of plotted 3D vectors) almost look white noise to me as they wildly change directions every timestep.

Is this something you would expect or I must be doing something wrong? How could I convince myself (e.g. through visualization code) that I am using the IMU acc readings correctly?

Thanks!

DLLs for setting up the Unity3D project

Hi there,

I try to setup the Unity3D project by following the instructions from the Setting up the Unity Project and I noticed that:

  • I had to make a copy of System.Windows.Forms.dll and WindowsBase.dll into the XSensStream directory which I created.
  • the files MasterInfo.cs and DeviceInfo.cs are missing from the directory: %MT_SOFTWARE_SUITE%/MT SDK/Examples/awindamonitor_csharp/. I copied them from %MT_SOFTWARE_SUITE%/MT SDK/Examples/xda_csharp/ and got stuck while trying to get Unity3D to recognize the xsensdeviceapi_csharp64.dll file.
  • all files DeviceInfo.cs, MasterInfo.cs, MyEventArgs.cs and MyXda.cs are present in the %MT_SOFTWARE_SUITE%/MT SDK/Examples/xda_csharp directory. I copied them from here and still got stuck while trying to get Unity3D to recognize the xsensdeviceapi_csharp64.dll file.

Do you have some tips for me on how to move on with setting up the Unity3D project?

Many thanks in advance!
Gheorghe

How to guarantee that the model has access to the whole motion sequence?

Dear DIP authors,

I am working on a re-implementation of your impressive paper. However, I am struggling when it comes to replicating your data feeding strategy during training.

Quoting from the paper:

At training-time our network has access to the whole sequence [...]

I am wondering how your implementation adheres to this. For training, you use a batch_size of 16 samples for a dataset of motion sequences (of varying length) split up into samples of 300 frames.

Since there are sequences in your dataset of more than 4800 frames (16 samples * 300 frames/sample) which do not fit into a single batch, how do you guarantee that the model has access to the whole motion sequence?

I tried to answer this question from your implementation but didn't find what I was looking for.

Thank you very much for your time!

AMASS

Thank you for your work. I would like to ask if the dataset you provide, AMASS, is already calibrated and pre-processed? Can I feed it directly into the network?

Wrist or Elbow?

In your live demo code, you specify the imu ids as elbows, knees, pelvis and head (also mapped in the SMPLPoseUpdater), whereas your video and the paper shows the locations as wrists, knees, pelvis and head. Is the code not up to date or did the neural network learn to cope with the wrong calibration?

// must match name in definition of SMPL model
_imuIdToBoneName.Add(_rArmId, "R_Elbow");
_imuIdToBoneName.Add(_rLegId, "R_Knee");
_imuIdToBoneName.Add(_lArmId, "L_Elbow");
_imuIdToBoneName.Add(_lLegId, "L_Knee");
_imuIdToBoneName.Add(_pelvisId, "Pelvis");
_imuIdToBoneName.Add(_headId, "Head");

Datasets and configuration

I'm currently replicating your work DIP and I found that the tensorflow version you provided is no longer available. I'm using Python 3.8.15 and I also need opencv, numpy-quaternion, and numba libraries. Can I successfully reproduce your DIP implementation with your codebase?
Besides, I have a question regarding the data_synthesis folder. In the md file, it says to download MoCap sequences from AMASS, and the code here uses a sample sequence from HumanEva. However, I noticed that there are two MoCap datasets in AMASS: TCDHands (TCD_handMocap) and Transitions (Transitions_mocap). I am unsure which one to download, and I also don't know where to place the dataset as there are no corresponding path variables in the .py files.
Also, what does "The code here uses a sample sequence from HumanEva" mean? Does it imply that I need to download the HumanEva dataset?
As a non-native speaker of English and a novice undergraduate student, I am sorry for my lack of understanding and I hope you will not be angry with me for asking such simple and foolish questions. Thank you very sincerely.

Synthetizing from AMASS dataset

I am attempting to utilize the genSynData.py file to synthesize data from the AMASS dataset. However, I've encountered an inconsistency between the official dataset and your code. The Jog_1 example file you're using is in .pkl format, whereas the files provided by AMASS are only available in .npz format. Consequently, I'm facing issues with the code because the load_model function from smpl_webuser.serialization does not support .npz files. I attempted to modify the load_model code, but encountered difficulties with the attributes of the files.

While .npz files contain these attributes:
['trans', 'gender', 'mocap_framerate', 'betas', 'dmpls', 'poses']

.pkl files have these attributes:
['heldout_err', 'resolution', 'all_obs', 'all_tls', 'frame_rate', 'can_mrk', 'poses', 'pose_est_weights', 'gender', 'traindir', 'betas', 'sv_fname', 'trans', 'all_betas']

How can I resolve this compatibility issue between formats, and what preprocessing did you apply to the AMASS data?

Can we apply the SMPL human poses to URDF avatar?

Hi,

First thank you so much for the wonderful work!
I'm currently working with DIP-IMU dataset. The learning targets are SMPL pose parameters, more specifically, the joints orientation w.r.t., their parent link. Is it possible to apply these joints' orientations to a URDF model? For instance, ignoring the deformation, we give the URDF model such a vector [root_pos, root_ori, joints_ori], such that the corresponding motions can also be retargeted to this URDF avatar. Or do we need some further mapping or analytical computations?

Many thanks for considering my request.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.