papagina/auto_conditioned_rnn_motion

Implementation of Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

acLSTM_motion

This folder contains an implementation of acRNN for the CMU motion database written in Pytorch.

See the following links for more background:

Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

CMU Motion Capture Database

Prequisite

You need to install python3.6 (python 2.7 should also be fine) and pytorch. You will also need to have transforms3d, which can be installed by using this command:

pip install transforms3d

Data Preparation

To begin, you need to download the motion data form the CMU motion database in the form of bvh files. I have already put some sample bvh files including "salsa", "martial" and "indian" in the "train_data_bvh" folder.

Then to transform the bvh files into training data, go to the folder "code" and run generate_training_data.py. You will need to change the directory of the source motion folder and the target motioin folder on the last line. If you don't change anything, this code will create a directory "../train_data_xyz/indian" and generate the training data for indian dances in this folder.

Training

After generating the training data, you can start to train the network by running the pytorch_train_aclstm.py. Again, you need to change some directories on the last few lines in the code, including "dances_folder" which is the location of the training data, "write_weight_folder" which is the location to save the weights of the network during training, "write_bvh_motion_folder" which is the location to save the temporate output of the network and the groundtruth motion sequences in the form of bvh, and "read_weight_path" which is the path of the network weights if you want to train the network from some pretrained weights other than from begining in which case it is set as "". If you don't change anything, this code will train the network upon the indian dance data and create two folders ("../train_weight_aclstm_indian/" and "../train_tmp_bvh_aclstm_indian/") to save the weights and temporate outputs.

Testing

When the training is done, you can use pytorch_test_synthesize_motion.py to synthesize motions. You will need to change the last few lines to set the "read_weight_path" which is the location of the weights of the network you want to test, "write_bvh_motion_folder" which is the location of the output motions, "dances_folder" is the where the code randomly picked up a short initial sequence from. You may also want to set the "batch" to determine how many motion clips you want to generate, the "generate_frames_numbers" to determine the length of the motion clips et al.. If you don't change anything, the code will read the weights from the 86000th iteration and generate 5 indian dances in the form of bvh to "../test_bvh_aclstm_indian/".

The output motions from the network usually have artifacts of sliding feet and sometimes underneath-ground feet. If you are not satisfied with these details, you can use fix_feet.py to solve it. The algorithm in this code is very simple and you are welcome to write a more complex version that can preserve the kinematics of the human body and share it to us.

For rendering the bvh motion, you can use softwares like MotionBuilder, Maya, 3D max or most easily, use an online BVH renderer for example: http://lo-th.github.io/olympe/BVH_player.html

Enjoy!

auto_conditioned_rnn_motion's Issues

Parsing bvh files

Thank you for your great work.

If I would like to use arbitrary bvh dataset, what should I do?

I already tried to convert not only CMU but also another bvh dataset, but When I converted another bvh dataset, it seems can not parse correctly.

When I visualize it, it seems strange moving. It's not only predicted result but also ground truth.

How did you get the results in Table 1 ?

Hi
Where can I find the code to regenerate the prediction results of your paper?

get_train_data and write_traindata_to_bvh

Hi

When I convert and restore BVH file, the area around End Site of restored motion is different.
Restored motion tremble.

The code is following:

from read_bvh import get_train_data, write_traindata_to_bvh

xyz_data = get_train_data("../train_data_bvh/indian/05.bvh")
write_traindata_to_bvh("restored_05.bvh", xyz_data)

How to solve this problem?

Can I experiment with another CMU data?

sorry maybe this is just my misundersood.
please delete this issue.

standard pose skeleton

Is the skeleton (standard.bvh) that is used to generate euclidean points based on a subject in the CMU dataset?

Assertion Error write bvh

Hello, I have been training the model in different data and when is time to write the bvh animation I get the following error from

rotation2xyz.py script:

"-->assert np.allclose(np.linalg.norm(children_xyz[i, :]), np.linalg.norm(children_orig[i, :]))
Assertion Error"

since there is no comment explaining this code I am having a really hard time figuring out how to solve it. I am using a small animation of shape:

[5,60,171]

The generate joint are rotated

Nice work!
I have a question~

The joint points in the data generated by training model are rotated. The joint of the original BVH data is not rotated, but when the original BVH data is converted to NPY and after the training is transferred out of the BVH, the joints begin to rotate. How to avoid this problem?
thank you！

How does the augment_train_data function work?

@papagina Hello, I am very interested in your work. But I have a question here when reading your code. Could you kindly explain how the augment_train_data function works below?

def get_train_data(bvh_filename):
    
    data=parse_frames(bvh_filename)
    train_data=get_training_format_data(data, non_end_bones,skeleton)
    center=get_motion_center(train_data) #get the avg position of the hip
    center[1]=0.0 #don't center the height

    new_train_data=augment_train_data(train_data, -center, [0,1,0, 0.0])
    return new_train_data

fix_feet.py

Hi,

the fix_feet.py doesnt seem to run of the shelf, has unitialized and unused variables.

can you provide comments for the code. its hard to understand.

Thanks,

Error 'float' object is not subscriptable

Hi I got an error when I run the file pytorch_train_aclstm.py

<ipython-input-13-4f37744e12e2> in train_one_iteraton(real_seq_np, model, optimizer, iteration, save_dance_folder, print_loss, save_bvh_motion)
    140     if(print_loss==True):
    141         print ("###########"+"iter %07d"%iteration +"######################")
--> 142         print ("loss: "+str(loss.data.tolist()[0]))
    143 
    144 

TypeError: 'float' object is not subscriptable

Network structure is different with presented in paper

I think the network consists three fc layers in paper while in your code, there is only one fc layer but 3 LSTM layers.

The first predict frame problem

I am very interested in your work, thank you very much for your work.

I have a problem. When I train the model, whether it is the result of the test or the tmp result in the training process will be like this picture in the first frame. How to avoid this problem, how to solve it?