sigal-raab/modi

Unconditional Motion Synthesis from Diverse Data

modi's Issues

how to test your denoising function？

Thank you for your outstanding work! But I don't know how to test your denoising function. How can I input a wild motion and use your algorithm to remove the noise?

How to evaluate on HumanAct12?

Hi, first of all, thank you for providing this excellent project! I am currently running code to evaluate the effectiveness of MoDi.
Currently, I have completed some of the testing, but have encountered some issues.

1. How to evaluate the performance of the model on the HumanAct12 dataset? I haven't found relevant data or prompts. The paper states that for evaluation on the Mixamo dataset, 2000 results will be sampled; Evaluate on the HumanAct12 dataset and sample 1000 results. The program I am currently running is:

   python evaluate.py --ckpt data/pretrained_model/ckpt.pt --path data/edge_rot_data.npy --act_rec_gt_path data/motion.npy
   

   python evaluate.py --ckpt data/pretrained_model/ckpt.pt --path data/train_edge_rot_data.npy --act_rec_gt_path data/motion.npy
   

   They display 2000 sampled results, so they are all from the Mixamo dataset. Results:
   

2. I completed some tests on the Mixamo dataset and used 2 different data:edge_rot_data.npy and train_edge_rot_data.npy, all of which were able to complete the tests. What are the differences between them?

How do you batch visualize these motions？

Your demo video is cool, and batch visualization is good for viewing the results. Can you tell me how to visualize these motions? What software is used? How does this work?

pretrain encode checkpoint mismatch model

Hi, thank you very much for this gorgeous project! I am testing the performance of Encoder task, but i meet a problem when loading the Pretrained Encoder(MoDi_encoder_f7c850_079999.pt) following readme. the error is below:

        size mismatch for convs.3.convs.1.1.mask: copying a param with shape torch.Size([128, 64, 4, 10, 3]) from checkpoint, the shape in current model is torch.Size([128, 64, 5, 10, 3]).
        size mismatch for convs.3.convs.1.1.scale: copying a param with shape torch.Size([1, 1, 4, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 5, 1, 1]).
        size mismatch for convs.3.convs.1.1.mask: copying a param with shape torch.Size([128, 64, 4, 10, 3]) from checkpoint, the shape in current model is torch.Size([128, 64, 5, 10, 3]).
        size mismatch for convs.3.convs.1.1.scale: copying a param with shape torch.Size([1, 1, 4, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 5, 1, 1]).
        size mismatch for convs.3.skip.1.weight: copying a param with shape torch.Size([128, 64, 4, 10, 1]) from checkpoint, the shape in current model is torch.Size([128, 64, 5, 10, 1]).
        size mismatch for convs.3.skip.1.mask: copying a param with shape torch.Size([128, 64, 4, 10, 1]) from checkpoint, the shape in current model is torch.Size([128, 64, 5, 10, 1]).
        size mismatch for convs.3.skip.1.scale: copying a param with shape torch.Size([1, 1, 4, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 5, 1, 1]).
        size mismatch for convs.4.convs.0.0.weight: copying a param with shape torch.Size([128, 128, 4, 4, 3]) from checkpoint, the shape in current model is torch.Size([128, 128, 5, 5, 3]).
        size mismatch for convs.4.convs.0.0.mask: copying a param with shape torch.Size([128, 128, 4, 4, 3]) from checkpoint, the shape in current model is torch.Size([128, 128, 5, 5, 3]).
        size mismatch for convs.4.convs.0.0.scale: copying a param with shape torch.Size([1, 1, 4, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 5, 1, 1]).
        size mismatch for convs.4.convs.1.1.weight: copying a param with shape torch.Size([256, 128, 2, 4, 3]) from checkpoint, the shape in current model is torch.Size([256, 128, 4, 5, 3]).
        size mismatch for convs.4.convs.1.1.mask: copying a param with shape torch.Size([256, 128, 2, 4, 3]) from checkpoint, the shape in current model is torch.Size([256, 128, 4, 5, 3]).
        size mismatch for convs.4.convs.1.1.scale: copying a param with shape torch.Size([1, 1, 2, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 4, 1, 1]).
        size mismatch for convs.4.skip.1.weight: copying a param with shape torch.Size([256, 128, 2, 4, 1]) from checkpoint, the shape in current model is torch.Size([256, 128, 4, 5, 1]).
        size mismatch for convs.4.skip.1.mask: copying a param with shape torch.Size([256, 128, 2, 4, 1]) from checkpoint, the shape in current model is torch.Size([256, 128, 4, 5, 1]).
        size mismatch for convs.4.skip.1.scale: copying a param with shape torch.Size([1, 1, 2, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 4, 1, 1]).
        size mismatch for final_conv.0.weight: copying a param with shape torch.Size([256, 257, 2, 2, 3]) from checkpoint, the shape in current model is torch.Size([256, 257, 4, 4, 3]).
        size mismatch for final_conv.0.mask: copying a param with shape torch.Size([256, 257, 2, 2, 3]) from checkpoint, the shape in current model is torch.Size([256, 257, 4, 4, 3]).
        size mismatch for final_conv.0.scale: copying a param with shape torch.Size([1, 1, 2, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 1, 4, 1, 1]).
        size mismatch for final_linear.0.weight: copying a param with shape torch.Size([256, 2048]) from checkpoint, the shape in current model is torch.Size([256, 4096]).
        size mismatch for latent_predictor.linear1.weight: copying a param with shape torch.Size([7168, 4096]) from checkpoint, the shape in current model is torch.Size([7168, 5120]).

ImportError: No module named 'fused'

(MoDi) jijiahui@ubuntu:/MoDi$ python generate.py --type sample --motions 18 --ckpt ./data/ckpt.pt --out_path ./output --path ./data/edge_rot_data.npy
Traceback (most recent call last):
File "generate.py", line 13, in
from utils.pre_run import GenerateOptions, load_all_form_checkpoint
File "/data/home/jijiahui/MoDi/utils/pre_run.py", line 5, in
from models.gan import Generator, Discriminator
File "/data/home/jijiahui/MoDi/models/gan.py", line 16, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/data/home/jijiahui/MoDi/op/init.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "/data/home/jijiahui/MoDi/op/fused_act.py", line 18, in
build_directory=os.path.join(os.path.expanduser(''),'tmp', 'stylegan_lock'),
File "/data/home/jijiahui/miniconda3/envs/MoDi/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 898, in load
is_python_module)
File "/data/home/jijiahui/miniconda3/envs/MoDi/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1097, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "/data/home/jijiahui/miniconda3/envs/MoDi/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1418, in _import_module_from_library
file, path, description = imp.find_module(module_name, [path])
File "/data/home/jijiahui/miniconda3/envs/MoDi/lib/python3.6/imp.py", line 297, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'fused'

Pre-trained model link require permissions to access.

Hi Sigal,

Really Appreciate the wonderful work you have open-sourced. When I want to test the model on my own dataset, I found the link for the pre-trained model requiring permission to access. I am wondering if it is possible that you can make it accessible. I would really appreciate that. Thank you so much for this.