carlosedubarreto/b3d_mocap_import

addon for blender to import mocap data from tools like easymocap, frankmocap and Vibe

b3d_mocap_import

addon for blender to import mocap data from tools like easymocap, frankmocap and Vibe

==================VIBE==================

To use VIBE you have to install it using

https://github.com/mkocabas/VIBE

if you are on windows, you can follow this repo

https://github.com/carlosedubarreto/vibe_win_install

explained in this tutorial

========================================

===============FRANKMOCAP===============

To use frankmocap you have to install it using

https://github.com/facebookresearch/frankmocap

if you are on windows, you can follow this repo

https://github.com/carlosedubarreto/frankmocap_win_install

explained in this tutorial

========================================

---

Version Beta 0.75

• ui rework
• mediapipe improvement

PS.: To use VIBE you must install a python package inside Blender.

To do that, you have to get your blender path and replace in the following command line (and execute in Windows CMD) if you have further problems, contact me on twitter(https://twitter.com/carlosedubarret)

The Commandline:

D:\Blender\blender-2.92.0-windows64\2.92\python\bin\python.exe D:\Blender\blender-2.92.0-windows64\2.92\python\lib\site-packages\pip install joblib

Version Beta 0.1 This version imports data from Easymocap only. (https://github.com/zju3dv/EasyMocap)

For contact you can reach me at:

https://twitter.com/carlosedubarret

If you find bugs, please report them here:

https://github.com/carlosedubarreto/b3d_mocap_import/issues

b3d_mocap_import's Issues

Traceback (most recent call last):

This happens when I click on Animation MOCAP Import

Mediapipe to retarget Daz Characters

Add option to retarget daz characters

Need some help on getting Frankmocap Data onto Auto-Rig Pro default biped rig

Dear Carlos,

first of all i'd like to thank you for implementing a pipe between Blender and Frankmocap! But because i'm way more familiar with Maya and a fresh noob to Blender i'd like to ask for some guidance on how to get this to work. So here is my progress so far:

1. Running Frankmocap on Linux and Windows -> exporting anim as .mkl sequence (Done)
2. Using Blender 2.90.1 (installed Auto-Rig Pro) and also your Add-on (Done)
3. Creating a default human biped-rig with ARP (Done)
4. Trying to import mkl-sequence via Mocap-UI (your add-on) Fails

When i browse to my folder with the mkl files and select the first one it got this error:

Traceback (most recent call last):
File "C:\Users\frank\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons\b3d_mocap_import-main\load_mocap.py", line 729, in execute
import joblib
ModuleNotFoundError: No module named 'joblib'

location: :-1

It's obvisously due to a missing "joblib"-package and i've already tried the button "Install Joblib" in the UI but no luck.
So again ... if you could give me small notes on how to get the data loaded to the rig i'd be super thankful.
Also maybe i was missing some steps inbetween.

Thank you very much!

No module named 'cv2' error installing package in Blender

Code to use for installation
Get it at
https://www.patreon.com/posts/54890147

Use a portable blender version for windows, if the code above doesnt work, follow the instructions on the link below

Solution on this issue
carlosedubarreto/vibe_win_install#17 (comment)

Easymocap demo

May I ask how do I fix this please?

How to install Mocap_PE_data_import for Blender?

Hi Carlos

Is there a link to the steps to install the Blender mocap importer called Mocap_PE_data_import?
I would like to open the output of EasyMocap in Blender.

The files I have are: _init_ helper load_mocap and test_panel but I am unsure where these files need to go for Blender or how to install.

Regards,
Jamal

Error when trying generating .pkl files from EasyMocap

Dear Carlos,

I've tried to follow your guidelines on how to properly render out .pkl-files from EasyMocap, therefor i was replacing the 2 files like you've mentioned. After running a batch-file i got this error:

Traceback (most recent call last):
File "apps/demo/mv1p.py", line 113, in
mv1pmf_smpl(dataset, args)
File "apps/demo/mv1p.py", line 72, in mv1pmf_smpl
weight_shape=weight_shape, weight_pose=weight_pose)
File "/media/frankfurt/Walhalla/Ubuntu/dev/EasyMocap/easymocap/pipeline/basic.py", line 58, in smpl_from_keypoints3d2d
params_init = body_model.init_params(nFrames=1)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 576, in getattr
type(self).name, name))
AttributeError: 'SMPLlayer' object has no attribute 'init_params'

For rendering i'm using the following command: (which is quite likely wrong but maybe you can help me how to set the parameters correct) ... one more note: I'm running the out-of-the box-easymocap example and i'm aiming for body and hands as outputs (you already told me that you've implemented body and hands within your great Blender-addon)

EasyMocap-run-command:
python3 apps/demo/mv1p.py ${data} --out ${out}/smplx --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male --vis_smpl

Thank you so much!

Edit: also tried this one with the same error:
python3 apps/demo/mv1p.py ${data} --out ${out}/b3d --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body bodyhand --model smplh --gender male --vis_smpl

Frankmocap and VIBE

Hello Carlos,

Thanks for the detailed info on each method! I ran Frankmocap but have no way to convert the output file(s) to fbx or bvh. Is that part completed? Seeing the VIBE method DID have fbx import to blender i ran this also but found that the hands did not capture. Did you find that or did i do something wrong on the VIBE installation? Hands captured fine on FRANK but not on VIBE

Spine, Neck And Head not Axis Rotating

Spine, neck, and head not rotating with the model. It took me a while to figure out what was happening. But I keep noticing that shoulder and legs are always twisting. Then I finally decided to add bone group to left and right side of the armature and assign them different color, and turned on axis of the bones. This is when I realize the spine Z axis is not rotating at all. I am not entirely sure yet but I think from Head down to the foot the z axis direction remains the same. This probably explains why the character mesh looks wonky when it starts to tun from front to back. And the foot is still facing the same direction even though it should be facing in a different direction.

os.sep instead of '\\' for portability in path string construction

Minor portability bug: In order to make the plugin work in Blender on Linux, I had to replace backslash filename separators with os.sep, which should resolve to backslash in Windows and forward slash in other OSes. This was limited to the load_mocap.py module, just replace '\\' with os.sep in 4 places.

Thanks for all the work you've put into this add-on!

Armature not recognized in blender, maybe I failed the installation.

Hello Carlos, thanks again for your patience,
this is what happens when I import the data:

There's a strange piece of armature and some little points moving.
What the hell? xD
How can I fix this?

Thank, thank you.

Problem importing SMPL generated by EasyMocap

I have this error when i'm trying to import with your plugin (my results are in: '1_output\sample\smpl' folder), what is the problem?

Python: Traceback (most recent call last):
  File "C:\Users\andrea\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 356, in execute
    #bpy.data.objects[armature].select_set(True)
  File "C:\Users\andrea\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 273, in load_smpl_params
    motions = load_motions(datapath)
  File "C:\Users\andrea\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 258, in load_motions
    infos = read_smpl(filename)
  File "C:\Users\andrea\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 230, in read_smpl
    data_pkl = read_pkl(filename+'.pkl')
  File "C:\Users\andrea\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 222, in read_pkl
    with open(path_pkl, 'rb') as pk:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\MOCAP\\EasyMocap\\1_output\\sample\\smpl\\000000.pkl'

location: <unknown location>:-1

Install version 0.0.722 on Blender fails

Hi Carlos,

when i try to install the latest version in Blender i got this error:

Traceback (most recent call last):
File "C:\Program Files\Blender Foundation\Blender 2.90\2.90\scripts\modules\addon_utils.py", line 351, in enable
mod = import(module_name)
File "C:\Users\frank\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons\b3d_mocap_import-main_init_.py", line 39, in
from . load_mocap import Import_Data_easymocap, OT_TestOpenFilebrowser,Import_Data_frankmocap,Import_Data_vibe,Mediapipe_Pose_estimation,
ImportError: cannot import name 'Audio2face_Export' from 'b3d_mocap_import-main.load_mocap' (C:\Users\frank\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons\b3d_mocap_import-main\load_mocap.py)

before i tried to install the new version i un-installed the older one (0.0.7) via the AddOn-UI in Blender.

Maybe i was missing some other steps.
Thanks again!

TypeError: Converting py args to operator properties: : keyword "scale" unrecognized

Hi I don't know how to solve the following problem
Traceback (most recent call last):
File "/home/dms/.config/blender/2.83/scripts/addons/b3d_mocap_import/load_mocap.py", line 1044, in execute
create_dots('Point',49)
File "/home/dms/.config/blender/2.83/scripts/addons/b3d_mocap_import/load_mocap.py", line 765, in create_dots
bpy.ops.mesh.primitive_plane_add(enter_editmode=True, align='WORLD', location=(0, 0, 0), scale=(1, 1, 1))
File "/home/dms/blender-2.83.19-linux-x64/2.83/scripts/modules/bpy/ops.py", line 201, in call
ret = op_call(self.idname_py(), None, kw)
TypeError: Converting py args to operator properties: : keyword "scale" unrecognized

location: :-1

The model gets distorted when retargeting

There is a problem when the bone is used on retargeting

error while using your tool

Hi, I am trying to import the easymocap output into blender using ur tool but i am getting an error. I tried using blender 3 and 2.8 on mac

Can't import as NLA Strip from OMNIVERSE

Hello
I am using blender 3 (i tried also with 2.93), after i exported the model trough the addon and i made all the set up in omniverse, i exported as maya cache the result. If i import the model with the modifier everything works well but id i try to import as NLA Strip i get this result, do you know why?
Thanks

PS the models are from Human Generator Addon

how to import bvh

thanks man for the help i am able to to bring in the mocap using smpl model in blender
I need to retarget it on make human model so i have to export it as a bvh file then retarget it can i avoid some of this steps to make the workflow faster ?

FrankMocap Development and Hands-On

Hi Carlos!. I am trying to complete the Frankmocap import algorithm. Could you suggest me how to quickly test the results? Create a separate test environment (a venv with bpy) and export the resulting fbx or directly it is better to work on the addon that imports into blender, reopen blender and test the import of Frankmocap (this process seems soooo slow to me). Thank you and greetings.

when i start realtime mediapip get that error

hello carlos ...thank you for ur amazing work and i faced that issues when i start mocap on blender

Python: Traceback (most recent call last):
File "C:\Users\azida\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\b3d_mocap_import-alpha_realtime\load_mocap.py", line 3351, in modal
msglen = int(msg[:HEADERSIZE])
ValueError: invalid literal for int() with base 10: b''

location: :-1

Hand Only Frankmocap

Is the addon capable of only converting PKL files exported from hand-only mocaps from frank mocap or it's for full-body mocap?

UnboundLocalError: local variable 'ob' referenced before assignment

Hi,

I am generating an smplfull folder from a monocular video using Easymocap and importing it to Blender, through this process:

• Generating openpose keypoints:
  python .\scripts\preprocess\extract_video.py .\0_input\test_02 --mode openpose --openpose ..\openpose\ --handface
• Generating the Easymocap output:
  python .\apps\demo\mocap.py 0_input/test_02 --fps 30 --mono --mode smpl
• Importing the 000000.json file to the "Easymocap SMPL" in the add-on with the "Monocular Import" enabled.

When I try to import it, I get this error message:

Python: Traceback (most recent call last):
  File "C:\Users\erent\AppData\Roaming\Blender Foundation\Blender\4.0\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 353, in execute
    ob, obname, arm_ob, cam_ob= init_scene(scene, params, obj_gender)
  File "C:\Users\erent\AppData\Roaming\Blender Foundation\Blender\4.0\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 333, in init_scene
    ob.data.use_auto_smooth = False  # autosmooth creates artifacts
UnboundLocalError: local variable 'ob' referenced before assignment

I looked into the load_mocap.py file, and tried to declare the ob in all cases, but that didn't seem to help.

I am using:

• Blender 4.0
• python 3.11
• MoCap Import 0.82

Traceback (most recent call last)

Whenever I try to import the .smpl files from easymocap, I get this error. Version 0.78 installed.

How to bring OpenPose output into Blender

Multiple cameras are problematic for me. I am wondering whether EasyMocap can produce BVH output from OpenPose .json output. Do you know? Or is the only workflow to output a model that can be converted to fbx and the animation then transferred to another character rig in Blender?

OpenPose is mentioned in the Git EasyMocap project, but it's not clear how to proceed. Haven't been able to find much written instruction on EasyMocap but will now see your videos on it and your plug-in too, which I understand can now import finger mocap.

Would text readme's for these projects be helpful? I submitted a workflow pull request for Forth MocapNet. BTW, that project will output a body animation .bvh for Blender using OpenPose input. The finger mocap is working too, but is not yet made public - something to do with the author's PhD work restriction.

ModuleNotFoundError: cv2 when generating mocap in Blender 2.91

cv2 and mediapipe aren't installed in Blender (checked through blender python console), despite mediapipe appearing in pip list -u in console.

Environment: Windows 10

Error message follows

location: :-1
Python: Traceback (most recent call last):
File "C:**\AppData\Roaming\Blender Foundation\Blender\2.91\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 1404, in execute
import cv2
ModuleNotFoundError: No module named 'cv2'

real time addon install

can u plz tell how to intsall the alpha version of real time mocap
shall i have to run any commands?

Problem loading some easymocap files

some easymocap data is giving problems, and to work again, I have to clean the scene and sometimes close and load blender again

here is a sample of the data tha rises error
keypoints3d.zip

EasyMocap errors

May I ask why when I extract the demo videos:

(easymocap) C:\Users\Samuel\mocap\EasyMocap>python scripts/preprocess/extract_video.py 0_input\sample

and it shows the json files in the annots folder are skipped:

cameras: 01 02 03 04 05 06 07 08 09 10 11 13 14 16 17 18 20 21 23 skip 0_input\sample\annots\01 skip 0_input\sample\annots\02 skip 0_input\sample\annots\03 skip 0_input\sample\annots\04 skip 0_input\sample\annots\05 skip 0_input\sample\annots\06 skip 0_input\sample\annots\07 skip 0_input\sample\annots\08 skip 0_input\sample\annots\09 skip 0_input\sample\annots\10 skip 0_input\sample\annots\11 skip 0_input\sample\annots\13 skip 0_input\sample\annots\14 skip 0_input\sample\annots\16 skip 0_input\sample\annots\17 skip 0_input\sample\annots\18 skip 0_input\sample\annots\20 skip 0_input\sample\annots\21 skip 0_input\sample\annots\23

Does it matter?

and i run the next code (#2.1 example for SMPL reconstruction):

(easymocap) C:\Users\Samuel\mocap\EasyMocap>python apps/demo/mv1p.py 0_input/sample --out 1_output/sample --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --vis_smpl

it shows an error:

Traceback (most recent call last): File "C:\Users\Samuel\mocap\EasyMocap\apps\demo\mv1p.py", line 11, in <module> from easymocap.pipeline import smpl_from_keypoints3d2d File "c:\users\samuel\mocap\easymocap\easymocap\pipeline\__init__.py", line 1, in <module> from .basic import smpl_from_keypoints3d, smpl_from_keypoints3d2d File "c:\users\samuel\mocap\easymocap\easymocap\pipeline\basic.py", line 8, in <module> from ..pyfitting import optimizeShape, optimizePose2D, optimizePose3D File "c:\users\samuel\mocap\easymocap\easymocap\pyfitting\__init__.py", line 9, in <module> from .optimize_mirror import optimizeMirrorDirect, optimizeMirrorSoft File "c:\users\samuel\mocap\easymocap\easymocap\pyfitting\optimize_mirror.py", line 10, in <module> from ..dataset.mirror import flipSMPLPoses, flipPoint2D, flipSMPLParams File "c:\users\samuel\mocap\easymocap\easymocap\dataset\__init__.py", line 11, in <module> from .mv1pmf_mirror import MV1PMF_Mirror File "c:\users\samuel\mocap\easymocap\easymocap\dataset\mv1pmf_mirror.py", line 14, in <module> from .mirror import calc_mirror_transform, flipSMPLParams, mirrorPoint3D, flipPoint2D, mirror_Rh File "c:\users\samuel\mocap\easymocap\easymocap\dataset\mirror.py", line 18, in <module> FLIP_SMPL_VERTICES = np.loadtxt(join(os.path.dirname(__file__), 'smpl_vert_sym.txt'), dtype=np.int) File "C:\Users\Samuel\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\__init__.py", line 284, in __getattr__ raise AttributeError("module {!r} has no attribute " AttributeError: module 'numpy' has no attribute 'int'. Did you mean: 'inf'?

May I ask how do i fix this?

Error trying to export to NVIDIA omniverse

Whenever I try to export my head from blender to a .usda file for nvidia omniverse, I get this error message in blender.

Use of Easymocap to obtain a output with file replacement

Hi!.
Thanks for the code. And trying to get the Easymocap output to import it into Blender, but I have had problems with the output even though I have replaced the 2 files indicated in the Git:
(env) laila@laila:~/Nexar/EasyMocapP$ python apps/demo/mv1p.py 0_input/sample --out 1_output/sample/smplx --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male --vis_smpl --write_smpl_full

Demo code for multiple views and one person:

- Input : 0_input/sample => 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23
- Output: 1_output/sample/smplx
- Body  : smplx=>male, bodyhandface

triangulation: 100%|██████████████████████████| 800/800 [05:54<00:00, 2.26it/s]
dump: 100%|█████████████████████████████████| 800/800 [00:00<00:00, 2730.57it/s]
loading: 100%|███████████████████████████████| 800/800 [00:05<00:00, 145.35it/s]
Traceback (most recent call last):
File "/home/laila/Nexar/EasyMocapP/apps/demo/mv1p.py", line 113, in
mv1pmf_smpl(dataset, args)
File "/home/laila/Nexar/EasyMocapP/apps/demo/mv1p.py", line 70, in mv1pmf_smpl
params = smpl_from_keypoints3d2d(body_model, kp3ds, keypoints2d, bboxes,
File "/home/laila/Nexar/EasyMocapP/easymocap/pipeline/basic.py", line 58, in smpl_from_keypoints3d2d
params_init = body_model.init_params(nFrames=1)
File "/home/laila/Nexar/EasyMocapP/env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 947, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'SMPLlayer' object has no attribute 'init_params'
(env) laila@laila:~/Nexar/EasyMocapP$

What error could be associated with this? I am using the June 18 version of Easymocap, when I got this error:
#14

Thanks!

Add realtime pose estimation using Mediapipe

Add it to the blender addon.

Put a windows to view the video result with the landmarks

Try to change the timer, to frame change to speed the process

Size of bones using MediaPipe

The bones are too big,
probably will need to create a reference bone sizes and then map the mediapipe result to it.

Report from Tintin
https://twitter.com/tintwotin/status/1379037852533714951

about getting joblib error when i try to import sk import frankmocap

Hi again, :)

I'm getting this error when i click on sk import frankmocap. I clicked on on the "install joblib" but still have the problem.
Here is what i get;( I took 3 screenshot for showing entirely about what i've done)

Importing EasyMocap finger motion

Today I try to import SMPL-X model into Blender 2.93.1 using EasyMoCap, I tried easymocap and successufully get the SMPL-full folder which contains a lot of json files, BTW I previously successfully import SMPL model into blender and I did this because I want to do the retargeting as the video https://www.youtube.com/watch?v=gvqGxQsPHuc&list=PLYzax7LPWaAAkTEXSWZanCsDmuZ8y3Bq7&index=4

But I got this error with Blender
FBX Import: start importing E:\3DProject\SMPL Series\smplx\SMPLX-Male.fbx
FBX version: 7400
FBX import: Prepare...
Done (0.000000 sec)

    FBX import: Templates...
            Done (0.000000 sec)

    FBX import: Nodes...
            Done (0.015625 sec)

    FBX import: Connections...
            Done (0.000000 sec)

    FBX import: Meshes...
            Done (0.187500 sec)

    FBX import: Materials & Textures...
            Done (0.000000 sec)

    FBX import: Cameras & Lamps...
            Done (0.000000 sec)

    FBX import: Objects & Armatures...
            Done (0.046875 sec)

    FBX import: ShapeKeys...
            Done (5.265625 sec)

    FBX import: Animations...
            Done (0.000000 sec)

    FBX import: Assign materials...
            Done (0.000000 sec)

    FBX import: Assign textures...
            Done (0.000000 sec)

    FBX import: Cycles z-offset workaround...
            Done (0.000000 sec)

    Done (5.859375 sec)

Import finished.
success load
Python: Traceback (most recent call last):
File "C:\Users\adminis\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\b3d_mocap_import-main\load_mocap.py", line 344, in execute
bpy.data.shape_keys["Key"].key_blocks[k].slider_min = -10
KeyError: 'bpy_prop_collection[key]: key "Basis" not found'

location: :-1

I'll appreciate it if someone can help me.

Error when import pkl file in blender

so iam using blender 2.93 and addon 0.7222
when iam using sk import Frankmocap

then hit import

and i got this error

the cmd line code i use for frankmocap is : "python -m demo.demo_bodymocap --input_path 1.mp4 --out_dir ./mocap_output --save_pred_pkl"
the pkl file can be find in the zip
1.zip

SMPL-X import

Hi, Thank you for very useful content, I had a question about importing smplx models using the addon mentioned here:
https://www.youtube.com/watch?v=-vvSjAyiK_g

I tried to import the result of smplify-x to blender using the addon but encounter an error
https://github.com/vchoutas/smplify-x

I was wondering is there any way to import the smplx models?
Thank you very much in advance!
I look forward to hearing from you

Best regards,
Leila

Vibe Colab with GUI link to Drive

I hope admin won't mind.

I would just like to share my version of VIBE Google Colab.

In this version, you can upload your video and save the result on google drive.

https://colab.research.google.com/drive/1AHiHkQ_SWOCFMnMx1WKGemNrPN8_2OH2?usp=sharing

Also, I have fixed the bug that prevents the model from downloading to Google Colab. Users might be able to use the same procedure if they want to run this on their pc.

Can I use it in ubuntu?

Hello, thank you for your work.
Can I use it in ubuntu?

Some problems with installing MOCAP PE Data Import

Blender version 2.93 is installed in Windows, and the installation of Mocap_PE_data_import_ALPHA is completed. When loading data, this error occurs. Should I modify the source code of Blender