jimwest/mefamo

mefamo's Issues

How to use android camera?

How to use android camera for this. i have Realme x2 . which has 2Mp depth camera . i think which will give more accurate movement than webcam.

black screen with obs virtual camera and other

i tested the first realease on window and i have a black screen with obs virtual camera and other

Is it a mistype error ?

In code of computing a stretch_max_right value, looks like there is a copy-paste mistype
stretch_max_right = -0.45 +
(0.45 * mouth_smile_right) + (0.36 * mouth_left)
Should it be mouth_right used here instead ?

MeFaMo/mefamo/blendshapes/blendshape_calculator.py

Line 170 in 341813f

(0.45 * mouth_smile_right) + (0.36 * mouth_left)

Face moving but not properly calibrated/synced way

It's amazing work!

I found that the LLink Face with Metahuman-generated human data doesn't 'sync' or 'calibrated' properly.
Below is the captured expression from video, the smiling person. mediapipe tracked it quite accurately.

The metahuman response to MeFaMo-transfered data.

Is there any parameter or step I should check or need to improve?

In your demo video, your smile is quite well-synced.
https://www.reddit.com/r/unrealengine/comments/r8wbe3/my_livelink_facetracking_without_an_apple_device/

Does the fresh-exported metahuman data need blueprint modification following below link you mentioned?
https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/FacialRecordingiPhone/

Another Strange example.
Just image input, and the result.

#`self._live_link_face.set_blendshape(FaceBlendShape.MouthClose, mouth_open)`

#self._live_link_face.set_blendshape(FaceBlendShape.MouthClose, mouth_open) in blendshape_calculator.py
if I delete this code,the program will run well.Could you help me explain the meaning of this code

Apple silicon requires mediapipe-silicon package

mediapipe package doesn't exist for Apple silicon. Use/install mediapipe-silicon instead.

This could probably be noted in the README.

How to convert the calculation of body posture to Unreal's format

Thanks to the livelink method you shared, I want to send the pose matrix of the body, but I don't know the pose format name that Livelink's bones need

Get arkit from blendshapes_calculator, and generate arkit csv file to drive metahuman

I generated the arkit csv file from blendshapes_calculator. By using the arkit csv file to drive the metahuman to generate animation of the metahuman. But the result of matahuman face animation performed unnormal shaking. However, I used live link face to drive the metahuman live face, the animation performed normally without shaking mentioned before. Could anyone help me, why this unnormal shaking happened? Thank you very much!

Is this repository cross platform or windows only?

Hello
I am trying to use this repository on my Ubuntu system with UE 5 and I am not able to find source for PyLiveLink python library. I have tried many things but I am not able to connect with udp server with LiveLink.

I am new to UE 5, so I don't know if it's UE issue or any other issue.

Thank You. Cheers.

what is get_metric_landmarks function for?

Hi I am reading your code. I don't know what is the purpose for get_metric_landmarks

MeFaMo/mefamo/custom/face_geometry.py

Line 2482 in 341813f

def get_metric_landmarks(screen_landmarks, pcf):

Very cool

I tried to use mediapipe with iClone. My major problem with mediapipe is the jitter. Even applying some filters I was still unable to get rid of the jitter enough to make it useful. Did you find a way to filter out the jitter?

Here is my attempt with the hand mocap. https://www.youtube.com/watch?v=j6JboJIlpfM

Using --input video file

eye movement / iris tracking

what would it take to enables the iris tracking? could it be enabled simply, or would it need some deeper implementation?

2nd thing that would be very cool is some kind of recroding data straight from the app, to some .csv or something.

rotate face to neutral pose first

I noticed that the results vary with different face rotation / head tilt, since the values used are tuned to the neutral / upright face rotation. I think you should first rotate the landmarks into a neutral pose before doing the calculations, so that results are rotation invariant. Are there plans on adding this feature?

Set which face to track when multiple faces are visible

Is there a way to set the detected face index to track?

Sometimes there are multiple faces on the image, and it would be useful to select the index (from left to right for example) of the target face.

Compile Python to Executable

Hi Jim, thanks for creating this repo, this combined with pylivelink opens a lot of a opportunities.

Can you share some details on how you created the stand along binary? Specifically how you've incorporate the libraries such as mediapipe and opencv.

Blend shape Config

Hello.
Super work. Thanks for sharing.
How did you work out the Blend shape type, min and max value.

MeFaMo/mefamo/blendshapes/blendshape_config.py

Line 45 in aed045f

# blend shape type, min and max value

Thanks 😄

No source detected in ue4.26/ue4.27

Hi, this is some really amazing work. I am trying to read the Live Link data in UE but I dont see any source detected there. I used your PyLiveLinkFace lib to verify if any data was actually being sent and it is receiving the data from MeFaMo correctly. Do you have any idea what the issue might be? Thanks

No video

When I start the mefamo.exe the screen remains black and there is no video input from the webcam.

Blackmagic software was the issue.

Which version of Python and of mediapipe do you use?

Hello! Your work here looks fun I'd love to try it out!
But I'm having problems getting 'mediapipe' installed on my system.

Which version of Python and of mediapipe do you use?

Cheers!

LiveLinkFace not showing up in Unreal

hi Hey I downloaded the EXE file from your github, it works and identifier my face, but I don't know how to get the data inside unreal. Can you help me?

Originally posted by @game-alle in #8 (comment)