Comments (12)
I will try. Thanks very much.
from nphm.
Hi @YiChenCityU,
thanks for your interest.
For the "dummy_dataset" as well as our proposed test set we provide single view data already.
If you want to change some properties of the input for inference you can play araound with scripts.data_processing.generate_single_view_observations
.
I used this script to generate the input. Per default it tries to do so for every subject in the test set. But you can simply specifiy what subject and expression you are interested in.
from nphm.
Thanks very much. What if I only have a point cloud captured from iphone, do I have to provide the expression of it?
from nphm.
This is the point cloud I used and the result was not similar to it. Do you have some suggestions? Ply files are below.
https://drive.google.com/file/d/1UYBbR-TkRtgSKJQbuNUnMu4dwdN1kx9a/view?usp=sharing https://drive.google.com/file/d/1A4EJbSUjuAfJ_k8FzmsimsSKBPi1QQ5k/view?usp=sharing
from nphm.
Hey, cool stuff.
The problem is very likely the coordinate system. NPHM only works if the input is in the expected coordinate system (FLAME coordinate system scaled by a factor of 4).
Therefore, you would first have to align the input point cloud with the FLAME coordinate system, e.g. a very simple approach would be a similarity transform from detected 3D landmarks to the landmarks of the FLAME template. Actually, you could also first fit FLAME and use the resulting Scale, Rotation, and Translation from the result. In that case, you can separate the head from the torso in the same way as in the preprocessing of NPHM. Having observations on the torso tends to confuse the inference optimization
Here is an example mesh from the dataset and one of the provided point clouds to show why the model fails:
from nphm.
Actually, the second Point Cloud aligns better, but is still noticeably off from the expected canonicalization.
from nphm.
Ive been trying to unravel the description as well, I didnt get as far as yichen, It would be wonderful if you could provide a full test example ...
If you are concerned about identity, maybe take a pointcloud of a statue...
from nphm.
Thank you so much @SimonGiebenhain for publishing the code and congrats for your great work!
Quick Q: I have a pointcloud in.obj format (lifted from a foreground RGB-D monocular image) that is transformed to be on the exact same space with FLAME as suggested above. How do you go about fitting NPHM to this particular pointcloud ?
I'm asking because the example provided uses existing identities (along with their expressions) from the dummy_data whereas I'm interested in preserving the identity of the pointcloud.
Thank you!
from nphm.
I am stuck in this project for 2 days. @YiChenCityU and @SimonGiebenhain could you please give more details? Could you provide a full test example? In paper it is mentioned that input is point cloud, so if i have point cloud, how can i get mesh?
from nphm.
@SimonGiebenhain
even with perfect alignment it did not resemble the identity
Original files are here.
https://drive.google.com/drive/folders/1cprPG_9AihL4HpYl0lOvZDz7kNbXv8kB?usp=sharing
from nphm.
The problem is very likely the coordinate system. NPHM only works if the input is in the expected coordinate system (FLAME coordinate system scaled by a factor of 4).
How did you get FLAME models? What solution did you employ?
Therefore, you would first have to align the input point cloud with the FLAME coordinate system, e.g. a very simple approach would be a similarity transform from detected 3D landmarks to the landmarks of the FLAME template.
Do you have this code of alignment or did you use another method to align point cloud and flame?
Actually, you could also first fit FLAME and use the resulting Scale, Rotation, and Translation from the result. In that case, you can separate the head from the torso in the same way as in the preprocessing of NPHM. Having observations on the torso tends to confuse the inference optimization
Do you mean that fitting Flame to point cloud can give us the same NPHM output?
from nphm.
What did use as FLAME model for this experiment? How did you align FLAME with your point cloud?
from nphm.
Related Issues (18)
- linting note: __pycache__
- How would you do the inverse fitting process (from Neural Parametric Head Models to photorealistic portrait) HOT 1
- how to
- how to get registration mesh from scan data๏ผ HOT 2
- problems on facescape data HOT 11
- Training on Facescape Dataset HOT 2
- RGB Images HOT 4
- Fitting and model latent code question HOT 2
- Are you plan to release pretrained models in paper HOT 1
- Can export 3D obj ? HOT 1
- Is it possible to get FLAME 51 landmark from the final Head Models? HOT 1
- How you make point clouds using depth map HOT 1
- straight longhair has bad results HOT 4
- Point Cloud - Custom Dataset HOT 2
- How do you fit the model into a 3090 GPU during training? HOT 4
- Identity with glasses
- FLAME model
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nphm.