Comments (7)
It seems there is some confusion about the mesh input to VOCA. As described in the readme, VOCA animates static templates in FLAME topology. Such templates can be obtained by fitting FLAME to scans, images, or by sampling the FLAME shape space. This means the input mesh must be in semantic correspondence to the FLAME template. Sampling the FLAME shape space for instance can give you an almost infinite amount of templates that can be animated by VOCA.
If you want to get a particular scan into FLAME topology (as we did for the scan on Winston Churchill), you can for instance read Section 4.2 of the FLAME paper how this is done. Long story short, this is an optimization problem where one minimizes the difference between the mesh surface of the FLAME surface and the scan.
We could potentially release VOCA compatible meshes for 1200 subjects obtained by scanning them. I would need to check with the dataset owners though if this is feasible. Would that help anybody?
from voca.
I downloaded a 3d model and tried it. It turned out to be ValueError: Cannot feed value of shape (357, 1499814, 3, 1) for Tensor 'VOCA/Inputs_decoder/template_placeholder:0', which has shape '(?, 5023, 3, 1)'.
from voca.
see #43
I have the same issue.
From my understanding, your 3D model must be registered to the FLAME topology.
There is no information on how to register a 3D model to the FLAME topology in this repo or paper.
EDIT: there is some info in section 4.2 of the paper, I am just not smart enough to make anything of it.
The only thing I found was to use Ringnet repo to pass an image and have it generate a 3D new model.
Not really a good solution.
We are in the realm of "programmers" vs "artists".
Classic in 3D :-)
Thx for this issue, it might bring more attention to the problem.
from voca.
@mbdash The only thing I found was to use Ringnet repo to pass an image and have it generate a 3D new model. Not really a good solution.
Does the model generated in this way meet the requirements of voca?Does this generated new 3D model contain information such as the texture and pose of the input image?Thanks.
from voca.
I havn't tried RingNet https://github.com/soubhiksanyal/RingNet
since the original texture and UVs would need to be redone and I am not an artist.
And the output would still be vtx animation which is not standard for a 3D engine / game.
(I am not saying it is not possible to use it in an engine such as Unreal, just that is is not standard for games)
I am also looking at this repo https://github.com/leventt/surat
which could instead output blendshape animation with a bit more community contribution.
If the community was to help create a dataset, maybe using ARkit from apple, that could provide enough data for the repo owner and contributor to push the code and models to output morph targets, which is way more 3D engine / game friendly.
The tricky part is to get enough people with access to an apple device with "TrueDepth Camera" (found on devices with chip A12) willing to start recording themselves.
from voca.
@TimoBolkart thank you for the information.
For the problem of using a hand modeled 3D mesh, I think what would help the most is any code related on how to accomplish 4.2:
Long story short, this is an optimization problem where one minimizes the difference between the mesh surface of the FLAME surface and the scan.
I think a good solution is to figure out how to solve the
optimization problem where one minimizes the difference between the mesh surface of the FLAME surface and
the scan.
But for an arbitrary 3D Model (obj, fbx, ply, etc) made in 3DSmax, Blender or other 3D software instead of a scan mesh.
Some ppl, (I mean myself) aren't smart enough to implement the beautiful equations in the section 4.2.
It would give an approximate FLAME version of the source mesh which should be acceptable.
(if the character is realistic enough)
A 3D artists would need to re-do UVs and maybe fix some textures, but that would be an acceptable process.
Especially if a mesh is "converted" to FLAME topology prior to texturing.
As for releasing the 1200 subject dataset, it might be interesting and give some options but with potential legal / license issues.
from voca.
@TimoBolkart I checked that the output of the voca model is an obj file containing only the values of v and f. If I have an obj file (containing only the values of v and f) and then convert it into a ply file, can I do anything about it You can rewrite the obj file containing vf, vn, etc. into the obj file output by the model. At this time, I also have the corresponding mtl file and texture png. Is there any method for reference? Thank you.
from voca.
Related Issues (20)
- Windows support and installing MPI-IS / mesh HOT 1
- Should --uv_template_fname be the same as --template_fname in run_voca.py?
- can you give pretrained_models?
- Initialization Of Decoder Layer
- Problem of training voca
- How to control expression in the edit_sequence.py? HOT 2
- Can I train this on custom dataset?
- Unknown mesh file format. HOT 1
- Training with new Tensorflow Version
- Unsolved reference tfbody
- Missing data on subj_seq_to_idx.pkl file HOT 1
- If I want to control the expression of the eyes, how should I set the parameters?
- I haven't found the 'output_graph.pb' file, where can I get it?
- dataset/voca_face_former/templates.pkl not found HOT 1
- Where to download npy and pkl files HOT 1
- Inference other .ply files
- Hot to get blink_exp_betas for another flame version?
- Is the the vertice's coordinates in registered data the same as it in data_verts.npy in training data? HOT 1
- How extract FLAME parameters (expression, jaw) from vertex offset. HOT 1
- After add eyeblink, the face twitches HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from voca.