lizhenwangt / faceverse Goto Github PK
View Code? Open in Web Editor NEWFaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset (CVPR2022)
License: BSD 2-Clause "Simplified" License
FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset (CVPR2022)
License: BSD 2-Clause "Simplified" License
Hi I noticed you have been online all day helping people including myself across your projects so I just wanted to say it's much appreciated as I know most of them and it's been a big help.
I wanted to ask about the quality of tracking as the outputs are all such a low resolution (256x256) I figure that's the reason I am getting sub-optimal results with lip-synchronization and if you had any plans to increase the quality, I understand this would come at a cost to speed but without better tracking I don't see much utility in StyleAvatar
please find the files "Driving.mp4" as the Lip-synced video
and "Lip-sync.mp4" the result of StyleAvatar
https://drive.google.com/drive/folders/1d0yzTnaDM-O4hYV3aXuHJiTMwFZZ10B2?usp=sharing
It could be the case I could just change a few settings or switch to linux for better tracking but I am unsure if this is the case
(styleavatar) C:\Users\Jack\FaceVerse\faceversev3_jittor>python tracking_offline_cuda.py --input C:\\Users\\Jack\\StyleAvatar\\styleavatar\\vids\\render.mp4 --res_folder C:\\Users\\Jack\\FaceVerse\\faceversev3_jittor\\output --save_for_styleavatar --crop_size 1536
[i 0830 00:07:09.335000 00 compiler.py:956] Jittor(1.3.8.5) src: c:\users\jack\anaconda3\envs\styleavatar\lib\site-packages\jittor
[i 0830 00:07:09.358000 00 compiler.py:957] cl at C:\Users\Jack\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe(19.29.30133)
[i 0830 00:07:09.358000 00 compiler.py:958] cache_path: C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default
[i 0830 00:07:09.363000 00 install_cuda.py:93] cuda_driver_version: [12, 2, 0]
[i 0830 00:07:09.384000 00 __init__.py:411] Found C:\Users\Jack\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe(11.2.67) at C:\Users\Jack\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe.
[i 0830 00:07:09.426000 00 compiler.py:1011] cuda key:cu11.2.67
[i 0830 00:07:09.428000 00 __init__.py:227] Total mem: 15.94GB, using 5 procs for compiling.
[i 0830 00:07:10.271000 00 jit_compiler.cc:28] Load cc_path: C:\Users\Jack\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe
[i 0830 00:07:10.278000 00 init.cc:62] Found cuda archs: [52,]
[w 0830 00:07:10.364000 00 compile_extern.py:203] CUDA related path found in LD_LIBRARY_PATH or PATH(['', 'C', '\\Users\\Jack\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\lib64', '', 'C', '\\Users\\Jack\\.cache\\jittor\\mkl\\dnnl_win_2.2.0_cpu_vcomp\\bin', '', 'C', '\\Users\\Jack\\.cache\\jittor\\mkl\\dnnl_win_2.2.0_cpu_vcomp\\lib', '', 'C', '\\Users\\Jack\\.cache\\jittor\\jt1.3.8\\cl\\py3.8.16\\Windows-10-10.x93\\IntelRCoreTMi7x72\\default', '', 'C', '\\Users\\Jack\\.cache\\jittor\\jt1.3.8\\cl\\py3.8.16\\Windows-10-10.x93\\IntelRCoreTMi7x72\\default\\cu11.2.67', '', 'C', '\\Users\\Jack\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\bin', '', 'C', '\\Users\\Jack\\.cache\\jittor\\jtcuda\\cuda11.2_cudnn8_win\\lib\\x64', '', 'C', '\\Users\\Jack\\.cache\\jittor\\msvc\\win10_kits\\lib\\ucrt\\x64', '', 'C', '\\Users\\Jack\\.cache\\jittor\\msvc\\win10_kits\\lib\\um\\x64', '', 'C', '\\Users\\Jack\\.cache\\jittor\\msvc\\VC\\lib', '', 'c', '\\users\\jack\\anaconda3\\envs\\styleavatar\\libs', 'C', '\\Users\\Jack\\.cache\\jittor\\msvc\\VC\\_\\_\\_\\_\\_\\bin', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar\\lib\\site-packages\\cv2\\../../x64/vc14/bin', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar\\Library\\mingw-w64\\bin', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar\\Library\\usr\\bin', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar\\Library\\bin', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar\\Scripts', 'C', '\\Users\\Jack\\anaconda3\\envs\\styleavatar\\bin', 'C', '\\Users\\Jack\\anaconda3\\condabin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.3\\bin', 'C', '\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.3\\libnvvp', 'C', '\\Windows\\system32', 'C', '\\Windows', 'C', '\\Windows\\System32\\Wbem', 'C', '\\Windows\\System32\\WindowsPowerShell\\v1.0', 'C', '\\Windows\\System32\\OpenSSH', 'C', '\\Program Files\\Microsoft SQL Server\\150\\Tools\\Binn', 'C', '\\Program Files\\Git\\cmd', 'C', '\\Program Files\\dotnet', 'C', '\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit', 'C', '\\Program Files\\NVIDIA Corporation\\Nsight Compute 2021.1.0', 'C', '\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common', 'C', '\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR', 'C', '\\Users\\Jack\\AppData\\Local\\Programs\\Python\\Python310\\Scripts', 'C', '\\Users\\Jack\\AppData\\Local\\Programs\\Python\\Python310', 'C', '\\Users\\Jack\\AppData\\Local\\Microsoft\\WindowsApps']), This path may cause jittor found the wrong libs, please unset LD_LIBRARY_PATH and remove cuda lib path in Path.
Or you can let jittor install cuda for you: `python3.x -m jittor_utils.install_cuda`
[i 0830 00:07:11.178000 00 cuda_flags.cc:49] CUDA enabled.
[e 0830 00:07:13.399000 00 log.cc:565] cl : Command line warning D9025 : overriding '/EHa' with '/EHs'
__opkey0_index__T_int32__DIM_2__XDIM_2__opkey1_binary__Tx_int32__Ty_int32__Tz_bool__OP_equ___hash_c68448bc674c5f27_op.cc
c:/users/jack/anaconda3/envs/styleavatar/lib/site-packages/jittor/src\misc/cuda_atomic.h(137): error: more than one conversion function from "__half" to a built-in type applies:
function "__half::operator float() const"
function "__half::operator short() const"
function "__half::operator unsigned short() const"
function "__half::operator int() const"
function "__half::operator unsigned int() const"
function "__half::operator long long() const"
function "__half::operator unsigned long long() const"
function "__half::operator __nv_bool() const"
c:/users/jack/anaconda3/envs/styleavatar/lib/site-packages/jittor/src\misc/cuda_atomic.h(137): error: more than one conversion function from "__half" to a built-in type applies:
function "__half::operator float() const"
function "__half::operator short() const"
function "__half::operator unsigned short() const"
function "__half::operator int() const"
function "__half::operator unsigned int() const"
function "__half::operator long long() const"
function "__half::operator unsigned long long() const"
function "__half::operator __nv_bool() const"
c:/users/jack/anaconda3/envs/styleavatar/lib/site-packages/jittor/src\misc/cuda_atomic.h(138): error: no instance of overloaded function "atomicCAS" matches the argument list
argument types are: (unsigned short *, unsigned short, unsigned short)
c:/users/jack/anaconda3/envs/styleavatar/lib/site-packages/jittor/src\misc/cuda_atomic.h(152): error: more than one conversion function from "__half" to a built-in type applies:
function "__half::operator float() const"
function "__half::operator short() const"
function "__half::operator unsigned short() const"
function "__half::operator int() const"
function "__half::operator unsigned int() const"
function "__half::operator long long() const"
function "__half::operator unsigned long long() const"
function "__half::operator __nv_bool() const"
c:/users/jack/anaconda3/envs/styleavatar/lib/site-packages/jittor/src\misc/cuda_atomic.h(152): error: more than one conversion function from "__half" to a built-in type applies:
function "__half::operator float() const"
function "__half::operator short() const"
function "__half::operator unsigned short() const"
function "__half::operator int() const"
function "__half::operator unsigned int() const"
function "__half::operator long long() const"
function "__half::operator unsigned long long() const"
function "__half::operator __nv_bool() const"
c:/users/jack/anaconda3/envs/styleavatar/lib/site-packages/jittor/src\misc/cuda_atomic.h(153): error: no instance of overloaded function "atomicCAS" matches the argument list
argument types are: (unsigned short *, unsigned short, unsigned short)
6 errors detected in the compilation of "C:/Users/Jack/.cache/jittor/jt1.3.8/cl/py3.8.16/Windows-10-10.x93/IntelRCoreTMi7x72/default/cu11.2.67/jit/__opkey0_index__T_int32__DIM_2__XDIM_2__opkey1_binary__Tx_int32__Ty_int32__Tz_bool__OP_equ___hash_c68448bc674c5f27_op.cc".
__opkey0_index__T_int32__DIM_2__XDIM_2__opkey1_binary__Tx_int32__Ty_int32__Tz_bool__OP_equ___hash_c68448bc674c5f27_op.cc
Traceback (most recent call last):
File "tracking_offline_cuda.py", line 222, in <module>
tracking = Tracking(args)
File "tracking_offline_cuda.py", line 27, in __init__
self.fvm, self.fvd = get_faceverse(batch_size=self.args.batch_size, focal=int(1315 / 512 * self.args.tar_size), img_size=self.args.tar_size)
File "C:\Users\Jack\FaceVerse\faceversev3_jittor\faceverse_cuda\__init__.py", line 9, in get_faceverse
faceverse_model = FaceVerseModel(faceverse_dict, **kargs)
File "C:\Users\Jack\FaceVerse\faceversev3_jittor\faceverse_cuda\FaceVerseModel.py", line 56, in __init__
self.mask_num = int(jt.sum(self.face_mask).numpy()[0])
RuntimeError: Wrong inputs arguments, Please refer to examples(help(jt.numpy)).
Types of your inputs are:
self = Var,
args = (),
The function declarations are:
ArrayArgs fetch_sync()
Failed reason:[f 0830 00:07:13.400000 00 parallel_compiler.cc:331] Error happend during compilation:
[Error] source file location:C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default\cu11.2.67\jit\__opkey0_index__T_int32__DIM_2__XDIM_2__opkey1_binary__Tx_int32__Ty_int32__Tz_bool__OP_equ___hash_c68448bc674c5f27_op.cc
Compile fused operator(0/8)failed:[Op(11:0:2:2:i0:o2:s0,index->...),Op(14:0:1:1:i2:o1:s0,binary.equal->15),Op(16:0:1:1:i1:o1:s0,unary.cast->17),]
Reason: [f 0830 00:07:13.400000 00 log.cc:608] Check failed ret(1) == 0(0) Run cmd failed: "C:\Users\Jack\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin\nvcc.exe" "C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default\cu11.2.67\jit\__opkey0_index__T_int32__DIM_2__XDIM_2__opkey1_binary__Tx_int32__Ty_int32__Tz_bool__OP_equ___hash_c68448bc674c5f27_op.cc" -shared -L"c:\users\jack\anaconda3\envs\styleavatar\libs" -lpython38 -Xcompiler -EHa -Xcompiler -MD -Xcompiler -utf-8 -I"C:\Users\Jack\.cache\jittor\msvc\VC\include" -I"C:\Users\Jack\.cache\jittor\msvc\win10_kits\include\ucrt" -I"C:\Users\Jack\.cache\jittor\msvc\win10_kits\include\shared" -I"C:\Users\Jack\.cache\jittor\msvc\win10_kits\include\um" -DNOMINMAX -L"C:\Users\Jack\.cache\jittor\msvc\VC\lib" -L"C:\Users\Jack\.cache\jittor\msvc\win10_kits\lib\um\x64" -L"C:\Users\Jack\.cache\jittor\msvc\win10_kits\lib\ucrt\x64" -I"c:\users\jack\anaconda3\envs\styleavatar\lib\site-packages\jittor\src" -I"c:\users\jack\anaconda3\envs\styleavatar\include" -DHAS_CUDA -DIS_CUDA -I"C:\Users\Jack\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\include" -I"c:\users\jack\anaconda3\envs\styleavatar\lib\site-packages\jittor\extern\cuda\inc" -lcudart -L"C:\Users\Jack\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\lib\x64" -L"C:\Users\Jack\.cache\jittor\jtcuda\cuda11.2_cudnn8_win\bin" -I"C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default\cu11.2.67" -L"C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default\cu11.2.67" -L"C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default" -l"jit_utils_core.cp38-win_amd64" -l"jittor_core.cp38-win_amd64" -x cu --cudart=shared -ccbin="C:\Users\Jack\.cache\jittor\msvc\VC\_\_\_\_\_\bin\cl.exe" --use_fast_math -w -I"c:\users\jack\anaconda3\envs\styleavatar\lib\site-packages\jittor\extern/cuda/inc" -arch=compute_52 -code=sm_52 -o "C:\Users\Jack\.cache\jittor\jt1.3.8\cl\py3.8.16\Windows-10-10.x93\IntelRCoreTMi7x72\default\cu11.2.67\jit\__opkey0_index__T_int32__DIM_2__XDIM_2__opkey1_binary__Tx_int32__Ty_int32__Tz_bool__OP_equ___hash_c68448bc674c5f27_op.dll" -Xlinker -EXPORT:"?jit_run@FusedOp@jittor@@QEAAXXZ"
(styleavatar) C:\Users\Jack\FaceVerse\faceversev3_jittor>
Nice work to learn an expressive face model from hybrid dataset. I wonder how the first 20 shape principal components is learned from the 3d scan dataset.
Does there any instructions on how to convert?
Thanks for your great work!
Different people have different eyes, the index of vertices belong to eyes is fixed or not. If this is fixed, your project can be used to reconstruct facial organs.
are there the raw rgb-d images in FaceVerse-Dataset?
your model use rgb image for input,but I can not find the raw rgb image in the data_sample folder
https://github.com/LizhenWangT/FaceVerse-Dataset
I see the value of the focal is fixed. Does that mean the intrinsics and extrinsics are not real? Does the project support exports intrinsics and extrinsics like https://github.com/sicxu/Deep3DFaceRecon_pytorch?
Dear author,
It is a perfect work. However, When i use FaceVerse with the code python tracking_online_cuda.py
,there are some errors. The window collected to camera pop up,but it is black and no reaction. Only my click turns to cross. The errors trouble me few days. How can i solve it?
the same error all in online,offline and StyleAvatar
`[e 1101 16:30:33.663000 20 log.cc:565] cl : Command line warning D9025 : overriding '/EHa' with '/EHs'
cub_where__Ti_bool__To_int32__NDIM_2__JIT_1__JIT_cuda_1__index_t_int32_hash_21a0093dfe861907_op.cc
c:\users\miniconda3\envs\styleavatar\include\cuda\std\detail\libcxx\include__type_traits/is_constant_evaluated.h(27): error: identifier "__builtin_is_constant_evaluated" is undefined
c:\users\miniconda3\envs\styleavatar\include\cuda\std\detail\libcxx\include__type_traits/is_constant_evaluated.h(32): error: identifier "__builtin_is_constant_evaluated" is undefined
2 errors detected in the compilation of "C:/Users/.cache/jittor/jt1.3.8/cl/py3.8.16/Windows-10-10.x78/13thGenIntelRCx8b/default/cu11.2.67/jit/cub_where__Ti_bool__To_int32__NDIM_2__JIT_1__JIT_cuda_1__index_t_int32_hash_21a0093dfe861907_op.cc".
cub_where__Ti_bool__To_int32__NDIM_2__JIT_1__JIT_cuda_1__index_t_int32_hash_21a0093dfe861907_op.cc`
Thanks you!
Hello author, would you like to ask if the 21 expressions in the detailed model have names?
Hi, I cann't find the arkit avatar in repository. Would you mind release the avatar, so that we can test the result of driving avatar?
Best,
The interocular distance of pca template is 0.55. How to covert it to unit(e.g. mm)?
Fitstly, many thanks for your excellent work!
I wonder how you fit the expression components to the 52 blendshapes defined by Apple, can you share the way to do that?
Thanks!
FaceVerse is a great work. And thanks for your open source. But I can't download the model files becaause google drive limited quota. Could you upload the model files onto Baidu netdisk. It's more convenient for Chinese users. That's the error :
抱歉,您目前无法查看或下载此文件。
最近查看或下载此文件的用户过多。请稍后再尝试访问此文件。如果您尝试访问的文件特别大或已与很多人共享,那么您最长可能需要等待 24 小时才能查看或下载该文件。如果您在 24 小时后仍然无法访问文件,请与您的网域管理员联系。
Do you have a schedule to release the 30fps version ? thank you for your work!
Hi again,
I am currently trying to use the offline tracking to render the face mesh in a static position and rotation while processing a video. So ideally I want the face to remain in the same location, and simply change expressions, rather than following the face movement and rotation. Do you have any advice on how I could implement this or where in the current code I could start looking and adjusting? I've been going through the different scripts, but some advice would be greatly appreciated. Thanks!
Nice work.
Can you share the shape of the base PCA model training code?
Is it possible to export mesh as obj sequence or alembic?
Would it help to improve in terms of quality or performance?
正如README.md中Fig.4文字上面显示的图片一样,获得:
['browDownLeft','browDownRight', 'browInnerUp', 'browOuterUpLeft', 'browOuterUpRight', 'cheekPuff', 'cheekSquintLeft', 'cheekSquintRight', 'eyeBlinkLeft', 'eyeBlinkRight', 'eyeLookDownLeft', 'eyeLookDownRight', 'eyeLookInLeft', 'eyeLookInRight', 'eyeLookOutLeft', 'eyeLookOutRight', 'eyeLookUpLeft', 'eyeLookUpRight', 'eyeSquintLeft', 'eyeSquintRight', 'eyeWideLeft', 'eyeWideRight', 'jawForward', 'jawLeft', 'jawOpen', 'jawRight', 'mouthClose', 'mouthDimpleLeft', 'mouthDimpleRight', 'mouthFrownLeft', 'mouthFrownRight', 'mouthFunnel', 'MouthLeft', 'mouthLowerDownLeft', 'mouthLowerDownRight', 'mouthPressLeft', 'mouthPressRight', 'mouthPucker', 'MouthRight', 'mouthRollLower', 'mouthRollUpper', 'mouthShrugLower', 'mouthShrugUpper', 'mouthSmileLeft', 'mouthSmileRight', 'mouthStretchLeft', 'mouthStretchRight', 'mouthUpperUpLeft', 'mouthUpperUpRight', 'noseSneerLeft', 'noseSneerRight',
'tongueOut']
对应的浮点数数组
Hi!
Really cool work! I was wondering if you're offline tracking does anything special in terms of stabilisation? I've noticed that using the image input demo across frames of a video leads to the mesh being more shakey compared to the specific tracking script and was wondering how you handled it.
Thanks!
Hi author,
I am looking to use only the face part and super impose it on the original image like the attached image and use it for styleavatar training. Can you help me how to get the face part instead of the full head as the 3DMM from Faceverse V3?
Also, if you could redirect me to any existing code how to superimpose the 3DMM on the original image, it would be greatly helpful for my research
Thank you so much!
Thanks for your great work, I'm wondering if I can export the extrinsics and intrinsics from this project?
hi,
Does the dataset contain 60K fused RGB-D images in paper? If not, could you tell me how to get it?
Great work~, and i am wondering if the training code will be released in the near future?
I have met some problems with the environment.
btw, can cuda11.2 works in faceverse?
nice job!I have a few questions, hope to get your reply
q1.Will the training part and the CUDA accelerated rendering version be announced soon?
q2.For example, how to calculate 52blendshapes for your V2 version of the design process?
q3.Do you have an idea for an end-to-end training and inference solution? Can you provide a case or open source project to learn from?
Hi, how to get the inference result of apple52 blendshapes? I saw the code only have a optimization based function, this could be every time-consuming.
I though it could a be single model outputs all blendshapes and rotation in a single forward.
Hello,
I'm trying to run tracking_offline_cuda.py
on macos and I get the following error:
File "/Users/xxxx/workspace/FaceVerse/faceversev3_jittor/fit_imgs_offline_cuda.py", line 113, in run
rendered_img_c = np.clip(self.pred_dict['rendered_img'].transpose((0, 2, 3, 1)).numpy(), 0, 255).astype(np.uint8)
RuntimeError: Wrong inputs arguments, Please refer to examples(help(jt.numpy)).
Types of your inputs are:
self = Var,
args = (),
The function declarations are:
ArrayArgs fetch_sync()
Failed reason:[f 0222 09:14:44.203245 00 op.cc:182] Check failed: flags.get(NodeFlags::_cpu) Something wrong... Could you please report this issue?
Op code doesn't have cpu version
Seems like the repo does not support CPU right? Is there a way to run it directly on macos (M1 or M3) ?
Thanks for your help!
I am using gtx 1660ti with 6gb vram.
HI, Whether it complies with the standard arkit specification. When I retarget to other standard arkit models, the effect is very bad. Is there a mistake for the expression base order of faceverse_simple_v2.npy? Thanks.
Hi, thank you for you solid work! Could you provide the eigen value about version 2. Because the 150 dimension ID is larger than other open source 3D models, I want to cut some dimensions through the eigen value to facilitate experimental comparison.
Hello author, I am interested in your work~. I just send you a e-mail with [[email protected]], could you send me the training code? If it can be shared will be great thanks~
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.