saic-violet / bilayer-model Goto Github PK

View Code? Open in Web Editor NEW

242.0 242.0 49.0 569 KB

License: Mozilla Public License 2.0

Python 96.56% Shell 3.44%

bilayer-model's People

Contributors

Stargazers

Watchers

Forkers

kakao014 kreativai peternara peterzs ak9250 chenchy aideveloper-oz liuguoyou janfschr ideaplexus avani17101 ml-and-ai-repo songguangchao noyami2033 blakecheng zhigaloff ludviqludviq michaelgrabinski dreyk taros1998 medical-projects ashourina ak391 khalednasser jiaxiangshang githubcrj pkarimib houlin hannhu jamesouy stronghold-nine darshan-20310597 felizang kangyeolk suzhenwang86 stevenhailin romakail sankarmurugan01 sal-dti techthiyanes bharat-mtr pk94 xinchen0521 slaustld howlin0523

bilayer-model's Issues

Did anyone manage to run the code?

ImportError Traceback (most recent call last)
/bilayer-model/examples/inference.ipynb in
8 from matplotlib import pyplot as plt
9
---> 10 from infer import InferenceWrapper

/bilayer-model/infer.py in
1 import argparse
----> 2 import torch
3 from torch import nn
4 from torchvision import transforms
5 from PIL import Image

~/anaconda3/lib/python3.7/site-packages/torch/init.py in
100 pass
101
--> 102 from torch._C import *
103
104 all += [name for name in dir(_C)

ImportError: /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize

Hey @egorzakharov @saic-violet
Much thanks for sharing such awesome work with the open-source community.
I have successfully used it for inference however the video produced has lot of jitter/shaking involved. For example -

out.mp4

Let me know what's the issue here.

loss in identity

First of all, thanks for great job!

I see some loss in identity, testing the code.
If the repository does not contain the latest weights, can I receive them somehow, please?

PackagesNotFoundError

I am unable to setup the environment

Linux 64 bits
Python 3.7.0

PackagesNotFoundError: The following packages are not available from current channels:

kiwisolver==1.1.0=pypi_0
pretrainedmodels==0.7.4=pypi_0
munch==2.5.0=pypi_0
tqdm==4.40.2=pypi_0
cycler==0.10.0=pypi_0
openssl==1.1.1=h7b6447c_0
matplotlib==3.1.2=pypi_0
pyparsing==2.4.5=pypi_0

Current channels:

If I remove "pypi_0", then there will be conflict error

Maybe an unnecessary operation

How to generate segment to compose the train dataset

Dear author,the bilayer-model is very useful,I'm using it to do some research.I want to train this model and I notice that the train dataset needs segment dataset.I dont' know how to generate it .Can you tell me how you get the segment dataset?

Finetuning the given pre-trained weights

I am a bit confused as to how to do the few-shot finetuning. Are there some specific files which need to be run for that?

Need VoxCeleb2-HQ contents.

@saic-violet @egorzakharov Hello! Thank you so much for sharing. I'm really impressed with the great work!

I noticed that you mentioned in the paper you make a "VoxCeleb2-HQ" dataset from original VoxCeleb2, can you provide the contents VoxCeleb2-HQ has, such as the videos id or youtube link in original VoxCeleb2. I want to reproduce the higher quality VoxCeleb2-HQ dataset.

Chapter 4 Experiments
We also use a high-quality version of the same dataset, additionally annotated with the segmentation masks (which were obtained using a model [15]), to measure how the performance of our model scales with a dataset of a significantly higher quality. We obtained this version by downloading the original videos via the links provided in the VoxCeleb2 dataset, and filtering out the ones with low resolution. This dataset is, therefore, significantly smaller and contains only 14859 videos of 4242 people, with each video having at most 250 frames (first 10 seconds). Lastly, we do ablation studies on both VoxCeleb2 and VoxCeleb2-HQ.

Wish for your reply. Thanks again.

怎么重现论文结果

你好，很高兴你能开源你的代码，目前我还没全看完，有个问题想问一下咨询咨询，我想重现论文中保留背景的图片，怎么保留生成图片中原图的背景？ @saic-violet

已创建中文的讨论组想加入的请添加微信xaaheng

colab

can you please add a google colab for inference?

unexpected key when testing examples/inference.ipynb

unexpected key "source_graph_2_fea.node_fea_for_res" in state_dict
unexpected key "source_graph_2_fea.node_fea_for_hidden" in state_dict
unexpected key "source_graph_2_fea.weight" in state_dict
unexpected key "source_skip_conv.0.weight" in state_dict
unexpected key "source_skip_conv.0.bias" in state_dict
unexpected key "source_semantic.weight" in state_dict
unexpected key "source_semantic.bias" in state_dict
unexpected key "middle_semantic.weight" in state_dict
unexpected key "middle_semantic.bias" in state_dict
unexpected key "middle_source_featuremap_2_graph.pre_fea" in state_dict
unexpected key "middle_source_featuremap_2_graph.weight" in state_dict
unexpected key "middle_source_graph_conv1.weight" in state_dict
unexpected key "middle_source_graph_conv2.weight" in state_dict
unexpected key "middle_source_graph_conv3.weight" in state_dict
unexpected key "middle_source_graph_2_fea.node_fea_for_res" in state_dict
unexpected key "middle_source_graph_2_fea.node_fea_for_hidden" in state_dict
unexpected key "middle_source_graph_2_fea.weight" in state_dict
unexpected key "middle_skip_conv.0.weight" in state_dict
unexpected key "middle_skip_conv.0.bias" in state_dict
unexpected key "transpose_graph_source2target.weight" in state_dict
unexpected key "transpose_graph_source2target.adj" in state_dict
unexpected key "transpose_graph_target2source.weight" in state_dict
unexpected key "transpose_graph_target2source.adj" in state_dict
unexpected key "transpose_graph_middle2source.weight" in state_dict
unexpected key "transpose_graph_middle2source.adj" in state_dict
unexpected key "transpose_graph_middle2target.weight" in state_dict
unexpected key "transpose_graph_middle2target.adj" in state_dict
unexpected key "transpose_graph_source2middle.weight" in state_dict
unexpected key "transpose_graph_source2middle.adj" in state_dict
unexpected key "transpose_graph_target2middle.weight" in state_dict
unexpected key "transpose_graph_target2middle.adj" in state_dict
unexpected key "fc_graph_source.weight" in state_dict
unexpected key "fc_graph_target.weight" in state_dict
unexpected key "fc_graph_middle.weight" in state_dict
missing keys in state_dict: "{'xception_features.block14.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block10.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block10.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block11.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block11.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block9.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.3.pointwise_bn.num_batches_tracked', 'transpose_graph.adj', 'xception_features.block12.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block5.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block12.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.bn1.num_batches_tracked', 'concat_projection_bn1.num_batches_tracked', 'xception_features.conv4.depthwise_bn.num_batches_tracked', 'xception_features.block19.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.conv4.pointwise_bn.num_batches_tracked', 'xception_features.block7.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block13.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block19.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block17.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block2.rep.3.pointwise_bn.num_batches_tracked', 'fc_graph.weight', 'xception_features.block3.skipbn.num_batches_tracked', 'xception_features.block14.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block15.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block17.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block8.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block10.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block6.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block9.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block12.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block6.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block4.rep.5.pointwise_bn.num_batches_tracked', 'aspp3.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block17.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block15.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block5.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.conv5.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block20.skipbn.num_batches_tracked', 'xception_features.block7.rep.5.pointwise_bn.num_batches_tracked', 'aspp2.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block12.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block13.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block4.rep.1.depthwise_bn.num_batches_tracked', 'aspp3.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block2.skipbn.num_batches_tracked', 'xception_features.block9.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block17.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.bn2.num_batches_tracked', 'xception_features.block11.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block19.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block9.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block1.rep.0.pointwise_bn.num_batches_tracked', 'decoder.0.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block6.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block19.rep.5.pointwise_bn.num_batches_tracked', 'decoder.1.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block15.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block16.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block2.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block2.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block9.rep.5.depthwise_bn.num_batches_tracked', 'global_avg_pool.2.num_batches_tracked', 'xception_features.block1.rep.0.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block20.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.conv3.pointwise_bn.num_batches_tracked', 'xception_features.block14.rep.3.pointwise_bn.num_batches_tracked', 'transpose_graph.weight', 'xception_features.block18.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block6.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block19.rep.5.depthwise_bn.num_batches_tracked', 'aspp1.bn.num_batches_tracked', 'xception_features.block15.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block12.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block17.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block4.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block6.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block3.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block6.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block1.rep.2.pointwise_bn.num_batches_tracked', 'xception_features.block7.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block1.rep.2.depthwise_bn.num_batches_tracked', 'xception_features.block11.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block11.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block1.skipbn.num_batches_tracked', 'xception_features.block18.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block20.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block1.rep.4.pointwise_bn.num_batches_tracked', 'xception_features.block14.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block2.block2_lastconv.1.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.3.depthwise_bn.num_batches_tracked', 'decoder.1.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block4.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block11.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block16.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block9.rep.5.pointwise_bn.num_batches_tracked', 'feature_projection_bn1.num_batches_tracked', 'xception_features.block15.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block18.rep.3.pointwise_bn.num_batches_tracked', 'aspp2.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.conv5.depthwise_bn.num_batches_tracked', 'xception_features.conv3.depthwise_bn.num_batches_tracked', 'decoder.0.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block19.rep.1.depthwise_bn.num_batches_tracked', 'aspp4.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block3.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block17.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block1.rep.4.depthwise_bn.num_batches_tracked', 'xception_features.block10.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block10.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block4.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block8.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block14.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block14.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block20.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block2.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block15.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block12.rep.1.depthwise_bn.num_batches_tracked', 'aspp4.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block2.block2_lastconv.1.pointwise_bn.num_batches_tracked', 'xception_features.block4.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block10.rep.3.depthwise_bn.num_batches_tracked'}"

Can't run the examples/inference.ipynb

When i run this .ipynb in the colab, it shows "ModuleNotFoundError: No module named 'infer'. When i run this .ipynb with local jupyter, it shows ——
unexpected key "source_graph_2_fea.node_fea_for_res" in state_dict
unexpected key "source_graph_2_fea.node_fea_for_hidden" in state_dict
unexpected key "source_graph_2_fea.weight" in state_dict
unexpected key "source_skip_conv.0.weight" in state_dict
unexpected key "source_skip_conv.0.bias" in state_dict
unexpected key "source_semantic.weight" in state_dict
unexpected key "source_semantic.bias" in state_dict
unexpected key "middle_semantic.weight" in state_dict
unexpected key "middle_semantic.bias" in state_dict
unexpected key "middle_source_featuremap_2_graph.pre_fea" in state_dict
unexpected key "middle_source_featuremap_2_graph.weight" in state_dict
unexpected key "middle_source_graph_conv1.weight" in state_dict
unexpected key "middle_source_graph_conv2.weight" in state_dict
unexpected key "middle_source_graph_conv3.weight" in state_dict
unexpected key "middle_source_graph_2_fea.node_fea_for_res" in state_dict
unexpected key "middle_source_graph_2_fea.node_fea_for_hidden" in state_dict
unexpected key "middle_source_graph_2_fea.weight" in state_dict
unexpected key "middle_skip_conv.0.weight" in state_dict
unexpected key "middle_skip_conv.0.bias" in state_dict
unexpected key "transpose_graph_source2target.weight" in state_dict
unexpected key "transpose_graph_source2target.adj" in state_dict
unexpected key "transpose_graph_target2source.weight" in state_dict
unexpected key "transpose_graph_target2source.adj" in state_dict
unexpected key "transpose_graph_middle2source.weight" in state_dict
unexpected key "transpose_graph_middle2source.adj" in state_dict
unexpected key "transpose_graph_middle2target.weight" in state_dict
unexpected key "transpose_graph_middle2target.adj" in state_dict
unexpected key "transpose_graph_source2middle.weight" in state_dict
unexpected key "transpose_graph_source2middle.adj" in state_dict
unexpected key "transpose_graph_target2middle.weight" in state_dict
unexpected key "transpose_graph_target2middle.adj" in state_dict
unexpected key "fc_graph_source.weight" in state_dict
unexpected key "fc_graph_target.weight" in state_dict
unexpected key "fc_graph_middle.weight" in state_dict
missing keys in state_dict: "{'xception_features.block12.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block10.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block20.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block17.rep.5.pointwise_bn.num_batches_tracked', 'decoder.0.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block2.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block11.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block19.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block3.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.conv4.depthwise_bn.num_batches_tracked', 'xception_features.block14.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block4.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block11.rep.5.pointwise_bn.num_batches_tracked', 'aspp4.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block11.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block9.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block19.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block12.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.conv5.pointwise_bn.num_batches_tracked', 'xception_features.block14.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block4.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block19.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block15.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block16.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block15.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block10.rep.3.depthwise_bn.num_batches_tracked', 'aspp1.bn.num_batches_tracked', 'aspp3.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block15.rep.5.depthwise_bn.num_batches_tracked', 'aspp3.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block8.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.5.depthwise_bn.num_batches_tracked', 'decoder.1.atrous_convolution.pointwise_bn.num_batches_tracked', 'decoder.0.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block11.rep.1.depthwise_bn.num_batches_tracked', 'aspp2.atrous_convolution.pointwise_bn.num_batches_tracked', 'xception_features.block12.rep.1.depthwise_bn.num_batches_tracked', 'feature_projection_bn1.num_batches_tracked', 'xception_features.block4.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block15.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.conv3.depthwise_bn.num_batches_tracked', 'xception_features.block1.skipbn.num_batches_tracked', 'xception_features.block2.block2_lastconv.1.pointwise_bn.num_batches_tracked', 'xception_features.block2.rep.1.depthwise_bn.num_batches_tracked', 'global_avg_pool.2.num_batches_tracked', 'xception_features.block16.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block6.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block12.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block7.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block17.rep.3.pointwise_bn.num_batches_tracked', 'aspp2.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block12.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block10.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block10.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block19.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block15.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.conv5.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.5.depthwise_bn.num_batches_tracked', 'fc_graph.weight', 'xception_features.block2.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block3.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block2.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block20.skipbn.num_batches_tracked', 'xception_features.block5.rep.3.pointwise_bn.num_batches_tracked', 'transpose_graph.adj', 'xception_features.block5.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block1.rep.2.pointwise_bn.num_batches_tracked', 'xception_features.block17.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block13.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block15.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block14.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block6.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block19.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block9.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block13.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block12.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.conv3.pointwise_bn.num_batches_tracked', 'xception_features.block10.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.conv4.pointwise_bn.num_batches_tracked', 'xception_features.block11.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block2.skipbn.num_batches_tracked', 'xception_features.block10.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block14.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block4.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block7.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block11.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block8.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.bn2.num_batches_tracked', 'xception_features.block20.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block9.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block4.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block14.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block3.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block9.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.bn1.num_batches_tracked', 'xception_features.block3.skipbn.num_batches_tracked', 'xception_features.block17.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block18.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block1.rep.0.depthwise_bn.num_batches_tracked', 'xception_features.block16.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block1.rep.4.pointwise_bn.num_batches_tracked', 'xception_features.block7.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block16.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block17.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block20.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block18.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block2.block2_lastconv.1.depthwise_bn.num_batches_tracked', 'xception_features.block17.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block9.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block1.rep.4.depthwise_bn.num_batches_tracked', 'aspp4.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block13.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block14.rep.3.depthwise_bn.num_batches_tracked', 'xception_features.block19.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block3.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block9.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block6.rep.5.depthwise_bn.num_batches_tracked', 'xception_features.block1.rep.0.pointwise_bn.num_batches_tracked', 'decoder.1.atrous_convolution.depthwise_bn.num_batches_tracked', 'xception_features.block1.rep.2.depthwise_bn.num_batches_tracked', 'transpose_graph.weight', 'concat_projection_bn1.num_batches_tracked', 'xception_features.block6.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block13.rep.5.pointwise_bn.num_batches_tracked', 'xception_features.block5.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block5.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block6.rep.1.pointwise_bn.num_batches_tracked', 'xception_features.block6.rep.1.depthwise_bn.num_batches_tracked', 'xception_features.block8.rep.3.pointwise_bn.num_batches_tracked', 'xception_features.block4.rep.5.pointwise_bn.num_batches_tracked'}"

'DistributedDataParallel' object has no attribute 'callback_queued'

Traceback (most recent call last): File "train.py", line 422, in <module> nets = m.train(args) File "train.py", line 337, in train loss = model(data_dict) File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/parallel/distributed.py", line 560, in forward result = self.module(*inputs, **kwargs) File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "/mnt/lustre/zhengchengyao/workspace/bilayer-model-master/runners/default.py", line 195, in forward self.data_dict = self.nets[net_name](self.data_dict, networks_to_train, self.nets) File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__ result = self.forward(*input, **kwargs) File "/mnt/lustre/zhengchengyao/workspace/bilayer-model-master/networks/texture_enhancer.py", line 149, in forward loss_enh.backward() File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/tensor.py", line 166, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward allow_unreachable=True) # allow_unreachable flag File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/parallel/distributed.py", line 392, in allreduce_hook if not self.callback_queued: File "/mnt/lustre/zhengchengyao/anaconda3/envs/py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__ type(self).__name__, name)) AttributeError: 'DistributedDataParallel' object has no attribute 'callback_queued'

when train enhancer, I meet this.
I found this may be caused by gradient operations—— NVIDIA/apex#107
Have you ever met this problem ?

Missed keys when finetuning with the given pretained weight

Hello,

I have a problem when finetuning on the dataset of the voxceleb2 with the given pretained weight.

untimeError: Error(s) in loading state_dict for NetworkWrapper:
Missing key(s) in state_dict: "frequencies".
size mismatch for net.mlp.0.weight_orig: copying a param with shape torch.Size([256, 136]) from checkpoint, the shape in current model is torch.Size([256, 1088]).
size mismatch for net.mlp.0.weight_v: copying a param with shape torch.Size([136]) from checkpoint, the shape in current model is torch.Size([1088]).

keypoints and segmentation extraction

Dear author,

Thank you so much for this wonderful work! In order to re-run your experiments, we need to extract the keypoints and the segmentation; however, it is not clear how to do it.

In the infer.py file, there is a sketch on how to generally extract the keypoints and the segmentation using the preprocess_data function, but some of the operations on the keypoints and segmentations in infer.py have overlap with the operations in voxceleb2.py file which is called during the training, thus the preprocess before training to save the keypoints and segmentations is different from infer.py. Furthermore, it's not clear if you have used the crop_data in the training process.

I will be most grateful if you provide us with the necessary code to generate the keypoints and segmentations from videos faithful to your design.

Thanks

video jitter

I very appreciate your creative work and sharing this code. But I found video jitter during the video test. The following is my result, which consists of source_img, target_img, target_pose, pred_img.

I've replaced the more stable face detector, but I have the same problem.
In https://github.com/saic-violet/bilayer-model/blob/master/infer.py, I found image and pose are center-aligned as the following, which I think may cause the video jitter problem, Can we remove the center-aligned operation in the data process and finetune the model again?

Wait for your reply. Thank you very much. @egorzakharov

image shaking

how I can fix it?

Syntax error in voxceleb2.py

In lines 40-41 in the voxceleb2.py, you repeated the help argument.

parser.add('--stickmen_thickness', default=2, type=int, help='thickness of lines in the stickman', help='thickness of lines in the stickman')

installation issue

when I try
conda install --file requirements.txt

Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

mkl_random==1.1.0=py37hb3f55d8_0
lame==3.100=h14c3975_1001
xorg-libice==1.0.10=h516909a_0
xorg-libxext==1.3.4=h516909a_0
x264==1!152.20180806=h14c3975_0
nettle==3.4.1=h1bed415_1002
fontconfig==2.13.1=h86ecdb6_1001
libxkbcommon==0.9.1=hebb1f50_0
glib==2.58.3=py37h6f030ca_1002
harfbuzz==2.4.0=h9f30f68_3
icu==64.2=he1b5a44_1
blas==2.14=openblas
libclang==9.0.0=default_hde54327_4
libllvm9==9.0.0=hc9558a2_3
xorg-xextproto==7.3.0=h14c3975_1002
bzip2==1.0.8=h516909a_2
nss==3.47=he751ad9_0
xorg-kbproto==1.0.7=h14c3975_1002
py-opencv==4.1.2=py37h5ca1d4c_2
pthread-stubs==0.4=h14c3975_1001
kiwisolver==1.1.0=pypi_0
xorg-libsm==1.2.3=h84519dc_1000
gstreamer==1.14.5=h36ae1b5_0
hdf5==1.10.5=nompi_h3c11f04_1104
nspr==4.24=he1b5a44_0
libwebp==1.0.2=h576950b_1
libopencv==4.1.2=py37_2
tqdm==4.40.2=pypi_0
torchvision==0.2.2=py_3
pytorch==1.0.1=py3.7_cuda10.0.130_cudnn7.4.2_2
cycler==0.10.0=pypi_0
libiconv==1.15=h516909a_1005
graphite2==1.3.13=hf484d3e_1000
gst-plugins-base==1.14.5=h0935bb2_0
mkl_fft==1.0.15=py37h516909a_1
pyparsing==2.4.5=pypi_0
xorg-libxdmcp==1.1.3=h516909a_0
pretrainedmodels==0.7.4=pypi_0
openssl==1.1.1=h7b6447c_0
pixman==0.38.0=h516909a_1003
expat==2.2.5=he1b5a44_1004
jasper==1.900.1=h07fcdf6_1006
dbus==1.13.6=he372182_0
munch==2.5.0=pypi_0
xorg-libxrender==0.9.10=h516909a_1002
xorg-renderproto==0.11.1=h14c3975_1002
libopenblas==0.3.7=h5ec1e0e_5
protobuf==3.11.1=py37he1b5a44_0
xorg-libxau==1.0.9=h14c3975_0
libxml2==2.9.10=hee79883_0
xorg-xproto==7.0.31=h14c3975_1007
liblapacke==3.8.0=14_openblas
tensorboardx==1.9=py_0
giflib==5.1.9=h516909a_0
libcblas==3.8.0=14_openblas
libxcb==1.13=h14c3975_1002
libuuid==2.32.1=h14c3975_1000
gnutls==3.6.5=hd3a4fd2_1002
xorg-libx11==1.6.9=h516909a_0
liblapack==3.8.0=14_openblas
libblas==3.8.0=14_openblas
gmp==6.1.2=hf484d3e_1000
matplotlib==3.1.2=pypi_0
opencv==4.1.2=py37_2
jpeg==9c=h14c3975_1001
cairo==1.16.0=hfb77d84_1002
pcre==8.43=he1b5a44_0
numpy==1.17.3=py37h95a1406_0
libprotobuf==3.11.1=h8b12597_0
qt==5.12.5=hd8c4c69_1
gettext==0.19.8.1=hc5be6a0_1002
ffmpeg==4.1.3=h167e202_0
openh264==1.8.0=hdbcaa40_1000

Current channels:

Is there any other way to install these packages?

Too few Fps

The speed of work does not correspond to the declared one, or I am doing something wrong.
GPU - 2080TI
Help with this pls.

Train on voxceleb2 dataset

I really appreciate your work, now I want to train the model from scratch, especially I want to train on voxceleb2 dataset, I want to know how do you process the data, have you take all frames of each video? or do you just take a few frames of each video?

Mobile

how did you launch this on your mobile phone? conversion to onnx? or how ?

Error in voxceleb2.py

In voxceleb2.py you have:

if self.args.output_stickmen: stickmen = utils.draw_stickmen(self.args, poses)

It should be:
if self.args.output_stickmen: stickmen = ds_utils.draw_stickmen(self.args, poses)

Because you imported from datasets import utils as ds_utils.

PyTorch version?

According to this it should be pytorch=1.0.1
https://github.com/saic-violet/bilayer-model/blob/master/requirements.txt#L101

But running with pytorch=1.0.1 produce error:

    from infer import InferenceWrapper
  File "../infer.py", line 15, in <module>
    from runners import utils as rn_utils
  File "../runners/utils.py", line 6, in <module>
    from networks import utils as nt_utils
  File "../networks/utils.py", line 288, in <module>
    'none': nn.Identity,
AttributeError: module 'torch.nn' has no attribute 'Identity'

Seems it's should be at least 1.1.0
huggingface/transformers#869

But running with 1.1.0 produce error:

  File "run_example.py", line 31, in <module>
    module = InferenceWrapper(args_dict)
  File "../infer.py", line 69, in __init__
    map_location='cpu'))
...
KeyError: 'net.enc.0.weight'

Seems weights name changed:

m = torch.load('../runs/vc2-hq_adrianb_paper_main/checkpoints/2225_identity_embedder.pth', map_location='cpu')
[l for l in m.keys() if 'net.enc.0' in l]
Output:
['net.enc.0.bias', 'net.enc.0.weight_orig', 'net.enc.0.weight_u', 'net.enc.0.weight_v']

Seems there is weights name mismatch for all networks:

------------------------------------------------------------
../runs/vc2-hq_adrianb_paper_main/checkpoints/2225_identity_embedder.pth
identity_embedder
'net.enc.0.weight'
------------------------------------------------------------
../runs/vc2-hq_adrianb_paper_main/checkpoints/2225_texture_generator.pth
texture_generator
'gen_tex.blocks.0.block_feats.3.weight'
------------------------------------------------------------
../runs/vc2-hq_adrianb_paper_main/checkpoints/2225_keypoints_embedder.pth
keypoints_embedder
'net.mlp.0.weight'
------------------------------------------------------------
../runs/vc2-hq_adrianb_paper_main/checkpoints/2225_inference_generator.pth
inference_generator
'gen_inf.blocks.0.block_feats.2.weight'