sahilg06 / emogen Goto Github PK
View Code? Open in Web Editor NEWPyTorch Implementation for Paper "Emotionally Enhanced Talking Face Generation" (ICCVW'23 and ACM-MMW'23)
License: Creative Commons Zero v1.0 Universal
PyTorch Implementation for Paper "Emotionally Enhanced Talking Face Generation" (ICCVW'23 and ACM-MMW'23)
License: Creative Commons Zero v1.0 Universal
Thank you for the great work!
The parameter's folder of you model is not provide, should i train the model in dataset before inference?
It's an interesting work, especially generating talking face with emotion. I do want to test it using your model weights. Thanks.
Hi Team,
That's a really good research avenue you have there. Is it possible for your team to share any of the pretrained models that can be directly used by us to test and see the inference? It would be quite useful.
Or
Would we have to train the models ourselves?
Thanks. Look forward to your response.
The uploaded model checkpoint "s3fd.pth" has no key called "state_dict". I understood from previous issues this for face_detection model not the whole model so the weights for whole model are available or not yet? I have a .mp4 video and .wav audio as I want to try your model.
can you advise on a workaround for this? I have been waiting for color_syncnet_train.py to run for over a few hours numerous times, and nothing is loading or running yet. you mentioned internal changes about tqdm, but what is the potential issue specifically with it?
I could run emotion_disc_train.py training, but not the color_syncnet_train.py training
code entered in cmd line:
python color_syncnet_train.py --data_root ./preprocessed_dataset/ --checkpoint_dir ./checkpoint/
Thank you for sharing your work. Is this method real-time and what is the time taken to generate one frame?
Traceback (most recent call last):
File "/workspace/Wav2Expression/tafreen/EmoGen-master/inference.py", line 298, in
main()
File "/workspace/Wav2Expression/tafreen/EmoGen-master/inference.py", line 265, in main
model = load_model(args.checkpoint_path)
File "/workspace/Wav2Expression/tafreen/EmoGen-master/inference.py", line 180, in load_model
s = checkpoint["state_dict"]
KeyError: 'state_dict'
I have trained till 1500 epochs while testing it is throwing this even though I have given the checkpoints path of main model itself.
Could you please tell me how to resolve this.
Thanks and regards
When I train the Wav2Lip model, adding the sync expert loss on top of the perceptual loss causes the training to collapse. I noticed that you also use perceptual loss in your training process, and I'm wondering if you have encountered a similar situation. If so, how did you solve it?
I was running the code on Colab and I got this error
Using cuda for inference.
Reading video frames...
Number of frames available for inference: 211
/content/EmoGen/audio.py:100: FutureWarning: Pass sr=16000, n_fft=800 as keyword args. From version 0.10 passing these as positional arguments will result in an error
return librosa.filters.mel(hp.sample_rate, hp.n_fft, n_mels=hp.num_mels,
(80, 2372)
Length of mel chunks: 885
0% 0/4 [00:00<?, ?it/s]
0% 0/14 [00:00<?, ?it/s]
7% 1/14 [00:12<02:41, 12.42s/it]
14% 2/14 [00:13<01:09, 5.82s/it]
21% 3/14 [00:15<00:42, 3.83s/it]
29% 4/14 [00:16<00:27, 2.79s/it]
36% 5/14 [00:18<00:21, 2.44s/it]
43% 6/14 [00:19<00:16, 2.11s/it]
50% 7/14 [00:20<00:12, 1.81s/it]
57% 8/14 [00:21<00:09, 1.62s/it]
64% 9/14 [00:23<00:07, 1.49s/it]
71% 10/14 [00:24<00:05, 1.40s/it]
79% 11/14 [00:25<00:03, 1.33s/it]
86% 12/14 [00:26<00:02, 1.29s/it]
93% 13/14 [00:27<00:01, 1.26s/it]
100% 14/14 [00:30<00:00, 2.18s/it]
Load checkpoint from: /content/EmoGen/checkpoint/pre.pth
0% 0/4 [00:31<?, ?it/s]
Traceback (most recent call last):
File "/content/EmoGen/inference.py", line 298, in
main()
File "/content/EmoGen/inference.py", line 265, in main
model = load_model(args.checkpoint_path)
File "/content/EmoGen/inference.py", line 180, in load_model
s = checkpoint["state_dict"]
KeyError: 'state_dict'
I was running the inference.py and I got this error
Traceback (most recent call last):
File "/sharefiles/zhaodan/projects/EmoGen-master/inference.py", line 298, in
main()
File "/sharefiles/zhaodan/projects/EmoGen-master/inference.py", line 233, in main
mel = audio.melspectrogram(wav)
File "/sharefiles/zhaodan/projects/EmoGen-master/audio.py", line 47, in melspectrogram
S = _amp_to_db(_linear_to_mel(np.abs(D))) - hp.ref_level_db
File "/sharefiles/zhaodan/projects/EmoGen-master/audio.py", line 95, in _linear_to_mel
_mel_basis = _build_mel_basis()
File "/sharefiles/zhaodan/projects/EmoGen-master/audio.py", line 100, in _build_mel_basis
return librosa.filters.mel(hp.sample_rate, hp.n_fft, n_mels=hp.num_mels,
TypeError: mel() takes 0 positional arguments but 2 positional arguments (and 3 keyword-only arguments) were give
Thanks and regards
F:\DaChuang\EmoGen>python inference.py --checkpoint_path face_detection/detection/sfd/s3fd.pth --face data/0.mp4 --audio data/1.mp3 --emotion HAP --resize_factor 2
Using cuda for inference.
Reading video frames...
Number of frames available for inference: 1997
Extracting raw audio...
Traceback (most recent call last):
File "inference.py", line 315, in
main()
File "inference.py", line 282, in main
model = load_model(args.checkpoint_path)
File "inference.py", line 184, in load_model
s = checkpoint["state_dict"]
KeyError: 'state_dict'
I prepare data 0.mp4 and 1.mp3, I try to inference on laptop with one 1650Ti. But something went wrong.
Could you please tell me how to resolve this.Thank you very much.
When I run:
python color_syncnet_train.py --data_root preprocessed_dataset/ --checkpoint_dir folder_to_save_checkpoints/
It only displays and nothing happen
Epoch: 0
0it [00:00, ?it/s]
Could anyone help me?
The interactive demo is not running now, will you fix this problem?
I encountered an issue when running the training code. The code requires the LRS2 training dataset as input, but there are problems when loading the CREMA-D dataset.
HI,
your work is very interesting. I would like to test the model with my videos and audios, but the link to the demo is not working. Could you share the pretrained checkpoint thus i can run the inference command?
Thatnk you in advance!
Great work! Could you please share the pre-trained model checkpoint so that we can test how it works?
Hello, when I was training the lip expert discriminator, the epoch got stuck at 0, but I made progress while training the emotion discriminator. Why is this?
Is pretrained weights are released or not
Hey hatss off to the wonderful research and work. I wanted to ask can you share the pretrained model as the online link https://midas.iiitd.edu.in/emo/ provided is down and we cant test it. If you guys can release the pretrained model it would be great to be used as a benchmark model for my research.
Thanks and regards
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.