Comments (5)
Hi!
AudioDec (ICASSP 2023 deadline: 22/10/26) and Encodec (arXiv, published on 22/10/24) were developed by different teams in Meta at almost the same time, so we haven’t compared AudioDec with Encodec. However, since the AudioDec project focuses on human sounds while Encodec focuses on general audio, there are several main differences.
- AudioDec only adopts the simplest single-resolution mel loss while Encodec adopts multi-resolution mel losses and waveform-based losses because only simple mel loss already modeled the speech well.
- AudioDec has a 2 stages architecture (encoder + vocoder) which is suitable for developing new encoders/decoders for different applications such as denoising or binaural rending.
- The autoencoder architecture of AudioDec is almost the same as SoundStream while Encodec’s architecture is a combination of SoundSteam and Demucs.
- The currently provided AudioDec pre-trained models were trained using speech-only corpora (VCTK or LibriTTs) while the Encodec models were trained using many different kinds of audio. Since training/testing data mismatch usually causes significant performance degradations for the data-driven models, we recommend you use the AudioDec pre-trained models only for speech and train new AudioDec models for other types of audio signals.
from audiodec.
Thanks for the suggestion!
I will add this question to README.
from audiodec.
Interesting. Thank you!
Also, the reason I've asked is because projects like NaturalSpeech2 are using neural codec as important part of their TTS pipeline, so it seems that maybe AudioDec would be better suited for replication of such a model!
from audiodec.
Yes, since the training script is provided, and the training is not very computation-consuming, it might be easy to train a new AudioDec using a new dataset for a new downstream task.
from audiodec.
Maybe the information in this issue would be a great addition for README. I assume, many people would have similar questions (on similarities/differences with encodec/soundstream/lyrav2)
from audiodec.
Related Issues (20)
- What data was used by the pretrained model of LibriTTS? HOT 1
- Some questions about CausalConvTranspose1d in conv_layer.py HOT 3
- The test results are different from those in the paper HOT 7
- Permissive license + patents
- VCTK dataset HOT 11
- vq_loss increase, not converge HOT 7
- projector(e)? HOT 7
- LOSS? HOT 8
- quantizer dropout HOT 3
- LibriTTS dataset HOT 2
- Baseline? HOT 4
- Fix the encoder HOT 4
- Trying to train a low bitrate version HOT 2
- Table 4 and 5 HOT 2
- downsample_scales? HOT 3
- Processing the same signal twice yields different results HOT 3
- Increasing batch length HOT 2
- The trained models seem overfitted to their training sets? HOT 3
- Perplexity blows up at start of training HOT 9
- Is it missing some activation functions between some layers? HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from audiodec.