Comments (3)
Hi there,
Thanks for reaching out.
1/
In your ast_models.py, you put cluster True as Default ... But if to use frame-level ssast, cluster should be False. Do I have to turn it off?
You are correct that cluster=True
is default in the model script, but we do pass cluster=False
for frame-level when we instantiate the model, please see here:
Lines 125 to 130 in a1a3eec
FYI, you can use cluster=True
for frame-level AST, but from my experience, it will lead to a performance drop.
2/
If I want to use your pretrained frame level ssast for audio token extraction, is the output of self.v.norm(x) except the first one what I have to use in finetuningcls function? because the first one is cls token
You are correct, but please be cautious on that cls_token_num
might not always be 1 for all models.
3/
One more thing I wonder....Could I get some part of fbank that is corresponding to the video frames? melspectrogram does but I don't know fbank could be....
This involves audio-visual learning while this paper is about pure audio research. But we do use fbank features as input in this paper, see:
Lines 126 to 127 in a1a3eec
Hope these help.
-Yuan
from ssast.
Have you tried other input types like melspectrograms or mfcc? @YuanGongND I am gonna try feeding melspectrograms to SSAST to extract audio feature....Is it okay?
Could I ask you why cls token number is not consistent as 1? It can be 2 because of dist_token? Then, what is dist_token?
from ssast.
Have you tried other input types like melspectrograms or mfcc? @YuanGongND I am gonna try feeding melspectrograms to SSAST to extract audio feature....Is it okay?
I have never tried other input features. You can pretrain your own model with other input feature, but if you plan to use our pretrained model to extract feature/embedding/token, then you have to use the same dataloader (which is fully released in this repo) with us, any input distribution shift could cause a dramatic performance difference.
Could I ask you why cls token number is not consistent as 1? It can be 2 because of dist_token? Then, what is dist_token?
dist_token
stands for distillation token, please read our AST paper for details. SSAST does not need this token, but our code is compatible with old AST models.
-Yuan
from ssast.
Related Issues (20)
- Bug with input_dim and pure inference HOT 11
- Stereo audio HOT 1
- Model fails to converge on transfer to audio backtesting problem HOT 5
- speech commands v1 HOT 3
- Extract audio representations for future use HOT 13
- sed HOT 1
- python and cuda version HOT 3
- Host on Huggingface HOT 2
- Which python version are you using? HOT 1
- For discriminative loss, is the true NCE batch size the number of masked patches? HOT 2
- Librispeech960 Pretrained Model HOT 3
- AttributeError: 'ASTModel' object has no attribute 'unfold' HOT 8
- I've been stuck on the first epoch in a tiny model for 8+ hours using 300 5s spectrograms for training. Is this normal? HOT 3
- Are the model weights loaded into the ASTModel with stochasticity? HOT 2
- Data Loading Logs HOT 2
- performance and loss of the frame-based model HOT 2
- audio file length HOT 4
- SSAST for embedding model
- question: example usage for inference, rather than training HOT 1
- Use SSAST pretrained model to inference
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ssast.