justin1904 / cmu-multimodalsdk-tutorials Goto Github PK
View Code? Open in Web Editor NEWThis is a short tutorial for using the CMU-MultimodalSDK.
This is a short tutorial for using the CMU-MultimodalSDK.
I'm trying to run through the experiments but i'm unable to get past the part where torch is imported.
I have Python: 3.7.4 installed. I'm currently running windows 10 on MacOs.
I have installed pytorch with the following command -> conda install pytorch torchvision cpuonly -c pytorch.
I get the following error running that segment.
Hi,
Thank you for this tutorials. Is there anywhere a description of the different database that have been created and relaesed in the SDK.
Fo example
in the part of the tutorial to define modalities. The files below are chosen:
visual_field = 'CMU_MOSI_VisualFacet_4.1'
acoustic_field = 'CMU_MOSI_COVAREP'
text_field = 'CMU_MOSI_ModifiedTimestampedWords'
But with the SDK we obtain the following list of CSD:
#Audio
CMU_MOSI_COVAREP.csd
CMU_MOSI_OpenSmile_EB10.csd
CMU_MOSI_openSMILE_IS09.csd
#Language
CMU_MOSI_TimestampedWords.csd
CMU_MOSI_TimestampedWordVectors.csd
CMU_MOSI_TimestampedWordVectors_1.1.csd
CMU_MOSI_TimestampedPhones.csd
#Video
CMU_MOSI_Visual_OpenFace_1.csd
CMU_MOSI_Visual_OpenFace_2.csd
CMU_MOSI_Visual_Facet_41.csd
CMU_MOSI_Visual_Facet_42.csd
is there any document where I could find what are the differences of those files/ How they were pre-processed?
Thank you in advance for your reply.
Regards,
Leticia Fernandez Moguel
Python exception :
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
Encountered in the definition of the load_emb function
We tried to overcome the issue by reading the file at WORD_EMB_PATH as a binary file but we just met another problem so we assume this is not the way to go.
Hi,
Thanks for this tutorials. When I downloaded dataset,I encountered this problem,which means the dataset wasn't be downloaded properly.
Here is the problem:
| Error | /data/mosei/CMU_MOSEI_VisualOpenFace2.csd resource is not a valid hdf5 computational sequence format ...
However,When I redownload the dataset without deleting the downloaded ones,I encountered:
“High-level features have been downloaded previously.”
Seems the downloader only checks the file name.
I wonder is there any way that I can download the proper dataset without deleting old ones?As u know,the dataset is enormous.
hi @Justin1904,
I am unable to install complete list of files, only following files are installed
CMU_MOSI_ModifiedTimestampedWords.csd
CMU_MOSI_Opinion_Labels.csd
CMU_MOSI_TimestampedPhones.csd
CMU_MOSI_TimestampedWords.csd
CMU_MOSI_TimestampedWordVectors_1.1.csd
Hi,
using the CMU-MOSI dataset:
the testing accuracy in the tutorial is around 71.6%.
I'm using the exactly same code from the tutorials, but I only get around 61.2% testing accuracy.
Any suggestion? or the author of CMU-MultimodalSDK has done something new to the dataset?
Thanks!
thank you for the tutorial, very helpful insights!
The Jupiter notebook works fine,
I saw a model.std as the final output;
Can you guide us in some experiments:
how we can test this model with other inputs?, I didn't found documentation about methods and this phase of "toy use of the model.."
how we correctly preprocess external datasets for test or dev environments?
thank you for your attention.
Pedro Moya
Thanks for your repo! I've a question about labels.
How to get Emotion labels from CMU_MOSEI_LabelsEmotions.csd?
I've written the next code:
dataset=mmdatasdk.mmdataset(mmdatasdk.cmu_mosei.labels,'./cmumosei')
In debug mode I see the next picture:
When we look into data, we get something like this:
I can not see emotion labels here (like sadness, neutral etc.)
What have I to do to get them for each frame in videos?
First of all, thank you very much for your tutorial! However, when I ran the following code, I encountered an error...So, can you tell me how to solve it? Thank you.
Code:
words.append(word2id[word[0].decode('utf-8')]) # SDK stores strings as bytes, decode into strings here
Error:
AttributeError: 'numpy.float64' object has no attribute 'decode'
Hello,
first of all thanks for the great work on the tutorial. Could you explain how I could use the trained model to let it run on my own data ?(for example a regular video of myself)
Hi @Justin1904
Thank You for the great tutorial for the CMU-MultimodalSDK. Its very well explained.
I was successful in training the LSTM defined in this tutorial code with the CMU provided .csd files.
I created my own csd for visual feature from openface. But when instead of facet visual features csd file i used this csd . Rest features are unchanged.
The function of alignment with labels is throwing error this below error :-
<Status>: Unify was called ....
<Warning>: 9qR7uwkblbs entry is not shared among all sequences, removing it ...
<Warning>: POKffnXeBds entry is not shared among all sequences, removing it ...
<Warning>: BioHAh1qJAQ entry is not shared among all sequences, removing it ...
.....**same warning followed for other 88 videos**
Traceback (most recent call last):
File "tutorial_interactive_demo.py", line 255, in <module>
_visual = dataset[visual_field].data[segment]['features']
KeyError: ['2WGyTLYerpo[51]'[]
Please guide me for the resolution for this ? (I have also attached a log file of the training)
I have used features for 91 videos in the CMU-MOSI raw dataset instead of 23 available .
I aim to train the model on my own videos.
log_train.txt
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.