eborboihuc/soundnet-tensorflow

TensorFlow implementation of "SoundNet".

SoundNet-tensorflow

TensorFlow implementation of "SoundNet" that learns rich natural sound representations.

Code for paper "SoundNet: Learning Sound Representations from Unlabeled Video" by Yusuf Aytar, Carl Vondrick, Antonio Torralba. NIPS 2016

Prerequisites

• Linux
• NVIDIA GPU + CUDA 8.0 + CuDNNv5.1
• Python 2.7 with numpy or Python 3.5
• Tensorflow 1.0.0 (up to 1.3.0)
• librosa

Getting Started

• Clone this repo:

git clone [email protected]:eborboihuc/SoundNet-tensorflow.git
cd SoundNet-tensorflow

• Pretrained Model

I provide pre-trained models that are ported from soundnet. You can download the 8 layer model here. Please place it as ./models/sound8.npy in your folder.

• Data

Prepare you input mp3 files and place them under ./data/

Generate a input file txt and place it under ./

./data/0001.mp3
./data/0002.mp3
./data/0003.mp3
...

Follow the steps in extract features

• NOTE

If you found out that some audio with offset value start in FFMPEG will cause a tremendous difference between torch audio and librosa, please convert it with following command.

sox {input.mp3} {output.mp3} trim 0

After this, the result might be much better.

Demo

For demo, you can follow the following steps

i) Download a converted npy file demo.npy and place it under ./data/

ii) To extract multiple features from a pretrained model with torch lua audio loaded sound track: The sound track is equivalent with torch version.

python extract_feat.py -m {start layer number} -x {end layer numbe} -s

Then you can compare the outputs with torch ones.

Feature Extraction

Minimum example

i) Download input file demo.mp3 and place it under ./data/

ii) Prepare a file list in txt format (demo.txt) that includes the input mp3 file(s) and place it under ./

./data/demo.mp3

iii) Then extract features from raw wave in demo.txt: Please put the demo mp3 under ./data/demo.mp3

python extract_feat.py -m {start layer number} -x {end layer numbe} -s -p extract -t demo.txt

More options

To extract multiple features from a pretrained model with downloaded mp3 dataset:

python extract_feat.py -t {dataset_txt_name} -m {start layer number} -x {end layer numbe} -s -p extract

e.g. extract layer 4 to layer 17 and save as ./sound_out/tf_fea%02d.npy:

python extract_feat.py -o sound_out -m 4 -x 17 -s -p extract

More details are in:

python extract_feat.py -h

Finetuning

To train from an existing model:

python main.py 

Training

To train from scratch:

python main.py -p train

To extract features:

python main.py -p extract -m {start layer number} -x {end layer numbe} -s

More details are in:

python main.py -h

TODOs

• Change audio loader to soundnet format
• Make it compatible to Python 3 format
• Batch Norm behaviour different from Torch
• Fix conv8 padding issue in training phase
• Change all config into tf.app.flags
• Change dummy distribution of scene and object to useful placeholder
• Add sound and feature loader from Data section

Known issues

• Loaded audio length is not consist in torch7 audio and librosa. Here is the issue
• Training with a short length audio will make conv8 complain about output size would be negative

FAQs

• Why my loaded sound wave is different from torch7 audio to librosa: Here is my WiKi

Acknowledgments

Code ported from soundnet. And Torch7-Tensorflow loader are from tf_videogan. Thanks for their excellent work!

Author

Hou-Ning Hu / @eborboihuc

soundnet-tensorflow's Issues

Extracting features in pool5

I have read in the paper that the best layer for feature extraction is 'pool5'. However, the feature sizes in that layer are h x w x 256.
Any idea how that 3D array has to be process for an SVM as it is said in the paper?

Using our pre-trained model, you can extract discriminative features for natural sound recognition. In our experiments, pool5 seems to work the best with a linear SVM

audio length

Thanks for your hard work on audio embedding. when I extract feature on my own dataset, I am wondering if the pretrained model can only accept the audio of which length has more than 8 or 9s, for example, if i use some audios which just have 3 to 5s, it can not be accepted and occur error like TypeError: 'float' object cannot be interpreted as an integer

question about kl loss

Thanks for your efforts!
The kl loss is implemented as:
tf.reduce_mean(-tf.nn.softmax_cross_entropy_with_logits(logits=dist_a, labels=dist_b))
I wonder whether there should be a negative indicator. The logits and labels definitions seem to be different from the first answer in http://stackoverflow.com/questions/41863814/kl-divergence-in-tensorflow. Did you use dist_a / dist_b as labels? I hope you could help me on this. Thanks!

Problem about division in python3

Thanks for the efforts!
However, there seems to be a minor problem with python3,
in line 59, utils
raw_audio = np.tile(raw_audio, length/raw_audio.shape[0] + 1
should be
raw_audio = np.tile(raw_audio, length//raw_audio.shape[0] + 1)
as python3 doesn't automatically convert float to int in division

The 8 layer model link gives 403 error.

It seems I don't have permission to open the link page. May you upload it again and give the access to download it? Please.

numpy tile parameter bug

Line 59 in util.py
raw_audio = np.tile(raw_audio, length/raw_audio.shape[0] + 1)
Should be
raw_audio = np.tile(raw_audio, length//raw_audio.shape[0] + 1)

Trivial python 3 incompatabilities

I think it would be a good idea to use python 3 syntax regarding print-statements, it's a minor thing that will probably allow this to be python 3 compatible.

pytorch pre-trained model

Thank you for your great work!
Do you have pre-trained a model of pytorch implement? Or is it possible to transform the .npy pre-trained model to .pth?

TypeError: unhashable type: 'numpy.ndarray'

When I run torchfile.load('./soundnet8_final.t7') on pycharm, I got this error, do you have any idea about it?
Thank you~