swagshaw/importantaug

1. Introduction

We introduced ImportantAug, a technique to augment training data for speech classification and recognition models by adding noise to unimportant regions of the speech and not to important regions.
This technique could also be used in computer vision tasks, such as image recognition by predicting importance maps for image

Trinh, Viet Anh, Hassan Salami Kavaki, and Michael I. Mandel. "ImportantAug: a data augmentation agent for speech." In ICASSP 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.

We compare our proposed ImportantAug with other methods: baseline (no augmentation), conventional noise augmentation. We also do ablation study by removing importance map and call this experiment Null ImportantAug.

In the baseline, we train a recognizer that does not utilize any data augmentation. All other methods are trained by initializing their parameters to those of this pre-trained baseline recognizer.

Additionally, we create a noisy test set version for Google Speech Commands, named GSC-Musan, and it could be downloaded here https://zenodo.org/record/6066174#.Yn7NPJPMLyU

2. Initial set up and data preparation

The best way to train and reprocedure the result is to use the same environment. The custom dataset GSC-QUT is different if create using different environment. conda env create -f environment.yml

After create the environment, please activate it: conda activate py39torch19

To prepare data: cd importantAug importantAug$./data/prepare_data.sh

3. To train from scratch

a. Baseline:

To train:
cd importantAug
importantAug$./scripts/baseline.sh

To test:
importantAug$./scripts/baseline_test.sh

b. Conventional noise:
First, find the best SNR on the dev set:
importantAug$./scripts/hyper_search_conventional_noise.sh
To test, use the best SNR in the previous step as the target_SNR_dB in conventional_noise_test.sh and execute:
importantAug$./scripts/conventional_noise_test.sh

c. ImportantAug:
To train the generator:
importantAug$./scripts/importantAug_train_generator.sh
To do augmentation:
importantAug$./scripts/importantAug.sh
To test:
.importantAug$/scripts/importantAug_test.sh

d. Null ImportantAug:
To train:
importantAug$./scripts/null_importantAug.sh
To test:
importantAug$./scripts/null_importantAug_test.sh

e. Binarized ImportantAug To train:
importantAug$./scripts/binarized_importantAug.sh

4. To reproduce the result

importantAug$./scripts/pretrained_baseline_test.sh
importantAug$./scripts/pretrained_conventional_noise_dev.sh
importantAug$./scripts/pretrained_conventional_noise_test.sh
importantAug$./scripts/pretrained_importantAugment_test.sh
importantAug$./scripts/pretrained_null_importantAug_test.sh
importantAug$./scripts/pretrained_binarized_importantAug_test.sh