Deep learning models have emerged as a powerful tool in avian bioacoustics to assess environmental health. To maximize the potential of cost-effective and minimal-invasive passive acoustic monitoring (PAM), models must analyze bird vocalizations across a wide range of species and environmental conditions. However, data fragmentation challenges a evaluation of generalization performance. Therefore, we introduce the $\texttt{BirdSet}$ dataset, comprising approximately 520,000 global bird recordings for training and over 400 hours PAM recordings for testing.

User Installation

The simplest way to install $\texttt{BirdSet}$ is to clone this repository and install it as an editable package using conda and pip:

conda create -n birdset python=3.10
pip install -e .

You can also use the devcontainer configured as as git submodule:

git submodule update --init --recursive

Or poetry.

poetry install
poetry shell

Reproduce Neurips2024 Baselines

First, you have to download the background noise files for augmentations

python resources/utils/download_background_noise.py

We provide all experiment YAML files used to generate our results in the path birdset/configs/experiment/birdset_neurips24. For each dataset, we specify the parameters for all training scenario: DT, MT, and LT

Dedicated Training (DT)

The experiments for DT with the dedicated subset can be easily run with a single line:

python birdset/train.py experiment="birdset_neurips24/DT/$Model"

Medium Training (MT) and Large Training (LT)

Experiments for training scenarios MT and LT are harder to reproduce since they require more extensive training times. Additionally, the datasets are quite large (90GB for XCM and 480GB for XCL). Therefore, we provide the best model checkpoints via Hugging Face in the experiment files to avoid the need for retraining. These checkpoints can be executed by running the evaluation script, which will automatically download the model and perform inference on the test datasets:

python birdset/eval.py experiment="birdset_neurips24/$EXPERIMENT_PATH"

As the model EAT is not implemented in Hugging Face transformer (yet), the checkpoints are available to download from the tracked experiments on Weights and Biases LT_XCL_eat.

If you want to start the large-scale trainings and download the big training datasets, you can also employ the XCM and XCL trainings via the experiment YAML files.

python birdset/train.py experiment="birdset_neurips24/$EXPERIMENT_PATH"

After training, the best model checkpoint is saved based on the validation loss and can then be used for inference:

python birdset/eval.py experiment="birdset_neurips24/$EXPERIMENT_PATH" module.model.network.local_checkpoint="$CHECKPOINT_PATH"

Example

Tutorial Notebook

Prepare Data

from birdset.datamodule.base_datamodule import DatasetConfig
from birdset.datamodule.birdset_datamodule import BirdSetDataModule

# initiate the data module
dm = BirdSetDataModule(
    dataset= DatasetConfig(
        data_dir='data_birdset/HSN', # specify your data directory!
        dataset_name='HSN',
        hf_path='DBD-research-group/BirdSet',
        hf_name='HSN',
        n_classes=21,
        n_workers=3,
        val_split=0.2,
        task="multilabel",
        classlimit=500,
        eventlimit=5,
        sampling_rate=32000,
    ),
)

# prepare the data (download dataset, ...)
dm.prepare_data()

# setup the dataloaders
dm.setup(stage="fit")

# get the dataloaders
train_loader = dm.train_dataloader()

Prepare Model and Start Training

from lightning import Trainer
min_epochs = 1
max_epochs = 5
trainer = Trainer(min_epochs=min_epochs, max_epochs=max_epochs, accelerator="gpu", devices=1)

from birdset.modules.multilabel_module import MultilabelModule
model = MultilabelModule(
    len_trainset=dm.len_trainset,
    task=dm.task,
    batch_size=dm.train_batch_size,
    num_epochs=max_epochs)

trainer.fit(model, dm)

Logging

Logs will be written to Weights&Biases by default.

Background noise

To enhance model performance we mix in additional background noise from downloaded from the DCASE18. To download the files and convert them to the correct format, run the notebook 'download_background_noise.ipynb' in the 'notebooks' folder.

Run experiments

Our experiments are defined in the configs/experiment folder. To run an experiment, use the following command in the directory of the repository:

python birdset/train.py experiment="EXPERIMENT_PATH"

Replace EXPERIMENT_PATH with the path to the disired experiment YAML config originating from the experiment directory. For example, here's a command for training an EfficientNet on HSN:

python bridset/train.py experiment="local/HSN/efficientnet.yaml"

Data pipeline

Our datasets are shared via HuggingFace Datasets in our BirdSet repository. First log in to HuggingFace with:

huggingface-cli login

For a detailed guide to using the BirdSet data pipeline and its many configuration options, see our comprehensive BirdSet Data Pipeline Tutorial.

Datamodule

The datamodules are defined in birdset/datamodule and configurations are stored under configs/datamodule. base_datamodule is the main class that can be inherited for specific datasets. It is responsible for preparing the data in the function prepare_data and loading the data in the function setup. prepare_data downloads the dataset, applies preprocessing, creates validation splits and saves the data to disk. setup initiates the dataloaders and configures data transformations.

The following steps are performed in prepare_data:

Data is downloaded from HuggingFace Datasets _load_data
Data gets preprocessed with _preprocess_data
Data is split into train validation and test sets with _create_splits
Length of the dataset gets saved to access later
Data is saved to disk with _save_dataset_to_disk

The following steps are performed in setup:

Data is loaded from disk with _get_dataset in which the transforms are applied

Transformations

Data transformations are referred to data transformations that are applied to the data during training. They include e.g. augmentations. The transformations are added to the huggingface dataset with set_transform.

Explore Baseline Models

Models for spectrograms:

ConvNeXT: A pure convolutional model (ConvNet), inspired by the design of Vision Transformers, that claims to outperform them. (https://huggingface.co/docs/transformers/model_doc/convnext#:~:text=ConvNeXT%20is%20a%20pure%20convolutional,art%20image )
Swin Transformer: A hierarchical vision transformer, that employs shifted windows to compute representations. (https://huggingface.co/docs/transformers/model_doc/swin )
Audio Spectrogram Transformer (AST): Applies a Vision Transformer to audio, by turning audio into an image (spectrogram) and obtains state-of-the-art results for audio classification. (https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer#:~:text=In%20this%20paper%2C%20we%20answer,50%2C%20and%2098.1%25 )
ResNet: A convolutional neural network that employs residual connections to train much deeper models, facilitating the training of networks with an unprecedented number of layers (up to 1000). (https://huggingface.co/docs/transformers/model_doc/resnet#:~:text=ResNet%20Overview,5%E2%80%9D )
EfficientNet: A model designed to uniformly scale the network width, depth, and resolution using a compound coefficient, which enables it to achieve state-of-the-art accuracy in image classification tasks while being smaller and faster than previous models (https://huggingface.co/docs/transformers/model_doc/efficientnet#:~:text=EfficientNet%20Overview,the%20paper%20is%20the%20following)

Models for waveforms:

Wav2Vec2: Learns powerful representations from speech audio, by solving a contrastive task, which can then be fine-tuned on transcribed speech. (https://huggingface.co/docs/transformers/model_doc/wav2vec2#:~:text=Wav2Vec2%20Overview,on%20transcribed%20speech%20can ). There is also one Wav2Vec2 model already trained on bird sounds (https://huggingface.co/Saads/bird_classification_model#:~:text=,Training%20procedure%20Training%20hyperparameters )
--> Nutzt eigenen Wav2Vec2FeatureExtractor
Wav2Vec2-Conformer: Follows the same architecture as Wav2Vec2, but replaces the Attention-block with a Conformer-block (https://huggingface.co/docs/transformers/model_doc/wav2vec2-conformer )
--> Nutzt AutoFeatureExtractor
Hubert: An approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. (https://huggingface.co/docs/transformers/model_doc/hubert#:~:text=Hubert%20Overview,by%20three%20unique%20problems )
--> Nutzt AutoFeatureExtractor
(Whisper): Pre-trained model for automatic speech recognition (ASR) and speech translation, trained on a large dataset and capable of generalizing to many datasets and domains without the need for fine-tuning. (https://huggingface.co/docs/transformers/model_doc/whisper )
--> WhisperFeatureExtractor extracts mel-filter bank features from raw speech
UniSpeech: Learns speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner. (https://huggingface.co/docs/transformers/model_doc/unispeech#:~:text=UniSpeech%20is%20a%20speech%20model,to%20be%20decoded%20using%20Wav2Vec2CTCTokenizer )
--> Nutzt AutoFeatureExtractor

Weitere mögliche Modelle für waveforms: Data2VecAudio, SEW, SEW-D, UniSpeechSat, WavLM (https://huggingface.co/docs/transformers/tasks/audio_classification#:~:text=The%20task%20illustrated%20in%20this,Conformer%E3%80%91%2C%20%E3%80%90114%E2%80%A0WavLM%E3%80%91%2C%20%E3%80%90115%E2%80%A0Whisper%E3%80%91 )

dbd-research-group / birdset Goto Github PK

birdset's Introduction

User Installation

Reproduce Neurips2024 Baselines

Dedicated Training (DT)

Medium Training (MT) and Large Training (LT)

Example

Prepare Data

Prepare Model and Start Training

Logging

Background noise

Run experiments

Data pipeline

Datamodule

Transformations

birdset's People

Contributors

Stargazers

Watchers

Forkers

birdset's Issues

Models for spectrograms:

Models for waveforms:

Todos

From CC, definitions:

Problems with publishing Datasets:

Possible Solutions

Todos

TODO

Recommend Projects

Recommend Topics

Recommend Org