ami-system/on_device_classifier

This repository contains the code to create on-device machine learning models for species classification.

Species Classifier Models

This repo creates PyTorch species classification models based on GBIF images (see the gbif_download_standalone repo for information and code to downloading images).

This model is built using pytorch. The user needs to run the following scripts in a sequence to train the model:

Training the Models for a Given Region

The easiest way to run this pipeline is to use the {region}_model.sh files.

To run this for a given species list with Slurm (e.g., on Baskerville). For example sbatch costarica_model.sh, which will output to cr_train.out.

Scripts

The pipeline is comprised of 4 scripts:

01-create_dataset_split.py

This creates training, validation and testing splits of the data downloaded from GBIF.

python 01_create_dataset_split.py \
    --data_dir /bask/homes/f/fspo1218/amber/data/gbif_download_standalone/gbif_images/ \
    --write_dir /bask/homes/f/fspo1218/amber/data/gbif_costarica/ \
    --species_list /bask/homes/f/fspo1218/amber/projects/gbif_download_standalone/species_checklists/costarica-moths-keys-nodup.csv \
    --train_ratio 0.75 \
    --val_ratio 0.10 \
    --test_ratio 0.15 \
    --filename 01_costarica_data

The description of the arguments to the script:

• --data_dir: Path to the root directory containing the GBIF data. Required.
• --write_dir: Path to the directory for saving the split files. Required.
• --train_ratio: Proportion of data for training. Required.
• --val_ratio: Proportion of data for validation. Required.
• --test_ratio: Proportion of data for testing. Required.
• --filename: Initial name for the split files. Required.
• --species_list: Path to the species list. Required.

02_calculate_taxa_statistics.py

This calculates information and statistics regarding the taxonomy to be used for model training.

python 02_calculate_taxa_statistics.py \
    --species_list /bask/homes/f/fspo1218/amber/projects/gbif_download_standalone/species_checklists/costarica-moths-keys-nodup.csv \
    --write_dir /bask/homes/f/fspo1218/amber/data/gbif_costarica/ \
    --numeric_labels_filename 01_costarica_data_numeric_labels \
    --taxon_hierarchy_filename 01_costarica_data_taxon_hierarchy \
    --training_points_filename 01_costarica_data_count_training_points \
    --train_split_file /bask/homes/f/fspo1218/amber/data/gbif_costarica/01_costarica_data-train-split.csv

The description of the arguments to the script:

• --species_list: Path to the species list. Required.
• --write_dir: Path to the directory for saving the information. Required.
• --numeric_labels_filename: Filename for numeric labels file. Required.

• --taxon_hierarchy_filename: Filename for taxon hierarchy file. Required.
• --training_points_filename: Filename for storing the count of training points. Required.
• --train_split_file: Path to the training split file. Required.

THEN after this is done you need to add the number fo families, genus, and species to the ./configs/01_uk_macro_data_config.json file. This is done manually.

03_create_webdataset.py

Creates webdataset from raw image data. It needs to be run individually for each of the train, validation and test sets.

So we will loop through each set:

for VARIABLE in 'train' 'val' 'test'
do
    echo '--' $VARIABLE
    mkdir -p /bask/homes/f/fspo1218/amber/data/gbif_costarica/$VARIABLE
    python 03_create_webdataset.py \
        --dataset_dir /bask/homes/f/fspo1218/amber/data/gbif_download_standalone/gbif_images/ \
        --dataset_filepath /bask/homes/f/fspo1218/amber/data/gbif_costarica/01_costarica_data-$VARIABLE-split.csv \
        --label_filepath /bask/homes/f/fspo1218/amber/data/gbif_costarica/01_costarica_data_numeric_labels.json \
        --image_resize 500 \
        --max_shard_size 100000000 \
        --webdataset_pattern "/bask/homes/f/fspo1218/amber/data/gbif_costarica/$VARIABLE/$VARIABLE-500-%06d.tar"
done

04_train_model.py

Training the Pytorch model. This step required the use of wandb. The user needs to create an account and login to the platform. The user will then need to set up a project and pass the entity (username) and project into the config file. This can be run with nohup:

nohup sh -c 'python 04_train_model.py  \
    --train_webdataset_url "$train_url" \
    --val_webdataset_url "$val_url" \
    --test_webdataset_url "$test_url" \
    --config_file ./configs/01_costarica_data_config.json \
    --dataloader_num_workers 6 \
    --random_seed 42' &

The description of the arguments to the script:

• --train_webdataset_url: path to webdataset tar files for training
• --val_webdataset_url: path to webdataset tar files for validation
• --test_webdataset_url: path to webdataset tar files for testing
• --config_file: path to configuration file containing training information
• --dataloader_num_workers: number of cpus available
• --random_seed: random seed for reproducible experiments

For setting up the config file: The total families, genuses, and species are spit out at the end of 02_calculate_taxa_statistics.py so you can use this info to fill in the config lines 5-7.

on_device_classifier's Issues

Provide list of UK moths for building regional classifier

determine accuracy of compressed model

and compare to the non-compressed model

compare accuracy of pytorch and tensorflow models

Some ideas from IAAD discussion

Just recording these somewhere (please edit/add to the list):

• Consider close crops/close ups of important features or partial moth images as another augmentation (c.f. whales)
• 'Open-world' classification - common for individual-level problems. Methods that produce an embedding of the image in a latent space can be convenient; visualisation of this
• https://github.com/facebookresearch/detr
• Megadetector: For detecting animals from camera traps. Used by WildLabs
• Networks used for facial recognition
• Grad-CAM: For explainable AI - to see which parts of the image the model focuses on

Set up pre-commits for species repo

• [] locally
• [] with git
• [] with github (actions)

Run models on baskerville with GPU/CUDA

Running the models on Baskerville crashes when requesting GPUs, despite torch.is_cuda_available() returning True and the set-up selecting multiple GPUs.

Using:

#SBATCH --gpus-per-task 3
#SBATCH --tasks-per-node 1
#SBATCH --nodes 1 

Returns error RuntimeError: CUDA error: no kernel image is available for execution on the device

Incorporate Crop details into modelling

I have observed a number of potential misclassifications which could be avoided if we incorporated features such as moth size, shape, etc.

For example Cryptocala acadiensis and Noctua pronuba are both in the Noctuidae family, have orange and black hindwings but are very different in size. Pixelflow could potentially be used in this case.

Add object detection to model

code to:

• identify if object is present
• crop images to object (bounding boxes)

Compress the trained species classifier model using tensorflow lite

Convert pytorch models to tensorflow lite following: https://github.com/omerferhatt/torch2tflite

populate data_download folder

Add code to download GBIF data for species of interest.
Scripts based on Rolnick's lab

Create classifier for Costa Rica moths

Provisional list for Costa Rica Lepidoptera
Lepidoptera of Costa Rica.xlsx

Create evaluation visualisations for each model run

General plots:

• confusion matrix (for family, genus and species)

For each species create:

• stacked bar confusion matrix
• rank bars from most TP+TN to least

Same for each family and each genus

GBIF returning genus results for some rank='species'

GBIF results are called with data = species_api.name_backbone(name=name, strict=True, rank="SPECIES")

However, some of the calls return 'rank': 'GENUS' results.

Examples:

data = species_api.name_backbone(name='Macaria notata', strict=True, rank="SPECIES")

gives:

{'usageKey': 3256294, 'scientificName': 'Macaria Curtis, 1826', 
    'canonicalName': 'Macaria', 'rank': 'GENUS', 'status': 'ACCEPTED', 
    'confidence': 98, 'matchType': 'HIGHERRANK', 'kingdom': 'Animalia', 
    'phylum': 'Arthropoda', 'order': 'Lepidoptera', 'family': 'Geometridae', 
    'genus': 'Macaria', 'kingdomKey': 1, 'phylumKey': 54, 'classKey': 216, 
    'orderKey': 797, 'familyKey': 6950, 'genusKey': 3256294, 
    'synonym': False, 'class': 'Insecta'}

Not sure why since https://api.gbif.org/v1/species?name=Macaria%20notata returns species as the first result

Create a classifier for Singapore moths

Re-creating this issue that David Roy initiated initially. The issue that documents the downloading of Singapore species images is separated out here

Fix file naming error

genus name and family name are mixed up in the 02a_fetch_gbif_metamorphic_data.py file

Set up TF Lite model with wandb

Test the species classification models on moth species which are not included in training data

We want to see how the model behaves when presented with a species it has not seen before.

And how does it work at the family, genus, species level ?

Catch gbif download asci errors

Set up project on Baskerville

Deploy compressed model on device

Train pytorch model for UK moths

Sub-tasks:

• #1
• #2
• #15
• #16
• #17
• #25

Issues arising:

• #7
• #18 (#21)
• #22 (#23)

Tag GBIF images containing multiple insects

@KatrionaGoldmann could you link the image that you showed at the meeting which had a moth and a non-moth in it?

Object detection with SAM

Using Segment anything model (SAM) to capture objects in the AMI images

Benchmark the TFLite models

Determine model benchmarking parameters using 1 thread and CPU.

Follow Tensorflow instructions here