drmmz/protonet

ProtoNet for Few-Shot Learning in TensorFlow2 and Applications

ProtoNet for Few-Shot Learning

This repository is a TensorFlow2 implementation of ProtoNet (Prototypical Network) and its applications, aiming for creating a tool in zero/few-shot learning task that can be easily extended to other datasets or used in building projects. It includes

1. source code of ProtoNet and its configuration (multiple GPUs training, inference and evaluation);
2. source code of data (ProtoNet's inputs) generator using multiple CPU cores;
3. source code of two backbones: conv4 (original in paper) and resnet;
4. source code of utilities such as image preprocessing and dataset.

Applications

• Recognize Jason Bourne! By using detections obtained from RetinaNet for Object Detection and the below face recognizer, we are able to track Jason Bourne.

bourne_540.mp4

Scenes are taken from The Bourne Ultimatum (2007 film) and the cover page is from The Bourne Identity (2002 film).

• By just learning few face images from a random person, the model is able to identify and recognize that person effectively from a group of people. Below are samples tested on the CelebA dataset.

In each sample, there are 3 face images learned by the model (under the text "Learning") and a group of 15 people face images to find that person (under the text 'Recognizing') where the correct recognization is labeled by "match" in green color and the wrong recognization has "ground-truth" and "predict" in red color.

The model is trained on the CelebA dataset following its default splitting and image size with ResNet50 backbone and Adam optimizer for 60 epochs over 2 GPUs. It achieves the following results after 10 epochs on the test set where query examples in each episode contains exact 1 person same as in support examples.

3-shot	time (second)	mean (F1-score)	median (F1-score)
1-way, 15-query	0.04	0.91	1.0
1-way, 100-query	0.17	0.82	0.83

Requirements

python 3.7.9, tensorflow 2.3.1, matplotlib 3.3.4, numpy 1.19.2, scikit-image 0.17.2 and scikit-learn 0.23.2

References

1. Snell et al., Prototypical Networks for Few-shot Learning, https://arxiv.org/abs/1703.05175, 2017