akshitac8/generative_mlzsl

[TPAMI 2023] Generative Multi-Label Zero-Shot Learning

Generative Multi-Label Zero-Shot Learning TPAMI 2023

Akshita Gupta^*, Sanath Narayan^*, Salman Khan, Fahad Shahbaz Khan, Ling Shao, Joost van de Weijer

(* denotes equal contribution)

Webpage: https://akshitac8.github.io/GAN_MLZSL/

Overview

This repository contains the implementation of Generative Multi-Label Zero-Shot Learning.

In this work, we tackle the problem of synthesizing multi-label features in the context of zero-shot setting for recognition all (un)seen labels with a novel training mechanism.

Installation

The codebase is built on PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.6, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

conda create -n mlzsl python=3.6
conda activate mlzsl
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image scikit-learn opencv-python yacs joblib natsort tqdm pandas h5py==2.10.0

Data Preparation

Training using NUS-WIDE dataset:

Download the NUS-WIDE features, tags and other required training files from the drive link shared below.

link: https://drive.google.com/drive/folders/1tCo-xawWrnGQGaWYJEKQOQ31ts__rAse?usp=sharing

Extract them in the ./datasets folder.

Training using custom dataset:

Download the custom dataset in the same data folder. Please make sure to convert your custom dataset in the same format as NUS-WIDE.

python preprocess.py --image_dir data/custom_data/ --output_dir data/custom_data_jsons/ --train_json custom_data_train  --test_json custom_data_test

Above preprocessing step will create train and test jsons with ZSL and GZSL requirements.

The train and test jsons are used as an input when running feature extraction code.

python extract_4096_features.py --train_json custom_data_train  --test_json custom_data_test --gpu

Above feature extraction will save features in .h5 format which is used for training for our CLF model.

Training and Evaluation

NUS-WIDE

To train and evaluate zero-shot learning model on full NUS-WIDE dataset, run:

sh ./scripts/train_nus_wide.sh

Model Checkpoint

We also include the checkpoint of the zero-shot generative model on NUS-WIDE for fast evaluation in weights folder. Please download the pretrained weights according to the intructions within the folder. To reproduce results, run:

sh ./scripts/eval_nus_wide.sh

Citation

If this code is helpful for your research, we would appreciate if you cite the work:

@article{gupta2021generative,
  title={Generative Multi-Label Zero-Shot Learning},
  author={Gupta, Akshita and Narayan, Sanath and Khan, Salman and Khan, Fahad Shahbaz and Shao, Ling and van de Weijer, Joost},
  journal={arXiv preprint arXiv:2101.11606},
  year={2021}

Acknowledgments

I thank Dat Huynh for discussions and feedback regarding the evaluation protocol and sharing details for the baseline zero-shot methods. I thank Aditya Arora for suggestions on the figure aesthetics.

generative_mlzsl's Issues

nus_zsl_test_vgg19.h5 vs s,nus_gzsl_test_vgg19.h5_features

Hi , i was wondering why the values of above mentioned extracted features are exactly similar in shape and values, Can you please verify from your training data in g drive please.

get_extract_data not defined

Hello!
I was using your 'extract_4095_features.py' binary when I realized that the 'get_extract_data' function is not imported or defined. could you please provide implementation or import for this function?

dataset issue

anyone else having issues accessing the datasets? Thank you

About nus-wide dataset

Hi
Thank you for the code. For a standard multi-label recognition dataset on the nuswide dataset, how to divide the training sample and the test sample.

Classification Results/Detection Results

Hi, I have run the code and now i am looking forward to reproduce the classification as well as detection results , Can you provide me access to code for classification results plotting as well as detection part. Also how to test it separately for only ALF scheme or FLF scheme without CLF .Thanks alot

Google drive link not working

Thank you for the wonderful codebase. However, the google drive link for getting the shared folders is not working and seems to have expired. I was struggling a bit with the word embeddings file essentially.

The NUS_WIDE_pretrained_w2v_glove-wiki-gigaword-300

Evaluation File

1. The dataset argument given in eval.py is not defined.
2. The HYBRID FUSION SELF ATTENTION module is not defined in file you need to change it to CLF.
3. When i run the code using given pre-trained weights the accuracies are very low
4. Without pre trained wieghts accuracies are in range of mAp 20 for zsl

Urgent Query Regarding datasets

Hello

Hope you are doing well. I have read your article named " Generative Multi-Label Zero shot Learning" and find it interesting. I have also checked its GitHub repository and get a deep understanding of your proposed model. In your repository you have share the whole process for custom dataset downloading as well as the process of preprocessing and features extraction. I need your help in one thing as I have limited resources available and downloading the "Open Images " and "MS COCO " dataset on my system is impossible for me because of my limited internet speed and storage limit. If you can provide me the google drive link as you have shared for NUS-WIDE dataset with nus_gzsl__test_vgg19.h5,nus_seen_train_vgg19.h5 and nus_zsl__test_vgg19.h5. If you can share the same for "MS COCO" and "Open Images" as you have also performed experiments on this so this sort of pre-processed dataset you would have. It would be a great help for me as my work deadline is already near and because of limited resources I have I wouldn't be able to complete it, If I have no access to it. Please help me in this

How much gpu memory does it costs?

I run train_nus_wide on a single 1080Ti, but it returns CUDA out of memory.

Training using NUS-WIDE dataset:gdrive not accessible

link: https://drive.google.com/file/d/1LaCDkLY4XnkB5901OwRTdjOxCRvtDbfP/view?usp=sharing

I am trying this but its not working please resolve if possible

Production of qualitative results

First of all, thanks for this repo. I have managed to train and evaluate successfully and am seeing some nice performance figures.

I now have a query regarding the production of some qualitative results, similar to those at (https://github.com/nasir6/zero_shot_detection#qualitative-results). I see these are featured in your paper:

I have tried manually reproducing these by opening up the model but as I see you have already produced some, I thought it was worth asking if code already exists to do this. I would be grateful if you could provide a link if it does, this would really help my research and demonstration of results for my academic report. Thanks in advance and apologies for all of these queries.

Memory problem

Thanks for code. How could i fix this problem? I also tried to reduce the number of batch size to 32 and 16 but the issue is the same.

DataLoading CPU Hangs

HI
To build jason files and extract features myself i am trying to upload data but when i run pre process files then cpu hangs. Any guideline for this issue

NUS_WIDE_pretrained_w2v_glove-wiki-gigaword-300

Hi,

First of all, thanks for the project, it's really cool. In trying to reproduce the results, I have so far been able to find all other required files except the NUS_WIDE_pretrained_w2v_glove-wiki-gigaword-300 dataset. I have reproduced it but I believe it is in the wrong format. If someone could provide me with a link to the file or steps to reproduce it, I would be massively grateful.

Cheers

Links for downloading datasets, NUS_WIDE and OPENIMage

Thanks for your work! could you please provide a link for downloading relevant datasets?

Why does memory keep increasing during training?

Dear author, thanks for your code.But when I reproduced this code, I found that the memory kept increasing, and finally caused the training failure of running out of memory.What is the possible reason?