Here is the official implementation of Pseudo Prompt Generating (PsPG).
Yaoqin Ye, Junjie Zhang and Hongwei Shi. 2024. Pseudo-Prompt Generating in Pre-trained Vision-Language Models for Multi-Label Medical Image Classification. In Pattern Recognition and Computer Vision: 7th Chinese Conference, PRCV 2024
Pseudo-Prompt Generating (PsPG) is designed to address Multi-Label Medical Image Classification task, capitalizing on the priori knowledge of multi-modal features. Featuring a RNN-based decoder, PsPG autoregressively generates class-tailored embedding vectors, i.e., pseudo-prompts.
We recommend you use conda and provide the environment.yaml. You can import and activate the environment by:
conda env create -f environment.yaml
conda activate pspgYou can use python train.py --help or python val.py --help to learn the usage of the python scripts for training and validating. Next section we'll introduce some details.
For reference, we provide two typical commands for training and validating, respectively:
# train
python train.py -nc configs/model/RN50.yaml -dc configs/dataset/example.yaml \
--dataset_dir datasets/example --checkpoint weights/PsPG.pth --decoder_hidden 512 \
--max_epochs 50 --output_dir output --test_file_path test.jsonl \
--val_file_path val.jsonl --train_file_path train.jsonl --start_afresh
# validate
python val.py -nc configs/model/RN50.yaml -dc configs/dataset/example.yaml \
--dataset_dir datasets/example --checkpoint weights/PsPG.pth --decoder_hidden 512 \
--output_dir output --test_file_path test.jsonl \
--model_name example-RN50 --save_predWe provide makefile to simplify the process of training and validating the PsPG model, and we recommend you use make instead of directly executing the python script.
To use make, you must have linux, msys2 (if in Windows) or OSX (not tested for OSX) environment with make and conda installed. And make sure that conda is in your PATH, or CONDA_EXE is set to the path of conda executable (usually path/to/Conda/Scripts/conda). Otherwise, the make will report that it cannot find conda.
Upon the first usage, you can establish the conda environment of running the PsPG model by this command:
make initNext, you can train the model with the default setting with this command:
make trainThen, you can validate the model with the weights you just trained with this command:
make valThe weights and predicting results will output to the output folder by default, use this command to clear the output folder (it will clear the __pycache__ as well):
make cleanWe provide an example dataset so that user can run for the first time quickly. For other datasets, please refer to Datasets.
To plug in a dataset, you should create a dataset folder named after the dataset in the datasets folder, create two (or three) datamap files, and place the datamap files (will be explained afterwards) and the pictures into the folder.
A datamap file consists of lines of jsons (jsonl), each decribing one data entry of a dataset. A data entry consists of:
- id (optional)
- path to the image (relative to the dataset root directory)
- text (optional)
- ground truth labels
The datamap files required by a dataset should be named as test.jsonl, val.jsonl (if need) and train.jsonl (or you could modify Makefile to use custom setting).
You can refer to datasets/example to further learn the format of a dataset.
Append DATASET=[dataset] to your make command to specify the [dataset], and the variable is set to example by default.
By default, make will check if there is any file of checkpoint output during training (to be exact if there is last.pth in the output folder). If so, make will select it as the checkpoint for training and validating.
Otherwise, make will next check if there is weights/PsPG.pth. If so, this will be used, or, make will try to train the model from scratch in case of train target is specified or report an error in case of val is specified.
If you want to specify a target checkpoint, append CHECKPOINT=[yourCheckpointPath] to your make command.
You can change the backbone type via appending MODEL=model to the make command. We use model=RN50 by default. The provided weights are also for resnet-50.
You can use other pre-trained clip provided by openai like RN50, RN101, ViT-B.
There are other variables in makefile for more flexible usage. If you want to modify more detailedly, please refer to utils/cfg_builder.py.
You can control the max train epochs by specifying the EPOCH variable.
The model name is used only when saving the validating result. Results will be output to output/$(MODEL_NAME) by default.
You can change the name of the output folder by appending OUTPUT_DIR=output_dir. This variable is useful when you want to execute two or more processes simultaneously.
You can add extra flags to the python script by specifying the EXTRA_FLAGS variable in Makefile (See Makefilefor details).
Or you can append EXTRAFLAGS="--option1 arg1 --option2 arg2..." (do not add extra spaces!!) to the make command.
If you wonder what flags are provided by the python script, execute:
make val VAL_FLAGS=--help(forval.py)make train TRAIN_FLAGS=--help(fortrain.py)
If you want to override the flag specified by make, use VAL_FLAG="..." or TRAIN_FLAGS="...".
We recommend you to refer to the source code.
Datasets for training or testing can be accessed via the following links:
In order to access to some of these datasets, you should be credentialed.
Notice that due to restriction, Private-CXR is not provided. For any research requests, please contact the corresponding author.
We provide some scripts can be run independently to preprocess datasets or calculate metrics. For more information, please refer to utils/independ_metrics and utils/preprocess.
We provide one pre-trained weight: PsPG. This weight was trained on CheXpert, use RN50 as backbone, with hidden size 512, seq length 16.
In addition, we provide another without parameters of the prompt learner: CLIP-finetuned. This weight was finetuned on MIMIC-CXR.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent