PLIP is a novel Language-Image Pre-training framework for generic Person representation learning which benefits a range of downstream person-centric tasks.
Also, we present a large-scale person dataset named SYNTH-PEDES to verify its effectiveness, where the Stylish Pedestrian Attributes-union Captioning method (SPAC) is proposed to synthesize diverse textual descriptions.
Experiments show that our model not only significantly improves existing methods on downstream tasks, but also shows great ability in the few-shot and domain generalization settings. More details can be found at our paper PLIP: Language-Image Pre-training for Person Representation Learning.
- ๐ฅ[06.1] The SYNTH-PEDES is released. Welcome to download and use!
- ๐ฅ[06.1] The code for CMPM/C fine-tuning is released! It leads to SOTA performance without bells and whistles!
- ๐ฅ[05.31] The pre-trained model and zero-shot inference code are released !
SYNTH-PEDES is by far the largest person dataset with textual descriptions without any human annotation effort. Every person image has 2 or 3 different texutal descriptions and 6 attribute annotations. The dataset is released at Baidu Yun.
Note that SYNTH-PEDES can only be used for research, any commercial usage is forbidden.
This is the comparison of SYNTH-PEDES with other popular datasets.
These are some examples of our SYNTH-PEDES dataset.
Annotation format:
{
"id": 7,
"file_path": "Part1/7/1.jpg",
"attributes": [
"man,black hair,black shirt,pink shorts,black shoes,unknown"
],
"captions": [
"A man in his mid-twenties with short black hair is wearing a black t-shirt over light pink trousers. He is also wearing black shoes.",
"The man with short black hair is wearing a black shirt and salmon pink shorts. He is also wearing black shoes."
],
"prompt_caption": [
"A man with black hair is wearing a black shirt with pink shorts and a pair of black shoes."
]
}
We utilize ResNet50 and Bert as our encoders. After pre-training, we fine-tune and evaluate the performance on three downstream tasks. The checkpoints have been released at Baidu Yun and Google Drive.
Pre-train | CMPM/C | SSAN | LGUR |
---|---|---|---|
IN sup | 54.81/83.22 | 61.37/86.73 | 64.21/87.93 |
IN unsup | 55.34/83.76 | 61.97/86.63 | 65.33/88.47 |
CLIP | 55.67/83.82 | 62.09/86.89 | 64.70/88.76 |
LUP | 57.21/84.68 | 63.91/88.36 | 65.42/89.36 |
LUP-NL | 57.35/84.77 | 63.71/87.46 | 64.68/88.69 |
PLIP(ours) | 69.23/91.16 | 64.91/88.39 | 67.22/89.49 |
Pre-train | CMPM/C | SSAN | LGUR |
---|---|---|---|
IN sup | 47.61/75.48 | 54.23/79.53 | 57.42/81.45 |
IN unsup | 48.34/75.66 | 55.27/79.64 | 59.90/82.94 |
CLIP | 48.12/75.51 | 53.58/78.96 | 58.35/82.02 |
LUP | 50.12/76.23 | 56.51/80.41 | 60.33/83.06 |
LUP-NL | 49.64/76.15 | 55.59/79.78 | 60.25/82.84 |
PLIP(ours) | 64.25/86.32 | 60.12/82.84 | 62.27/83.96 |
Methods | Market1501 | DukeMTMC |
---|---|---|
BOT | 85.9/94.5 | 76.4/86.4 |
BDB | 86.7/95.3 | 76.0/89.0 |
MGN | 87.5/95.1 | 79.4/89.0 |
ABDNet | 88.3/95.6 | 78.6/89.0 |
PLIP+BOT | 88.0/95.1 | 77.0/86.5 |
PLIP+BDB | 88.4/95.7 | 78.2/89.8 |
PLIP+MGN | 90.6/96.3 | 81.7/90.3 |
PLIP+ABDNet | 91.2/96.7 | 81.6/90.9 |
Methods | PETA | PA-100K | RAP |
---|---|---|---|
DeepMAR | 80.14/83.56 | 78.28/84.32 | 76.81/78.94 |
Rethink | 83.96/86.35 | 80.21/87.40 | 79.27/79.95 |
VTB | 84.12/86.63 | 81.02/87.31 | 81.43/80.63 |
Label2Label | 84.08/86.57 | 82.24/87.08 | 81.82/80.93 |
PLIP+DeepMAR | 82.46/85.87 | 80.33/87.24 | 78.96/80.12 |
PLIP+Rethink | 85.56/87.63 | 82.09/88.12 | 81.87/81.53 |
PLIP+VTB | 86.03/88.14 | 83.24/88.57 | 83.64/81.78 |
PLIP+Label2Label | 86.12/88.08 | 84.36/88.63 | 83.77/81.49 |
we use 4 RTX3090 24G GPU for training and evaluation.
Create conda environment.
conda create --name PLIP --file requirements.txt
conda activate PLIP
Download the CUHK-PEDES dataset from here and ICFG-PEDES dataset from here.
Organize them in data
folder as follows:
|-- data/
| |-- <CUHK-PEDES>/
| |-- imgs
| |-- cam_a
| |-- cam_b
| |-- ...
| |-- reid_raw.json
|
| |-- <ICFG-PEDES>/
| |-- imgs
| |-- test
| |-- train
| |-- ICFG_PEDES.json
|
| |-- <SYNTH-PEDES>/
| |-- Part1
| |-- ...
| |-- Part11
| |-- synthpedes_dataset.json
Our pre-trained model can directly be transfered to downstream tasks, especially text-based Re-ID.
- Run the python file and generate train/test/valid json files respectively.
python dataset_split.py
- Then you can evaluate by running:
python zs_inference.py
Almost all existing downstream person-centric methods can be improved through replacing the backbone with our pre-trained model. Taking CMPM/C as example:
- Go to the CMPM/C root:
cd Downstreams/CMPM-C
- Run the following to train and test. Note that you can modify the code yourself for single GPU training:
python dataset_split.py
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 train.py
python test.py
By simply replacing the visual backbone with our pre-trained model, almost all existing methods on downstream tasks make significant improvements. For example, you can try by the following repositories:
Image-based Re-ID: BOT, MGN, ABD-Net
Person Attribute Recognition: Rethink, Label2label, VTB
If you use PLIP in your research, please cite it by the following BibTeX entry:
@misc{zuo2023plip,
title={PLIP: Language-Image Pre-training for Person Representation Learning},
author={Jialong Zuo and Changqian Yu and Nong Sang and Changxin Gao},
year={2023},
eprint={2305.08386},
archivePrefix={arXiv},
primaryClass={cs.CV}
}