vita-epfl/detection-attributes-fields

PyTorch implementation of "Detecting 32 Pedestrian Attributes for Autonomous Vehicles"

Object Detection and Attribute Recognition with Fields

PyTorch implementation of paper Detecting 32 Pedestrian Attributes for Autonomous Vehicles by Taylor Mordan (EPFL/VITA), Matthieu Cord (Sorbonne Université, valeo.ai), Patrick Pérez (valeo.ai) and Alexandre Alahi (EPFL/VITA).

Abstract

Detecting 32 Pedestrian Attributes for Autonomous Vehicles

Pedestrians are arguably one of the most safety-critical road users to consider for autonomous vehicles in urban areas. In this paper, we address the problem of jointly detecting pedestrians and recognizing 32 pedestrian attributes from a single image. These encompass visual appearance and behavior, and also include the forecasting of road crossing, which is a main safety concern. For this, we introduce a Multi-Task Learning (MTL) model relying on a composite field framework, which achieves both goals in an efficient way. Each field spatially locates pedestrian instances and aggregates attribute predictions over them. This formulation naturally leverages spatial context, making it well suited to low resolution scenarios such as autonomous driving. By increasing the number of attributes jointly learned, we highlight an issue related to the scales of gradients, which arises in MTL with numerous tasks. We solve it by normalizing the gradients coming from different objective functions when they join at the fork in the network architecture during the backward pass, referred to as fork-normalization. Experimental validation is performed on JAAD, a dataset providing numerous attributes for pedestrian analysis from autonomous vehicles, and shows competitive detection and attribute recognition results, as well as a more stable MTL training.

The model MTL-Fields learns multiple fields for both object detection and attribute recognition in a Multi-Task Learning way. Learning is done on full images with dedicated field and image-wise loss function for each task, and predictions are obtained at inference through a post-processing instance-wise decoding step that yields a bounding box and all attributes for each detected instance. This model is applied on dataset JAAD to detect up to 32 pedestrian attributes in an autonomous vehicle scenario.

The model MTL-Fields also contains a normalization of gradients during backward to solve gradient scale issues when learning numerous tasks.

Table of Contents

• Installation
• Dataset
• Interfaces
• Training
• Evaluation
• Project structure
• License
• Citation
• Acknowledgements

Installation

Clone this repository in order to use it.

# To clone the repository using HTTPS
git clone https://github.com/vita-epfl/detection-attributes-fields
cd detection-attributes-fields/

All dependencies can be found in the requirements.txt file.

# To install dependencies
pip3 install -r requirements.txt

This project has been tested with Python 3.7.7, PyTorch 1.9.1, CUDA 10.2 and OpenPifPaf 0.13.0.

Dataset

This project uses dataset JAAD for training and evaluation.

Please refer to JAAD documentation to download the dataset.

Interfaces

This project is implemented as an OpenPifPaf plugin module. As such, it benefits from all the core capabilities offered by OpenPifPaf, and only implements the additional functions it needs.

All the commands can be run through OpenPifPaf's interface using subparsers. Help can be obtained for any of them with option --help. More information can be found in OpenPifPaf documentation.

Training

Training is done using subparser openpifpaf.train.

Training on JAAD with all attributes can be run with the command:

python3 -m openpifpaf.train \
  --output <path/to/model.pt> \
  --dataset jaad \
  --jaad-root-dir <path/to/jaad/folder/> \
  --jaad-subset default \
  --jaad-training-set train \
  --jaad-validation-set val \
  --log-interval 10 \
  --val-interval 1 \
  --epochs 5 \
  --batch-size 4 \
  --lr 0.0005 \
  --lr-warm-up-start-epoch -1 \
  --weight-decay 5e-4 \
  --momentum 0.95 \
  --basenet fn-resnet50 \
  --pifpaf-pretraining \
  --detection-bias-prior 0.01 \
  --jaad-head-upsample 2 \
  --jaad-pedestrian-attributes all \
  --fork-normalization-operation power \
  --fork-normalization-duplicates 35 \
  --lambdas 7.0 7.0 7.0 7.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 \
  --attribute-regression-loss l1 \
  --attribute-focal-gamma 2 \
  --auto-tune-mtl

Arguments should be modified appropriately if needed.

More information about the options can be obtained with the command:

python3 -m openpifpaf.train --help

Evaluation

Evaluation of a checkpoint is done using subparser openpifpaf.eval.

Evaluation on JAAD with all attributes can be run with the command:

python3 -m openpifpaf.eval \
  --output <path/to/outputs> \
  --dataset jaad \
  --jaad-root-dir <path/to/jaad/folder/> \
  --jaad-subset default \
  --jaad-testing-set test \
  --checkpoint <path/to/checkpoint.pt> \
  --batch-size 1 \
  --jaad-head-upsample 2 \
  --jaad-pedestrian-attributes all \
  --head-consolidation filter_and_extend \
  --decoder instancedecoder:0 \
  --decoder-s-threshold 0.2 \
  --decoder-optics-min-cluster-size 10 \
  --decoder-optics-epsilon 5.0 \
  --decoder-optics-cluster-threshold 0.5

Arguments should be modified appropriately if needed.

Using option --write-predictions, a json file with predictions can be written as an additional output.

Using option --show-final-image, images with predictions displayed on them can be written in the folder given by option --save-all <path/to/image/folder/>. To also display ground truth annotations, add option --show-final-ground-truth.

More information about the options can be obtained with the command:

python3 -m openpifpaf.eval --help

Project structure

The code is organized as follows:

openpifpaf_detection_attributes/
├── datasets/
│   ├── jaad/
│   ├── (+ common files for datasets)
│   └── (add new datasets here)
└── models/
    ├── mtlfields/
    ├── (+ common files for models)
    └── (add new models here)

License

This project is built upon OpenPifPaf and shares the AGPL Licence.

This software is also available for commercial licensing via the EPFL Technology Transfer Office (https://tto.epfl.ch/, [email protected]).

Citation

If you use this project in your research, please cite the corresponding paper:

@article{mordan2021detecting,
  title={Detecting 32 Pedestrian Attributes for Autonomous Vehicles},
  author={Mordan, Taylor and Cord, Matthieu and P{\'e}rez, Patrick and Alahi, Alexandre},
  journal={IEEE Transactions on Intelligent Transportation Systems (T-ITS)},
  year={2021},
  doi={10.1109/TITS.2021.3107587}
}

Acknowledgements

We would like to thank Valeo for funding our work, and Sven Kreiss for the OpenPifPaf Plugin architecture.

detection-attributes-fields's Issues

Pretrained model

@taylormordan thanks for sharing the code base and the wonderful work is it possible to share the pretrained model to test the results ??
Thanks in advance

Can you provide a trained model?

I found it too slow to train a new model. Can you share a trained model? Thanks a lot.

where is the training script?

where is the training script?

How can I plot detected attributions on the image?

How can I plot detected attributions on the image?

how's the result and performance?

how's the result and performance?

Segmentation from compositie fields

@svenkreiss @taylormordan any idea how to extend this composite fields for segmentation based tasks along with multi label classification of the preson
Please share your thoughts

Extenstion to other objects

@svenkreiss @taylormordan thanks for sharing the source code , can we extend this work for car detection with attributes along with person and its attributes , if so how can we do it ? we can annotated vehicle with bounding box and attributes can be brand , #wheels ,color . what all changes have to be made in the source code for extending the source code

Thanks in advance

How to fix this error? Plz help

usage: python3 -m openpifpaf.train [options]
python3 -m openpifpaf.train: error: unrecognized arguments: --jaad-root-dir C:/Users/VR/Desktop/JAAD_clips.zip --jaad-subset default --jaad-training-set train --jaad-validation-set val --pifpaf-pretraining --detection-bias-prior 0.01 --jaad-head-upsample 2 --jaad-pedestrian-attributes all --fork-normalization-operation power --fork-normalization-duplicates 35 --attribute-regression-loss l1 --attribute-focal-gamma 2

I just gave the path to the output of the model and the data set this way.
Please take a look:

!python3 -m openpifpaf.train
--output C:/Users/VR/Desktop/model
--dataset jaad
--jaad-root-dir C:/Users/VR/Desktop/JAAD_clips.zip
--jaad-subset default
--jaad-training-set train
--jaad-validation-set val
--log-interval 10
--val-interval 1
--epochs 5
--batch-size 4
--lr 0.0005
--lr-warm-up-start-epoch -1
--weight-decay 5e-4
--momentum 0.95
--basenet fn-resnet50
--pifpaf-pretraining
--detection-bias-prior 0.01
--jaad-head-upsample 2
--jaad-pedestrian-attributes all
--fork-normalization-operation power
--fork-normalization-duplicates 35
--lambdas 7.0 7.0 7.0 7.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
--attribute-regression-loss l1
--attribute-focal-gamma 2
--auto-tune-mtl

Reduce the number of classes

@svenkreiss @taylormordan can we train the whole model for only few classes like the general attributes from 32 to 10 . what all changes have to be made ? so how will be a change in accuracy of the model ?
THanks in advance

loss become negative after 100 epochs

INFO:openpifpaf.network.trainer:{'type': 'train', 'epoch': 65, 'batch': 840, 'n_batches': 1266, 'time': 0.948, 'data_time': 0.0, 'lr': 0.0005, 'loss': -31.85, 'head_losses': [0.003, 0.097, 0.076, 0.111, 0.02, 0.25, 0.016, 0.027, 0.001, 0.0, 0.014, 0.015, 0.026, 0.019, 0.073, 0.021, 0.013, 0.004, 0.0, 0.039, 0.09, 0.088, 0.037, 0.006, 0.0, 0.003, 0.014, 0.003, 0.012, 0.011, 0.002, 0.002, 0.001, 0.002, 0.012], 'mtl_sigmas': [0.002, 0.301, 0.275, 0.367, 0.081, 0.518, 0.089, 0.171, 0.016, 0.0, 0.1, 0.092, 0.124, 0.122, 0.218, 0.046, 0.09, 0.027, 0.0, 0.176, 0.262, 0.26, 0.157, 0.037, 0.0, 0.003, 0.09, 0.003, 0.087, 0.065, 0.006, 0.01, 0.0, 0.002, 0.072]}

is this normal?

Increasing the accuracy of the model

@svenkreiss @taylormordan i test the model on certain images , for few images it works well and for others its not work well

1. Bad
   

2. Good

The observation is the nearby person has multiple bounding boxes , pls share your thoughts

Thanks in advance

To Run Decoder on GPU

Hi @svenkreiss @taylormordan.

Is there a way that we can run decoder on GPU?
Evaluation and Validation is consuming more time during the decoder run as it runs on CPU.

Thanks in Advance