alfredqin/fiery

PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

FIERY

This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in:

FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, ‪Vijay Badrinarayanan, Roberto Cipolla and Alex Kendall

preprint (2021)
Blog post

_{Multimodal future predictions by our bird’s-eye view network.
Top two rows: RGB camera inputs. The predicted future trajectories and segmentations are projected to the ground plane in the images.
Bottom row: future instance prediction in bird’s-eye view in a 100m×100m capture size around the ego-vehicle, which is indicated by a black rectangle in the center.}

If you find our work useful, please consider citing:

@inproceedings{fiery2021,
  title     = {{FIERY}: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras},
  author    = {Anthony Hu and Zak Murez and Nikhil Mohan and Sofía Dudas and 
               Jeffrey Hawke and Vijay Badrinarayanan and Roberto Cipolla and Alex Kendall},
  booktitle = {arXiv preprint},
  year = {2021}
}

⚙ Setup

• Create the conda environment by running conda env create.

🏄 Prediction

Visualisation

In a colab notebook:

Or locally:

• Download pre-trained weights.
• Run python visualise.py --checkpoint ${CHECKPOINT_PATH}. This will render predictions from the network and save them to an output_vis folder.

Evaluation

• Download the NuScenes dataset.
• Download pre-trained weights.
• Run python evaluate.py --checkpoint ${CHECKPOINT_PATH} --dataroot ${NUSCENES_DATAROOT}.

🔥 Pre-trained models

All the configs are in the folder fiery/configs

Config	Dataset	Past context	Future horizon	BEV size	IoU	VPQ
baseline.yml	NuScenes	1.0s	2.0s	100mx100m (50cm res.)	36.7	29.9
lyft/baseline.yml	Lyft	0.8s	2.0s	100mx100m (50cm res.)	36.3	29.2
literature/pon_setting.yml	NuScenes	0.0s	0.0s	100mx50m (25cm res.)	40.0	-
literature/lift_splat_setting.yml	NuScenes	0.0s	0.0s	100mx100m (50cm res.)	37.8	-
literature/fishing_setting.yml	NuScenes	1.0s	2.0s	32.0mx19.2m (10cm res.)	59.4	-

🏊 Training

To train the model from scratch on NuScenes:

• Run python train.py --config fiery/configs/baseline.yml DATASET.DATAROOT ${NUSCENES_DATAROOT}.

To train on single GPU add the flag GPUS [0], and to change the batch size use the flag BATCHSIZE ${DESIRED_BATCHSIZE}.

🙌 Credits

Big thanks to Piotr Sokólski (@pyetras) for the panoptic metric implementation, and to Hannes Liik (@hannesliik) for the awesome future trajectory visualisation on the ground plane.