vamoko/ultradet

UltraDet

This is the official implementation of Mining Negative Temporal Contexts For False Positive Suppression In Real-Time Ultrasound Lesion Detection. Our paper has been early accepted by MICCAI 2023!

[Arxiv] [PDF] [Code]

Abstract

During ultrasonic scanning processes, real-time lesion detection can assist radiologists in accurate cancer diagnosis. However, this essential task remains challenging and underexplored. General-purpose real-time object detection models can mistakenly report obvious false positives (FPs) when applied to ultrasound videos, potentially misleading junior radiologists. One key issue is their failure to utilize negative symptoms in previous frames, denoted as negative temporal contexts (NTC). To address this issue, we propose to extract contexts from previous frames, including NTC, with the guidance of inverse optical flow. By aggregating extracted contexts, we endow the model with the ability to suppress FPs by leveraging NTC. We call the resulting model UltraDet. The proposed UltraDet demonstrates significant improvement over previous state-of-the-arts and achieves real-time inference speed.

Result

The NTCA module leverages RoI-level NTC which are crucial for radiologists but ignored in previous works, thereby effectively improving the detection performance in a reliable and interpretable way. We plug the NTCA module into a basic real-time detector to form UltraDet. Experiments on CVA-BUS dataset demonstrate that UltraDet, with real-time inference speed, significantly outperforms previous works, reducing about 50% FPs at a recall rate of 0.90.

Quantitative results of real-time lesion detection on CVA-BUS.

Model	Type	Pr80	Pr90	FP80	FP90	AP50	R@16	FPS	Model Weights
One-Stage Detectors
YOLOX	Image	69.73.7	43.47.7	23.84.8	87.624.5	80.41.6	97.50.5	59.8	-
RetinaNet	Image	75.72.5	57.22.9	9.32.0	32.86.5	84.51.0	95.10.6	53.6	-
FCOS	Image	87.22.2	72.25.1	11.02.4	23.03.7	89.51.4	98.80.3	56.1	-
DeFCN	Image	81.51.8	67.52.3	21.13.2	33.44.3	86.41.3	99.30.3	51.2	-
Track-YOLO	Video	75.12.7	47.03.1	18.11.9	74.214.7	80.11.0	94.70.9	46.0	-
DETR-Based Detectors
DeformDETR	Image	90.13.2	72.710.6	5.62.2	37.820.9	90.52.0	98.70.3	33.8	-
TransVOD	Video	92.52.2	77.57.2	3.11.3	23.711.5	90.11.8	98.40.4	24.2	-
CVA-Net	Video	92.32.6	80.26.1	4.72.6	19.65.6	91.61.9	98.60.8	23.1	-
PTSEFormer	Video	93.31.9	85.46.0	2.81.1	12.59.8	91.51.6	97.91.2	9.1	-
FasterRCNN-Based Detectors
FasterRCNN	Image	91.30.9	75.23.6	6.91.4	34.46.7	88.01.4	92.41.0	49.2	-
RelationNet	Image	91.41.3	79.22.9	6.22.0	24.45.6	87.61.7	92.40.9	42.7	-
FGFA	Video	92.91.5	82.24.1	4.41.6	13.33.7	90.51.1	93.60.9	33.8	-
SELSA	Video	91.61.7	80.22.5	7.51.5	23.35.5	89.21.1	92.60.8	43.8	-
MEGA	Video	93.91.5	86.92.3	3.11.7	11.73.0	90.91.0	93.60.7	40.2	-
BasicDet (RDN)	Video	92.41.0	83.62.2	3.81.2	13.43.2	88.71.4	92.70.6	42.2	-
UltraDet (Ours)	Video	95.71.2	90.81.4	1.90.4	5.71.6	91.61.6	93.81.3	30.4	Link

Citing UltraDet

If you find UltraDet useful in your research, please consider citing:

@article{yu2023mining,
  title={Mining Negative Temporal Contexts For False Positive Suppression In Real-Time Ultrasound Lesion Detection},
  author={Yu, Haojun and Li, Youcheng and Wu, QuanLin and Zhao, Ziwei and Chen, Dengbo and Wang, Dong and Wang, Liwei},
  journal={arXiv preprint arXiv:2305.18060},
  year={2023}
}

Requirments

The whole project is based on detectron2, which is used as a package.

Installation

# install anaconda3 with python3.9
conda create -n <env_name> python=3.9
conda activate <env_name>

# use tuna source for pypi (optional)
python -m pip install --upgrade pip

# install packages
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html
pip install pandas
pip install path
pip install sqlmodel
pip install coloredlogs
pip install opencv-p
pip install mmcv
pip install prettytable
pip install fire
pip install scipy
pip install timm
pip install pytorch_metric_learning

# install ultrasound_vid
git clone https://github.com/HaojunYu1998/UltraDet.git
cd UltraDet
pip install -e .

# issue shooting
# AttributeError: module 'distutils' has no attribute 'version'
pip uninstall setuptools
pip install setuptools==58.0.4

# Visualizaition
pip install vidgear

Checkpoints

We are pleased to share the weights of the models required for UltraDet inference. You can access them through the following links:

Weigths
flownet.ckpt
r34.pkl
ultradet.pth

After downloading the weights, please store them in the following path:

UltraDet/
  pretrained_models/
    flownet.ckpt
    r34.pkl
    ultradet.pth

Dataset Preparation

We provide high quality annotation files. You can access them through the following links:

Json File
test.json
trainval.json

Please download PNG images from https://github.com/jhl-Det/CVA-Net and arrange the images and labels in the COCO format.

Note: We are using the initial version of the data provided by the authors of CVA-Net. The authors later made corrections to the annotations (please see their Github repository). Our high-quality annotations reference the correction content, and the video and image paths in the annotations are from the paths in the initial version of the data.

UltraDet/
  datasets/
    bus_data_cva/
      rawframes/
        benign/
          2cda21c3aab26332/
            000000.png
            000001.png
            ...
          ...
        malignant/
          1dc9ca2f1748c2ec/
            000000.png
            000001.png
            ...
          ...
      test.json
      trainval.json

Training and Inference

You can train or infer with UltraDet by running the following code:

# Training
python train_net.py --num-gpus {GPU_NUM} \
--config-file {CONFIG_FILES}

# Inference
python train_net.py --num-gpus {GPU_NUM} --eval-only --resume \
--config-file {CONFIG_FILES} \
# MODEL.WEIGHTS 'pretrained_models/ultradet.pth' # If you wish to load the pre-trained weights of UltraDet that we have released, please include this line of code