bailvwangzi/repulsion_loss_ssd

Repulsion Loss: Detecting Pedestrians in a Crowd. https://arxiv.org/abs/1711.07752

Repulsion Loss implemented with SSD

Forked from PyTorch-SSD, which is a PyTorch implementation of Single Shot MultiBox Detector from the 2016 paper by Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang, and Alexander C. Berg. The official and original Caffe code can be found here.

Table of Contents

• Installation
• Datasets
• Train
• Evaluate
• Performance
• Demos
• Future Work
• Reference

       

Installation

• Install PyTorch by selecting your environment on the website and running the appropriate command.
• Clone this repository.
  • Note: We currently only support Python 3+.
• Then download the dataset by following the instructions below.
• We now support Visdom for real-time loss visualization during training!
  • To use Visdom in the browser:

  # First install Python server and client
  pip install visdom
  # Start the server (probably in a screen or tmux)
  python -m visdom.server

  • Then (during training) navigate to http://localhost:8097/ (see the Train section below for training details).
• Note: For training, we currently support VOC and COCO, and aim to add ImageNet support soon.

Datasets

To make things easy, we provide bash scripts to handle the dataset downloads and setup for you. We also provide simple dataset loaders that inherit torch.utils.data.Dataset, making them fully compatible with the torchvision.datasets API.

COCO

Microsoft COCO: Common Objects in Context

Download COCO 2014

# specify a directory for dataset to be downloaded into, else default is ~/data/
sh data/scripts/COCO2014.sh

VOC Dataset

PASCAL VOC: Visual Object Classes

Download VOC2007 trainval & test

# specify a directory for dataset to be downloaded into, else default is ~/data/
sh data/scripts/VOC2007.sh # <directory>

Download VOC2012 trainval

# specify a directory for dataset to be downloaded into, else default is ~/data/
sh data/scripts/VOC2012.sh # <directory>

Training SSD

• First download the fc-reduced VGG-16 PyTorch base network weights at: https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth
• By default, we assume you have downloaded the file in the ssd.pytorch/weights dir:

mkdir weights
cd weights
wget https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth

• To train SSD using the train script simply specify the parameters listed in train.py as a flag or manually change them.

python train.py

• Note:
  • For training, an NVIDIA GPU is strongly recommended for speed.
  • For instructions on Visdom usage/installation, see the Installation section.
  • You can pick-up training from a checkpoint by specifying the path as one of the training parameters (again, see train.py for options)

Evaluation

To evaluate a trained network:

python eval.py

You can specify the parameters listed in the eval.py file by flagging them or manually changing them.

Example

SSD:

SSD + repulsion loss：

Performance

VOC2007 Test

mAP

Method	mAP	mAP on Crowd
SSD	77.52%	48.24%
SSD+RepGT	77.43%	50.12%

Demos

Use a pre-trained SSD network for detection

Download a pre-trained network

• We are trying to provide PyTorch state_dicts (dict of weight tensors) of the latest SSD model definitions trained on different datasets.
• Currently, we provide the following PyTorch models:
  • SSD300 trained on VOC0712 (newest PyTorch weights)
    • https://s3.amazonaws.com/amdegroot-models/ssd300_mAP_77.43_v2.pth
  • SSD300 trained on VOC0712 (original Caffe weights)
    • https://s3.amazonaws.com/amdegroot-models/ssd_300_VOC0712.pth
• Our goal is to reproduce this table from the original paper

Try the demo notebook

• Make sure you have jupyter notebook installed.
• Two alternatives for installing jupyter notebook:

  1. If you installed PyTorch with conda (recommended), then you should already have it. (Just navigate to the ssd.pytorch cloned repo and run): jupyter notebook

  2. If using pip:

# make sure pip is upgraded
pip3 install --upgrade pip
# install jupyter notebook
pip install jupyter
# Run this inside ssd.pytorch
jupyter notebook

• Now navigate to demo/demo.ipynb at http://localhost:8888 (by default) and have at it!

Try the webcam demo

• Works on CPU (may have to tweak cv2.waitkey for optimal fps) or on an NVIDIA GPU
• This demo currently requires opencv2+ w/ python bindings and an onboard webcam
  • You can change the default webcam in demo/live.py
• Install the imutils package to leverage multi-threading on CPU:
  • pip install imutils
• Running python -m demo.live opens the webcam and begins detecting!

TODO

We have accumulated the following to-do list, which we hope to complete in the near future

• Still to come:
  • Support for the MS COCO dataset
  • Support for SSD512 training and testing
  • Support for training on custom datasets
  • Support for RepBox term
  • Support for selecting the second largest IoU from the same class

Authors

References

• Xinlong Wang, et al. "Repulsion Loss: Detecting Pedestrians in a Crowd." CVPR2018.
• Wei Liu, et al. "SSD: Single Shot MultiBox Detector." ECCV2016.
• Pytorch-SSD.
• Original Implementation (CAFFE).

repulsion_loss_ssd's Issues

Question about the code

Hi. I read your code and only find the RepGT loss. Is there RepBox loss that I overlook?

Can the code run on the pytorch 0.4x ?

How can I do to make the code suitable for pytorch 0.4+? Anyone can give me some advices? Thank you a lot.

Use own data to train this model

I try to train my own data using this model, but there is easy to be NaN loss. I have modified my learning rate enough small and double the batch size. It does not work. And I want to transfer the loss to Yolo model. Is it feasible?

a doubt about "Line 97: priors = priors[pos_idx].view(-1, 4)" in multibox_loss.py

hi, i have a doubt, such as the title. the shape of priors is (num_priors, 4), and pos_idx is (num, num_priors, 4). this code how to run success? thank u!

Why use iog instead of smooth_ln(iog) as the loss?

Hello, this is a great job. Is it better to use IOG as a loss function than smoothln (iog)? Can you explain it to me? Thank you in advance.

IndexError: too many indices for tensor of dimension 2

Setting up a new session...
E:\MyPaper\TargetDetection\Ours_SSD\Code\SSD\ssd_-repulsion_-loss\ssd.py:35: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
self.priors = Variable(self.priorbox.forward(), volatile=True)
E:\MyPaper\TargetDetection\Ours_SSD\Code\SSD\ssd_-repulsion_-loss\layers\modules\l2norm.py:17: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
init.constant(self.weight,self.gamma)
Loading base network...
Initializing weights...
Loading the dataset...
Training SSD on: VOC0712
Using the specified args:
Namespace(basenet='vgg16_reducedfc.pth', batch_size=16, cuda=True, dataset='VOC', dataset_root='E:\MyPaper\TargetDetection\Ours_SSD\Code\SSD\ssd_-repulsion_-loss\data/VOCdevkit/', gamma=0.1, lr=0.0001, momentum=0.9, num_workers=0, resume=None, save_folder='weights/', start_iter=0, visdom=False, weight_decay=0.0005)
E:\MyPaper\TargetDetection\Ours_SSD\Code\SSD\ssd_-repulsion_-loss\utils\augmentations.py:238: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
mode = random.choice(self.sample_options)
E:/MyPaper/TargetDetection/Ours_SSD/Code/SSD/ssd_-repulsion_-loss/train.py:178: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
targets = [Variable(ann.cuda(), volatile=True) for ann in targets]
Traceback (most recent call last):
File "E:/MyPaper/TargetDetection/Ours_SSD/Code/SSD/ssd_-repulsion_-loss/train.py", line 265, in
train()
File "E:/MyPaper/TargetDetection/Ours_SSD/Code/SSD/ssd_-repulsion_-loss/train.py", line 187, in train
loss_l, loss_l_repul, loss_c = criterion(out, targets)
File "D:\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "E:\MyPaper\TargetDetection\Ours_SSD\Code\SSD\ssd_-repulsion_-loss\layers\modules\multibox_loss.py", line 97, in forward
priors = priors[pos_idx].view(-1, 4)
IndexError: too many indices for tensor of dimension 2

The MOT19 detection benchmark may be a good benchmark for evaluating this work.

when I run train.py, I meet runtime error

RuntimeError: $ Torch: not enough memory: you tried to allocate 0GB. Buy new RAM! at D:\pytorch\pytorch\torch\lib\TH\THGeneral.c:246

When running train.py, I meet this error

Loading the dataset...
Training SSD on: VOC0712
Using the specified args:
Namespace(basenet='vgg16_reducedfc.pth', batch_size=32, cuda=True, dataset='VOC', dataset_root='C:\Users\Admin\data/VOCdevkit/', gamma=0.1, lr=0.001, momentum=0.9, num_workers=4, resume=None, save_folder='weights/', start_iter=0, visdom=False, weight_decay=0.0005)
train.py:181: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
targets = [Variable(ann.cuda(), volatile=True) for ann in targets]
C:\Users\Admin\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
warnings.warn(warning.format(ret))
Traceback (most recent call last):
File "train.py", line 269, in
train()
File "train.py", line 190, in train
loss_l, loss_l_repul, loss_c = criterion(out, targets)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "D:\wang\repulsion_loss_ssd\layers\modules\multibox_loss.py", line 101, in forward
loss_l_repul = repul_loss(loc_p, loc_g, priors)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "D:\wang\repulsion_loss_ssd\layers\modules\repulsion_loss.py", line 26, in forward
decoded_boxes = decode_new(loc_data, Variable(prior_data.data, requires_grad=False), self.variance)
File "D:\wang\repulsion_loss_ssd\layers\box_utils.py", line 212, in decode_new
priors[:, :2] + loc[:, :2] * variances[0] * priors[:, 2:],
RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 2

The training is easy to collapse

Thanks for your sharing. I have one question, after I have made some changes for running on my environment (Python 3.6, Pytorch 4.1), the training is easy to collapse （ loss be NAN), have you encountered it ?
Best,

RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered

RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered
repulsion_loss_ssd/layers/modules/multibox_loss.py", line 110, in forward
_, loss_idx = loss_c.sort(1, descending=True)

Plz clarify definition of 'crowd' in Performance Evaluation

First, thanks for your sharing.

Plz clarify the definition of Crowd from "mAP on crowd" in VOC2007 Test. To what degree, the object is defined as " crowd"?

update_vis_plot function called with incorrect # of arguments

I am running the train.py with the following cmd command:
python train.py --dataset VOC --dataset_root /home/alyoussef/repulsionLoss/repulsion_loss_ssd/data/VOCdevkit/ --basenet vgg16_reducedfc.pth --batch_size 32 --start_iter 10 --num_workers 0 --cuda False --lr 0.01 --momentum 0.9 --weight_decay 0.0005 --gamma 0.1 --visdom True --save_folder weights/

Error:
File "train.py", line 164, in train
update_vis_plot(epoch, loc_loss, repul_loss, conf_loss, epoch_plot, None,'append', epoch_size)
TypeError: update_vis_plot() takes from 6 to 7 positional arguments but 8 were given

This is how it is being called:
update_vis_plot(epoch, loc_loss, repul_loss, conf_loss, epoch_plot, None,'append', epoch_size)

Function definition is:
def update_vis_plot(iteration, loc, conf, window1, window2, update_type, epoch_size=1):
print('update vist plot called')
viz.line(
X=torch.ones((1, 3)).cpu() * iteration,
Y=torch.Tensor([loc, conf, loc + conf]).unsqueeze(0).cpu() / epoch_size,
win=window1,
update=update_type
)
# initialize epoch plot on first iteration
if iteration == 0:
print('passed if iteration = 0 condition')
viz.line(
X=torch.zeros((1, 3)).cpu(),
Y=torch.Tensor([loc, conf, loc + conf]).unsqueeze(0).cpu(),
win=window2,
update=True
)

TypeError: 'module' object is not subscriptable

File "train.py", line 268, in
train()
File "train.py", line 85, in train
transform=SSDAugmentation(cfg['min_dim']

How to train the model on CityPersons and Caltech datasets?

I want to train the model on CityPersons and Caltech datasets. How should I modify the config.py or some other files?

demo issue

when I run the demo, I got a error,please help me,thank you.

the error information as following:
y = net(xx)
File "/home/bupt-sse3/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/bupt-sse3/repulsion_loss_ssd-master/ssd.py", line 103, in forward
self.priors.type(type(x.data)) # default boxes
File "/home/bupt-sse3/repulsion_loss_ssd-master/layers/functions/detection.py", line 54, in forward
ids, count = nms(boxes, scores, self.nms_thresh, self.top_k)
ValueError: not enough values to unpack (expected 2, got 0)

model of Repulsion loss

Hi, can you provide a model trained with repulsion loss, Thank you. @bailvwangzi

get Train.py Run problem

!python train.py

Traceback (most recent call last):
  File "train.py", line 2, in <module>
    from data import *
  File "/_zhangyongleth4/repulsion_loss_ssd/data/__init__.py", line 1, in <module>
    from .voc0712 import VOCDetection, VOCAnnotationTransform, VOC_CLASSES, VOC_ROOT
ImportError: cannot import name 'VOC_ROOT'

when I followed your steps,I got an error

What's the use of these three lines of code?

I can't understand the use of these three lines, it seems that they didn't change the value of any tensor.

map in voc

Why is the map on voc lowered?Is it because the inter-class iog join loss function is wrong?

Pytorch version

Can anyone please tell me the supported version of Pytorch? 0.3 or 0.4? Thanks