linksense/lightnetplusplus

LightNet++: Boosted Light-weighted Networks for Real-time Semantic Segmentation

LightNet++

!!!New Repo.!!! ⇒ EfficientNet.PyTorch: Concise, Modular, Human-friendly PyTorch implementation of EfficientNet with Pre-trained Weights

!!!New Repo.!!! ⇒ MixNet-Pytorch: Concise, Modular, Human-friendly PyTorch implementation of MixNet with Pre-trained Weights

This repository contains the code (PyTorch-1.0+, W.I.P.) for: "LightNet++: Boosted Light-weighted Networks for Real-time Semantic Segmentation" by Huijun Liu.
LightNet++ is an advanced version of LightNet, which purpose to get more concise model design, smaller models, and better performance.

• MobileNetV2Plus: Modified MobileNetV2 (backbone)^[1,8] + DSASPPInPlaceABNBlock^[2,3] + Parallel Bottleneck Channel-Spatial Attention Block (PBCSABlock)^[6] + UnSharp Masking (USM) + Encoder-Decoder Arch.^[3] + InplaceABN^[4].

• ShuffleNetV2Plus: Modified ShuffleNetV2 (backbone)^[1,8] + DSASPPInPlaceABNBlock^[2,3] + Parallel Bottleneck Channel-Spatial Attention Block (PBCSABlock)^[6]+ UnSharp Masking (USM) + Encoder-Decoder Arch.^[3] + InplaceABN^[4].

• MixSeg-MixBiFPN: Modified MixNet (backbone)^[1,8] + MixBiFPNBlock^[2,3] + Encoder-Decoder Arch.^[3]

More about USM(Unsharp Mask)-Operator Block see Repo: SharpPeleeNet

Dependencies

• Python3.6
• PyTorch(1.0.1+)
• inplace_abn
• apex: Tools for easy mixed precision and distributed training in Pytorch
• tensorboard
• tensorboardX
• tqdm

Datasets for Autonomous Driving

• Cityscapes
• Mapillary Vistas Dataset
• Berkeley Deep Drive (BDD100K)
• ApolloScape

Results

Results on Cityscapes (Pixel-level/Semantic Segmentation)

Model	mIoU (S.S* Mixed Precision)	Model Weight
MobileNetV2Plus X1.0	71.5314 (WIP)	cityscapes_mobilenetv2plus_x1.0.pkl (14.3 MB)
ShuffleNetV2Plus X1.0	69.0885-72.5255 (WIP)	cityscapes_shufflenetv2plus_x1.0.pkl (8.59 MB)
MixSeg+MixBiFPN ArchS	72.2321 (WIP)	cityscapes_mixseg_archs_mixbifpn.pkl (16.4 MB)

• S.S.: Single Scale (1024x2048)

Feature Visualization

lightnetplusplus's Issues

"RuntimeError: CUDA out of memory."

Thanks for your job!!!
I got this error when using "train_shuffle.py" to train model on citysacpes dataset

Traceback (most recent call last):    
  File "/home/wcgu/code/RTSeg/LightNetPlusPlus/scripts/train_shuffle.py", line 382, in <module>    
    train(train_args, data_path, save_path)    
  File "/home/wcgu/code/RTSeg/LightNetPlusPlus/scripts/train_shuffle.py", line 290, in train    
    val_loss = loss_fn(input=net_out, target=labels_val, K=topk, weight=None)    
  File "/home/wcgu/code/RTSeg/LightNetPlusPlus/scripts/loss.py", line 111, in bootstrapped_cross_entropy2d    
    size_average=size_average)    
  File "/home/wcgu/code/RTSeg/LightNetPlusPlus/scripts/loss.py", line 88, in _bootstrap_xentropy_single    
    log_p = log_p[target.view(n * h * w, 1).repeat(1, c) >= 0]   
RuntimeError: CUDA out of memory. Tried to allocate 304.00 MiB (GPU 0; 10.72 GiB total capacity; 9.30 GiB already allocated; 226.62 MiB free; 59.54 MiB cached)

The batch_size is 1, and the training step is ok. Could you help me to solve it?

Training setting

Hi, @ansleliu , thanks for your work, can you share the training protocal, lr, batchsize, optimizer, epochs etc., for training on Cityscapes? Thanks a lot.

one question

how to use the weight file for test in video？

Missing keys in state dict

Only the deepdrive checkpoint file seems to have the usm.norm_act.weight

...
> # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
> 1. Setting up Model...
Traceback (most recent call last):
  File "deploy/cityscapes/evaluation/ss.py", line 253, in <module>
    model.load_state_dict(pre_weight)
  File "/home/ben/LightNetPlusPlus/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 777, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
	Missing key(s) in state_dict: "module.usm.norm_act.weight", "module.usm.norm_act.bias", "module.usm.norm_act.running_mean", "module.usm.norm_act.running_var". 

(env) ben@mercury:~/LightNetPlusPlus$ grep -r norm_act.weight ./*
Binary file ./checkpoint/cityscapes_mobilenetv2plus_x1.0.pkl matches
Binary file ./checkpoint/cityscapes_shufflenetv2plus_x0.5.pkl matches
Binary file ./checkpoint/cityscapes_shufflenetv2plus_x1.0.pkl matches
Binary file ./checkpoint/deepdrive_shufflenetv2plus_x1.0.pkl matches

(env) ben@mercury:~/LightNetPlusPlus$ grep -r usm.norm_act.weight ./*
Binary file ./checkpoint/deepdrive_shufflenetv2plus_x1.0.pkl matches

Request a reference of your work !!!

Thanks for sharing the code! it real helpful for our work.We want to quote your LightNet in our work, can you provide a reference ?

Have you published the paper

Hi, thanks for sharing the code. Is the paper "LightNet++: Boosted Light-weighted Networks for Real-time Semantic Segmentation" published? Because of I couldn't find it anywhere!

pretrained model doesn't match model

I tried 'MobileNetV2Plus X1.0'. But its OrderedDict keys don't match model's keys. I'm confused.

train

Hey, @ansleliu
First of all I thank you for good job.
How I can start to training this models?
Usually repository has been file 'train.py'
If I need to take this file in thr old repository of LightNet then what needs to be changed in it.