swoook/ddrnet

Cloned from chenjun2hao/DDRNet (https://github.com/chenjun2hao/DDRNet.pytorch).

❗ This is cloned repository!

This repository is cloned from chenjun2hao/DDRNet.pytorch and modified for research

Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes

Introduction

This is the unofficial code of Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes. which achieve state-of-the-art trade-off between accuracy and speed on cityscapes and camvid, without using inference acceleration and extra data!on single 2080Ti GPU, DDRNet-23-slim yields 77.4% mIoU at 109 FPS on Cityscapes test set and 74.4% mIoU at 230 FPS on CamVid test set.

The code mainly borrows from HRNet-Semantic-Segmentation OCR and the official repository, thanks for their work.

^{A comparison of speed-accuracy trade-off on Cityscapes test set.}

Requirements

torch>=1.7.0
cudatoolkit>=10.2

Cityscapes Data Preparation

1. Download two files below from Cityscapes.to the ${CITYSCAPES_ROOT}

   • leftImg8bit_trainvaltest.zip
   • gtFine_trainvaltest.zip

2. Unzip them

3. Rename the folders like below

   └── cityscapes
     ├── leftImg8bit
         ├── test
         ├── train
         └── val
     └── gtFine
         ├── test
         ├── train
         └── val
   

4. Update some properties in {REPO_ROOT}/experiments/cityscapes/${MODEL_YAML} like below

   DATASET:
     DATASET: cityscapes
     ROOT: ${CITYSCAPES_ROOT}
     TEST_SET: 'cityscapes/list/test.lst'
     TRAIN_SET: 'cityscapes/list/train.lst'
     ...

Pretrained Models

• The official repository provides pretrained models for Cityscapes

1. Download the pretrained model to ${MODEL_DIR}

2. Update MODEL.PRETRAINED and TEST.MODEL_FILE in {REPO_ROOT}/experiments/cityscapes/${MODEL_YAML} like below

   ...
   MODEL:
     ...
     PRETRAINED: "${MODEL_DIR}/${MODEL_NAME}.pth"
     ALIGN_CORNERS: false
     ...
   TEST:
     ...
     MODEL_FILE: "${MODEL_DIR}/${MODEL_NAME}.pth"
     ...

Validation

• Execute the command below to evaluate the model on Cityscapes-val

cd ${REPO_ROOT}
python tools/eval.py --cfg experiments/cityscapes/ddrnet23_slim.yaml

model	OHEM	Multi-scale	Flip	mIoU	FPS	E2E Latency (s)	Link
DDRNet23_slim	Yes	No	No	77.83	91.31	0.062	official
DDRNet23_slim	Yes	No	Yes	78.42	TBD	TBD	official
DDRNet23	Yes	No	No	79.51	TBD	TBD	official
DDRNet23	Yes	No	Yes	79.98	TBD	TBD	official

mIoU denotes an mIoU on Cityscapes validation set.

FPS is measured by following the test code provided by SwiftNet. (Refer to speed_test from lib/utils/utils.py for more details.)

E2E Latency denotes an end-to-end latency including pre/post-processing.

FPS and latency are measured with batch size 1 on RTX 2080Ti GPU and Threadripper 2950X CPU.

Note

• with the ALIGN_CORNERS: false in ***.yaml will reach higher accuracy.

TRAIN

download the imagenet pretrained model, and then train the model with 2 nvidia-3080

cd ${PROJECT}
python -m torch.distributed.launch --nproc_per_node=2 tools/train.py --cfg experiments/cityscapes/ddrnet23_slim.yaml

the own trained model coming soon

OWN model

model	Train Set	Test Set	OHEM	Multi-scale	Flip	mIoU	Link
DDRNet23_slim	train	eval	Yes	No	Yes	77.77	Baidu/password:it2s
DDRNet23_slim	train	eval	Yes	Yes	Yes	79.57	Baidu/password:it2s
DDRNet23	train	eval	Yes	No	Yes	~	None
DDRNet39	train	eval	Yes	No	Yes	~	None

Note

• set the ALIGN_CORNERS: true in ***.yaml, because i use the default setting in HRNet-Semantic-Segmentation OCR.
• Multi-scale with scales: 0.5,0.75,1.0,1.25,1.5,1.75. it runs too slow.
• from ydhongHIT, can change the align_corners=True with better performance, the default option is False

Reference

[1] HRNet-Semantic-Segmentation OCR branch

[2] the official repository

ddrnet's Issues

RuntimeError: Error(s) in loading state_dict for DualResNet: ...

Backgrounds

• Faster SOD model required

1. 10 FPS on CPU
2. 30 FPS on 1080Ti GPU

• No public detector found that satisfies the requirements
• Recall the 1904.09569 (arxiv.org)
• SOTA architectures for SOD follow pixel-wise prediction
• Actually, most of them were proposed for segmentation tasks
• U-Net is one of them which showed impressive results on segmentation tasks
• And PoolNet employed it as a backbone
• Maybe we can use real-time segmentation models for SOD
• Refer to #6 from swoook/ucnet (github) for more details

Issue description

• Confirmed the error messages below when trying to execute ${REPO_ROOT}/main.py

RuntimeError: Error(s) in loading state_dict for DualResNet:

Code example

• configuration in launch.json:

        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", //"${file}",
            "console": "integratedTerminal",
            "args": [
                "--nproc_per_node", "1",
                "${workspaceRoot}/main.py",
                "--mode", "train",
                "--cfg_path", "./experiments/duts/ddrnet23_slim_poolnet_train_scheme.yaml",
                ]
        },

• Error messages and stack traces:

Traceback (most recent call last):
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3293, in <module>
    main()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3286, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2360, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2367, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydev_imps/_pydev_execfile.py", line 25, in execfile
    exec(compile(contents + "\n", file, 'exec'), glob, loc)
  File "/data/swook/repos/chenjun2hao/ddrnet/main.py", line 115, in <module>
    main(args)
  File "/data/swook/repos/chenjun2hao/ddrnet/main.py", line 98, in main
    solver = Solver(train_loader, None, config, args)
  File "/data/swook/repos/chenjun2hao/ddrnet/sod/solver.py", line 26, in __init__
    self.build_model()
  File "/data/swook/repos/chenjun2hao/ddrnet/sod/solver.py", line 74, in build_model
    self.net.load_state_dict(model_dict, strict=False)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DualResNet:
        size mismatch for seghead_extra.conv2.weight: copying a param with shape torch.Size([19, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 64, 1, 1]).
        size mismatch for seghead_extra.conv2.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([1]).
        size mismatch for final_layer.conv2.weight: copying a param with shape torch.Size([19, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 64, 1, 1]).
        size mismatch for final_layer.conv2.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([1]).
Killing subprocess 53003
Traceback (most recent call last):
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/data/swook/miniconda3/envs/torch18csnet/bin/python', '-u', '/data/swook/repos/chenjun2hao/ddrnet/main.py', '--local_rank=0', '--mode', 'train', '--cfg_path', './experiments/duts/ddrnet23_slim_poolnet_train_scheme.yaml']' returned non-zero exit status 1.

System Info

PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.3 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.1

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py38h27cfd23_1  
[conda] mkl_fft                   1.3.0            py38h42c9631_2  
[conda] mkl_random                1.2.1            py38ha9443f7_2  
[conda] numpy                     1.20.2           py38h2d18471_0  
[conda] numpy-base                1.20.2           py38hfae3a4d_0  
[conda] pytorch                   1.8.1           py3.8_cuda10.2_cudnn7.6.5_0    pytorch
[conda] torchvision               0.9.1                py38_cu102    pytorch

Increase the resolution of training images in DUTS-TR without scaling

Is your feature request related to a problem? Please describe.

• We usually scale images when we need to increase the resolution
• However, upscaling can bring pixelation
• It would be critical to saliency masks
• And the size of the images in the DUTS-TR is distributed like:

• Assume we'd like to maintain aspect ratios while increasing the resolution to 448 x 448
• Then the scale factors should be small (1.12 at maximum)
• If so, can we just use padding?

Describe the solution you'd like

• Increase the resolution with padding, not scaling

Describe alternatives you've considered

• None

Additional context

• DUTS-TR is a dataset for salient object detection
• We'd like to train DDRNet on this dataset for real-time salient object detection
• Refer to #6 from swoook/ucnet (github) for more details

Study factors of unusual patterns found from results of DDRNet23-slim trained on DUTS-TR

Is your feature request related to a problem? Please describe.

• Recall that we trained DDRNet23-slim on DUTS-TR

^{Figure 01. An example of input images}

^{Figure 02. Inference results of DDRNet23-slim for Figure 1}

• However, we've confirmed that it creates two types of unusual patterns

1. diamond
2. rectangle

• At first, we expected DAPPM might be a factor
• But there's no significant improvement even after modifications on DAPPM
• Refer to here for more details
• We highly suspect the down-sampling rates in backbone is the reason
• We'd like to change it to output feature-maps with similar size for (512, 512)
• I.e. Decrease down-sampling rates to 2, 4, 8, 16 and 32

Describe the solution you'd like

• Increase a resolution (r) of feature-maps in hidden layers

Describe alternatives you've considered

• None

Additional context

• None

Train DDRNet on multiple famous public datasets for salient object detection

Is your feature request related to a problem? Please describe.

• We trained DDRNet23-slim on DUTS-TR
• However, we'd like to train it on multiple famous datasets

1. HKU-IS
2. MSRA-B
3. BSDS
4. PASCAL
5. ...

Describe the solution you'd like

• We can train jointly if we convert the formats of other datasets to the one of DUTS-TR

Describe alternatives you've considered

• Train on other datasets sequentially
• However, it's not optimal 😞

Additional context

• None

Request a feature for training DDRNet-23-slim on datasets for salient object detection like DUTS

Is your feature request related to a problem? Please describe.

• We'd like to train DDRNet23-slim on datasets for salient object detection like DUTS
• Currently, chenjun2hao/DDRNet.pytorch supports training on:

1. Cityscapes
2. ADE20K
3. COCO stuff
4. ...

Describe the solution you'd like

• Implement a Dataset class based on lib.datsets.Cityscapes

Describe alternatives you've considered

• Use CSNet

Transfer learning for salient object detection

Is your feature request related to a problem? Please describe.

• We'd like to employ the DDRNet23-slim pre-trained on the Cityscapes for salient object detection
• How should we perform transfer-learning for salient object detection?

Describe the solution you'd like

• Pick a strategy by referring to Transfer learning from pre-trained models (towardsdatascience) (korean)

Describe alternatives you've considered

• None

Additional context

• DUTS-TR is a dataset for salient object detection
• We'd like to train DDRNet on this dataset for real-time salient object detection
• Refer to #6 from swoook/ucnet (github) for more details

Future work for transfer learning on salient object detection

Is your feature request related to a problem? Please describe.

• We trained DDRNet23-slim on DUTS-TR and it achieved 0.47 MAE on DUTS-TE
• And we'd like to improve the accuracy of DDRNet23-slim on salient object detection

Describe the solution you'd like

• Research BoFs and BoSs for related tasks, especially segmentation

Describe alternatives you've considered

• None

Additional context

• None

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])

Issue description

• Confirmed the error messages below when trying to execute ${REPO_ROOT}/main.py with an image of arbitrary size

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])

Code example

• configuration in launch.json:

        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", //"${file}",
            "console": "integratedTerminal",
            "args": [
                "--nproc_per_node", "1",
                "${workspaceRoot}/main.py",
                "--mode", "train",
                "--cfg_path", "./experiments/duts/ddrnet23_slim_poolnet_train_scheme.yaml",
                ]
        },

• Error messages and stack traces:

Traceback (most recent call last):
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3293, in <module>
    main()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3286, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2360, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2367, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydev_imps/_pydev_execfile.py", line 25, in execfile
    exec(compile(contents + "\n", file, 'exec'), glob, loc)
  File "/data/swook/repos/chenjun2hao/ddrnet/main.py", line 115, in <module>
    main(args)
  File "/data/swook/repos/chenjun2hao/ddrnet/main.py", line 99, in main
    solver.train()
  File "/data/swook/repos/chenjun2hao/ddrnet/sod/solver.py", line 123, in train
    sal_pred = self.net(sal_image)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/swook/repos/chenjun2hao/ddrnet/lib/models/ddrnet_23_slim.py", line 346, in forward
    self.spp(self.layer5(self.relu(x))),
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/swook/repos/chenjun2hao/ddrnet/lib/models/ddrnet_23_slim.py", line 179, in forward
    x_debug = self.scale3(x)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 537, in forward
    return F.batch_norm(
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/functional.py", line 2147, in batch_norm
    _verify_batch_size(input.size())
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/functional.py", line 2114, in _verify_batch_size
    raise ValueError("Expected more than 1 value per channel when training, got input size {}".format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])
Killing subprocess 43746
Traceback (most recent call last):
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/data/swook/miniconda3/envs/torch18csnet/bin/python', '-u', '/data/swook/repos/chenjun2hao/ddrnet/main.py', '--local_rank=0', '--mode', 'train', '--cfg_path', './experiments/duts/ddrnet23_slim_poolnet_train_scheme.yaml']' returned non-zero exit status 1.

System Info

PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.3 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.1

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py38h27cfd23_1  
[conda] mkl_fft                   1.3.0            py38h42c9631_2  
[conda] mkl_random                1.2.1            py38ha9443f7_2  
[conda] numpy                     1.20.2           py38h2d18471_0  
[conda] numpy-base                1.20.2           py38hfae3a4d_0  
[conda] pytorch                   1.8.1           py3.8_cuda10.2_cudnn7.6.5_0    pytorch
[conda] torchvision               0.9.1                py38_cu102    pytorch

Train

I just need to split the two categories. How do I change it

RuntimeError: The size of tensor a (38) must match the size of tensor b (37) at non-singleton dimension 2

Issue description

• Confirmed the error messages below when trying to execute ${REPO_ROOT}/main.py with an image of arbitrary size

RuntimeError: The size of tensor a (34) must match the size of tensor b (33) at non-singleton dimension 3

Code example

• configuration in launch.json:

        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", //"${file}",
            "console": "integratedTerminal",
            "args": [
                "--nproc_per_node", "1",
                "${workspaceRoot}/main.py",
                "--mode", "train",
                "--cfg_path", "./experiments/duts/ddrnet23_slim_poolnet_train_scheme.yaml",
                ]
        },

• Error messages and stack traces:

Traceback (most recent call last):
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3293, in <module>
    main()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 3286, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2360, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2367, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydev_imps/_pydev_execfile.py", line 25, in execfile
    exec(compile(contents + "\n", file, 'exec'), glob, loc)
  File "/data/swook/repos/chenjun2hao/ddrnet/main.py", line 115, in <module>
    main(args)
  File "/data/swook/repos/chenjun2hao/ddrnet/main.py", line 99, in main
    solver.train()
  File "/data/swook/repos/chenjun2hao/ddrnet/sod/solver.py", line 123, in train
    sal_pred = self.net(sal_image)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/swook/repos/chenjun2hao/ddrnet/lib/models/ddrnet_23_slim.py", line 324, in forward
    x_ = x_ + F.interpolate(
RuntimeError: The size of tensor a (34) must match the size of tensor b (33) at non-singleton dimension 3
Killing subprocess 6217
Traceback (most recent call last):
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/home/swook/.vscode-server/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/data/swook/miniconda3/envs/torch18csnet/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/data/swook/miniconda3/envs/torch18csnet/bin/python', '-u', '/data/swook/repos/chenjun2hao/ddrnet/main.py', '--local_rank=0', '--mode', 'train', '--cfg_path', './experiments/duts/ddrnet23_slim_poolnet_train_scheme.yaml']' returned non-zero exit status 1.

System Info

PyTorch version: 1.8.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.3 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.5.1

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torchvision==0.9.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py38h27cfd23_1  
[conda] mkl_fft                   1.3.0            py38h42c9631_2  
[conda] mkl_random                1.2.1            py38ha9443f7_2  
[conda] numpy                     1.20.2           py38h2d18471_0  
[conda] numpy-base                1.20.2           py38hfae3a4d_0  
[conda] pytorch                   1.8.1           py3.8_cuda10.2_cudnn7.6.5_0    pytorch
[conda] torchvision               0.9.1                py38_cu102    pytorch

Inference feature for random data

Is your feature request related to a problem? Please describe.

• We need an inference feature for deployment
• Currently, chenjun2hao/DDRNet.pytorch provides a demo
• However, it's based on the specific datasets
• E.g. It requires a cityscapes meta-data including filename list
• It's not appropriate for deployment
• We need the inference feature w/o any specific datasets

Describe the solution you'd like

1. Implement a class based on torch.utils.data.TensorDataset
   • Assume that we follow official documentation
   • We have to initialize a instance every time new data arrives
2. Implement an inference feature w/o any torch.utils.data.Dataset

Describe alternatives you've considered

1. Use official repository for DDRNet
   • It only provide model codes

Modify the architecture of DAPPM considering the spatial size of the feature maps from generated by DAPPM

Is your feature request related to a problem? Please describe.

• DAPPM performs large pooling kernels and exponential strides to:

1. enlarge effective receptive fields
2. fuse multi-scale context

• See four different sub-branches which include pooling layer (red rectangle) from Fig. 5.

  name	operation	down-sampling rate
  scale1	AvgPool2d(kernel_size=5, stride=2, padding=2)	128
  scale2	AvgPool2d(kernel_size=9, stride=4, padding=4)	256
  scale3	AvgPool2d(kernel_size=17, stride=8, padding=8)	512
  scale4	AdaptiveAvgPool2d((1, 1))	(H, W)

• DAPPM down-samples an input image by factors of 128, 256 and 512

• However, recall that DDRNet is trained and benchmarked on two datasets:

1. Cityscapes
2. CamVid

• The resolutions of their frames are (2048, 1024) and (1024, 1024), respectively
• In those cases, scale3 outputs feature-maps with spatial size of (4, 2) and (2, 2), respectively
• But assume the input images are <= (512, 512)
• Then, scale3 and scale4 outputs feature-maps with spatial size of (1, 1)
• It means they have almost same receptive fields
• I.e. Redundant maybe?
• If the input images are small, wouldn't it be better to eliminate some poolings in DAPPM considering the receptive field?
• Decide after seeing if the existing one is trained well

Describe the solution you'd like

• If the input images are small, wouldn't it be better to eliminate some poolings in DAPPM considering the receptive field?
• Decide after seeing if the existing one is trained well

Describe alternatives you've considered

• None

Additional context

• DUTS-TR is a dataset for salient object detection
• We'd like to train DDRNet on this dataset for real-time salient object detection
• Refer to #6 from swoook/ucnet (github) for more details