Performance Issue

[wuhuikai]

Performance Issue

[wuhuikai]

In the same environment, I got 0.5105 for single-scale.
Can you provide your scripts, running log and sha1sum of the model?

[tonysy]

The sha1sum of the model_best.pth.tar is: d581b28ac711fda74100529319d4853041ba0e2b
I use the script in your project:

#train
CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py --dataset pcontext \
    --model encnet --jpu --aux --se-loss \
    --backbone resnet50 --checkname encnet_res50_pcontext

#test [single-scale]
CUDA_VISIBLE_DEVICES=0,1,2,3 python test.py --dataset pcontext \
    --model encnet --jpu --aux --se-loss \
    --backbone resnet50 --resume {MODEL} --split val --mode testval

#test [multi-scale]
CUDA_VISIBLE_DEVICES=0,1,2,3 python test.py --dataset pcontext \
    --model encnet --jpu --aux --se-loss \
    --backbone resnet50 --resume {MODEL} --split val --mode testval --ms

The script didn't store the log and I'am re-training it again for saving the logs.

Maybe you can provide a Dockerfile to make sure we can get the same performance for verification.

[tonysy]

Another issue about the FPS measurement.
I have tried the test_fps_params.py for FPS measurement.
I use

CUDA_VISIBLE_DEVICES=0 python test_fps_params.py --dataset pcontext \
    --model encnet --jpu --aux --se-loss \
    --backbone resnet50

get the following results:

which is similar to your reported FPS.
But when I remove the --jpu parameters in the same machine, I got:

where the result is 78 FPS, different from the 18.77 FPS. I wonder is there anything wrong in my experiment? Or you use different repo to report the FPS of the original EncNet?

[wuhuikai]

1. For the original EncNet, the script is --model encnet --dilated --lateral.
2. For the performance, can you first download the pre-trained model and verify its performence ?
3. I'll test the training script myself and tell you the result later.

[wuhuikai]

@tonysy I realized that there's a bug in this version's code. This bug will be fixed soon.

[wuhuikai]

I got a mIoU 51.27% with the updated code.
Here's the log for training and evaluation.

[tonysy]

Thanks for your update. While, I think the performance of EncNet reported in your work is not accurate, which is not fair for comparison. EncNet-Res50 is 51.0% and EncNet-Res101 is 54.1% reported by the author.
https://hangzhang.org/PyTorch-Encoding/experiments/segmentation.html

[wuhuikai]

We cannot reproduce the performance reported in https://hangzhang.org/PyTorch-Encoding/experiments/segmentation.html. The performance I report is reproduced with the official code.

[HolmesShuan]

• @tonysy In https://hangzhang.org/PyTorch-Encoding/experiments/segmentation.html#test-pre-trained-model

The test script is in the experiments/segmentation/ folder. For evaluating the model (using MS), for example Encnet_ResNet50_PContext:

using MS may explain the performance gap.

• @wuhuikai Could you please further release the code to reproduce the results of Table 3, i.e., mIoU measured on 60 classes w/ background. Thanks in advance.

[wuhuikai]

@HolmesShuan For 60 classes, see here.

[HolmesShuan]

Table 1 results should be higher than Table 3, right? But 51.2 (Table 3) is higher than 51.05 (Table 1). Did I miss anything?

[wuhuikai]

@HolmesShuan Table 3 is evaluated with MS.