khurramhashmi/featenhancer

[ICCV 2023] FeatEnHancer: Enhancing Hierarchical Features for Object Detection and Beyond Under Low-Light Vision

[ICCV 2023] FeatEnHancer: Enhancing Hierarchical Features for Object Detection and Beyond Under Low-Light Vision

---

Enhancing representation of a low-light image.

---

Installation

Please refer to low-light-object-detection-detectron2 for installation requirements

Datasets

ExDark

Create a new folder named "exdark" in the "low-light-object-detection-detectron2/data" folder. Create a new folder named "exdark" in the "low-light-object-detection-mmdetection/data" folder.

Download the ExDark dataset and copy the images into "low-light-object-detection-detectron2/data/exdark/images/" and "low-light-object-detection-mmdetection/data/exdark/images/" folders.

DARK FACE

Create a new folder named "darkface" in the "low-light-object-detection-detectron2/data" folder. Create a new folder named "darkface" in the "low-light-object-detection-mmdetection/data" folder.

Download the DARK FACE dataset and copy the images into "low-light-object-detection-detectron2/data/darkface/images/" and "low-light-object-detection-mmdetection/data/darkface/images/" folders.

Train

To train the ExDark and DARK FACE using FeatEnHancer based Featurized Query R-CNN run the following commands: The training utilizes 2 GPU's

sh low-light-object-detection-detectron2/train_exdark.sh
sh low-light-object-detection-detectron2/train_darkface.sh

To train the ExDark and DARK FACE using FeatEnHancer based RetinaNet run the following commands: The training utilizes 6 GPU's

sh low-light-object-detection-mmdetection/exec_script_exdark.sh
sh low-light-object-detection-mmdetection/exec_script_darkface.sh

Results and Checkpoints

ExDark

Model	mAP	Config
FeatEnHancer + Featurized Query R-CNN	86.3	config

DARK FACE

Model	mAP	Config
FeatEnHancer + Featurized Query R-CNN	69.0	config

Reproducing Results on Other Downstream Vision Tasks:

• The models developed for other downstream tasks, such as Semantic Segmentation and Video Object Detection, utilize distinct frameworks (MMDet, MMSeg, and MMTracking). Hence, it was not possible to release a unified repository at this time. However, to facilitate reproducibility of results, the same FeatEnHancer script can be employed across these different tasks.

Acknowledgment

This work would not be possible without the following codebases. We gratefully thank the authors and collaborators for their wonderful works:
Featurized Query R-CNN, detectron2, mmdetection, mmsegmentation, and mmtracking

License

The proposed FeatEnHancer is released under the Creative Commons Attribution-NonCommercial 4.0 International Licence.

Citation

If you find FeatEnHancer useful in your research or applications, please consider giving us a star ⭐ and citing it by the following BibTeX entry.

@InProceedings{FeatEnHancer_Hashmi_ICCV23,
    author    = {Hashmi, Khurram Azeem and Kallempudi, Goutham and Stricker, Didier and Afzal, Muhammad Zeshan},
    title     = {FeatEnHancer: Enhancing Hierarchical Features for Object Detection and Beyond Under Low-Light Vision},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {6725-6735}
}

featenhancer's Issues

more details about the data

Hi, the format of the dataset used in this project is different to the original one. Would you please share the annotation files. Thanks a lot !

about datasets

Hi,bro,how can i get the train sets and tests sets?i dont know the right sets image list.can you provide the train and test sets json file? or the datasets splits image list.
thank you!

"self.scale_aware_aggregation = ScaleAwareFeatureAggregation(channels=32).to('cuda')"

Hi,
Thanks for your insteresting work! But when I run the "FeatEnHancer", the "channels" of"self.scale_aware_aggregation = ScaleAwareFeatureAggregation(channels=32).to('cuda')" is wrong? Because the "out_channels=32". And the result of "self.quarter_conv" is [B, int_out_channels, H/4-1,W/4-1]?

Best wishes!

This is a very interesting job

could you share the checkpoints of model_darkface

Confusing training efficiency

This study is intriguing, and I am reproducing it using the provided training scripts. However, the training efficiency of FQ R-CNN is surprisingly low due to the absence of pre-trained weight files. Below is a log file documenting my training from scratch on the exdark dataset.
my environment

-------------------------------  --------------------------------------------------------------------------------------------
sys.platform                     linux
Python                           3.7.16 (default, Jan 17 2023, 22:20:44) [GCC 11.2.0]
numpy                            1.21.6
detectron2                       0.6 @/home/jjk/workspace/REF/detectron2/detectron2
Compiler                         GCC 11.4
CUDA compiler                    not available
DETECTRON2_ENV_MODULE            <not set>
PyTorch                          1.10.0+cu111 @/home/jjk/miniconda3/envs/featenhancer/lib/python3.7/site-packages/torch
PyTorch debug build              False
torch._C._GLIBCXX_USE_CXX11_ABI  False
GPU available                    Yes
GPU 0                            NVIDIA GeForce RTX 3080 Ti Laptop GPU (arch=8.6)
Driver version                   546.80
CUDA_HOME                        None - invalid!
Pillow                           9.5.0
torchvision                      0.11.0+cu111 @/home/jjk/miniconda3/envs/featenhancer/lib/python3.7/site-packages/torchvision
torchvision arch flags           /home/jjk/miniconda3/envs/featenhancer/lib/python3.7/site-packages/torchvision/_C.so
fvcore                           0.1.5.post20221221
iopath                           0.1.9
cv2                              3.4.2
-------------------------------  --------------------------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

i use coco format exdark dataset. download from https://github.com/cs-chan/Exclusively-Dark-Image-Dataset
after 39999 iter it got

|  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|
| 0.004 | 0.019  | 0.000  | 0.000 | 0.000 | 0.005 |
[08/18 22:05:31] d2.evaluation.coco_evaluation INFO: Per-category bbox AP: 
| category   | AP    | category   | AP    | category     | AP    |
|:-----------|:------|:-----------|:------|:-------------|:------|
| person     | 0.016 | bicycle    | 0.000 | car          | 0.000 |
| motorcycle | 0.000 | bus        | 0.000 | boat         | 0.000 |
| cat        | 0.000 | dog        | 0.000 | bottle       | 0.000 |
| cup        | 0.000 | chair      | 0.010 | dining table | 0.019 |

it has low mAP after hours of training. I also tried another dataset and get similar result.
I am very interested in the implementation of this paper, it helped me a lot
Additionally, I would appreciate it if you could provide the pre-trained weights for exdark.

RuntimeError: CUDA error: device-side assert triggered

when i try to run train_exdark.sh, it occurred:

line 51, in generalized_box_iou
assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Can you provide some suggestion to fix it?

installation prob

bro, I clicked the "installation" instruction link, but got 404 error. So I cant reproduce your code, I'm confused

can you provide the right link so that I can reproduce your code.

thank you