naver-ai/rdnet

[ECCV2024] Official implementation of paper, "DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs".

🎉🎉 Our paper has been accepted at ECCV 2024! Stay tuned for more updates !! 🎉🎉

DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs

Donghyun Kim^1*, Byeongho Heo², Dongyoon Han^2*

¹NAVER Cloud AI, ²NAVER AI Lab

We revitalize Densely Connected Convolutional Networks (DenseNets) and reveal their untapped potential to challenge the prevalent dominance of ResNet-style architectures. Our research indicates that DenseNets were previously underestimated, primarily due to conventional design choices and training methods that underexploited their full capabilities.

Tradeoff with RDNet (ours) and SOTA models

Tradeoff with RDNet (ours) and mainstream models

Key Highlights:

• Pilot study (§5.1) reveals concatenations' effectiveness.
• We have meticulously upgraded various aspects of DenseNets (§3.2) through architectural tweaks and block redesigns.
• Our revitalized DenseNets (RDNets) outperform mainstream architectures such as Swin Transformer, ConvNeXt, and DeiT-III (§4.1).

Our work aims to reignite interest in DenseNets by demonstrating their renewed relevance and superiority in the current architectural landscape. We encourage the community to explore and build upon our findings, paving the way for further innovative contributions in deep learning architectures.

We believe that various architectural designs that have been popular recently would be combined with dense connections successfully.

Easy to use

RDNet is available on timm. You can easily use RDNet by installing the timm package.

import timm

model = timm.create_model('rdnet_large', pretrained=True)

For detailed usage, please refer to the huggingface model card.

Updates

• (2024.07.24): Pip installable pacakge added.
• (2024.04.19): Initial release of the repository.
• (2024.03.28): Paper is available on arXiv.

Coming Soon

• More ImageNet-22k Pretrained Models.
• More ImageNet-1k fine-tuned models.
• Cascade Mask R-CNN with RDNet.
• Transfer Learning with RDNet (with cifar10, cifar100, stanford cars, ...).

RDNet for Image Classification

For details on object detection and instance segmentation, please refer to detection/README.md.

For details on semantic segmentation, please refer to segmentation/README.md.

Model Zoo

We provide the pretrained models for RDNet. You can download the pretrained models from the links below.

ImageNet-1K (pre-)trained models

Model	IMG Size	Params	FLOPs	Top-1	Model Card	url
RDNet-T	224	22M	5.0G	82.8	model_card	HFHub
RDNet-S	224	50M	8.7G	83.7	model_card	HFHub
RDNet-B	224	87M	15.4G	84.4	model_card	HFHub
RDNet-L	224	186M	34.7G	84.8	model_card	HFHub

ImageNet-1K fine-tuned models

Model	fine-tune from	IMG Size	Params	FLOPs	Top-1	Model Card	url
RDNet-L (384)	RDNet-L	384	186M	101.9G	85.8	model_card	HFHub

Training

We provide the graphs of the training procedure. The graph is generated by the Weights & Biases service. You can check the graph by clicking the link below.

https://api.wandb.ai/links/dhkim0225/822w2zsj

For training commands, please refer to the TRAINING.md.

Acknowledgement

This repository is built using the timm, MMDetection, and MMSegmentation.

Citation

@misc{kim2024densenets,
    title={DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs}, 
    author={Donghyun Kim and Byeongho Heo and Dongyoon Han},
    year={2024},
    eprint={2403.19588},
    archivePrefix={arXiv},
}

License

Copyright (c) 2024-present NAVER Cloud Corp.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

License for Dataset

ImageNet - ImageNet Terms of access, https://image-net.org/download

Images from ADE20K - ADE20K Terms of Use, https://groups.csail.mit.edu/vision/datasets/ADE20K/terms/

MS COCO images dataset - Creative Commons Attribution 4.0 License, https://viso.ai/computer-vision/coco-dataset/

rdnet's Issues

Pretrained model in timm

Thanks for the great work!

Any plans to have pretrained models in timm repo so it can be used as an arbitrary backbone/embedder?

Where can I find the CIFAR-10 training code?

I discovered this repository through the "Papers with Code" page for CIFAR-10 image classification. However, it appears that this repository primarily addresses the ImageNet dataset. Could you provide the CIFAR-10 training code to assist with my research?Thanks.

To many Total mult-adds nums.

Thank you for great research!

model_t = timm.create_model('rdnet_tiny.nv_in1k', pretrained=False)
summary(model_t, input_size=(1, 3, 224, 224), device='cpu')

Total params: 23,541,024
Trainable params: 23,541,024
Non-trainable params: 0
Total mult-adds (Units.GIGABYTES): 78.87
Input size (MB): 0.60
Forward/backward pass size (MB): 2651.23
Params size (MB): 94.15
Estimated Total Size (MB): 2745.99

your paper assert that tiny model's FLOPs is 5GB. but it has 78GB.
Of course, i might not estimate right method.
But, i compare FLOPs with another models. than rdnet has much more.

I think, this code has some junk behavior. What do you think about this cause?

Thanks.

Please provide the versions of all installation packages, especially timm

Please provide the versions of all installation packages, especially timm

About downstream task and DropPath

Thank you for your great ideas and work.

I have currently re-implemented RDNet Backbone with Tensorflow, and I am trying to apply it to Human Pose Estimation(HPE) as a downstream task and I am asking you because I have the following questions:

1. From what I understand after reading RDNet paper, RDNet backbone uses Stochastic Depth(DropPath), but RDNet's DenseBlock class only declares DropPath and doesn't seem to use it. Is this the part that was intended? If yes, please explain why you didn't use it.
2. As a downstream task, I am trying to train a model by applying RDNet to the HPE task, but I have confirmed that the model is not trained well. I have tried several times by modifying hyperparameters such as batch size, LR, etc., but the model is not trained well. As a curiosity about this, could the training be sensitive or not trained well when applying RDNet Backbone to the downstream task?
   The conditions used in training are as follows:
   • Backbone: RDNet-Tiny (pretrained: X)
   • LR Scheduler: MultiStepLR
   • LR: 0.001 (also tried lower LR)
   • batch size: 64 (also tried 16, 32)
   • optimizer: Adam

---

한국어 버전(Korean Translation)

RDNet이라는 좋은 아이디어와 결과물을 공유 해주셔서 감사합니다.

현재 저는 RDNet Backbone을 Tensorflow로 재 구현하여, downstream task로 Human Pose Estimation(HPE)에 적용하고자 하는 상황 중에 궁금한 점이 생겨 아래의 질문들을 문의드립니다:

1. RDNet 논문을 이해하기로 RDNet Backbone은 Stochastic Depth(DropPath)를 사용하는 것으로 알고 있습니다. 하지만 RDNet의 DenseBlock 클래스에서 DropPath를 선언만 하고 사용하지 않는 것으로 보입니다. 이 부분이 의도된 것이 맞는 문의드립니다. 만약, 의도된 사항이 맞다면 그 이유도 설명 부탁드립니다.
2. downstream task로 RDNet Backbone을 HPE task에 적용하고자 시도한 결과, 모델의 학습이 잘 이루어지지 않음을 확인했습니다. batch size나 LR과 같은 hyperparamter도 수정하여 여러 번 시도 해봤지만 이 역시 학습이 잘 되지 않았습니다. 이에 대한 궁금증으로 RDNet Backbone을 downstream task에 적용할 경우, 학습이 민감해지거나(hyperparameter 등으로 인해), 학습이 잘 안될 수도 있을지 궁금합니다:
   학습 때 적용한 조건들은 다음과 같습니다.
   • Backbone: RDNet-Tiny (pretrained: X)
   • LR Scheduler: MultiStepLR
   • LR: 0.001 (also tried lower LR)
   • batch size: 64 (also tried 16, 32)
   • optimizer: Adam