snoop2head/elimnet

🔪 Elimination based Lightweight Neural Net with Pretrained Weights

Capture Questions, Answer with Code

elimnet's Issues

Create WandB Log to compare between models' performances

Divide model directory by model

Now: model/resnet18.yaml

-> Proposal: model/resnet18/resnet18.yaml

Illustrate performance gain of each pretrained models & how each models got lighter

For each pretrained models, display

• wandb log
• table which consists of maximum f1 score in 100 epochs and FLOPS(either inference time)
• percentage gain in both metrics of f1 score and FLOPS(either inference time)

Apply FP16 for pretrained efficientnetb0 experimentation

Test on pretrained mobilenetv3 and on vgg11

• Eliminate conv blocks from pretrained mobilenet_v3_large
• Eliminate conv blocks from pretrained vgg11
  • Set baseline vgg11 as vgg11 where fully connected layers are dropped

Experiment conditions are the same as the following:

• Augment: None
• Optimizer: SGDP(momentum = 0.9)
• (Initial) Learning Rate: 0.01
• Scheduler: OneCycleLR
• Epochs: 100
• Best Model Selection(Save strategy): Validation Loss
• FP16: True

Use torchvision version 0.11.0

• Project needs to use torchvision 0.11.0 where it provides follwowing pretrained models.

import torchvision.models as models

mobilenet_v3_large = models.mobilenet_v3_large()
mobilenet_v3_small = models.mobilenet_v3_small()
efficientnet_b0 = models.efficientnet_b0()

• torchvision 0.11.0 requires version above 1.9.0.
• However, problem arose in inference.py when using torch version above 1.8.0
  https://github.com/snoop2head/ELimNet/blob/590152df6fd0cfa63718af581ca7c2d1956acb50/inference.py#L22-L25
• Currently using torch==1.7.0 and torchvision==0.10.0 as a workaround

Check the discrepancy between two resnet18 implementations

Performance between these code blocks are different.
https://github.com/snoop2head/ELimNet/blob/799a5c25199b925dc3bf035d95c81a5a9765ba0c/src/modules/resnet18.py#L9-L52

https://github.com/snoop2head/ELimNet/blob/799a5c25199b925dc3bf035d95c81a5a9765ba0c/train_resnet_elim.py#L31-L59

Enable layer numbers based elimination from input arguments

• Currently supports output channel based elimination
• In order to fulfill the goal of lightweight model(less layers and less FLOPS), enable layer numbers based elimination for each pretrained models
• It is observed that deleting more than 4 convolutional blocks on pretrained efficientnetb0 and on mobilenet_v3_large degrades models' performance.

Repeat inference experiments to obtain confidence interval of elapsed time

For the inference experiments, get following times for each pretrained models

• Data Loading
• Profiling
• Forward passing

Repeat the inference experiments since time elapsed has discrepancy according to CPU cores utilized through time.
Suggested by @hihellohowareyou

Display distribution of primary indices for lightweight models

• inference time(CUDA and CPU time)
• FLOPS
• number of hyperparams
• number of layers