eidoslab/simplify

Simplification of pruned models for accelerated inference | SoftwareX https://doi.org/10.1016/j.softx.2021.100907

simplify's Issues

Output dimension of nn.Linear doesn't stay pinned

I am considering using your simplify package in one of my project. However, I run into an issue during a simple test of mine. I cannot get the output to stay the same size. Do you know how I need to adapt my example to get it working?

My code:

class MyNetwork(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.first = torch.nn.Linear(5, 5)
        self.second = torch.nn.Linear(5, 5)

    def forward(self, inputs):
        return self.second(torch.relu(self.first(inputs)))

model = MyNetwork()

# set some rows of the weight matrices to zero
for m in model.modules():
    if isinstance(m, torch.nn.Linear):
        w = m.weight.data
        mask = torch.bernoulli(0.8 * torch.ones(size=(w.shape[0], 1), device=w.device))
        m.weight.data.mul_(mask)

dummy_input = torch.zeros([1,5])  # Tensor shape is that of a standard input for the given model
model(dummy_input)

to_be_modified_model = copy.deepcopy(model) # deep copy so that simplify doesn't change the original model
simplified_model = simplify(to_be_modified_model, dummy_input, pinned_out=['second', to_be_modified_module.second]) # docs are unclear about what `pinned_out` should be, so just add module name and module object

Expected behavior

Both the simplified model and the original model give the same result all the time, i.e.

print(simplified_module(torch.Tensor([2,2,2,2,2])))
print(module(torch.Tensor([2,2,2,2,2])))

matches in dimension and size.

Actual behavior

print(simplified_module(torch.Tensor([2,2,2,2,2])))
print(module(torch.Tensor([2,2,2,2,2])))

gives inconsistent results. The simplified module gives the same results for all indices corresponding to nonzero rows of the second weight matrix, but the indices corresponding to the zero rows of the second weight matrix are missing. As an example:

for a second layer having

as weight matrix and bias vector.

Accuracy drop after simplify

Hi guys, thank you for your library. It is quite cool.

However, I have met a problem when I try to use Simplify with Resnet18 on CIFAR10 dataset. I have used global_unstructured pruning in Pytorch and the sparsity was set to 0.875. Then I have transfer the weights of a list of neurons in layer4.1.Conv1 (Penultimate layer) to zero. In the next, I implemented the Simplify. The test_acc after “pruning” is 92.6%, after “pruning + transfer neurons’ weight to 0” is 92.24%, after “pruning + transfer neurons’ weight to 0 + simplify” is 92.21%.

I think there was supposed to be no test_acc decrease after the Simplify process. Could you please help me understand what happens here? I hope I have described my problem well.

If you want to reproduce, here is the link of my code and required files.
https://partage.imt.fr/index.php/s/9jemn7WfkBWS5tx

Runtime error when simplifying model

Hi! I'm working on model compression, and when running simplify on a pretrained efficientnet_lite0 model, I have run into some issues. I'm trying to run it all on Conv2D layers after pruning with pytorch. It seems like when I use simplify, it does reduce the number of parameters, but doesn't adjust the number of input/output channels correctly when compressing. I know there is a pinned_out argument, but it seems that almost every conv layer would have to be put in that list based on what I'm seeing, which defeats the purpose of calling simplify, as it would skip over those layers. An example image is shown below of the difference between the models before and after calling simplify:

The error I get is this:

That is when I call summary on the simplified model, when passing in the same input size. The same error occurs when trying to pass data into the model. I tried reshaping the input, but wasn't able to do so successfully to make the model happy. I was wondering if I was using the library wrong, or if this is just something that is not covered. Any help would be great, thanks!

A copy of the colab notebook that can be run and edited is here: https://colab.research.google.com/drive/1AdVbpT-wyhMD_wcAqrL42rshtGc8wRFa?usp=sharing

Simplify with Squeeze Net

Hi guys, congrats for the work, nice library.

I am trying to use Simplify with SquezeetNet.
I fine-tuned a pre-trained SqueezetNet from torchvision, fused some layers and pruned it with structured mode.

But when I use the library, it seems nothing changes.
For example, after structured prune (50%), the model has 361_216 parameters equals to zero. After simplify, it still has 361_216.

Do you have any advice to use the library with SquezeeNet?

Big accuracy drop after simplify

Hi guys, congrats for the work, nice library.

I am trying to use Simplify with ResNet on CIFAR-100 dataset. I use prune.ln_structured method from Pytorch with amount value is 0.8 and then apply simplify and do fine-tune of the model. As a result I have got the inference time speed up by two times. But I have big accuracy drop: 75% accuracy before prune.ln_structured + simplify + fine-tune and 65% after prune.ln_structured + simplify + fine-tune. Is it expected result? Have you checked accuracy before and after simplify in your cases?

Not work for Swin_T model

Thank you guys for your library. It's a quite convenient tool.

However, when I want to implement simplify with Swin_T model, it seems it does not work.

To illustrate the problem, I've shared a simplified version of the code in the following link. Could you kindly review it and assist me in identifying a solution to rectify this issue? Your support would be greatly appreciated.
https://partage.imt.fr/index.php/s/9jemn7WfkBWS5tx

Thanks a lot.

How to speed up the inference time?

My code is:

model = models.resnet18(pretrained=True)
dummy_input = torch.zeros(64, 3, 224, 224)  # Tensor shape is that of a standard input for the given model
simplified_model = simplify(model, dummy_input)

def run_model(model, dummy_input):
    start = time.time()
    for _ in tqdm(range(100)):
        model(dummy_input)
    end = time.time()
    print("Time taken: ", end - start)

print("Original model")
run_model(model, dummy_input)

print("Simplified model")
run_model(simplified_model, dummy_input)

The output is:

Original model
100%|██████████| 100/100 [01:37<00:00,  1.03it/s]
Time taken:  97.2786557674408
Simplified model
100%|██████████| 100/100 [01:39<00:00,  1.01it/s]
Time taken:  99.11441326141357

It seems there is no acceleration in inference. Maybe there are not zero channels to be pruned in the pre-trained model.
I have a question about how to speed up the inference time.
Should I use the prune.ln_structured in torch.nn.utils.prune to prune the pre-trained model at first?
I think this is a good project to do the following work behind the torch prune.
Can you provide an entire example for accelerated inference?

Does transformer layer supported??

i am trying to pass transformer model, but encountering issues while passing model.
simplified_model = simplify(model, dummy_input,fuse_bn=False)

Am getting AssertionError
Does transformer layer is supported at the time??