woosukkwon/retraining-free-pruning

[NeurIPS 2022] A Fast Post-Training Pruning Framework for Transformers

retraining-free-pruning's Issues

About the speedup performance of the code

Hi @WoosukKwon , in your paper. You use the code outputs = model(head_mask=head_mask, **batch) to get the output. So model is the same model, we change the head_mask to do pruning right? But does it really save time? I mean we actually do all the forward of bert right? So we do not change the model architecture?

Many thanks!

Can this be used for detection, segmentation, and other tasks?

dependency package versions

Hi, thanks for the great work! When I tried out to reproduce your work, some errors appeared. I suppose it due to the incorrect package versions. Could you specify the versions in the 'requirements.txt' file to help me out?

Question about the pruned model.

Hi @WoosukKwon,

Thanks for your excellent work.
I have a question about the smaller dense model, BertMHA and BertFFN in generate_lut.py are initialized via config, config = AutoConfig.from_pretrained(args.model_name), whether the args.model_name is the smaller dense model? If so, could you please point out how can I get the pruned model with redundant structure removed?

Looking for your reply!

What is the purpose of setting "encoder.layers" and how does it differ from "encoder.layer" ?

https://github.com/WoosukKwon/retraining-free-pruning/blob/806ac5a6ff53b4978e5330d45a3de692493e4d0b/utils/arch.py#LL107C9-L107C36

Real-time inference results

Hi @WoosukKwon,

Thanks for your code and your help for my previous question. I have another question about the real-time inference speed.

Like shown in Fig.3 in paper, the efficiency has been improved. I am wondering if this figure is plotted using some python functions like "time.time()" to record the operation time consumption? or estimated by some calculation functions like "MAC" or "Latency" provided in this repo?

Thanks!

Missing datasets file?

Hi Woosuk!

Thanks for sharing your code! I tried to run the example script but got:

from datasets import load_dataset
ModuleNotFoundError: No module named 'datasets'

It looks like the 'datasets' file is missed? Could you please help with it?

Thanks!

Test Accuracy function is a bit too slow

retraining-free-pruning/evaluate/nlp.py

Line 10 in 806ac5a

def test_accuracy(model, head_mask, neuron_mask, tokenizer, task_name):

Hi there, thanks for sharing the coding for retraining-free-pruning, I tried to pruning bert-large model, and found the test_accuracy function is too slow for the squad dataset, any suggestion or guidance how can we speed it up?

Use your Code for other classification datasets

Hi,

I want to test your code for some of my models that are fine tuned to a german classification dataset. What would be your idea to get your code/results to work for other classification datasets with a different language?

Thank you very much in advice!

Good Job! Can this framework work for the Transformer-based vision models?

Hi, can this "the three-stage decomposition of the pruning process" be applied to GPT-X or any other NLG task? And how this could be done?

GLUE & SQuAD Metadata

GLUE Benchmark Metadata

Task	MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE
Train set size	393K	364K	105K	67K	8.5K	7K	3.7K	2.5K
Dev set size	20K	40K	5.4K	0.8K	1K	1.5K	0.4K	0.3K
Test set size	20K	391K	5.4K	1.8K	1K	1.4K	1.7K	3K
Med. sequence length (train)	38	28	48	11		24	54	57
Med. sequence length (dev)	37	28	45	25		29	54	54

SQuAD Benchmark Metadata

Task	SQuAD V1.1	SQuAD 2.0

GLUE dev set Evaluation time (in sec)

Task	MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE
V100 (FP32, batch 128)	24.39	76.39	14.43	1.41		2.31	1.09

SQuAD Evaluation time (in sec)

Task	SQuAD V1.1	SQuAD 2.0
V100 (FP32, batch 128)	110.11	115.59

SlowFast and TimesFormer?

Can this model be used for pruning training with SlowFast and TimesFormer?

Any experiments on NLG tasks?

Hi, I just notice only NLU tasks are exposed. I am wondering if there was or will be any generation tasks like LAMBDA/WIKITEXT?

why bert-base-uncased model set constraint to 0.5, qqp test accuracy only 0.3743

I download the bert-base-uncased from https://huggingface.co/bert-base-uncased
and I execute the main.py like the example

python3 main.py --model_name bert-base-uncased \
                --task_name qqp \
                --ckpt_dir <your HF ckpt directory> \
                --constraint 0.5

But I just get the qqp Test accuracy is only 0.3743

And the log like this
12/13/2022 08:27:08 - INFO - main - Pruned Model MAC: 50.00 %
12/13/2022 08:29:00 - INFO - main - qqp Pruning time (s): 133.1617510318756
12/13/2022 08:34:51 - INFO - main - qqp Test accuracy: 0.3743