arexh/bert-pruning

Implement pruning algorithm on BERT.

bert-pruning's Issues

Prune parameters in a given layer

Hi,

I am trying remove some filters in a Conv2D layer based on their l2 norm values.

Ex:
If a Conv2D layer has 32 filters with size (3,3,3,32) 3x3 is size of the kernel, 3 is repeating gray scale image and 32 is number of filters

Can i remove some filters and make the layer shape (3,3,3,30) with your codebase

I dint really go through your codebase

Thanks

help me

Hi,
Thank you so much for publishing your code. I download your code and run it step by step (run_all.sh, factorize.sh, run.sh, remove_mask.sh) on the sst-2 task. The last step, 'finetune again', is not provided and I follow the settings in run_all.sh. But the result is unsatisfying (dev acc =83%). And when I use your method on some smaller datasets (e.g., RTE, CoLA), the actual_compact_rate (after removing mask) is larger than 1. (RTE=1.36). Could your please provide more details of sst-2 task about the last step or tell me the most possible reasons, if it is convenient for you. I would really appreciate it.
Sincerely.

BertConfig object has no attribute pruned_layers_dim

I ran

bash factorize.sh

factorize.sh is:

python ./flop/factorize.py \
  --bert_config_file=./uncased_L-12_H-768_A-12/bert_config.json \
  --checkpoint=/workspace/BERT-Pruning/output/fine_tune_outputs/SST-2/lr_3e-5/model.ckpt-6313 \
  --output_dir=./uncased_L-12_H-768_A-12_SST-2_f/bert_model_f.ckpt \
  --finetuned

I got the following error

Traceback (most recent call last):
  File "./flop/factorize.py", line 135, in <module>
    finetuned=args.finetuned)
  File "./flop/factorize.py", line 69, in save_factorized_model
    finetuned=finetuned)
  File "./flop/factorize.py", line 46, in create_model
    factorize=True)
  File "/workspace/BERT-Pruning/flop/modeling_flop.py", line 178, in __init__
    pruned_layers_dim=config.pruned_layers_dim)
AttributeError: 'BertConfig' object has no attribute 'pruned_layers_dim

Any comments / help would be really appreciated :)