zju-vipa / cmi Goto Github PK
View Code? Open in Web Editor NEW[IJCAI-2021] Contrastive Model Inversion for Data-Free Knowledge Distillation
[IJCAI-2021] Contrastive Model Inversion for Data-Free Knowledge Distillation
Greetings,
Is it possible to use your codebase on BDD100K dataset ? I want to use teacher as Resnet-50 and my own architecture for Student.
Thanks
Hi, thank you for this great job
Can you publish the training details about the teacher model? For example, ResNet34. I can't get a very clear inversion picture using my own trained ResNet18 and 34. Is it because you used special training techniques when training the pre-training model or the hyperparameters of CMI need to be adjusted accordingly. If the latter, can you provide some guidance on how to tune hyperparameters?
Thanks for your impressive work and code. Could you provide pretrained model about cifar100?
I replayed the code from pre-training and got the following results:
Model | Method | Dataset | top1-accuracy |
---|---|---|---|
ResNet34* | Teacher | Cifar-10 | 93.94(95.70) |
wrn_40_2 | Teacher | Cifar-10 | 92.01(94.87) |
For DFQ algorithm, the results are as follows:
Model | Data-Free Method | Student-Loss | Generative-Loss | Dataset | top1-accuracy |
---|---|---|---|---|---|
ResNet34-ResNet18 | DFQ(Baseline) | KL | adv+bn+oh | Cifar-10 | 88.89(94.61) |
ResNet34-ResNet18 | DFQ(Baseline) | KL | adv+bn+oh | Cifar-100 | 1.89(77.01) |
For the above results, I reproduce DFQ based on your code for some reason. But the results are poor. I would be grateful if you could kindly give me your advice.
I ran the code according to the "Scripts" of DFQ.
Teacher - wrn_40_2, Student - wrn_16_1, dataset - CIFAR10
But the result is too much outstanding. The top1 accuracy in testset is more than 92%, which is even better than the result of cmi in the paper. I'm wondering if this is a coincidence or there are something wrong in the code?
Thx!
Hi~
Thank you for this great work.
My question is about the TV loss. Could you give the reason why you took mean while calculating the TV loss?
The paper about that work did not mention the 'mean'. Thank you
Line 48 in 9e79fa9
Can you provide the training super parameters of resnet34 and resnet18 on cifar10/cifar100?
This part is not mentioned in the paper. Again, there is no code in the repo to generate this data. Did I miss something?
Hi there~
Thank you for the fine work.
Can I know the exact torch version, please? I saw the "requirements.txt" file, but versions weren't specified.
I'm having some issues regarding it.
Thank you
I try to repruduce your impressive work. I just simply run 'bash scripts/cmi/adv_cmi_cifar10_wrn402_wrn161.sh' but get lower accuracy than paper:
Use GPU: 0 for training
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: method: cmi
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: adv: 0.5
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: bn: 1.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: oh: 0.5
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: act: 0.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: balance: 0.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: save_dir: run/adv_cmi_cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cr: 0.8
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cr_T: 0.1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: cmi_init: run/cmi-preinverted-wrn402
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: data_root: data
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: teacher: wrn40_2
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: student: wrn16_1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dataset: cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr: 0.1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr_decay_milestones: 25,30,35
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: lr_g: 0.001
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: T: 20.0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: epochs: 40
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: g_steps: 200
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: kd_steps: 2000
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: ep_steps: 2000
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: resume:
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: evaluate_only: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: batch_size: 128
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: synthesis_batch_size: 256
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: gpu: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: world_size: -1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: rank: -1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dist_url: tcp://224.66.41.62:23456
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: dist_backend: nccl
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: multiprocessing_distributed: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: fp16: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: seed: None
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: log_tag: -adv_cmi_cifar10
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: workers: 4
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: start_epoch: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: momentum: 0.9
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: weight_decay: 0.0001
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: print_freq: 0
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: pretrained: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: distributed: False
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: ngpus_per_node: 1
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: autocast: <function dummy_ctx at 0x7f61b2802560>
[08/27 20:15:02 cifar10-wrn40_2-wrn16_1]: logger: <Logger cifar10-wrn40_2-wrn16_1 (DEBUG)>
Files already downloaded and verified
Files already downloaded and verified
CMI dims: 2704
[08/27 20:20:43 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=0 Acc@1=23.6500 Acc@5=76.6300 Loss=4.2118 Lr=0.1000
[08/27 20:24:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=1 Acc@1=31.9100 Acc@5=81.9000 Loss=3.3887 Lr=0.0998
[08/27 20:27:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=2 Acc@1=36.4000 Acc@5=85.6800 Loss=3.0814 Lr=0.0994
[08/27 20:30:02 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=3 Acc@1=42.9800 Acc@5=88.9000 Loss=2.7402 Lr=0.0986
[08/27 20:32:29 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=4 Acc@1=45.4600 Acc@5=91.3300 Loss=2.6151 Lr=0.0976
[08/27 20:34:49 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=5 Acc@1=51.9300 Acc@5=93.0500 Loss=2.3906 Lr=0.0962
[08/27 20:37:04 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=6 Acc@1=52.5100 Acc@5=91.0300 Loss=2.4513 Lr=0.0946
[08/27 20:39:13 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=7 Acc@1=56.1100 Acc@5=94.8800 Loss=2.0773 Lr=0.0926
[08/27 20:41:20 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=8 Acc@1=58.4500 Acc@5=95.4500 Loss=2.1580 Lr=0.0905
[08/27 20:43:24 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=9 Acc@1=64.0700 Acc@5=96.5000 Loss=1.6752 Lr=0.0880
[08/27 20:45:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=10 Acc@1=61.5900 Acc@5=96.7800 Loss=1.9675 Lr=0.0854
[08/27 20:47:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=11 Acc@1=65.8900 Acc@5=96.1100 Loss=1.6139 Lr=0.0825
[08/27 20:49:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=12 Acc@1=66.5500 Acc@5=96.0800 Loss=1.6001 Lr=0.0794
[08/27 20:51:26 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=13 Acc@1=69.8900 Acc@5=97.6900 Loss=1.3662 Lr=0.0761
[08/27 20:53:24 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=14 Acc@1=66.6300 Acc@5=97.1500 Loss=1.5617 Lr=0.0727
[08/27 20:55:21 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=15 Acc@1=72.4200 Acc@5=98.0800 Loss=1.1806 Lr=0.0691
[08/27 20:57:19 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=16 Acc@1=72.5100 Acc@5=98.1300 Loss=1.2018 Lr=0.0655
[08/27 20:59:15 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=17 Acc@1=68.6100 Acc@5=98.2000 Loss=1.5399 Lr=0.0617
[08/27 21:01:11 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=18 Acc@1=74.9700 Acc@5=98.4700 Loss=1.0825 Lr=0.0578
[08/27 21:03:06 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=19 Acc@1=70.5000 Acc@5=97.9700 Loss=1.4037 Lr=0.0539
[08/27 21:05:01 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=20 Acc@1=74.3100 Acc@5=98.4500 Loss=1.1729 Lr=0.0500
[08/27 21:06:55 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=21 Acc@1=76.1000 Acc@5=98.3600 Loss=1.0403 Lr=0.0461
[08/27 21:08:49 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=22 Acc@1=76.4900 Acc@5=98.1800 Loss=1.0029 Lr=0.0422
[08/27 21:10:42 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=23 Acc@1=77.2500 Acc@5=98.5500 Loss=1.0258 Lr=0.0383
[08/27 21:12:34 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=24 Acc@1=76.9900 Acc@5=98.7200 Loss=0.9811 Lr=0.0345
[08/27 21:14:27 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=25 Acc@1=77.7800 Acc@5=98.5100 Loss=0.9825 Lr=0.0309
[08/27 21:16:19 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=26 Acc@1=77.1600 Acc@5=98.5800 Loss=1.0215 Lr=0.0273
[08/27 21:18:09 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=27 Acc@1=78.0100 Acc@5=98.4900 Loss=0.9878 Lr=0.0239
[08/27 21:20:00 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=28 Acc@1=77.0600 Acc@5=98.7700 Loss=1.0611 Lr=0.0206
[08/27 21:21:52 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=29 Acc@1=78.3700 Acc@5=98.7500 Loss=1.0098 Lr=0.0175
[08/27 21:23:42 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=30 Acc@1=78.0200 Acc@5=98.8700 Loss=1.0001 Lr=0.0146
[08/27 21:25:32 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=31 Acc@1=79.6000 Acc@5=99.0600 Loss=0.8993 Lr=0.0120
[08/27 21:27:23 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=32 Acc@1=80.0800 Acc@5=99.1700 Loss=0.8516 Lr=0.0095
[08/27 21:29:13 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=33 Acc@1=80.1500 Acc@5=99.1100 Loss=0.8723 Lr=0.0074
[08/27 21:31:03 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=34 Acc@1=81.4100 Acc@5=99.0800 Loss=0.8326 Lr=0.0054
[08/27 21:32:53 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=35 Acc@1=81.4900 Acc@5=99.1500 Loss=0.8322 Lr=0.0038
[08/27 21:34:43 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=36 Acc@1=81.2300 Acc@5=99.1200 Loss=0.8257 Lr=0.0024
[08/27 21:36:32 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=37 Acc@1=81.7300 Acc@5=99.2600 Loss=0.8082 Lr=0.0014
[08/27 21:38:21 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=38 Acc@1=81.6900 Acc@5=99.2500 Loss=0.7939 Lr=0.0006
[08/27 21:40:10 cifar10-wrn40_2-wrn16_1]: [Eval] Epoch=39 Acc@1=81.8800 Acc@5=99.2300 Loss=0.7991 Lr=0.0002
[08/27 21:40:10 cifar10-wrn40_2-wrn16_1]: Best: 81.8800
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.