ibm / fedma Goto Github PK
View Code? Open in Web Editor NEWCode for Federated Learning with Matched Averaging, ICLR 2020.
License: MIT License
Code for Federated Learning with Matched Averaging, ICLR 2020.
License: MIT License
Good job! I am very interested in this work and I tried run the experiments mentioned in the paper. My questions are:
Thank you.
Hi,
Thank you for your wonderful work and making the code public.
I have a question regarding the code in language_fedma.py. In line 302-303, why is there a 'pass' if layer_index ==2 for example (please refer to attached screenshot)? What happens to the case where we have more layers?
Can I use the code from lines 309-313 again for layer_index ==2 instead of a pass?
Thank you very much!
Hi,
Very good job! I love your method.
I tried to run FedMA using the "simple-cnn" model, but there is a miss-match in the size of matched_cnn (which is bigger than the original model) and the weights after alignment.
The error is:
File "../main.py", line 97, in trans_next_conv_layer_backward
reshaped = layer_weight.reshape(reconstructed_next_layer_shape).transpose(1, 0, 2, 3).reshape(next_layer_shape[0], -1)
ValueError: cannot reshape array of size 3750 into shape (15,25,5,5)
When I try to change the NUM_LAYERS in lstm_fedma_with_comm.py,it will have some problem,so i change the RNNmodel's hyper parameters nlayers.but still can't debug.
for example,when i change the NUM_LAYERS = 4(2-layer LSTM (4 layers: encoder|hidden LSTM1|hidden LSTM2|decoder)),so i change the nlayers=2.
Traceback (most recent call last):
File "language_oneshot_matching.py", line 504, in
matching_shapes=matching_shapes)
File "/home/hx/github/FedMA/language_modeling/language_fedma.py", line 258, in layerwise_fedma
reconstructed_bias = [split_bias(batch_weights[j][layer_index+3+2]) for j in range(J)]
File "/home/hx/github/FedMA/language_modeling/language_fedma.py", line 258, in
reconstructed_bias = [split_bias(batch_weights[j][layer_index+3+2]) for j in range(J)]
IndexError: list index out of range
When i wangt to use the model of simple-cnn under heterogeneous data, there are some errors , my commond is :
python dist_skew_main.py --model=simple-cnn --dataset=cifar10 --lr=0.01 --retrain_lr=0.01 --batch-size=64 --epochs=10 --retrain_epochs=10 --n_nets=10 --partition=hetero-dir --comm_type=fedma --comm_round=10 --retrain=True --rematching=True
and the error is :
Traceback (most recent call last):
File "dist_skew_main.py", line 1181, in
args.partition, args.n_nets, args_alpha, args=args)
File "/home/wjj/three/FedMA-master/utils.py", line 274, in partition_data_dist_skew
traindata_cls_counts = record_net_data_stats(y_train, net_dataidx_map, logdir)
UnboundLocalError: local variable 'net_dataidx_map' referenced before assignment
Can you tell me how to solve this problem
Thanks for sharing your great work. When I reimplemented your code, I met the following error:
UnboundLocalError: local variable 'shape_estimator' referenced before assignment
I used LeNet on Mnist.
Could you tell me the reason may cause such error?
Thanks.
Hi, i have something that confused me:
Dear authors,
Your work is very impressive and thanks for open-sourcing the code!
Please correct me if I got it wrong - In each round of FedMA, you need to retrain (num_layers * retrain_epoch) total epochs, not including (retrain_epoch) epochs for the whole model to be updated. So the extra computation will be very intensive if you set a relatively large "retrain_epoch" (say 20 as shown in the sample command). Could you share the specific number for this hyper-parameter to reproduce your results on Cifar-10?
Thanks in advance and looking forward to your reply.
Hi @hwang595, a few weeks ago I made some questions in another issue thread about I problem that I had when trying to train a model with input image shape greater or equal to 224x224. Since then, I tried to reduce the dimensions of my problem to the default size, i.e. 32x32, and it worked well! But when I run using 224x224, I'm still locked in this training part.
So I'm gonna ask my questions here again:
Obs.: As I'm working with medical images it is critical resize them.
Thanks for the great work!
Hello,
Thank you for the great work. I am studying federated learning in NLP. I tried to reproduce the results in the paper (mainly LSTM on Shakespeare dataset) but results seem very off from what it should be. Please help me recheck what I missed in my experiments.
(1) The Shakespeare data preprocessing is noted like below in the paper:
So I use the command like this to preprocess the data:
./preprocess.sh -s niid --sf 1.0 -k 0 -t sample -tf 0.8 -k 10000
(2) It is indicated the the paper that experiments were done with 1-Layer LSTM.
Anyway, reading from the code, I believe it is equal to setting:
NUM_LAYERS=3
As it will have one input layer, one output layer and one hidden LSTM layer (where the invariant permutation problem is addressed by FedMA)
(3) It is noted in the paper that FedAvg and FedProx awere trained with 33 communication rounds, while FedMA was trained with 11 communication rounds (because each round of FedMA requires 3 communication rounds correspoding to number of LSTM layers). I actually used 30 for FedAvg and FedProx and 10 for FedMA like these:
For FedAvg
python language_main.py --mode=fedavg --comm-round=30
For FedProx
python language_main.py --mode=fedprox --comm-round=30
For FedMA
python language_main.py --mode=fedma --comm-round=10
(I do not think --comm-round has any effect in FedMA anyway because the code perform single round of FedMA)
Then I performed the rest of FedMA communication round by running
python lstm_fedma_with_comm.py
(The lstm_fedma_with_comm.py has 10 communication rounds hard-coded)
(4) The results seem not aligned with what indicated in the paper. While FedProx got lower test accuracy than FedAvg, but FedMA also got lower accuracy than FedAvg too.
For FedMA
Result from the first step (language_main.py)
Result from the second step (lstm_fedma_with_comm.py)
Thank you in advance for your help.
Lot of intersting ideas in the paper.
I am trying to reproduce the results for fedavg using "run_dist_skew.sh". In Fig 4 in paper the quoted accuracy is around 66%. But in my runs, accuracy hardly increases beyond 50%. Could you please help with the settings needed to reproduce the results in Fig4 for fedavg ??
PC
Hi, thank you for sharing your outstanding work! I have some questions about the settings in the paper, could you please support some more details about the experiment settings?
python main.py --model=moderate-cnn \ --dataset=cifar10 \ --lr=0.01 \ --retrain_lr=0.01 \ --batch-size=64 \ --epochs=150 \ --retrain_epochs=150 \ --n_nets=16 \ --partition=hetero-dir \ --comm_type=fedma \ --comm_round=10 \ --retrain=True \ --rematching=True
Thanks very much
I want to implement FedMA in a FML framework, is there a function inside this repo that i can use as a black box?
I want it to have input lets say two layers from client cnn and one from the global and to return the matched output.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.