Comments (14)
but for patho and diri the results are sometimes growing sometimes lower than the previous iteration
Yes, it is a very NATURAL phenomenon!
(a.k.a. statistical heterogeneity issue in federated learning, in a technical term)
That is why federated learning in the wild is very difficult and is also sensitive to hyperparameters I mentioned.
(e.g., C
, E
, B
, R
, lr
, optimizer
, lr_decay
, lr_decay_step
)
You may try other advanced method, e.g., FedProx
, FedAdam
, FedYogi
, FedAdagrad
instead of FedAvg
, or even personalized FL methods, if you are interested in.
from federated-learning-in-pytorch.
Dear @bbw251,
Thank you for your warm comments!
Could you please share example commands (IID & Patho & Diri with Dataset name you used, K, R, E, B, etc.) so that I can reproduce the unexpected result?
Thank you!
from federated-learning-in-pytorch.
@bbw251
Hi, I have just tried with following commands with MNIST dataset, K=100 clients, R=50 rounds, E=5, B=10, and C=0.1:
- IID
python3 main.py \
--exp_name "FedAvg_MNIST_2NN_IID" --seed 42 --device cuda \
--dataset MNIST \
--split_type iid --test_size 0.1 \
--model_name TwoNN --resize 28 --hidden_size 200 \
--algorithm fedavg --eval_fraction 1 --eval_type both --eval_every 1 --eval_metrics acc1 acc5 \
--K 100 --R 50 --E 5 --C 0.1 --B 10 --beta1 0 \
--optimizer SGD --lr 0.01 --lr_decay 0.99 --lr_decay_step 25 --criterion CrossEntropyLoss
- Pathological (with
mincls=2
)
python3 main.py \
--exp_name "FedAvg_MNIST_2NN_PATHO" --seed 42 --device cuda \
--dataset MNIST \
--split_type patho --mincls 2 --test_size 0.1 \
--model_name TwoNN --resize 28 --hidden_size 200 \
--algorithm fedavg --eval_fraction 1 --eval_type both --eval_every 1 --eval_metrics acc1 acc5 \
--K 100 --R 50 --E 5 --C 0.1 --B 10 --beta1 0 \
--optimizer SGD --lr 0.01 --lr_decay 0.99 --lr_decay_step 25 --criterion CrossEntropyLoss
- Dirichlet (with
cncntrtn=0.01
)
python3 main.py \
--exp_name "FedAvg_MNIST_2NN_DIRI" --seed 42 --device cuda \
--dataset MNIST \
--split_type diri --cncntrtn 0.01 --test_size 0.1 \
--model_name TwoNN --resize 28 --hidden_size 200 \
--algorithm fedavg --eval_fraction 1 --eval_type both --eval_every 1 --eval_metrics acc1 acc5 \
--K 100 --R 50 --E 5 --C 0.1 --B 10 --beta1 0 \
--optimizer SGD --lr 0.01 --lr_decay 0.99 --lr_decay_step 25 --criterion CrossEntropyLoss
Here are corresponding results:
- Central evaluation: loss: 0.1455 | acc1: 0.9553 | acc5: 0.9987
- Local evaluation:
- Loss: Avg. (0.1426) Std. (0.0785) | Top 10% (0.3214) Std. (0.0337) | Bottom 10% (0.0349) Std. (0.0179)
- Acc1: Avg. (0.9555) Std. (0.0236) | Top 10% (0.9926) Std. (0.0082) | Bottom 10% (0.9130) Std. (0.0070)
- Acc5: Avg. (0.9987) Std. (0.0045) | Top 10% (1.0000) Std. (0.0000) | Bottom 10% (0.9869) Std. (0.0070)
- Pathological FedAvg_MNIST_2NN_PATHO_231013_122126.log
- Central evaluation: loss: 0.5953 | acc1: 0.7880 | acc5: 0.9932
- Local evaluation:
- Loss: Avg. (0.6345) Std. (0.4361) | Top 10% (1.4864) Std. (0.3184) | Bottom 10% (0.1114) Std. (0.0312)
- Acc1: Avg. (0.7779) Std. (0.1661) | Top 10% (0.9816) Std. (0.0119) | Bottom 10% (0.4675) Std. (0.0888)
- Acc5: Avg. (0.9939) Std. (0.0125) | Top 10% (1.0000) Std. (0.0000) | Bottom 10% (0.9628) Std. (0.0132)
- Dirichlet FedAvg_MNIST_2NN_DIRI_231013_122148.log
- Central evaluation: loss: 0.8174 | acc1: 0.7071 | acc5: 0.9865
- Local evaluation:
- Loss: Avg. (0.4397) Std. (0.3672) | Top 10% (1.0558) Std. (0.2559) | Bottom 10% (0.0301) Std. (0.0086)
- Acc1: Avg. (0.8454) Std. (0.1938) | Top 10% (1.0000) Std. (0.0000) | Bottom 10% (0.4545) Std. (0.0962)
- Acc5: Avg. (0.9956) Std. (0.0042) | Top 10% (1.0000) Std. (0.0000) | Bottom 10% (0.9865) Std. (0.0113)
According to these demo experiments, I found expected behaviors that the performance of IID split is better than Dirichlet distribution-based non-IID split or pathological non-IID split.
If it is not the case for you, I presume it might be due to some configurations like local batch size, client sampling size, optimizer, learning rate scheduling, and so forth.
Please let me know if there exist other issues that I didn't catch up with.
Thank you!
from federated-learning-in-pytorch.
Thank you for your fast response, i appreciate your dedication Adam!. The results you reproduced above is the expected one. let me provide my parameter settings may be it will be the case. here is one example of the command:
for e in 1
do
for s in 'patho' 'iid' 'diri'
do
python main.py \
--exp_name "FedAvg_MNIST_CNN_BN_${s}_e${e}" --seed 42 --device cpu \
--dataset MNIST \
--split_type $s --test_fraction 0 --mincls 2\
--model_name OneCNN --resize 28 --hidden_size 5 \
--algorithm fedavg --eval_fraction 1 --eval_type local --eval_every 1 --eval_metrics acc1 f1 precision recall \
--K 5 --R 100 --E $e --C 1 --B 30 --beta 0 \
--optimizer SGD --lr 0.01 --lr_decay 0.95 --lr_decay_step 20 --criterion CrossEntropyLoss
done
done
Note that: i have also tried for B = 10, 20 and C=0.1, there is test_size in the previous code, may be you mean test_fraction
and other parameters are the default settings you provided in the main function.
I thank you !
from federated-learning-in-pytorch.
@bbw251
TL; DR: pull the newest version of this repository and run again, please!
Seems like you customized the model (i.e., named OneCNN
in your command) which has batchnorm layer in it, right?
Since I don't have OneCNN
architecture right now, I cannot assert the main cause of your situation.
Instead, I STRONGLY recommend you to pull the newest version of this repository first, and run your code again.
There were bugs in previous version of this repository, specifically occurred when using a model having bathcnorm layers.
Both running mean and variance should also be federated on top of BN layer's weight and bias (commonly denoted as gamma and beta), but in the previous version, only gamma and beta are communicated with the server.
Thank you.
Best,
Adam
from federated-learning-in-pytorch.
from federated-learning-in-pytorch.
for the new version could you mention the modified parts since i have changed many parts to my own research purpose. i can see you have modified the fedavgserver.py file and some lines of fedavg.py and model.py
from federated-learning-in-pytorch.
Since many parts are updated, I cannot pinpoint which parts are different from yours. (plus, I have no idea which version you are working on)
You can find all changes in commit histories.
(https://github.com/vaseline555/Federated-Learning-in-PyTorch/commits/main)
Thank you.
from federated-learning-in-pytorch.
The new version is not stable especially for the patho, i tried without any change except the device to cpu:
for b in 0 10
do
for c in 0.0 0.1 0.2 0.5 1.0
do
"C:/Users/A0923/AppData/Local/Programs/Python/Python311/python.exe" main.py
--exp_name "FedAvg_MNIST_CNN_Patho_C${c}_B${b}" --seed 42 --device cpu
--dataset MNIST
--split_type patho --test_size 0
--model_name TwoCNN --resize 28 --hidden_size 200
--algorithm fedavg --eval_fraction 1 --eval_type both --eval_every 1 --eval_metrics acc1 acc5
--K 100 --R 1000 --E 5 --C $c --B $b --beta1 0
--optimizer SGD --lr 0.1 --lr_decay 0.99 --lr_decay_step 10 --criterion CrossEntropyLoss
done
done
i am not sure if device can make that much difference, and i watched for about 20 iterations, iid starts from accuracy of 60 and patho starts from accuracy of 55 from the first epoch. i am confused!
from federated-learning-in-pytorch.
What do you mean 'unstable', and what do you refer to 'accuracy', is it about central evaluation or averaged local evaluations?
I don't get the exact problem in this statement: "iid starts from accuracy of 60 and patho starts from accuracy of 55 from the first epoch".
What did you expect from each setting (IID and Patho)?
The convergence speed can vary across the combination of hyperparameters (e.g., C, E, B, lr, lr_decay, lr_decay_step).
from federated-learning-in-pytorch.
I still cannot understand exact problem you faced...
You said,
iid starts from accuracy of 60 and patho starts from accuracy of 55 from the first epoch
Then, the initial performance of IID setting is better than that of Pathological setting, right?
Is this problematic? Isn't this what you originally expected(i.e., IID > Patho)?
-
If it is, could you clarify what is the problem, in detail?
-
If it is not, please also provide me detailed information. (what results you originally expected & what you had with commands)
Per your uploaded log,
Here is the example trained for 30 epochs, i am concerned on the "server_evaluated": acc1
What is the issue about the metric "server_evaluated": acc1
?
Is it problematic because it showed too low accuracies(~10%)?
-
If it is, it is a very natural result since you have set
C=0
andB=0
.
In theory, you should setC
not to be too small or too big (see Li et al., 2020), and the batch size (B
) should also be appropriately tuned along withlr
. -
If it the low accuracy is NOT the issue, please also clarify your problem with details.
from federated-learning-in-pytorch.
I think the server evaluation result is expected to be growing in every iteration. i have trained for three of the data splitting types for 20 global epochs each, i found training with iid is growing, but for patho and diri the results are sometimes growing sometimes lower than the previous iteration. i have attached the results for patho and dir below. that was i mean.
FedAvg_MNIST_2NN_diri.json
FedAvg_MNIST_2NN_PATHO.json
note: the parameter settings are as you provided 3 days before.
I thank you
from federated-learning-in-pytorch.
Yes, it is a very NATURAL phenomenon!
(a.k.a. statistical heterogeneity issue in federated learning, in a technical term), thank you very much. I want to train in HFL, hierarchical FL, would you share with me valuable resources or code repositories designed for that, especially for wireless communications?
I am thankful indeed dear Adam!
from federated-learning-in-pytorch.
Sorry, I am not familiar with the concept HFL, you may find related resources in Google.
For example, hierarchical federated learning github
may act as a good search keyword for your case.
Hope this helps!
Thank you.
Best,
Adam
from federated-learning-in-pytorch.
Related Issues (19)
- when num_clients =1000, shards_per_clients=0, and there is a ValueError! HOT 1
- Implemented aggregation is different from the description in McMahan 2016 paper HOT 2
- Set a different number of Clients when using LEAF dataset HOT 1
- AttributeError: 'FedavgServer' object has no attribute 'lr_scheduler' HOT 5
- Type error for `_request()` HOT 2
- Client Multithreading is actually not Working. HOT 5
- About Dirichlet non-iid Partition HOT 3
- Clients won't download server model under the global evaluation HOT 2
- Is there any plan to implement DITTO Algorithm?
- maximum recursion depth exceeded while calling a Python object error HOT 3
- Sent140 Dataset
- 我运行cifar10数据集出现错误,该怎么解决 HOT 1
- undefined variable error HOT 1
- AssertionError: Invalid device id HOT 2
- Looking forward to the introduction, clear code on the CIFAR dataset! HOT 2
- The accuracy on Cifar10 may be low. HOT 3
- Datasets location HOT 2
- Instructions to run on separate server/client devices? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from federated-learning-in-pytorch.