jywa / fednova Goto Github PK
View Code? Open in Web Editor NEWPyTorch implementation of FedNova (NeurIPS 2020), and a class of federated learning algorithms, including FedAvg, FedProx.
PyTorch implementation of FedNova (NeurIPS 2020), and a class of federated learning algorithms, including FedAvg, FedProx.
Hi,
Congratulation for your great work. I've a question.
In the case that the local epoch is fixed for each user, vector ai and it's norm ||ai|| is unchanged across rounds?
For example, consider SGDMomentum-Nova, vector ai is as following:
[1+m+m2+m3+...m^(ti-1), ..., 1+m, 1]
where ti is the number of round's steps. When ti is fixed, ai is always equal to [1+m+m2+m3+...m^(ti-1), ..., 1+m, 1]
So, the value ||ai|| is known in advance as a certain function f(number of steps). It can be calculated once and reused for every round?
Thank you in advance for your feedback.
Hello,
I am not sure about this, but it seems there is a slight mismatch between the implementation of FedNova, and it's description from the paper. It seems that in the current implementation, you compute the cumulative sum of gradient, without multiplying by weights a_{i}, even when etamu is not zero (see below)
# update accumalated local updates
if 'cum_grad' not in param_state:
param_state['cum_grad'] = torch.clone(d_p).detach()
param_state['cum_grad'].mul_(local_lr)
else:
param_state['cum_grad'].add_(local_lr, d_p)
p.data.add_(-local_lr, d_p)
Probably it is needed to add a line in order to multiply d_p
by a_i
(this variable should be tracked as well), before updating param_state['cum_grad']
.
Is what I said correct, or is there anything I am missing?
Thanks in advance for your help
I'm sorry to bother you, but can you provide me with the source code for this paper?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.