Could you add more servers in FedAvg for faster training speed?

[fedml-ai]

Could you add more servers in FedAvg for faster training speed?

[chaoyanghe]

FedML supports multiple parameter servers for the communication efficiency via hierarchical FL and decentralized FL .
In hierarchical FL, there are group parameter servers that split the total client set into multiple client subsets.
In decentralized FL, each client acts as a parameter server.

Please refer to the following links for details.
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/hierarchical_fl
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/decentralized

@prosopher Thanks. But I guess he was discussing the distributed computing setting, not the standalone version.

[wizard1203]

@chaoyanghe Thanks for your detailed explanation. Maybe I can try to complete it by myself, and when I finish it I would like to push it to your master branch.

[chaoyanghe]

BytePS is for data center-based distributed training, while FedML (e.g., FedAvg) is edge-based distributed training. The particular assumptions of FL include:

1. heterogeneous data distribution cross devices (non-I.I.D.)
2. resource constrained edge devices (memory, computational, and communication)
3. label deficiency (harder to label the data points because of privacy)
4. concern the security and privacy

So what do you mean "adding more servers in FedAvg"?

[wizard1203]

BytePS is for data center-based distributed training, while FedML (e.g., FedAvg) is edge-based distributed training. The particular assumptions of FL include:

1. heterogeneous data distribution cross devices (non-I.I.D.)
2. resource constrained edge devices (memory, computational, and communication)
3. label deficiency (harder to label the data points because of privacy)
4. concern the security and privacy

So what do you mean "adding more servers in FedAvg"?

I mean adding more parameter servers to improve the communication efficiency. Maybe this can be used suitably only in cluster environment but not true Federated Learning environment with resource constrained edge devices. However, it is still can accelerate the training when doing research.

[prosopher]

FedML supports multiple parameter servers for the communication efficiency via hierarchical FL and decentralized FL .
In hierarchical FL, there are group parameter servers that split the total client set into multiple client subsets.
In decentralized FL, each client acts as a parameter server.

Please refer to the following links for details.
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/hierarchical_fl
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/decentralized

[chaoyanghe]

@wizard1203 Thanks for your suggestion. As for acceleration, FedML is the only research-oriented FL framework that supports cross-machine multiple GPU distributed training. To further accelerate, we can definitely use many techniques from traditional distributed training (very mature with much less research attention). I elaborate a few here:

1. AllReduce-based GPU-GPU communication using InfiniBand. However, this is not a real FL setting. As you said, it is only useful to evaluate algorithms or modeling which are not insensitive to the training speed.
2. Hybrid parallelism (model + data parallelism) + pipeline
3. Bucking BP (as introduced in PyTorch VLDB 2020)
4. low-bit (half precision)
5. pruning
   ....

As @prosopher pointed out, you can design any topology as you like. Our topology configuration is very flexible. In distributed computing setting, you can refer the following algorithms with different topologies:
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/distributed

In addition, I have to point out that "adding more parameter servers to improve the communication efficiency" is a bit confusing conceptually. We cannot say using more computation resources improves communication efficiency. Normally, the relationship between computation and communication is a trade-off. Using more parallel computation cannot change the communication itself, and it also does not mean we can speed up the training since the communication cross machines may dominate the training time. But I agree with your idea of using traditional techniques in distributed computing to accelerate FL research. Thanks.

[wizard1203]

FedML supports multiple parameter servers for the communication efficiency via hierarchical FL and decentralized FL .
In hierarchical FL, there are group parameter servers that split the total client set into multiple client subsets.
In decentralized FL, each client acts as a parameter server.

Please refer to the following links for details.
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/hierarchical_fl
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/decentralized

@prosopher Thanks for this, I will carefully read them.

[chaoyanghe]

@chaoyanghe Thanks for your detailed explanation. Maybe I can try to complete it by myself, and when I finish it I would like to push it to your master branch.

Thanks. Looking forward to your contribution.

[chaoyanghe]

@wizard1203 Do you mean modifying based on this code?
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/distributed/fedavg

[wizard1203]

@wizard1203 Do you mean modifying based on this code?
https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/distributed/fedavg

@chaoyanghe No, maybe it needs to base on those codes on fedml_core. Whatever, I may try to do it many days later. In fact, I have some other algorithms that I want to implement more urgently than this.