shaoxiongji/fed-att

Attentive Federated Learning for Private NLM

Attentive Federated Learning

This repository contains the code for the paper Learning Private Neural Language Modeling with Attentive Aggregation, which is an attentive extention of federated aggregation. A brief introductionary blog is avaiable here.

Further reference: a universal federated learning repository implemented by PyTorch - Federated Learning - PyTorch.

Run

Refer to the README.md under the data folder and download the datasets into their corresponding folders. Enter the source code folder to run the scripts with arguments assigned using argparse package.

cd src
python run.py

See configs in src/utils/options.py

Requirements

Python 3.6
PyTorch 0.4.1

Cite

@inproceedings{ji2019learning,
  title={Learning Private Neural Language Modeling with Attentive Aggregation},
  author={Ji, Shaoxiong and Pan, Shirui and Long, Guodong and Li, Xue and Jiang, Jing and Huang, Zi},
  booktitle={International Joint Conference on Neural Networks (IJCNN)},
  year={2019}
}

fed-att's Issues

RuntimeError: each element in list of batch should be of equal size

Hi there, thank you for the awesome code! I'm running into the following runtime error (PFA the screenshot of the error), and will really appreciate any help!

hi, where is the model?or how to download?

E:\Documents\fed-att-master\src>python run.py --gpu -1

Existing from training early
Traceback (most recent call last):
File "run.py", line 104, in
with open(model_saved, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '../log/wikitext-2/model_50_att_0.1.pt'

Issue in dataloader -- total perplexity in paper incorrect

Hi, your dataloader uses the default collate_fn in pytorch 0.4.1. This truncates every batch to the length of the shortest sentence in that batch. By extension your experiments are only operating on about 20% of the total datasets.

When fixed, this increases perplexity to about 300 for both FedAtt and FedAvg on the Penn Treebank dataset.

Some questions about the code

Thank you very much for your work. When running your code, I found such a problem: The data format read by the dataloader is as follows. Is this format correct? I understand that your thought is to predict the last word of the sentence based on the sentence missing the last word. Is this a mistake, or is my understanding incorrect?

sent:
[tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), tensor([ 17, 721, 26116, 114, 2163, 246, 19139, 27, 17, 1]), tensor([ 1117, 148, 8592, 17984, 8559, 78, 129, 2441, 18529, 1]), tensor([8712, 1509, 3303, 7059, 548, 518, 1627, 16, 37, 1]), tensor([ 625, 14, 22, 19632, 12964, 27, 43, 10, 548, 988]), tensor([ 3611, 27, 332, 3554, 39, 3448, 10, 129, 12964, 37]), tensor([ 564, 1718, 17, 83, 17, 22, 14, 1299, 10614, 247]), tensor([ 14, 3743, 26117, 22, 59, 14633, 18160, 713, 502, 1]), tensor([ 1563, 284, 3006, 17388, 52, 30, 5, 39, 112, 1])]
x:
tensor([ 0, 17, 1117, 8712, 625, 3611, 564, 14, 1563, 0,
721, 148, 1509, 14, 27, 1718, 3743, 284, 0, 26116,
8592, 3303, 22, 332, 17, 26117, 3006, 0, 114, 17984,
7059, 19632, 3554, 83, 22, 17388, 0, 2163, 8559, 548,
12964, 39, 17, 59, 52, 0, 246, 78, 518, 27,
3448, 22, 14633, 30, 0, 19139, 129, 1627, 43, 10,
14, 18160, 5, 0, 27, 2441, 16, 10, 129, 1299,
713, 39, 0, 17, 18529, 37, 548, 12964, 10614, 502,
112])
torch.Size([81])
out:
tensor([[ -3.0746, -10.4770, -10.3294, ..., -10.4987, -10.5410, -10.4060],
[ -9.7837, -10.1357, -10.2601, ..., -10.0260, -10.2036, -10.3559],
[-10.5659, -10.2263, -10.2304, ..., -10.2324, -10.3104, -10.2632],
...,
[-11.1314, -10.1663, -10.3698, ..., -10.1777, -10.1619, -10.2236],
[-10.0721, -10.2309, -10.0301, ..., -10.1624, -10.0988, -10.6142],
[-11.0124, -10.2391, -10.6138, ..., -10.4047, -10.1974, -10.5044]],
grad_fn=)
torch.Size([81, 28913])
y.data:
tensor([ 0, 721, 148, 1509, 14, 27, 1718, 3743, 284, 0,
26116, 8592, 3303, 22, 332, 17, 26117, 3006, 0, 114,
17984, 7059, 19632, 3554, 83, 22, 17388, 0, 2163, 8559,
548, 12964, 39, 17, 59, 52, 0, 246, 78, 518,
27, 3448, 22, 14633, 30, 0, 19139, 129, 1627, 43,
10, 14, 18160, 5, 0, 27, 2441, 16, 10, 129,
1299, 713, 39, 0, 17, 18529, 37, 548, 12964, 10614,
502, 112, 0, 1, 1, 1, 988, 37, 247, 1,
1])

Two methods results same

Dear,
Thank you for your contribution.
I read your paper and run your code.
But I get a same test perplexity by fed-avg and your method fed-att.
All parameters are the default, and the dataset is wikitext-2.
Could you tell me whether some initiatives are wrong?

Best wishes.
2020.4.13