jasperzhong/gnnflow

Distributed Deep Graph Learning Framework for Dynamic Graphs

gnnflow's Issues

need the most recent version of the README file

I apologize for any inconvenience, but it appears that the README file may not be the most up-to-date version. Would it be possible for you to provide me with the latest version of the README file?

distributed training crashed

When training TGN on four machines, rank 1 and rank 3 crashed during sampling after training for 3 epochs.

./run_offline_multi_node_g4_4.sh TGN REDDIT LRUCache 0.2 hash 

INFO:root:Start training...
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
INFO:root:Epoch 1/100 | Iter 100/196 | Throughput 54889.16 samples/s | Loss 1761.0231 | Cache node ratio 0.0000 | Cache edge ratio 0.8370
INFO:root:Epoch 1/100 | Validation ap 0.9322 | Validation auc 0.9334 | Train time 25.95 s | Validation time 4.94 s | Train Throughput 54419.25 samples/s | Cache node ratio 0.0000 | Cache edge ratio 0.7388
INFO:root:Epoch 2/100 | Iter 100/196 | Throughput 63050.01 samples/s | Loss 1191.9175 | Cache node ratio 0.0000 | Cache edge ratio 0.8392
INFO:root:Epoch 2/100 | Validation ap 0.9572 | Validation auc 0.9588 | Train time 23.20 s | Validation time 4.39 s | Train Throughput 60872.94 samples/s | Cache node ratio 0.0000 | Cache edge ratio 0.7647
INFO:root:Epoch 3/100 | Iter 100/196 | Throughput 61008.39 samples/s | Loss 1002.1672 | Cache node ratio 0.0000 | Cache edge ratio 0.8479
INFO:root:Epoch 3/100 | Validation ap 0.9685 | Validation auc 0.9684 | Train time 22.69 s | Validation time 4.47 s | Train Throughput 62237.72 samples/s | Cache node ratio 0.0000 | Cache edge ratio 0.8491
INFO:root:Best val AP: 0.9685 & val AUC: 0.9684
INFO:root:Epoch 4/100 | Iter 100/196 | Throughput 63978.52 samples/s | Loss 891.6926 | Cache node ratio 0.0000 | Cache edge ratio 0.8873