Non-Blocking Methodology on ChainerMN

[fengyuan14]

Non-Blocking Methodology on ChainerMN

[keisukefukuda]

@arthuryuan1987 , thanks you for raising an issue, and thank you for your contribution!

It's great to hear that you implemented a layer-wise computation-communication overlap feature and obtained a significant performance improvement.

Actually, the feature is in our scope, but we have no concrete short-term plan. This is because the overlapping feature is not straightforward as it looks.

• What should the API look like?
• How should it work with recurrent network such as LSTM?
• How should it treat a dynamic network? i.e. a layer may not be computed depending on a dynamic condition (if statement). The current ChainerMN pads zero for all un-evaluated layers and works well.

As you may know, ChainerMN already has a "double-buffering" feature. Although it comes with a cost of accuracy degradation, it is general and can work with any network.

In fact, ChainerMN is integrated into Chainer (see chainer/chainer#5226), and will be more tightly integrated with Chainer. There may be more opportunities to support such a corner cases with more advanced Chainer features and we will start considering the feature.

Thanks,

[fengyuan14]

Thanks for your elaborate explanation. We take deep dive for a completed view. Thanks a lot.

[keisukefukuda]

I'm closing the issue for now. Feel free to reopen it if you like to have more discussion on this. Thanks!