Time taken to data loading increased in newer builds (ARM)

[dilililiwhy]

Time taken to data loading increased in newer builds (ARM)

[malfet]

I think your analysis is correct, but i wonder if builds after pytorch/builder#1787 would revert to the original behavior.

@snadampal can we add some sort of a regression test to the release process to avoid such flaws in the future?

[malfet]

Ok, taking for myself as pytorch/builder#1787 actually broke nightly builds, need to restart the process..

[dilililiwhy]

Ok, taking for myself as pytorch/builder#1787 actually broke nightly builds, need to restart the process..

@malfet Thanks.
Will there be a nightly version of 2.3.1?

[snadampal]

Hi @dilililiwhy , can you please try with the latest nightly wheel, something like this
pip3 install --pre torch==2.4.0.dev20240426 -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html

this has the correct openBLAS and also correct libgomp. This was the main difference went into previous builder patch you had pointed out.

[snadampal]

Hi @malfet , yes, we need some regression tests around openblas and openmp. will check what can be included in the CD test.

[dilililiwhy]

Hi @dilililiwhy , can you please try with the latest nightly wheel, something like this pip3 install --pre torch==2.4.0.dev20240426 -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html

this has the correct openBLAS and also correct libgomp. This was the main difference went into previous builder patch you had pointed out.

Time taken is return to normal by using dev20240426.

Step 0 : 4.790385484695435
Step 1 : 0.009980916976928711
Step 2 : 0.0010445117950439453
Step 3 : 0.0010247230529785156
Step 4 : 0.0011034011840820312
Step 5 : 0.0011696815490722656
... ...
Step 25 : 0.0009491443634033203
Step 26 : 0.0009517669677734375
Step 27 : 0.0009474754333496094
Step 28 : 0.000934600830078125
Step 29 : 0.0009233951568603516
Step Avg : 0.17011735439300538

torch 2.4.0.dev20240426
torchvision 0.19.0.dev20240426

@snadampal Thanks for your help. Will there be a nightly version of torch 2.3.1?

[snadampal]

great! I have raised this PR to fix this issue on pytorch/2.3 release
pytorch/builder#1805

I think once it is merged to 2.3 release branch, there may be a nightly wheel.