unexpected performance when using COSMA with GPU (single node)

[eth-cscs]

unexpected performance when using COSMA with GPU (single node)

[kabicm]

Great that it works now!

Can you check also without overlapping communication and computation? And can you also try some larger matrix sizes, e.g. 32k or so? Basically, 16k case can be run on a rank with a single GPU.

I will check this testcase on our system and then we will see.

[rohany]

I see slightly better performance without overlap:
1 GPU:
COSMA TIMES [ms] = 1996 2453 3166 4126 4613
2 GPU:
COSMA TIMES [ms] = 1934 2348 2505 3602 5486
4 GPU:
COSMA TIMES [ms] = 1041 1370 1530 1584 1905

At matrix size 30000 without overlap i see:
1 GPU:
COSMA TIMES [ms] = 8987 9811 9840 13445 13833
2 GPU:
COSMA TIMES [ms] = 7181 7182 7227 7839 9772
4 GPU:
COSMA TIMES [ms] = 4282 4327 6609 8345 23970

[rohany]

I'm going to close this since things are working as expected for me now.

[kabicm]

Hi Rohan,

What was the main problem? Was this also related to limited memory that we yesterday discussed?

Also, did you check adding just -s "sm2" or just -s "sm2,sn2" instead of splitting all three dimensions beforehand?

Thanks for your feedback!

[rohany]

The main problem at this time iirc was that I was strong scaling instead of weak scaling, as well as on relatively small problem sizes.