Segmentation fault in 3D classification in relion

[SepidehV]

Segmentation fault in 3D classification in relion

[biochem-fan]

The program appears to be dying inside the CUDA runtime. Very puzzling indeed.

For a quick check, make a subset with very few particles (let say 1000) and try the followings:

• Run as usual (to make sure the problem is reproducible with fewer particles)
• Run without MPI (e.g. 1 process, 16 threads)
• Run without CUDA
• Run without MPI and CUDA

[SepidehV]

Hi, Thanks for your reply. It only runs without CUDA with small subset, otherwise I receive the same error that I had for the whole data set. Could you please let me know what is the problem and the solution? Thanks, Sepideh Sepideh Valimehr | PhD Research Fellow Ian Holmes Imaging Center | Bio21 Molecular Science & Biotechnology Institute |The University of Melbourne ARC Centre for Cryo-electron Microscopy of Membrane Proteins (CCeMMP) 30 Flemington Road, Parkville Victoria 3052 T: [phone] E: ***@***.******@***.***> I acknowledge the Traditional Owners of the land on which I work, and pay my respects to the Elders, past and present. ***@***.*** From: biochem_fan ***@***.***> Sent: Tuesday, February 6, 2024 5:27 PM To: 3dem/relion ***@***.***> Cc: Sepideh Valimehr ***@***.***>; Author ***@***.***> Subject: Re: [3dem/relion] Segmentation fault in 3D classification in relion (Issue #1078) The program appears to be dying inside the CUDA runtime. Very puzzling indeed. For a quick check, make a subset with very few particles (let say 1000) and try the followings: * Run as usual (to make sure the problem is reproducible with fewer particles) * Run without MPI (e.g. 1 process, 16 threads) * Run without CUDA * Run without MPI and CUDA — Reply to this email directly, view it on GitHub<#1078 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BC7CPQGI3YHTDXFZZ6IILATYSHEJ3AVCNFSM6AAAAABC3MBX22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRYHA3DGNZTGM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[biochem-fan]

Thanks for testing. This indicates that the problem is really with CUDA.
If you can share this small subset privately to us, we will investigate further.

[SepidehV]

Hi, How can I share the subset privately? I tried to run the whole data set without GPU. The iteration 1 took around 4 hours and when it reached to iteration 2 the estimated time was 89 hours. I used 45 MPIs for that. Bests, Sepideh Sepideh Valimehr | PhD Research Fellow Ian Holmes Imaging Center | Bio21 Molecular Science & Biotechnology Institute |The University of Melbourne ARC Centre for Cryo-electron Microscopy of Membrane Proteins (CCeMMP) 30 Flemington Road, Parkville Victoria 3052 T: [phone] E: ***@***.******@***.***> I acknowledge the Traditional Owners of the land on which I work, and pay my respects to the Elders, past and present. ***@***.*** From: biochem_fan ***@***.***> Sent: Thursday, February 8, 2024 1:01 PM To: 3dem/relion ***@***.***> Cc: Sepideh Valimehr ***@***.***>; Author ***@***.***> Subject: Re: [3dem/relion] Segmentation fault in 3D classification in relion (Issue #1078) Thanks for testing. This indicates that the problem is really with CUDA. If you can share this small subset privately to us, we will investigate further. — Reply to this email directly, view it on GitHub<#1078 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BC7CPQCN2WOVFOFXZSJZLY3YSQWXNAVCNFSM6AAAAABC3MBX22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZTGI2DQMZWGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[biochem-fan]

Please make the test case as small as possible. Make a new directory and copy minimum required files (STAR file, particles, mask, reference etc) and make sure the problem can be reproduced there. Compress the folder and upload it to Google Drive or Drop Box and send the link to Takanori Nakane, whose email address can be found in the CCPEM mailing list. Don't forget to write the full command line to reproduce the issue.