program freezes at nerfacc/grid.py/ray_aabb_intersect

[IamMohitM]

program freezes at nerfacc/grid.py/ray_aabb_intersect

[hesom]

You have probably solved it on your own by now, but what helped me with the same problem was just deleting ~/.cache/torch_extensions/

[IamMohitM]

@hesom Thanks a lot! I actually kinda moved on from this issue for a while but deleting that folder solved the issue.

[IamMohitM]

The issue persisted because I had multiple nerfstudio projects (and by extension nerfacc). They were in conflict with the extensions folder and causing the freezing again. For each project, I updated the environment variable TORCH_EXTENSIONS_DIR to a different folder which solved the freezing issue.

[juuso-oskari]

Hi @IamMohitM, would you mind telling in more detail what you did? reinstalling nerfacc, or removing the folder ~/.cache/torch_extensions/, or running "export TORCH_EXTENSIONS_DIR="/home/juuso/miniconda3/envs/nerfstudio/torch_extension" does not remove the issue for me and the run still gets killed at the same point.

[IamMohitM]

By any chance, are you running nerfacc from multiple projects?

I had two projects using nerfacc (different venvs), but they shared the same TORCH_EXTENSIONS_DIR, so I changed the location of the extensions dir to not conflict with each other and that seemed to work.

Particularly, when running each project I set the the export TORCH_EXTENSIONS_DIR=temp/torch_extensions, and it would create this directory relative to the project you are working on.

I ran into this issue, when there was a conflict with two different nerfacc installations accessing the same extension dir. I hope this helps.

[juuso-oskari]

I actually found out that the issue was for me running out of GPU memory, due to instant-ngp-bounded method using a dataparser that does not have downscaling for the images. So it ended up loading the full resolution images from mip-nerf-360 dataset to the GPU which are pretty large. That instant_ngp_dataparser.py sure would make use of downscaling functionality.