In CuArrays v2.0, GPU operation takes hours to run for the first time

[lihebi]

In CuArrays v2.0, GPU operation takes hours to run for the first time

[lihebi]

Thanks, looks like it is downloading artifacts. Probably my network is slow for this. I just tried a proxy, and it finished pretty fast. Then I tried a new Julia session, and this time the delay is a few seconds as expected.

Closing this. Not sure why previously a new Julia session did not yield this behavior. Maybe the artifacts cache somehow expired after several hours? I'll monitor this and reopen it if I still have problems.

[staticfloat]

Just to confirm; Pkg.gc() won't delete lazy artifacts. It treats lazy and non-lazy artifacts the same; it will only delete them once all Artifacts.toml files referring to those artifacts are removed. (And only if they have been unreachable for a certain amount of time, too)

[maleadt]

Try running with JULIA_DEBUG=all, interrupting the operation to see a stack trace, etc. It's probably downloading artifacts, but that should only happen once.

[maleadt]

Maybe the artifacts cache somehow expired after several hours?

I don't think that happens, although you can GC them. @staticfloat, is the way we use artifacts here (multiple lazy ones, one @artifact at first use) compatible with that mechanism (i.e. won't downloaded CUDA artifacts get removed too soon)?

[lihebi]

Just to confirm that it still works fine after a day. It should be something else happened. I tried to re-install NVIDIA driver several times, maybe that invalidated the artifacts at some point.