freeing up temporary CUDA tensors w/o Caml.Gc.full_major() ?

[laurentmazare]

freeing up temporary CUDA tensors w/o Caml.Gc.full_major() ?

[LaurentMazare]

Indeed that's actually an issue and you will notice that [Gc.full_major] is actually called pretty often in the code. I'm not sure whether this is actually necessary, a [Gc.minor] may be enough and quite faster.
I would not expect this to be very long compared to running some backward step on your gpu but maybe in some use case it's significant.
Anyway there is no perfect way around it, rust has these linear/affine types that help a lot but there is nothing like this in ocaml. That said I think that some recent version of ocaml added a way to tell the gc about how much 'resource' some opaque pointer is consuming and that should work well for this use case, I haven't played with it yet but that would probably be a nice thing to have.

[zeroexcuses]

I would not expect this to be very long compared to running some backward step on your gpu but maybe in some use case it's significant.

You're right, currently "Gc.full_major" seems slow, but only relative to doing backpass on MNIST. I'm sure on larger datasets it'll be better.

Do you know how PyTorch handles this problem? My OCaml code, without manual Gc.full_major, will often throw exception of unable to allocate mem on CUDA. I haven't run into this issue in PyTorch. Does PyTorch catch the exception, run a full GC, and continue? [I'm not sure, just guessing.]

[zeroexcuses]

Speaking of which -- can the OCaml bindings do this? Where if something fails to allocate it does:

fn handle_fail_alloc =
  run full gc;
  try alloc again;
  if still fail, die; else continue;

[LaurentMazare]

Python uses ref counting for memory management rather than only a gc (there may be a gc involved to find cycles?) so this is not an issue there.
Handling failed allocation by running the gc is probably not very straightforward do I'm afraid. Most code invariants break when you don't have memory. When it's gpu memory that may be less of an issue but it's still probably quite difficult.

[zeroexcuses]

1. Right, IIRC Python is RefCount + cycle detector.

2. I know close to nothing about OCaml VM and defer to you on this issue.

Thanks for the detailed response, looks like Caml.Gc.full_major it is.