Error when saving Flux models/CuArrays from GPU

[juliaio]

Error when saving Flux models/CuArrays from GPU

[jonas-eschmann]

I would say model = model |> gpu is not a solution because it corrupts the correspondence in stateful optimisers. For example the Adam optimiser uses an IdDict to keep track of the momentum for different params. After |> gpu the object ids change and the optimiser state has to start from scratch. This means in the end we are not able to resume training. The BSON saving and loading can only be used for training => saving and then loading => inference. Restarting training with a blank optimiser state would corrupt reproducability

[jpsamaroo]

Right, you can't save CuArrays with BSON.jl. Doing data = data |> Flux.cpu before saving your model should fix this (of course, when you load it again it will just be a regular array, not a CuArray).

[theimperior]

Right, you can't save CuArrays with BSON.jl.
Okay, but then doing this should at least produce some error. Like in many other applications, no output (error/warning message) means everything went as expected!
You can get in big trouble if you are not aware of this issue and try to save your model after a time consuming learning phase...

Doing data = data |> Flux.cpu before saving your model should fix this
This is what I am doing now too, but I think the documentation of Flux should be more clear about this

[jpsamaroo]

Agreed. If you have time, it would be great if you can submit a Flux PR to make this very clear in the docs.

[dominusmi]

Agreed, just had the same problem and it's written nowhere in the Flux documentation

[ali-ramadhan]

Sounds like this issue has been resolved?

Out of curiosity, it is possible to overload some method so that when someone tries to save CuArray to BSON, it copies the data into an Array and saves that? And maybe it's possible it's possible to save a tiny bit of metadata so that when loading a "CuArray" from disk, it creates an Array and copies the data over into a newly created CuArray?

[Moelf]

If you have time, it would be great if you can submit a Flux PR to make this very clear in the docs.

@jpsamaroo a fellow student hit this bug last week, I'm thinking throw an info and automatically moved to CPU for them.

Where should this be,
https://github.com/JuliaGPU/CUDA.jl/blob/603edb87891da8fd5b2623f17544aebe9706069a/src/array.jl#L68

unfortunately there's no interface package defining this type, so I'm thinking adding a Requires.jl here at BSON?