Comments (7)
I would say model = model |> gpu
is not a solution because it corrupts the correspondence in stateful optimisers. For example the Adam optimiser uses an IdDict to keep track of the momentum for different params. After |> gpu
the object ids change and the optimiser state has to start from scratch. This means in the end we are not able to resume training. The BSON saving and loading can only be used for training => saving and then loading => inference. Restarting training with a blank optimiser state would corrupt reproducability
from bson.jl.
Right, you can't save CuArrays with BSON.jl. Doing data = data |> Flux.cpu
before saving your model should fix this (of course, when you load it again it will just be a regular array, not a CuArray
).
from bson.jl.
Right, you can't save CuArrays with BSON.jl.
Okay, but then doing this should at least produce some error. Like in many other applications, no output (error/warning message) means everything went as expected!
You can get in big trouble if you are not aware of this issue and try to save your model after a time consuming learning phase...
Doing data = data |> Flux.cpu before saving your model should fix this
This is what I am doing now too, but I think the documentation of Flux should be more clear about this
from bson.jl.
Agreed. If you have time, it would be great if you can submit a Flux PR to make this very clear in the docs.
from bson.jl.
Agreed, just had the same problem and it's written nowhere in the Flux documentation
from bson.jl.
Sounds like this issue has been resolved?
Out of curiosity, it is possible to overload some method so that when someone tries to save CuArray to BSON, it copies the data into an Array and saves that? And maybe it's possible it's possible to save a tiny bit of metadata so that when loading a "CuArray" from disk, it creates an Array and copies the data over into a newly created CuArray?
from bson.jl.
If you have time, it would be great if you can submit a Flux PR to make this very clear in the docs.
@jpsamaroo a fellow student hit this bug last week, I'm thinking throw an info
and automatically moved to CPU for them.
Where should this be,
https://github.com/JuliaGPU/CUDA.jl/blob/603edb87891da8fd5b2623f17544aebe9706069a/src/array.jl#L68
unfortunately there's no interface package defining this type, so I'm thinking adding a Requires.jl
here at BSON?
from bson.jl.
Related Issues (20)
- This BSON does not seem compatible with external programs HOT 1
- Convert types to / from BSON automatically based on a list of types or a package name HOT 2
- Cannot save Flux gpu model weights HOT 1
- ERROR: too many parameters for type when loading Flux model HOT 3
- Problem Loading HOT 3
- Problem saving function in BSON HOT 5
- BSON fails with import as HOT 5
- `setfield!: const field .names of type TypeName cannot be changed` on Julia 1.8 HOT 17
- Trying to load data from summer 2020 HOT 1
- Document `@save "filename" key=value`
- reading back SVector fails with StaticArraysCore not defined
- Loading step ranges from Julia < 1.7 causes segmentation fault in Julia >= 1.7
- BSON error after project packages update, Type problem
- Name of local custom types not resolved
- Store PkgId for packages?
- Padding of type UInt8 is not compatible
- Add README message to point users to Serialization stdlib, JLD2, and LightBSON HOT 4
- Loading BSON saved dataframe raises "import errors" in Pluto
- Saving Array{Union{Missing, ...}}(...) is slow HOT 1
- Can't load binary when not working in Main HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bson.jl.