juliaio / bson.jl Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
bson.jl's People
Contributors
Stargazers
Watchers
Forkers
tsurherman roboneet zgornel richiejp jondeuce timotaularson eulerkochy mauro3 jonasisensee batmanabcdefg afqueiruga jeffbezanson simondanisch oxinabox kristofferc codyk12 aviks ayushk4 goretkin juliatagbot modelzookeeper dhairyalgandhi martinusr stamp1993 stjordanis racinmat bozenkhaa carlolucibello darsnack standardgalactic daviehh playfloor proteanblank ranocha chengchingwen ericphanson sharanry topolaritybson.jl's Issues
Cannot recover types of type-parameterized objects
Hi,
I tried to serialize an object with a type parameter and deserialize it from a BSON file. However, it doesn't work because BSON.jl cannot recover the parameter type in a case like below:
using BSON
struct Foo{T}
data::Vector{T}
end
struct Bar
data::Vector{Foo{T}} where T
end
obj = Bar([Foo([1])])
buf = IOBuffer()
BSON.bson(buf, Dict(:data => obj.data))
seekstart(buf)
@show typeof(BSON.load(buf)[:data]) # typeof((BSON.load(buf))[:data]) = Array{Any,1}
seekstart(buf)
Bar(BSON.load(buf)[:data]) # This won't work.I'm using BSON.jl v0.2.1 and Julia 1.0.1 on macOS.
Is this a bug or a limitation of this package? Thank you in advance.
Compatibility with FileIO
Would it be possible to make BSON compatible with FileIO like JLD or JLD2? Right now I can save/load model files in different formats relying on FileIO figuring out the format looking at file extension and/or magic numbers and would like to do the same with BSON.
UndefRefError: access to undefined reference
Nice package, Mike! I found this obscure error message when saving this custom AbstractArray type from SASLib.jl.
julia> using SASLib
julia> p = SASLib.ObjectPool{String,UInt8}("", 3)
3-element SASLib.ObjectPool{String,UInt8}:
""
""
""
julia> BSON.@save "test.bson" p
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] lower(::Array{String,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/extensions.jl:60
[2] _lower_recursive(::Array{String,1}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[3] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:25
[4] _lower_recursive(::Array{Any,1}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[5] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:18
[6] _lower_recursive(::Dict{String,UInt8}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[7] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:25
[8] _lower_recursive(::Array{Any,1}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[9] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:18
[10] _lower_recursive(::SASLib.ObjectPool{String,UInt8}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[11] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:18
[12] _lower_recursive(::Dict{Symbol,SASLib.ObjectPool{String,UInt8}}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[13] lower_recursive(::Dict{Symbol,SASLib.ObjectPool{String,UInt8}}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:73
[14] open(::BSON.##16#17{Dict{Symbol,SASLib.ObjectPool{String,UInt8}}}, ::String, ::String) at ./iostream.jl:152
[15] bson(::String, ::Dict{Symbol,SASLib.ObjectPool{String,UInt8}}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:83
BSON.@save has side effects
Hi,
I just encountered following side effect (in a more complex scenario, boiled down to the minimum):
using BSON
a = Dict{Symbol, Any}(:a => Float32.([1]))
BSON.@save "/tmp/deleteme.bson" a
After that a is changed to contain all the type information that is probably generated to write the output.
Dict{Symbol,Any} with 1 entry:
:t => Dict{Symbol,Any}(:tag=>"array",:type=>Dict{Symbol,Any}(:tag=>"datatype",:params=>Any[],:name=>Any["Core", "Float32"]),:size=>Any[3],:data=>UInt8[0x00, 0x00, 0x80, 0x3f, 0x00, 0x00, 0x00, 0x40, 0x00, 0…
After loading the dict again:
BSON.@load "/tmp/deleteme.bson" a
A is restored in its original form. To me this was unexpected behaviour and I did not find any documentation that there may be side effects when using @save.
System:
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: AMD Ryzen 7 4800H with Radeon Graphics
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, znver2)
BSON:
[fbb218c0] BSON v0.2.6
Apart from that minor issue, thanks for this fabulous library!
ERROR loading Ordered Collections
It seems, Using BSON library I can save Ordered collections but can't load them.
I saved one variable and when I tried to load it again . It showed me error.
ERROR: UndefVarError: OrderedCollections not defined
gzip compress and uncompress a .bson file?
Hello,
I have a large .bson file (468 MB). After gzipping it to .bson.gz, the file size is reduced to 169 MB (~3x compression). How feasible would it be to have a macro that automatically saves/loads a gz compressed bson file? Or is there a way to do so already using something like CodecZlib.jl?
Interestingly, using the same data I tried an experiment with JLD2 files (see issue), and their file size is 615 MB (.jld2) and 187 MB (.jld2.gz). It seems like .bson has better compression than .jld2, although the loading speed for .jld2 files are 3-4x faster (1.5 sec vs 5 sec). Is this a correct interpretation?
Reconstruct Types
Hello, one of the features for using BSON you mention is the backwards compatibility. But I have a very general question about this.
Suppose I save a type (struct) which is designed in a certain way and comes from a package, like this:
struct Foo
foo::String
bar::Int
endAfter some time the design of this structure changes due to some improvements of the package or whatever...
struct Foo
foo::String
bar::Int
added_feature::Float64
endNow I will not be able to load my old saved BSON file anymore because I cannot reproduce the right struct in my environment and I cannot recreate it from the saved data because BSON does not load anything.
I do not expect that a reconstructed struct will work properly but at least it would be great if it still would be possible to load it and, e.g., define a conversion method or just extract some data.
compression while storing BSON
Is it possible to specify compression while storing BSON files?
`ERROR: type Method has no field sparam_syms` when saving Flux Chain
When saving a struct composed of Flux Chain object with anonymous function BSON returns the following error:
ERROR: type Method has no field sparam_syms
It seems to be due to the presence of anonymous function.
I am using Julia 1.2, BSON v2.3, Flux v0.8.3
The same code is working in Julia 1.1 (and same Flux and BSON version)
Here is a MWE:
using Flux
using BSON
model = Chain(Conv((5,), 1 => 32, relu, stride=5),
x -> view(x, :, :, :, :),
x -> permutedims(x, (1, 2, 4, 3)),
Conv((1,32), 1 => 32, relu),
x -> reshape(x, (8, 32, :)),
x -> maximum(x, dims=1),
x -> reshape(x, (32, :)))
bson("model.bson", model=model)Thanks for your help
Loading VGG19 weights is quite slow
julia> using Metalhead
julia> @time Metalhead.weights("vgg19.bson")
29.465177 seconds (433.86 M allocations: 13.256 GiB, 47.36% gc time)Split out Julia serialiser.
This package is poorly designed (it's ok, I'm allowed to say that) in that it tries to support both plain BSON and Julia data structures in one go. This creates annoying corner cases (do we preserve the type tag in Dict{String,Int} or store it in a compatible way?) and also makes it hard to safely load BSON anyway, because of the obvious security issues around loading Julia data (#50).
BSON.jl should strip out all serialisation utilities and just have a solid reader/writer. Other formats (model weights, Julia data) should be defined on top in separate packages. The new package could be a fork of BSON.jl, or we could just recommend JLSO, which is built on BSON anyway.
I don't know if JLSO supports closures or cycles, which are important to Flux, but it certainly has other nice features, and if its maintainers are open to those things it'd be my preferred option.
DimensionMismatch on load of Unitful vector
I am not sure where the following is coming from. It happened only with Unitful.rad and Unitful.°, so maybe the problem is in Unitful?
import Unitful
import BSON
save_test = rand(ComplexF64, 100) .* Unitful.rad
BSON.@save "test.bson" save_test
BSON.@load "test.bson"
Stack trace:
DimensionMismatch("new dimensions (1,) must be consistent with array size 0")
Stacktrace:
[1] _throw_dmrsa(::Tuple{Int64}, ::Int64) at ./array.jl:322
[2] reshape at ./array.jl:304 [inlined]
[3] reshape(::Array{UInt8,1}, ::Int64) at ./reshapedarray.jl:99
[4] (::getfield(BSON, Symbol("##41#42")))(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/extensions.jl:71
[5] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[7] (::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[8] applychildren!(::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[9] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[10] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[11] (::getfield(BSON, Symbol("##25#26")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:86
[12] applychildren!(::getfield(BSON, Symbol("##25#26")){IdDict{Any,Any}}, ::Array{Any,1}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:28
[13] raise_recursive(::Array{Any,1}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:86
[14] (::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}})(::Array{Any,1}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[15] applychildren!(::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[16] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[17] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[18] (::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[19] applychildren!(::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[20] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[21] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[22] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:74
[23] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[24] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:74
[25] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[26] raise_recursive at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[27] load(::String) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:96
[28] top-level scope at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:51
[29] top-level scope at In[43]:3
Deserialization is restricted to the Main namespace?
I am having problems deserialising BSON objects from within a package.
See this Julia discourse thread
bson(path, plot) once => Plots broken thereafter
This one's a little spooky, and took me a loooong time to debug, because the Stacktraces are completely misleading.
julia> using Plots, BSON
julia> plot(rand(10)); # now it works
julia> bson("/tmp/test.bson", p = plot(rand(10)))
julia> plot(rand(10)); # now it doesn't
ERROR: MethodError: no method matching plot_color(::Dict{Symbol,Any})
Closest candidates are:
plot_color(::Any, ::Nothing) at /home/ubuntu/.julia/packages/PlotUtils/NS6Tf/src/colors.jl:20
plot_color(::Any, ::Number) at /home/ubuntu/.julia/packages/PlotUtils/NS6Tf/src/colors.jl:23
plot_color(::Symbol) at /home/ubuntu/.julia/packages/PlotUtils/NS6Tf/src/colors.jl:7
...
Stacktrace:
[1] _update_plot_args(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/args.jl:1419
[2] _plot_setup(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Array{Dict{Symbol,Any},1}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/pipeline.jl:183
[3] plot_setup! at /home/ubuntu/.julia/packages/Plots/ey5o3/src/pipeline.jl:156 [inlined]
[4] recipe_pipeline!(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{Array{Float64,1}}) at /home/ubuntu/.julia/packages/RecipesPipeline/qM4Ea/src/RecipesPipeline.jl:86
[5] _plot!(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{Array{Float64,1}}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/plot.jl:167
[6] plot(::Array{Float64,1}; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/plot.jl:57
[7] plot(::Array{Float64,1}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/plot.jl:51
[8] top-level scope at REPL[11]:1All attempts to plot thereafter fail similarly.
That's with BSON v0.2.6, Plots v1.6.1, but I've had the same happen with Plots 1.5.3+.
Any idea what BSON.jl is doing that might cause this? Type piracy?
Support large BSONs by using Int64 in case Int32 size range is exceeded.
Proposal how to fix #67 .
Based on discussion https://julialang.slack.com/archives/C67TK21LJ/p1596027666030200
Test if data fit to Int32 range and use it, othwerwise use 64bit range. #67 (comment) described where specifically the cast is happening.
Rationale beihind this: BSON.jl already violates BSON specification by allowing saving arbitrary types.
This follows same ides, as long as user makes sure input data follow BSON specification, it is compliant with the specification, but it also allows to persist data that don't follow BSON specification if it's required.
saving with bson modifies dict argument
Writing with bson(file, dict) seems to modify the dict argument.
Is this the intended behaviour?
Julia-1.1.0> using BSON
Julia-1.1.0> d = Dict(:a => [1+2im, 3+4im], :b => "Hello, World!")
Dict{Symbol,Any} with 2 entries:
:a => Complex{Int64}[1+2im, 3+4im]
:b => "Hello, World!"
Julia-1.1.0> d2 = copy(d)
Dict{Symbol,Any} with 2 entries:
:a => Complex{Int64}[1+2im, 3+4im]
:b => "Hello, World!"
Julia-1.1.0> bson("test.bson", d)
Julia-1.1.0> d
Dict{Symbol,Any} with 2 entries:
:a => Dict{Symbol,Any}(:tag=>"array",:type=>Dict{Symbol,Any}(:tag=>"datatype",:params=>Any[Dict{Symbol,Any}(:tag=>"datatype",:params=>Any[],:name=>Any["Main", "Core", "Int64"])],:name=>Any["Main", …
:b => "Hello, World!"
Julia-1.1.0> d3 = BSON.load("test.bson")
Dict{Symbol,Any} with 2 entries:
:a => Complex{Int64}[1+2im, 3+4im]
:b => "Hello, World!"
Julia-1.1.0> d3 == d
false
Julia-1.1.0> d3 == d2
true
Array type not preserved when loading. Saved as `Array{String,1}` but loaded as `Array{Any,1}`.
julia> using BSON
julia> x = ["foo", "bar", "baz"]
3-element Array{String,1}:
"foo"
"bar"
"baz"
julia> typeof(x)
Array{String,1}
julia> bson("test.bson", Dict(:x => x))Now close Julia and open a new Julia session.
julia> using BSON
julia> data = BSON.load("test.bson")
Dict{Symbol,Any} with 1 entry:
:x => Any["foo", "bar", "baz"]
julia> data[:x]
3-element Array{Any,1}:
"foo"
"bar"
"baz"
julia> typeof(data[:x])
Array{Any,1}MultiDim Arrays stored in a BSON file saved on 64-bit not loads on 32-bit system
This is somewhat similar to #41 .
Here's an MWE - Assume using BSON and using BSON:@save
julia> a = ones(Float32, 2, 2)
2×2 Array{Float32,2}:
1.0 1.0
1.0 1.0
julia> @save "saved_on_64bit.bson" a
On a 32-bit system @test BSON.load("saved_on_64bit.bson")[:a] == ones(Float32, 2, 2) gives the following stack trace https://gist.github.com/Ayushk4/2376db8679caa5e9f70c13abe89c981e
This was narrowed down to
Line 80 in a58c88a
This seems to be happening because on a 64-bit reshape(ones(Float32, 1, 4), 2, 2) works whereas the following doesn't. Similarly, Int64 doesn't work on 32-bit-
julia> reshape(ones(Float32, 1, 4), Int32(2), Int32(2))
ERROR: MethodError: no method matching reshape(::Array{Float32,2}, ::Int32, ::Int32)
It can be fixed by changing the line to reshape(reinterpret_(d[:type], d[:data]), map(normalize_typeparams, d[:size])...) :.
Versions -
BSON v0.2.4
Julia 1.3
Came here from JuliaText/TextAnalysis.jl#180
Error tagging new release
The URL of this package does not match that stored in METADATA.jl.
cc: @julia-tagbot[bot]
cannot round trip custom type (DataFrame) inside a module
I discovered that BSON.jl can't round-trip a custom data type inside a module. MWE:
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.0.3 (2018-12-18)
_/ |\__'_|_|_|\__'_| |
|__/ |
(v1.0) pkg> activate .
julia> module A
using DataFrames, BSON
d = DataFrame(a = 1:10, b = rand(10))
bson("d.bson", Dict(:d=>d))
d2 = BSON.load("d.bson")[:d]
end
ERROR: UndefVarError: DataFrames not defined
Stacktrace:
[1] (::getfield(BSON, Symbol("##33#34")))(::Module, ::String) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:20
[2] mapfoldl_impl(::typeof(identity), ::getfield(BSON, Symbol("##33#34")), ::NamedTuple{(:init,),Tuple{Module}}, ::Array{Any,1}) at ./reduce.jl:43
[3] #mapfoldl#170(::Base.Iterators.Pairs{Symbol,Module,Tuple{Symbol},NamedTuple{(:init,),Tuple{Module}}}, ::Function, ::Function, ::Function, ::Array{Any,1}) at ./reduce.jl:70
[4] #mapfoldl at ./none:0 [inlined]
[5] _mapreduce_dim at ./reducedim.jl:303 [inlined]
[6] #mapreduce#536 at ./reducedim.jl:301 [inlined]
[7] #mapreduce at ./none:0 [inlined]
[8] #reduce#537 at ./reducedim.jl:345 [inlined]
[9] (::getfield(Base, Symbol("#kw##reduce")))(::NamedTuple{(:init,),Tuple{Module}}, ::typeof(reduce), ::Function, ::Array{Any,1}) at ./none:0
[10] resolve(::Array{Any,1}) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:20
[11] (::getfield(BSON, Symbol("##37#38")))(::Dict{Symbol,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:48
[12] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:71
[13] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:81
[14] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:122
[15] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:80
[16] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:74
[17] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[18] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:74
[19] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:81
[20] raise_recursive at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[21] load(::String) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:96
[22] top-level scope at none:0
[23] run_backend(::REPL.REPLBackend) at /home/dave/.julia/packages/Revise/EuQoV/src/Revise.jl:771
[24] (::getfield(Revise, Symbol("##58#60")){REPL.REPLBackend})() at ./task.jl:259This works fine if it's run outside of the module A ... end:
julia> using DataFrames, BSON
julia> d = DataFrame(a = 1:10, b = rand(10))
10×2 DataFrame
│ Row │ a │ b │
│ │ Int64 │ Float64 │
├─────┼───────┼───────────┤
│ 1 │ 1 │ 0.0805191 │
│ 2 │ 2 │ 0.44521 │
│ 3 │ 3 │ 0.126867 │
│ 4 │ 4 │ 0.466975 │
│ 5 │ 5 │ 0.782517 │
│ 6 │ 6 │ 0.904739 │
│ 7 │ 7 │ 0.928574 │
│ 8 │ 8 │ 0.268764 │
│ 9 │ 9 │ 0.822993 │
│ 10 │ 10 │ 0.0755515 │
julia> bson("d.bson", Dict(:d=>d))
julia> d2 = BSON.load("d.bson")[:d]
10×2 DataFrame
│ Row │ a │ b │
│ │ Int64 │ Float64 │
├─────┼───────┼───────────┤
│ 1 │ 1 │ 0.0805191 │
│ 2 │ 2 │ 0.44521 │
│ 3 │ 3 │ 0.126867 │
│ 4 │ 4 │ 0.466975 │
│ 5 │ 5 │ 0.782517 │
│ 6 │ 6 │ 0.904739 │
│ 7 │ 7 │ 0.928574 │
│ 8 │ 8 │ 0.268764 │
│ 9 │ 9 │ 0.822993 │
│ 10 │ 10 │ 0.0755515 │and it also works fine if you do using DataFrames in Main first. I think the issue is https://github.com/MikeInnes/BSON.jl/blob/master/src/extensions.jl#L20 which hard codes the start of the search path for datatypes as Main when it should be the current module (could be wrong about this though).
For context, I was hoping to use BSON.jl in a Weave.jl document with a sandbox module, so that the code in the notebook is evaled in a separate module, insulating it from the global workspace while still allowing for a long-running session (to avoid startup costs). JLD2 fails at this in a similar way (only looks for loaded packages in Main)
`lower` failure
When I try to save an instance of TimerOutput (from the package TimerOutputs), I get
ERROR: MethodError: no method matching lower(::TimerOutputs.TimerOutput).
Do the types that attempt to get saved need to provide this function?
loading custom types (DESolution) from packages
I tried this on the DifferentialEquations.jl solution type and it failed. Here's an MWE:
using OrdinaryDiffEq
f_2dlinear = (du,u,p,t) -> du.=1.01u
prob = ODEProblem(f_2dlinear,rand(2,2),(0.0,1.0))
sol1 =solve(prob,Euler();dt=1//2^(4))
using BSON
bson("test.bson",Dict(:sol1=>sol1))
# New session
using OrdinaryDiffEq
using BSON
BSON.load("test.bson")which gives
ERROR: type ODESolution has no field module
Stacktrace:
[1] newstruct!(::DiffEqBase.ODESolution{Float64,3,Array{Array{Float64,2},1},Void,Void,Array{Float64
,1},Array{Array{Array{Float64,2},1},1},DiffEqBase.ODEProblem{Array{Float64,2},Float64,true,Void,BSON
.__deserialized_types__.##3#4,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem},Ordinar
yDiffEq.Euler,OrdinaryDiffEq.InterpolationData{BSON.__deserialized_types__.##3#4,Array{Array{Float64
,2},1},Array{Float64,1},Array{Array{Array{Float64,2},1},1},OrdinaryDiffEq.EulerCache{Array{Float64,2
},Array{Float64,2}}}}, ::Array{Any,1}, ::Void, ::Void, ::Array{Float64,1}, ::Array{Any,1}, ::DiffEqB
ase.ODEProblem{Array{Float64,2},Float64,true,Void,BSON.__deserialized_types__.##3#4,Void,Void,Unifor
mScaling{Int64},DiffEqBase.StandardODEProblem}, ::OrdinaryDiffEq.Euler, ::OrdinaryDiffEq.Interpolati
onData{BSON.__deserialized_types__.##3#4,Array{Array{Float64,2},1},Array{Float64,1},Array{Array{Arra
y{Float64,2},1},1},OrdinaryDiffEq.EulerCache{Array{Float64,2},Array{Float64,2}}}, ::Bool, ::Int64, :
:Symbol) at C:\Users\Chris\.julia\v0.6\BSON\src\anonymous.jl:12
[2] newstruct_raw(::ObjectIdDict, ::Type{T} where T, ::Dict{Symbol,Any}) at C:\Users\Chris\.julia\v
0.6\BSON\src\extensions.jl:102
[3] (::BSON.##53#54)(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\ext
ensions.jl:117
[4] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\read
.jl:80
[5] (::BSON.##24#26{ObjectIdDict})(::Dict{Symbol,Any}) at C:\Users\Chris\.julia\v0.6\BSON\src\read.
jl:74
[6] applychildren!(::BSON.##24#26{ObjectIdDict}, ::Dict{Symbol,Any}) at C:\Users\Chris\.julia\v0.6\
BSON\src\BSON.jl:18
[7] _raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\rea
d.jl:74
[8] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\read
.jl:81
[9] load(::String) at C:\Users\Chris\.julia\v0.6\BSON\src\read.jl:96
[10] eval(::Module, ::Any) at .\boot.jl:235
[11] eval(::Any) at .\boot.jl:234
[12] macro expansion at C:\Users\Chris\.julia\v0.6\Atom\src\repl.jl:117 [inlined]
[13] anonymous at .\<missing>:?Serializing anonymous method fails on julia 1.6-beta1
Failing example:
using BSON
f = x -> 2x
BSON.@save "test.bson" f
Stacktrace:
ERROR: LoadError: type Method has no field ambig
Stacktrace:
[1] getproperty(x::Method, f::Symbol)
@ Base ./Base.jl:33
[2] structdata(meth::Method)
@ BSON ~/.julia/packages/BSON/XAts7/src/anonymous.jl:15
[3] lower(x::Method)
@ BSON ~/.julia/packages/BSON/XAts7/src/extensions.jl:91
[4] _lower_recursive(x::Method, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[5] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Method)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[6] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:28
[7] _lower_recursive(x::Vector{Method}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[8] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Vector{Method})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[9] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:28
[10] _lower_recursive(x::Vector{Any}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[11] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[12] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:28
[13] _lower_recursive(x::Vector{Any}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[14] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[15] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[16] _lower_recursive(x::Core.TypeName, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[17] (::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}})(x::Core.TypeName)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:53
[18] applychildren!(f::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[19] (::BSON.var"#_lower#6"{IdDict{Any, Any}, Vector{Any}})(x::Type)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:53
[20] _lower_recursive(x::Type, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:54
[21] (::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}})(x::Type)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:53
[22] applychildren!(f::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[23] _lower
@ ~/.julia/packages/BSON/XAts7/src/write.jl:53 [inlined]
[24] _lower_recursive(x::Function, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:54
[25] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Function)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[26] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[27] _lower_recursive(x::Dict{Symbol, var"#1#2"}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[28] lower_recursive(y::Dict{Symbol, var"#1#2"})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:73
[29] bson(io::IOStream, doc::Dict{Symbol, var"#1#2"})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:81
[30] #12
@ ~/.julia/packages/BSON/XAts7/src/write.jl:83 [inlined]
[31] open(::BSON.var"#12#13"{Dict{Symbol, var"#1#2"}}, ::String, ::Vararg{String, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Base ./io.jl:330
[32] open
@ ./io.jl:328 [inlined]
[33] bson(path::String, doc::Dict{Symbol, var"#1#2"})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:83
[34] top-level scope
@ ~/test.jl:3
Not being able to save Flux models
using Flux
using BSON: @save, @load
m = Chain(Dense(23, 10))
@save "model.bson" mThis produces the following error:
StackOverflowError:
Stacktrace:
[1] modpath(::Module) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/extensions.jl:24 (repeats 43609 times)
[2] typepath at /home/avik-pal/.julia/packages/BSON/zmGJP/src/extensions.jl:33 [inlined]
[3] lower(::DataType) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/extensions.jl:37
[4] (::getfield(BSON, Symbol("#_lower#8")){IdDict{Any,Any},Array{Any,1}})(::Type) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:53
[5] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:54
[6] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[7] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/BSON.jl:20
[8] _lower_recursive(::Chain, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[9] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Chain) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[10] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/BSON.jl:20
[11] _lower_recursive(::Dict{Symbol,Chain}, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[12] lower_recursive(::Dict{Symbol,Chain}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:73
[13] bson(::IOStream, ::Dict{Symbol,Chain}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:81
[14] #14 at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:83 [inlined]
[15] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Chain}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
[16] open at ./iostream.jl:367 [inlined]
[17] bson(::String, ::Dict{Symbol,Chain}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:83
[18] top-level scope at In[23]:2
loading large dataset is very slow (compared to JLD2)
I tried BSON.jl with a large dataset. Cf JLD2:
julia> @time load("/tmp/test.jld2")
5.885191 seconds (21.85 M allocations: 2.056 GiB, 60.13% gc time)
Dict{String,Any} with 2 entries:
"levels" => 2247195×6 DataFrames.DataFrame…
"transitions" => 3775512×7 DataFrames.DataFrame…whereas BSON.load("/tmp/test.bson") never terminates (I interrupted after 5 minutes). The file itself is around 600M (slightly smaller than the JLD2 file which is around 700M). I can make an MWE if the issue is not known.
Error when trying to reload a BSON file
Up until today (10/30/2020) I have run the following code numerous times with no error. Today I get the following error:
using BSON: @load, @save
using AxisArrays
using CSV, DataFrames, Plots, StatsPlots, Distributions
using Turing, LinearAlgebra
@load "heterog_model_hmc_chains_02prior.bson" cc
ERROR: TypeError: in Type, in A, expected A<:(AxisArray{Union{Missing, Real},3,D,Ax} where Ax where D), got Type{Missing}
Stacktrace:
[1] constructtype(::Type{T} where T, ::Array{Any,1}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\extensions.jl:50
[2] (::BSON.var"#33#34")(::Dict{Symbol,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\extensions.jl:53
[3] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:79
[4] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:89
[5] (::BSON.var"#43#44")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\extensions.jl:138
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:88
[7] (::BSON.var"#18#20"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:82
[8] applychildren!(::BSON.var"#18#20"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\BSON.jl:21
[9] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:82
[10] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:89
[11] raise_recursive at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:99 [inlined]
[12] load(::String) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:104
[13] top-level scope at C:\Users\millsjf.julia\packages\BSON\XAts7\src\BSON.jl:52
I have tried updating the packages. The same error occurs for different BSON files, so it's not a corrupted file.
It takes several hours to estimate the model, so I really need to be able to load the MCMC chains rather than keep running the model again. Any help would be greatly appreciated.
Thanks,
Jeff Mills
Unknown loading error Dict of model, loss, params from Flux
Hi,
I have a problem that on server julia v1.2 I save oos predictions into a .bson. When I load those at this computer I successfully do. If I download them to my PC (Mac) julia v1.3.1 I get error see below. However, if I save, download, and load test.bson containing basic Dict() with randn(10,10) load works.
The file contains Dict{Symbol, Any} with 4 entries: ... and looks like this:
julia> aa = BSON.load("output/tt/asset_model_1_oos.bson")
Dict{Symbol,Any} with 4 entries:
:a_oos_res => Any[Any[[2952], 0.00335798, Float32[0.121619; 0.191023; … ;…
:a_model => Chain(Dense(11, 128, relu), Dense(128, 256, relu), Dense(25…
:asset => Any[1, :aapl, [-0.010887, 0.0324042, -0.0108924, 0.0177172,…
:a_params_dict => Dict{Symbol,Any}(:loss=>0.000304821,:pl=>0.0009687,:lr=>0.0…But when I download it to my computer and try to load it I get the error. I don't understand this behaviour and it is really frustraiting. I do not get the point why it works on one machine and not on the other one with this particular file and it does work with different new file on both.
BSON.load("output/tt/asset_model_1_oos.bson")
ERROR: InexactError: check_top_bit(Int64, -1)
Stacktrace:
[1] throw_inexacterror(::Symbol, ::Type{Int64}, ::Int64) at ./boot.jl:560
[2] check_top_bit at ./boot.jl:574 [inlined]
[3] toUInt64 at ./boot.jl:685 [inlined]
[4] UInt64 at ./boot.jl:715 [inlined]
[5] convert(::Type{UInt64}, ::Int64) at ./number.jl:7
[6] newstruct!(::Core.CodeInfo, ::Array{Any,1}, ::Array{Int32,1}, ::Int64, ::Array{UInt8,1}, ::Nothing, ::Array{Any,1}, ::Array{Any,1}, ::Array{UInt8,1}, ::Nothing, ::Type, ::Nothing, ::Int64, ::Vararg{Any,N} where N) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:98
[7] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:125
[8] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:140
[9] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[10] (::BSON.var"#41#42"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:124
[11] collect_to!(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./generator.jl:47
[12] collect_to!(::Array{Module,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:675
[13] collect_to_with_first!(::Array{Module,1}, ::Module, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Int64) at ./array.jl:646
[14] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:640
[15] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}) at ./array.jl:564
[16] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:2073
[17] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:124
[18] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:140
[19] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[20] (::BSON.var"#23#24"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[21] applychildren!(::BSON.var"#23#24"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:28
[22] raise_recursive at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94 [inlined]
[23] (::BSON.var"#23#24"{IdDict{Any,Any}})(::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[24] applychildren!(::BSON.var"#23#24"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:28
[25] raise_recursive(::Array{Any,1}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[26] newstruct_raw(::IdDict{Any,Any}, ::Type{Core.TypeName}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/anonymous.jl:71
[27] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:140
[28] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[29] (::BSON.var"#19#21"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[30] applychildren!(::BSON.var"#19#21"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:21
[31] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[32] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:89
[33] (::BSON.var"#23#24"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[34] applychildren!(::BSON.var"#23#24"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:28
[35] raise_recursive at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94 [inlined]
... (the last 7 lines are repeated 1 more time)
[43] (::BSON.var"#19#21"{IdDict{Any,Any}})(::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[44] applychildren!(::BSON.var"#19#21"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:21
[45] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[46] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:89
[47] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:138
[48] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[49] (::BSON.var"#20#22"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:82
[50] applychildren!(::BSON.var"#20#22"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:21
[51] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:82
[52] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:89
... (the last 4 lines are repeated 1 more time)
[57] raise_recursive at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:99 [inlined]
[58] load(::String) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:104FileIO `Stream` Interface not supported
julia> import FileIO
julia> FileIO.save(FileIO.Stream(FileIO.format"BSON", IOBuffer()), Dict(:hey=>3))
Error encountered while saving nothing.
Fatal error:
ERROR: MethodError: no method matching bson(::Nothing, ::Dict{Symbol,Int64})
Closest candidates are:
bson(::IO, ::AbstractDict) at /Users/goretkin/.julia/dev/BSON/src/write.jl:81
bson(::String, ::AbstractDict) at /Users/goretkin/.julia/dev/BSON/src/write.jl:83
Stacktrace:
[1] handle_error(::MethodError, ::Stream{DataFormat{:BSON},Base.GenericIOBuffer{Array{UInt8,1}}}) at /Users/goretkin/.julia/dev/FileIO/src/error_handling.jl:82
[2] handle_exceptions(::Array{Any,1}, ::String) at /Users/goretkin/.julia/dev/FileIO/src/error_handling.jl:77
[3] #save#30(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(save), ::Formatted, ::Any) at /Users/goretkin/.julia/dev/FileIO/src/loadsave.jl:211
[4] save(::Formatted, ::Any) at /Users/goretkin/.julia/dev/FileIO/src/loadsave.jl:194
[5] top-level scope at none:0Transfer to JuliaIO
In Slack, Mike mentioned that he'd be open to transferring the repository to a central location. I've given you the appropriate permissions to do so for JuliaIO, @MikeInnes.
UndefVarError: Flux not defined when creating the project
Built own project, the current code is:
module NewProject
import Flux
import BSON
using Flux, Tracker, BSON
function read_ANN()
@BSON.load "Processmodel2.bson" m
return m
end
m = read_ANN()
end # module
Project.toml and Manifest.toml are OK, when I'm running this code directly - everything goes fine, when I'm tryin to import this project - I get the ERROR: LoadError: UndefVarError: Flux not defined caused by BSON
Cannot save array with undefined values
I am opening this issue so we can keep track of our progress in solving this bug.
Currently, BSON is unable to save an array that has one or more undefined values. Here is the minimum working example:
using BSON
x = Array{String, 1}(undef, 5)
x[1] = "a"
x[4] = "d"
println(x)
bson("test.bson", Dict(:x => x))Expected behavior: BSON saves the data as requested.
Actual behavior: the following error is thrown:
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] lower at /Users/dilum/.julia/packages/BSON/XPZLD/src/extensions.jl:66 [inlined]
[2] _lower_recursive(::Array{String,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:62
[3] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{String,1}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:62
[4] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/BSON.jl:21
[5] _lower_recursive(::Dict{Symbol,Array{String,1}}, ::IdDict{Any,Any}, ::Array{Any,1}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:62
[6] lower_recursive(::Dict{Symbol,Array{String,1}}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:73
[7] bson(::IOStream, ::Dict{Symbol,Array{String,1}}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:81
[8] #14 at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:83 [inlined]
[9] #open#310(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Array{String,1}}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:375
[10] open at ./iostream.jl:373 [inlined]
[11] bson(::String, ::Dict{Symbol,Array{String,1}}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:83
[12] top-level scope at none:0
compress bson file with CodecZstd.jl ?
Can I save a compressed BSON file?
such as using CodecZstd.jl ?
e.g. Pandas has a parameter compression='infer'
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
Can't save large file due to conversion to Int32
https://github.com/MikeInnes/BSON.jl/blob/7ed00f318f39cfa90a929927132ec252338745a6/src/write.jl#L32
Hi!
I'm using BSON to save a DataFrame with some custom types. For a large DataFrame (about 3.2 GB) this fails at the line indicated because, I think, 3.2GB is larger than typemax(Int32) (about 2.2GB).
Is BSON unable to save larger files or is the cast to Int32 unnecessary and could also be changed to Int64?
My exact trace is:
ERROR: LoadError: InexactError: trunc(Int32, 2397876411)
Stacktrace:
[1] throw_inexacterror(::Symbol, ::Any, ::Int64) at ./boot.jl:567
[2] checked_trunc_sint at ./boot.jl:589 [inlined]
[3] toInt32 at ./boot.jl:626 [inlined]
[4] Type at ./boot.jl:716 [inlined]
[5] bson_doc(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Pair{String,_1} where _1,1}) at [...]/.julia/ packages/BSON/kxdIr/src/write.jl:32
[6] bson_primitive(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Any,1}) at [...]/.julia/packages/BSON/kxdIr/ src/write.jl:37
[7] bson_pair(::Base.GenericIOBuffer{Array{UInt8,1}}, ::String, ::Array{Any,1}) at [...]/.julia/packages/BSON/ kxdIr/src/write.jl:22
[8] bson_doc(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Pair{String,_1} where _1,1}) at [...]/.julia/ packages/BSON/kxdIr/src/write.jl:28
[9] bson_primitive(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Any,1}) at [...]/.julia/packages/BSON/kxdIr/ src/write.jl:37
[10] bson_pair(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Symbol, ::Array{Any,1}) at [...]/.julia/packages/BSON/ kxdIr/src/write.jl:22
[11] bson_doc(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Dict{Symbol,Any}) at [...]/.julia/packages/BSON/kxdIr/src/ write.jl:28
[12] bson_primitive at [...]/.julia/packages/BSON/kxdIr/src/write.jl:36 [inlined]
[13] bson_pair(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Symbol, ::Dict{Symbol,Any}) at [...]/.julia/packages/ BSON/kxdIr/src/write.jl:22
[14] bson_doc(::IOStream, ::Dict{Symbol,Any}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:28
[15] bson_primitive(::IOStream, ::Dict{Symbol,Any}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:36
[16] bson(::IOStream, ::Dict{Symbol,DataFrame}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:81
[17] #14 at [...]/.julia/packages/BSON/kxdIr/src/write.jl:83 [inlined]
[18] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,DataFrame}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
[19] open at ./iostream.jl:367 [inlined]
[20] bson(::String, ::Dict{Symbol,DataFrame}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:83
[21] top-level scope at none:0
[22] include at ./boot.jl:317 [inlined]
[23] include_relative(::Module, ::String) at ./loading.jl:1044
[24] include(::Module, ::String) at ./sysimg.jl:29
[25] exec_options(::Base.JLOptions) at ./client.jl:231
[26] _start() at ./client.jl:425
in expression starting [...]/runheadless.jl:67
edit:
So it seems like in general all write and read cast the length of the object to write or read to an Int32.
Maybe that's reasonable and I should just split up my DataFrame instead?
Cheers
Strange error while saving large BSON
I am getting a InexactError: trunc(Int32, 8431957686) error while saving a large BSON. I was wondering if you know what could be causing it and how to get around it. Unfortunately, I am not sure how to replicate the error.
Error on anonymous functions with keyword arguments
Looks like serializing functions with keyword arguments doesn't work:
julia> using BSON
julia> f = (;x)->x^2
#7 (generic function with 1 method)
julia> BSON.@save "test.bson" f
julia> BSON.@load "test.bson" f
julia> f(x=2)
ERROR: function #7 does not accept keyword arguments
Stacktrace:
[1] kwfunc(::Any) at ./boot.jl:330
[2] top-level scope at none:0
Replacing the above with a positional argument works as expected. Would be great if this could be fixed. I would use JLD2 but while that works with keyword arguments, it can't do closures unlike BSON. I'm on Julia 1.1.0 and BSON 0.2.3.
BSON file saves on 64bit system can not be loaded on 32bit system
When is an Core.Array{UInt8,1} not a Core.Array{UInt8,1} ?
Answer: When Core.Array{UInt8,1}).uid is not 27.
How does it get that way?
idk exactly but it looks like when BSON deserializing Dict{String, UInt8} that was serialized
on a 64 bit julia, on to a 32 bit julia,
BSON seems to be creating a new type with the same name in the same module.
but a different uid the true type.
This is from invenia/JLSO.jl#11
and is why the AppVeyor 32 bit tests are failing (but are not showing as I said allow failures)
but a JSLO file is just a BSON file anyway.
WE:
On 32 bit system only.
using BSON
filename = download("https://github.com/invenia/JLSO.jl/blob/4386519551912c54936ca89e61c4a352e0a2ebf8/test/specimens/v2_julia_serialize_none.jlso?raw=true")
begin
try
BSON.load(filename)
catch err
global err=err
@show typeof(err)
@show err.f
@show typeof.(err.args)
@show methods(err.f, map(typeof, err.args))
end
end
Output:
typeof(err) = MethodError
err.f = Array{UInt8,1}
typeof.(err.args) = (Array{UInt8,1},)
methods(err.f, map(typeof, err.args)) = # 0 methods for generic function "(::Type)":
# 0 methods for generic function "(::Type)":
I think a smaller working example can be created by running on a 64 bit computer
bson("mwe.bson", obj=Dict("name"=>[0x1, 0x2]))
but I have not tested.
world age issue in loading closure
While working at FluxML/model-zoo#209, I encountered the following problem when loading a saved closure within the function train:
julia> import BSON
julia> function train()
model = x -> x
BSON.@save "model.bson" model
model = BSON.load("model.bson")[:model]
ŷ = model(1)
end
train (generic function with 1 method)
julia> train()
ERROR: MethodError: no method matching (::BSON.__deserialized_types__.var"#94#95")(::Int64)
The applicable method may be too new: running in world age 27203, while current world is 27204.
Closest candidates are:
#94(::Any) at REPL[73]:2 (method too new to be called from this world context.)
Stacktrace:
[1] train() at ./REPL[73]:5
[2] top-level scope at REPL[74]:1Notice that if I just execute the body of the function in REPL, everything works fine:
julia> import BSON
julia> model = x -> x
#96 (generic function with 1 method)
julia> BSON.@save "model.bson" model
julia> model = BSON.load("model.bson")[:model]
#96 (generic function with 1 method)
julia> ŷ = model(1)
1Package compatibility caps
Error when saving Flux models/CuArrays from GPU
Saved Flux models or CuArrays while they are in GPU Memory can only be loaded again in the same julia session. Once this session is terminated and a new session is started, loading this data will either result in random values or in a CUDA error.
This can make trained and saved models useless since most of the time they will be loaded in a new julia session...
MWE (for CuArrays):
using BSON: @save, @load
using CUDAdrv
using CuArrays
using Flux
data = [1 2 3; 4 5 6]
data = data |> gpu
@show data
@save "data.bson" data
@load "data.bson" data
@show data
this gives me the correct output:
data = Float32[1.0 2.0 3.0; 4.0 5.0 6.0]
data = Float32[1.0 2.0 3.0; 4.0 5.0 6.0]
2×3 CuArray{Float32,2}:
1.0 2.0 3.0
4.0 5.0 6.0
Loading the data in a new session
using BSON: @load
using CUDAdrv
using CuArrays
using Flux
@load "data.bson" data
@show data
will result in an error:
ERROR: CUDA error: invalid argument (code #1, ERROR_INVALID_VALUE)
Stacktrace:
[1] macro expansion at /home/user/.julia/packages/CUDAdrv/WVU1H/src/base.jl:147 [inlined]
[2] #copy!#10(::Nothing, ::Bool, ::Function, ::Ptr{Float32}, ::CUDAdrv.Mem.DeviceBuffer, ::Int64) at /home/user/.julia/packages/CUDAdrv/WVU1H/src/memory.jl:344
[3] copy! at /home/user/.julia/packages/CUDAdrv/WVU1H/src/memory.jl:335 [inlined]
[4] copyto!(::Array{Float32,2}, ::Int64, ::CuArray{Float32,2}, ::Int64, ::Int64) at /home/user/.julia/packages/CuArrays/PwSdF/src/array.jl:194
[5] show(::Base.GenericIOBuffer{Array{UInt8,1}}, ::CuArray{Float32,2}) at /home/user/.julia/packages/GPUArrays/fAX0Q/src/abstractarray.jl:101
[6] #sprint#340(::Nothing, ::Int64, ::Function, ::Function, ::CuArray{Float32,2}) at ./strings/io.jl:101
[7] #sprint at ./none:0 [inlined]
[8] #repr#341 at ./strings/io.jl:208 [inlined]
[9] repr(::CuArray{Float32,2}) at ./strings/io.jl:208
[10] top-level scope at show.jl:555
The error occurs during the show command and not during loading!
I experienced the same issue when I tried to save Flux models. Saving and loading worked without errors but the loaded model had not the trained weights but random values.
The documentation of Flux only says that GPU support needs to be available when loading models which where in GPU memory when saved.
Explicitly document that loading of files is insecure
We had some discussion on slack that loading of adversarial BSON files is insecure by design and intention. Hence, this is not really a security issue, and requires no responsible disclosure. However, I think we could document this better, and maybe keep this issue open as catch-all discussion thread for security aspects of BSON.jl.
On slack, @oxinabox asked for a PoC. Find a PoC attached (expected behavior: parse should work without issue and allow you to inspect the file; load runs cat /etc/passwd before erroring out; you might consider using a VM for that).
Code execution is obtained by triggering a convert of a SubArray, which in turn triggers a getindex on a Base.Broadcast.Broadcasted(run, ([cat /etc/passwd],)).
Can't load Set's
Loading any type of Set seems to fail; the code
using BSON
x = Set(rand(3)) # as far as I can tell, this can be any type of Set
bson("test.bson", x=x)
BSON.load("test.bson")throws the following error:
ERROR: DivideError: integer division error
Stacktrace:
[1] rem at ./int.jl:233 [inlined]
[2] reinterpret at ./reinterpretarray.jl:39 [inlined]
[3] reinterpret_(::Type{Nothing}, ::Array{UInt8,1}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:62
[4] (::getfield(BSON, Symbol("##41#42")))(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:71
[5] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:71
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:81
[7] (::getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[8] iterate at ./generator.jl:47 [inlined]
[9] collect_to!(::Array{Array{Float64,1},1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:656
[10] collect_to_with_first!(::Array{Array{Float64,1},1}, ::Array{Float64,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Int64) at ./array.jl:643
[11] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:637
[12] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}) at ./array.jl:561
[13] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:1987
[14] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[15] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:124
[16] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:80
[17] (::getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[18] iterate at ./generator.jl:47 [inlined]
[19] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:632
[20] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}) at ./array.jl:561
[21] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:1987
[22] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[23] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:124
[24] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:80
[25] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:74
[26] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[27] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:74
[28] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:81
[29] raise_recursive at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[30] load(::String) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:96
[31] top-level scope at none:0This is on v1.0.2 on a Linux machine, in case it's relevant:
julia> versioninfo()
Julia Version 1.0.2
Commit d789231e99 (2018-11-08 20:11 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-3610QM CPU @ 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)[FR] Alter contents of a BSON file?
Nice work, BSON.jl is very useful.
Suppose one is running simulations that generate lots of data, perhaps more than can fit in RAM. In that case one would like to be able to save the data incrementally as it is generated.
Would it be feasible to add functionality to alter the contents of a BSON file? For example,
- append a variable and its value to a file, or
- change the value of a top-level variable, or
- change the value of a nested variable
Based on my limited understand of the BSON format, I imagine (1) might not be too hard, but (2) and (3) might be difficult since the storage needed for a variable in the middle of a file might change.
I could also imagine the answer might be, "In this case don't use BSON, use (some other storage approach)".
Save failed with the error: " expected Type, got Type{Vararg{Any,N} where N}"
I use BSON in Flux to save/load my model, it works well when I save the origin model. But it failed when I save the model which load from file.
Below is the code, the last line failed.
using BSON: @save, @load
a=1
@save "a.bson" a
@load "a.bson" a
@save "a.bson" a # works well
using Flux
m = Dense(10, 5)
@save "m.bson" m # works well
@load "m.bson" m
@save "m.bson" m # TypeError: in Type, in parameter, expected Type, got Type{Vararg{Any,N} where N}
Stack trace:
Stacktrace:
[1] typeof_ at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:50 [inlined]
[2] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\kx
dIr\src\write.jl:54
[3] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.julia\p
ackages\BSON\kxdIr\src\write.jl:62
[4] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}
) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[5] _lower_recursive(::Array{UnionAll,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\pac
kages\BSON\kxdIr\src\write.jl:62
[6] (::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}})(::Array{UnionAll,1}) at D:\ju
lia\.julia\packages\BSON\kxdIr\src\write.jl:53
[7] applychildren!(::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,A
ny}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[8] (::getfield(BSON, Symbol("#_lower#8")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.juli
a\packages\BSON\kxdIr\src\write.jl:53
[9] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\kx
dIr\src\write.jl:54
[10] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.julia\
packages\BSON\kxdIr\src\write.jl:62
[11] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[12] _lower_recursive(::Array{DataType,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\pa
ckages\BSON\kxdIr\src\write.jl:62
[13] (::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}})(::Array{DataType,1}) at D:\j
ulia\.julia\packages\BSON\kxdIr\src\write.jl:53
[14] applychildren!(::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,
Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[15] (::getfield(BSON, Symbol("#_lower#8")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.jul
ia\packages\BSON\kxdIr\src\write.jl:53
[16] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\k
xdIr\src\write.jl:54
[17] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.julia\
packages\BSON\kxdIr\src\write.jl:62
[18] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[19] _lower_recursive(::Flux.Tracker.Call{Nothing,Tuple}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\
julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[20] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Flux.Tracker.Call{Nothing
,Tuple}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[21] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[22] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\package
s\BSON\kxdIr\src\write.jl:62
[23] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at D:\julia
\.julia\packages\BSON\kxdIr\src\write.jl:62
[24] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[25] _lower_recursive(::Flux.Tracker.Tracked{Array{Float64,2}}, ::IdDict{Any,Any}, ::Array{Any,1})
at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[26] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Flux.Tracker.Tracked{Arra
y{Float64,2}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[27] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[28] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\package
s\BSON\kxdIr\src\write.jl:62
[29] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at D:\julia
\.julia\packages\BSON\kxdIr\src\write.jl:62
[30] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[31] _lower_recursive(::TrackedArray{…,Array{Float64,2}}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:
\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[32] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::TrackedArray{…,Array{Flo
at64,2}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[33] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[34] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\package
s\BSON\kxdIr\src\write.jl:62
[35] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at D:\julia
\.julia\packages\BSON\kxdIr\src\write.jl:62
[36] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[37] _lower_recursive(::Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},TrackedArray{…,Ar
ray{Float64,1}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\kxdIr\src\writ
e.jl:62
[38] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Dense{typeof(identity),Tr
ackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}) at D:\julia\.julia\packages\BSON
\kxdIr\src\write.jl:62
[39] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[40] _lower_recursive(::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},Tracke
dArray{…,Array{Float64,1}}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\k
xdIr\src\write.jl:62
[41] lower_recursive(::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},Tracked
Array{…,Array{Float64,1}}}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:73
[42] bson(::IOStream, ::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},Tracke
dArray{…,Array{Float64,1}}}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:81
[43] #14 at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:83 [inlined]
[44] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function,
::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64
,2}},TrackedArray{…,Array{Float64,1}}}}}, ::String, ::Vararg{String,N} where N) at .\iostream.jl:36
9
[45] open at .\iostream.jl:367 [inlined]
[46] bson(::String, ::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},TrackedA
rray{…,Array{Float64,1}}}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:83
[47] top-level scope at none:0
Freeze while trying to save custom struct
I am able to save a custom struct using BSON when one of the data members is a character but not when it is a string (it seems to hang). It is a nested struct, and the change that I am making is in the inner struct. I am not sure if this is related to padding (I did also at one point get ERROR: LoadError: Padding of type UInt8 is not compatible with type) or another known issue. I have been able to save the struct using JSON.
e.g. a::Int64 b::Int64 c::Char d::Bool saves but not a::Int64 b::Int64 c::String d::Bool
TypeError when deserialising `ReshapedArray`
using BSON
data = reinterpret(UInt64, zeros(9))
img = reshape(data, (3, 3))
BSON.save("img.bson", Dict(:img => img))
julia> BSON.load("img.bson")
ERROR: TypeError: in ReshapedArray, in MI, expected MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N}, got Type{Tuple}
Stacktrace:
[1] constructtype(::Type, ::Type, ::Vararg{Any,N} where N) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/extensions.jl:46
[2] (::getfield(BSON, Symbol("##37#38")))(::Dict{Symbol,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/extensions.jl:48
[3] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:71
[4] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:81
[5] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/extensions.jl:122
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:80
[7] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:74
[8] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[9] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:74
[10] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:81
[11] raise_recursive at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[12] load(::String) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:96
[13] top-level scope at none:0
Similar to x-ref: #18
SimpleVector not found
Is it too optimistic to expect this package to work with 0.7? SimpleVector was not found when I tried…
Recover saved Flux model from Julia 1.0 in Julia 1.1
Hi,
I have saved a trained Flux model in julia 1.0 a while ago and would like to retrieve it in julia 1.1 but I am getting the following error:
julia> BSON.load("model_car_1.bson")
ERROR: BoundsError: attempt to access Core.TypeofBottom
at index [1]
Stacktrace:
[1] newstruct!(::Type, ::Array{Any,1}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/extensions.jl:91
[2] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/extensions.jl:109
[3] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/extensions.jl:124
[4] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:80
[5] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:74
[6] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[7] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:74
[8] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:81
[9] raise_recursive at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[10] load(::String) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:96
[11] top-level scope at none:0In julia 1.0 I can load the model fine. I have the same version of Flux (v0.7.2) and BSON (v0.2.1) on both installations. This is I believe the versions that were used to train and save the old model.
Do you have any suggestions on how to load the model without error in julia 1.1?
Thanks
Array{Array{T, N}, 1} saved as Array{Any, 1}?
First, thanks for this awesome package!
I was wondering if the following is expected behavior:
using BSON
z = [ones(2, 3) for i in 1:2]
typeof(z) # Array{Array{Float64,2},1}
BSON.bson("test.bson", z=z)
d = BSON.load("test.bson")
typeof(d[:z]) # Array{Any,1}I would have expected typeof(d[:z]) == typeof(z). But that is not the case.
This seems to happen when typeof(z) is an array of arrays. But it doesn't happen with arrays. i.e.,
a = ones(3)
BSON.bson("a.bson", a=a)
d2 = BSON.load("a.bson")
typeof(d2[:a]) # Array{Float64, 1}returns Array{Float64, 1} as expected.
I came across #26, but that seems like a different issue.
Thanks.
BSON load Array{typeof(σ),1} problem
Hi,
I have a setup for modelling using Flux and I change activation functions. When I have my activation functions in an array and those are different:
julia> [relu, sigmoid]
2-element Array{Function,1}:
relu (generic function with 2 methods)
σ (generic function with 4 methods)the type of the array is Function and when saved in BSON after working with the array then the type can be Any and this is fine for loading .bson.
But, when the array of activation functions is only sigmoids then the type of the array is typeof(σ), such as:
julia> [sigmoid, sigmoid]
2-element Array{typeof(σ),1}:
σ (generic function with 4 methods)
σ (generic function with 4 methods)This works fine when having the dictionary with these but it does not work well when saved and then loaded. I got this error:
(Also posted on slack: https://julialang.slack.com/archives/C7LFJTXV5/p1610360933246600 )
julia> BSON.load(file_with_sigmoids)
ERROR: MethodError: no method matching typeof(σ)()
Stacktrace:
[1] (::BSON.var"#37#38")(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:76
[2] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:79
[3] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:89
[4] (::BSON.var"#39#40"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[5] iterate at ./generator.jl:47 [inlined]
[6] collect_to!(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:732
[7] collect_to!(::Array{Real,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:740 (repeats 3 times)
[8] collect_to_with_first!(::Array{Int64,1}, ::Int64, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64) at ./array.jl:710
[9] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:704
[10] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}) at ./array.jl:628
[11] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:2162
[12] newstruct_raw(::IdDict{Any,Any}, ::Type{T} where T, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[13] (::BSON.var"#43#44")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:140
[14] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:88
[15] (::BSON.var"#21#22"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:94
[16] applychildren!(::BSON.var"#21#22"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/BSON.jl:28
[17] raise_recursive at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:94 [inlined]
[18] (::BSON.var"#39#40"{IdDict{Any,Any}})(::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[19] iterate at ./generator.jl:47 [inlined]
[20] collect_to!(::Array{Array{Any,1},1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:732
[21] collect_to_with_first!(::Array{Array{Any,1},1}, ::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64) at ./array.jl:710
[22] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:704
[23] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}) at ./array.jl:628
[24] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:2162
[25] newstruct_raw(::IdDict{Any,Any}, ::Type{T} where T, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[26] (::BSON.var"#43#44")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:140
[27] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:88
[28] raise_recursive at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:99 [inlined]
[29] load(::String) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:104
[30] top-level scope at REPL[20]:1I have figured a workaround for my case and I pushed the array to be saved as Any, but with the results I already have I don't know what to do. (Will recompute all.) Maybe I don't see my own mistake there in loading or saving, but, isn't there just an easy fix with the raise_recursive ?
Thanks.
BSON error when used to load weights in module / package
I am trying something like the following and the error I get is ERROR: UndefVarError: Tracker not defined
Steps to replicate this -
- Let
ExampleModule.jlfile be the following
module ExampleModule
using BSON, Tracker, Flux
include("./file1.jl")
end
- And here is the code in
file1.jl
function load_weights(path) # Loading weights
c = BSON.load(path)
return c
end
- Create and save weights like
using Flux; using BSON: @save;,w = param(rand(2,2))and then@save "wt.bson" w. - In a new REPL session , if I do the following -
julia> include("ExampleModule.jl")
julia> ExampleModule.load_weights("wt.bson")
ERROR: UndefVarError: Tracker not defined
It throws an error. The same thing happens if w in step 3. is replaced by Flux.Conv as well.
Also if I do using Tracker, Flux at the top level (i.e. in REPL session - step 4.) this starts working fine. Exporting Tracker and Flux also work fine, except for the tests.
The above example is a simplified version to recreate the error I am facing in JuliaText/TextAnalysis.jl#167
Other Info
Flux v0.8.3
BSON v0.2.3
Julia 1.0.3
OS - Ubuntu 18.04.2 LTS - 64-bit although with Travis CI, doesn't work on osx.
From the looks of it, BSON searches for Tracker and Flux at the topmost level. Am I missing something here?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.