juliaio / bson.jl Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Hi,
I tried to serialize an object with a type parameter and deserialize it from a BSON file. However, it doesn't work because BSON.jl cannot recover the parameter type in a case like below:
using BSON
struct Foo{T}
data::Vector{T}
end
struct Bar
data::Vector{Foo{T}} where T
end
obj = Bar([Foo([1])])
buf = IOBuffer()
BSON.bson(buf, Dict(:data => obj.data))
seekstart(buf)
@show typeof(BSON.load(buf)[:data]) # typeof((BSON.load(buf))[:data]) = Array{Any,1}
seekstart(buf)
Bar(BSON.load(buf)[:data]) # This won't work.
I'm using BSON.jl v0.2.1 and Julia 1.0.1 on macOS.
Is this a bug or a limitation of this package? Thank you in advance.
Would it be possible to make BSON compatible with FileIO like JLD or JLD2? Right now I can save/load model files in different formats relying on FileIO figuring out the format looking at file extension and/or magic numbers and would like to do the same with BSON.
Nice package, Mike! I found this obscure error message when saving this custom AbstractArray type from SASLib.jl.
julia> using SASLib
julia> p = SASLib.ObjectPool{String,UInt8}("", 3)
3-element SASLib.ObjectPool{String,UInt8}:
""
""
""
julia> BSON.@save "test.bson" p
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] lower(::Array{String,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/extensions.jl:60
[2] _lower_recursive(::Array{String,1}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[3] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:25
[4] _lower_recursive(::Array{Any,1}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[5] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:18
[6] _lower_recursive(::Dict{String,UInt8}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[7] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:25
[8] _lower_recursive(::Array{Any,1}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[9] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:18
[10] _lower_recursive(::SASLib.ObjectPool{String,UInt8}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[11] applychildren!(::BSON.##7#11{ObjectIdDict,Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/tomkwong/.julia/v0.6/BSON/src/BSON.jl:18
[12] _lower_recursive(::Dict{Symbol,SASLib.ObjectPool{String,UInt8}}, ::ObjectIdDict, ::Array{Any,1}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:62
[13] lower_recursive(::Dict{Symbol,SASLib.ObjectPool{String,UInt8}}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:73
[14] open(::BSON.##16#17{Dict{Symbol,SASLib.ObjectPool{String,UInt8}}}, ::String, ::String) at ./iostream.jl:152
[15] bson(::String, ::Dict{Symbol,SASLib.ObjectPool{String,UInt8}}) at /Users/tomkwong/.julia/v0.6/BSON/src/write.jl:83
Hi,
I just encountered following side effect (in a more complex scenario, boiled down to the minimum):
using BSON
a = Dict{Symbol, Any}(:a => Float32.([1]))
BSON.@save "/tmp/deleteme.bson" a
After that a is changed to contain all the type information that is probably generated to write the output.
Dict{Symbol,Any} with 1 entry:
:t => Dict{Symbol,Any}(:tag=>"array",:type=>Dict{Symbol,Any}(:tag=>"datatype",:params=>Any[],:name=>Any["Core", "Float32"]),:size=>Any[3],:data=>UInt8[0x00, 0x00, 0x80, 0x3f, 0x00, 0x00, 0x00, 0x40, 0x00, 0…
After loading the dict again:
BSON.@load "/tmp/deleteme.bson" a
A is restored in its original form. To me this was unexpected behaviour and I did not find any documentation that there may be side effects when using @save.
System:
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: AMD Ryzen 7 4800H with Radeon Graphics
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, znver2)
BSON:
[fbb218c0] BSON v0.2.6
Apart from that minor issue, thanks for this fabulous library!
It seems, Using BSON library I can save Ordered collections but can't load them.
I saved one variable and when I tried to load it again . It showed me error.
ERROR: UndefVarError: OrderedCollections not defined
Hello,
I have a large .bson
file (468 MB). After gzipping it to .bson.gz
, the file size is reduced to 169 MB (~3x compression). How feasible would it be to have a macro that automatically saves/loads a gz compressed bson file? Or is there a way to do so already using something like CodecZlib.jl
?
Interestingly, using the same data I tried an experiment with JLD2 files (see issue), and their file size is 615 MB (.jld2) and 187 MB (.jld2.gz). It seems like .bson
has better compression than .jld2
, although the loading speed for .jld2
files are 3-4x faster (1.5 sec vs 5 sec). Is this a correct interpretation?
Hello, one of the features for using BSON you mention is the backwards compatibility. But I have a very general question about this.
Suppose I save a type (struct) which is designed in a certain way and comes from a package, like this:
struct Foo
foo::String
bar::Int
end
After some time the design of this structure changes due to some improvements of the package or whatever...
struct Foo
foo::String
bar::Int
added_feature::Float64
end
Now I will not be able to load my old saved BSON file anymore because I cannot reproduce the right struct in my environment and I cannot recreate it from the saved data because BSON does not load anything.
I do not expect that a reconstructed struct will work properly but at least it would be great if it still would be possible to load it and, e.g., define a conversion method or just extract some data.
Is it possible to specify compression while storing BSON files?
When saving a struct composed of Flux Chain object with anonymous function BSON returns the following error:
ERROR: type Method has no field sparam_syms
It seems to be due to the presence of anonymous function.
I am using Julia 1.2, BSON v2.3, Flux v0.8.3
The same code is working in Julia 1.1 (and same Flux and BSON version)
Here is a MWE:
using Flux
using BSON
model = Chain(Conv((5,), 1 => 32, relu, stride=5),
x -> view(x, :, :, :, :),
x -> permutedims(x, (1, 2, 4, 3)),
Conv((1,32), 1 => 32, relu),
x -> reshape(x, (8, 32, :)),
x -> maximum(x, dims=1),
x -> reshape(x, (32, :)))
bson("model.bson", model=model)
Thanks for your help
julia> using Metalhead
julia> @time Metalhead.weights("vgg19.bson")
29.465177 seconds (433.86 M allocations: 13.256 GiB, 47.36% gc time)
This package is poorly designed (it's ok, I'm allowed to say that) in that it tries to support both plain BSON and Julia data structures in one go. This creates annoying corner cases (do we preserve the type tag in Dict{String,Int}
or store it in a compatible way?) and also makes it hard to safely load BSON anyway, because of the obvious security issues around loading Julia data (#50).
BSON.jl should strip out all serialisation utilities and just have a solid reader/writer. Other formats (model weights, Julia data) should be defined on top in separate packages. The new package could be a fork of BSON.jl, or we could just recommend JLSO, which is built on BSON anyway.
I don't know if JLSO supports closures or cycles, which are important to Flux, but it certainly has other nice features, and if its maintainers are open to those things it'd be my preferred option.
I am not sure where the following is coming from. It happened only with Unitful.rad
and Unitful.°
, so maybe the problem is in Unitful
?
import Unitful
import BSON
save_test = rand(ComplexF64, 100) .* Unitful.rad
BSON.@save "test.bson" save_test
BSON.@load "test.bson"
Stack trace:
DimensionMismatch("new dimensions (1,) must be consistent with array size 0")
Stacktrace:
[1] _throw_dmrsa(::Tuple{Int64}, ::Int64) at ./array.jl:322
[2] reshape at ./array.jl:304 [inlined]
[3] reshape(::Array{UInt8,1}, ::Int64) at ./reshapedarray.jl:99
[4] (::getfield(BSON, Symbol("##41#42")))(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/extensions.jl:71
[5] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[7] (::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[8] applychildren!(::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[9] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[10] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[11] (::getfield(BSON, Symbol("##25#26")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:86
[12] applychildren!(::getfield(BSON, Symbol("##25#26")){IdDict{Any,Any}}, ::Array{Any,1}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:28
[13] raise_recursive(::Array{Any,1}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:86
[14] (::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}})(::Array{Any,1}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[15] applychildren!(::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[16] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[17] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[18] (::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[19] applychildren!(::getfield(BSON, Symbol("##21#23")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[20] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:71
[21] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[22] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:74
[23] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[24] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:74
[25] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:81
[26] raise_recursive at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[27] load(::String) at /home/ubuntu/.julia/packages/BSON/kxdIr/src/read.jl:96
[28] top-level scope at /home/ubuntu/.julia/packages/BSON/kxdIr/src/BSON.jl:51
[29] top-level scope at In[43]:3
I am having problems deserialising BSON objects from within a package.
See this Julia discourse thread
This one's a little spooky, and took me a loooong time to debug, because the Stacktraces are completely misleading.
julia> using Plots, BSON
julia> plot(rand(10)); # now it works
julia> bson("/tmp/test.bson", p = plot(rand(10)))
julia> plot(rand(10)); # now it doesn't
ERROR: MethodError: no method matching plot_color(::Dict{Symbol,Any})
Closest candidates are:
plot_color(::Any, ::Nothing) at /home/ubuntu/.julia/packages/PlotUtils/NS6Tf/src/colors.jl:20
plot_color(::Any, ::Number) at /home/ubuntu/.julia/packages/PlotUtils/NS6Tf/src/colors.jl:23
plot_color(::Symbol) at /home/ubuntu/.julia/packages/PlotUtils/NS6Tf/src/colors.jl:7
...
Stacktrace:
[1] _update_plot_args(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/args.jl:1419
[2] _plot_setup(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Array{Dict{Symbol,Any},1}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/pipeline.jl:183
[3] plot_setup! at /home/ubuntu/.julia/packages/Plots/ey5o3/src/pipeline.jl:156 [inlined]
[4] recipe_pipeline!(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{Array{Float64,1}}) at /home/ubuntu/.julia/packages/RecipesPipeline/qM4Ea/src/RecipesPipeline.jl:86
[5] _plot!(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{Array{Float64,1}}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/plot.jl:167
[6] plot(::Array{Float64,1}; kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/plot.jl:57
[7] plot(::Array{Float64,1}) at /home/ubuntu/.julia/packages/Plots/ey5o3/src/plot.jl:51
[8] top-level scope at REPL[11]:1
All attempts to plot thereafter fail similarly.
That's with BSON v0.2.6, Plots v1.6.1, but I've had the same happen with Plots 1.5.3+.
Any idea what BSON.jl
is doing that might cause this? Type piracy?
Proposal how to fix #67 .
Based on discussion https://julialang.slack.com/archives/C67TK21LJ/p1596027666030200
Test if data fit to Int32 range and use it, othwerwise use 64bit range. #67 (comment) described where specifically the cast is happening.
Rationale beihind this: BSON.jl already violates BSON specification by allowing saving arbitrary types.
This follows same ides, as long as user makes sure input data follow BSON specification, it is compliant with the specification, but it also allows to persist data that don't follow BSON specification if it's required.
Writing with bson(file, dict)
seems to modify the dict
argument.
Is this the intended behaviour?
Julia-1.1.0> using BSON
Julia-1.1.0> d = Dict(:a => [1+2im, 3+4im], :b => "Hello, World!")
Dict{Symbol,Any} with 2 entries:
:a => Complex{Int64}[1+2im, 3+4im]
:b => "Hello, World!"
Julia-1.1.0> d2 = copy(d)
Dict{Symbol,Any} with 2 entries:
:a => Complex{Int64}[1+2im, 3+4im]
:b => "Hello, World!"
Julia-1.1.0> bson("test.bson", d)
Julia-1.1.0> d
Dict{Symbol,Any} with 2 entries:
:a => Dict{Symbol,Any}(:tag=>"array",:type=>Dict{Symbol,Any}(:tag=>"datatype",:params=>Any[Dict{Symbol,Any}(:tag=>"datatype",:params=>Any[],:name=>Any["Main", "Core", "Int64"])],:name=>Any["Main", …
:b => "Hello, World!"
Julia-1.1.0> d3 = BSON.load("test.bson")
Dict{Symbol,Any} with 2 entries:
:a => Complex{Int64}[1+2im, 3+4im]
:b => "Hello, World!"
Julia-1.1.0> d3 == d
false
Julia-1.1.0> d3 == d2
true
julia> using BSON
julia> x = ["foo", "bar", "baz"]
3-element Array{String,1}:
"foo"
"bar"
"baz"
julia> typeof(x)
Array{String,1}
julia> bson("test.bson", Dict(:x => x))
Now close Julia and open a new Julia session.
julia> using BSON
julia> data = BSON.load("test.bson")
Dict{Symbol,Any} with 1 entry:
:x => Any["foo", "bar", "baz"]
julia> data[:x]
3-element Array{Any,1}:
"foo"
"bar"
"baz"
julia> typeof(data[:x])
Array{Any,1}
This is somewhat similar to #41 .
Here's an MWE - Assume using BSON
and using BSON:@save
julia> a = ones(Float32, 2, 2)
2×2 Array{Float32,2}:
1.0 1.0
1.0 1.0
julia> @save "saved_on_64bit.bson" a
On a 32-bit system @test BSON.load("saved_on_64bit.bson")[:a] == ones(Float32, 2, 2)
gives the following stack trace https://gist.github.com/Ayushk4/2376db8679caa5e9f70c13abe89c981e
This was narrowed down to
Line 80 in a58c88a
This seems to be happening because on a 64-bit reshape(ones(Float32, 1, 4), 2, 2)
works whereas the following doesn't. Similarly, Int64 doesn't work on 32-bit-
julia> reshape(ones(Float32, 1, 4), Int32(2), Int32(2))
ERROR: MethodError: no method matching reshape(::Array{Float32,2}, ::Int32, ::Int32)
It can be fixed by changing the line to reshape(reinterpret_(d[:type], d[:data]), map(normalize_typeparams, d[:size])...) :
.
Versions -
BSON v0.2.4
Julia 1.3
Came here from JuliaText/TextAnalysis.jl#180
The URL of this package does not match that stored in METADATA.jl.
cc: @julia-tagbot[bot]
I discovered that BSON.jl can't round-trip a custom data type inside a module. MWE:
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.0.3 (2018-12-18)
_/ |\__'_|_|_|\__'_| |
|__/ |
(v1.0) pkg> activate .
julia> module A
using DataFrames, BSON
d = DataFrame(a = 1:10, b = rand(10))
bson("d.bson", Dict(:d=>d))
d2 = BSON.load("d.bson")[:d]
end
ERROR: UndefVarError: DataFrames not defined
Stacktrace:
[1] (::getfield(BSON, Symbol("##33#34")))(::Module, ::String) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:20
[2] mapfoldl_impl(::typeof(identity), ::getfield(BSON, Symbol("##33#34")), ::NamedTuple{(:init,),Tuple{Module}}, ::Array{Any,1}) at ./reduce.jl:43
[3] #mapfoldl#170(::Base.Iterators.Pairs{Symbol,Module,Tuple{Symbol},NamedTuple{(:init,),Tuple{Module}}}, ::Function, ::Function, ::Function, ::Array{Any,1}) at ./reduce.jl:70
[4] #mapfoldl at ./none:0 [inlined]
[5] _mapreduce_dim at ./reducedim.jl:303 [inlined]
[6] #mapreduce#536 at ./reducedim.jl:301 [inlined]
[7] #mapreduce at ./none:0 [inlined]
[8] #reduce#537 at ./reducedim.jl:345 [inlined]
[9] (::getfield(Base, Symbol("#kw##reduce")))(::NamedTuple{(:init,),Tuple{Module}}, ::typeof(reduce), ::Function, ::Array{Any,1}) at ./none:0
[10] resolve(::Array{Any,1}) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:20
[11] (::getfield(BSON, Symbol("##37#38")))(::Dict{Symbol,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:48
[12] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:71
[13] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:81
[14] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/extensions.jl:122
[15] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:80
[16] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:74
[17] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[18] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:74
[19] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:81
[20] raise_recursive at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[21] load(::String) at /home/dave/.julia/packages/BSON/kxdIr/src/read.jl:96
[22] top-level scope at none:0
[23] run_backend(::REPL.REPLBackend) at /home/dave/.julia/packages/Revise/EuQoV/src/Revise.jl:771
[24] (::getfield(Revise, Symbol("##58#60")){REPL.REPLBackend})() at ./task.jl:259
This works fine if it's run outside of the module A ... end
:
julia> using DataFrames, BSON
julia> d = DataFrame(a = 1:10, b = rand(10))
10×2 DataFrame
│ Row │ a │ b │
│ │ Int64 │ Float64 │
├─────┼───────┼───────────┤
│ 1 │ 1 │ 0.0805191 │
│ 2 │ 2 │ 0.44521 │
│ 3 │ 3 │ 0.126867 │
│ 4 │ 4 │ 0.466975 │
│ 5 │ 5 │ 0.782517 │
│ 6 │ 6 │ 0.904739 │
│ 7 │ 7 │ 0.928574 │
│ 8 │ 8 │ 0.268764 │
│ 9 │ 9 │ 0.822993 │
│ 10 │ 10 │ 0.0755515 │
julia> bson("d.bson", Dict(:d=>d))
julia> d2 = BSON.load("d.bson")[:d]
10×2 DataFrame
│ Row │ a │ b │
│ │ Int64 │ Float64 │
├─────┼───────┼───────────┤
│ 1 │ 1 │ 0.0805191 │
│ 2 │ 2 │ 0.44521 │
│ 3 │ 3 │ 0.126867 │
│ 4 │ 4 │ 0.466975 │
│ 5 │ 5 │ 0.782517 │
│ 6 │ 6 │ 0.904739 │
│ 7 │ 7 │ 0.928574 │
│ 8 │ 8 │ 0.268764 │
│ 9 │ 9 │ 0.822993 │
│ 10 │ 10 │ 0.0755515 │
and it also works fine if you do using DataFrames
in Main
first. I think the issue is https://github.com/MikeInnes/BSON.jl/blob/master/src/extensions.jl#L20 which hard codes the start of the search path for datatypes as Main
when it should be the current module (could be wrong about this though).
For context, I was hoping to use BSON.jl in a Weave.jl document with a sandbox module, so that the code in the notebook is evaled in a separate module, insulating it from the global workspace while still allowing for a long-running session (to avoid startup costs). JLD2 fails at this in a similar way (only looks for loaded packages in Main)
When I try to save an instance of TimerOutput
(from the package TimerOutputs
), I get
ERROR: MethodError: no method matching lower(::TimerOutputs.TimerOutput)
.
Do the types that attempt to get saved need to provide this function?
I tried this on the DifferentialEquations.jl solution type and it failed. Here's an MWE:
using OrdinaryDiffEq
f_2dlinear = (du,u,p,t) -> du.=1.01u
prob = ODEProblem(f_2dlinear,rand(2,2),(0.0,1.0))
sol1 =solve(prob,Euler();dt=1//2^(4))
using BSON
bson("test.bson",Dict(:sol1=>sol1))
# New session
using OrdinaryDiffEq
using BSON
BSON.load("test.bson")
which gives
ERROR: type ODESolution has no field module
Stacktrace:
[1] newstruct!(::DiffEqBase.ODESolution{Float64,3,Array{Array{Float64,2},1},Void,Void,Array{Float64
,1},Array{Array{Array{Float64,2},1},1},DiffEqBase.ODEProblem{Array{Float64,2},Float64,true,Void,BSON
.__deserialized_types__.##3#4,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem},Ordinar
yDiffEq.Euler,OrdinaryDiffEq.InterpolationData{BSON.__deserialized_types__.##3#4,Array{Array{Float64
,2},1},Array{Float64,1},Array{Array{Array{Float64,2},1},1},OrdinaryDiffEq.EulerCache{Array{Float64,2
},Array{Float64,2}}}}, ::Array{Any,1}, ::Void, ::Void, ::Array{Float64,1}, ::Array{Any,1}, ::DiffEqB
ase.ODEProblem{Array{Float64,2},Float64,true,Void,BSON.__deserialized_types__.##3#4,Void,Void,Unifor
mScaling{Int64},DiffEqBase.StandardODEProblem}, ::OrdinaryDiffEq.Euler, ::OrdinaryDiffEq.Interpolati
onData{BSON.__deserialized_types__.##3#4,Array{Array{Float64,2},1},Array{Float64,1},Array{Array{Arra
y{Float64,2},1},1},OrdinaryDiffEq.EulerCache{Array{Float64,2},Array{Float64,2}}}, ::Bool, ::Int64, :
:Symbol) at C:\Users\Chris\.julia\v0.6\BSON\src\anonymous.jl:12
[2] newstruct_raw(::ObjectIdDict, ::Type{T} where T, ::Dict{Symbol,Any}) at C:\Users\Chris\.julia\v
0.6\BSON\src\extensions.jl:102
[3] (::BSON.##53#54)(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\ext
ensions.jl:117
[4] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\read
.jl:80
[5] (::BSON.##24#26{ObjectIdDict})(::Dict{Symbol,Any}) at C:\Users\Chris\.julia\v0.6\BSON\src\read.
jl:74
[6] applychildren!(::BSON.##24#26{ObjectIdDict}, ::Dict{Symbol,Any}) at C:\Users\Chris\.julia\v0.6\
BSON\src\BSON.jl:18
[7] _raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\rea
d.jl:74
[8] raise_recursive(::Dict{Symbol,Any}, ::ObjectIdDict) at C:\Users\Chris\.julia\v0.6\BSON\src\read
.jl:81
[9] load(::String) at C:\Users\Chris\.julia\v0.6\BSON\src\read.jl:96
[10] eval(::Module, ::Any) at .\boot.jl:235
[11] eval(::Any) at .\boot.jl:234
[12] macro expansion at C:\Users\Chris\.julia\v0.6\Atom\src\repl.jl:117 [inlined]
[13] anonymous at .\<missing>:?
Failing example:
using BSON
f = x -> 2x
BSON.@save "test.bson" f
Stacktrace:
ERROR: LoadError: type Method has no field ambig
Stacktrace:
[1] getproperty(x::Method, f::Symbol)
@ Base ./Base.jl:33
[2] structdata(meth::Method)
@ BSON ~/.julia/packages/BSON/XAts7/src/anonymous.jl:15
[3] lower(x::Method)
@ BSON ~/.julia/packages/BSON/XAts7/src/extensions.jl:91
[4] _lower_recursive(x::Method, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[5] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Method)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[6] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:28
[7] _lower_recursive(x::Vector{Method}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[8] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Vector{Method})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[9] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:28
[10] _lower_recursive(x::Vector{Any}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[11] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[12] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:28
[13] _lower_recursive(x::Vector{Any}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[14] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[15] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[16] _lower_recursive(x::Core.TypeName, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[17] (::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}})(x::Core.TypeName)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:53
[18] applychildren!(f::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[19] (::BSON.var"#_lower#6"{IdDict{Any, Any}, Vector{Any}})(x::Type)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:53
[20] _lower_recursive(x::Type, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:54
[21] (::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}})(x::Type)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:53
[22] applychildren!(f::BSON.var"#3#7"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[23] _lower
@ ~/.julia/packages/BSON/XAts7/src/write.jl:53 [inlined]
[24] _lower_recursive(x::Function, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:54
[25] (::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}})(x::Function)
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[26] applychildren!(f::BSON.var"#5#9"{IdDict{Any, Any}, Vector{Any}}, x::Dict{Symbol, Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/BSON.jl:21
[27] _lower_recursive(x::Dict{Symbol, var"#1#2"}, cache::IdDict{Any, Any}, refs::Vector{Any})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:62
[28] lower_recursive(y::Dict{Symbol, var"#1#2"})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:73
[29] bson(io::IOStream, doc::Dict{Symbol, var"#1#2"})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:81
[30] #12
@ ~/.julia/packages/BSON/XAts7/src/write.jl:83 [inlined]
[31] open(::BSON.var"#12#13"{Dict{Symbol, var"#1#2"}}, ::String, ::Vararg{String, N} where N; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Base ./io.jl:330
[32] open
@ ./io.jl:328 [inlined]
[33] bson(path::String, doc::Dict{Symbol, var"#1#2"})
@ BSON ~/.julia/packages/BSON/XAts7/src/write.jl:83
[34] top-level scope
@ ~/test.jl:3
using Flux
using BSON: @save, @load
m = Chain(Dense(23, 10))
@save "model.bson" m
This produces the following error:
StackOverflowError:
Stacktrace:
[1] modpath(::Module) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/extensions.jl:24 (repeats 43609 times)
[2] typepath at /home/avik-pal/.julia/packages/BSON/zmGJP/src/extensions.jl:33 [inlined]
[3] lower(::DataType) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/extensions.jl:37
[4] (::getfield(BSON, Symbol("#_lower#8")){IdDict{Any,Any},Array{Any,1}})(::Type) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:53
[5] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:54
[6] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[7] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/BSON.jl:20
[8] _lower_recursive(::Chain, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[9] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Chain) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[10] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/BSON.jl:20
[11] _lower_recursive(::Dict{Symbol,Chain}, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:62
[12] lower_recursive(::Dict{Symbol,Chain}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:73
[13] bson(::IOStream, ::Dict{Symbol,Chain}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:81
[14] #14 at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:83 [inlined]
[15] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Chain}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
[16] open at ./iostream.jl:367 [inlined]
[17] bson(::String, ::Dict{Symbol,Chain}) at /home/avik-pal/.julia/packages/BSON/zmGJP/src/write.jl:83
[18] top-level scope at In[23]:2
I tried BSON.jl
with a large dataset. Cf JLD2:
julia> @time load("/tmp/test.jld2")
5.885191 seconds (21.85 M allocations: 2.056 GiB, 60.13% gc time)
Dict{String,Any} with 2 entries:
"levels" => 2247195×6 DataFrames.DataFrame…
"transitions" => 3775512×7 DataFrames.DataFrame…
whereas BSON.load("/tmp/test.bson")
never terminates (I interrupted after 5 minutes). The file itself is around 600M (slightly smaller than the JLD2 file which is around 700M). I can make an MWE if the issue is not known.
Up until today (10/30/2020) I have run the following code numerous times with no error. Today I get the following error:
using BSON: @load, @save
using AxisArrays
using CSV, DataFrames, Plots, StatsPlots, Distributions
using Turing, LinearAlgebra
@load "heterog_model_hmc_chains_02prior.bson" cc
ERROR: TypeError: in Type, in A, expected A<:(AxisArray{Union{Missing, Real},3,D,Ax} where Ax where D), got Type{Missing}
Stacktrace:
[1] constructtype(::Type{T} where T, ::Array{Any,1}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\extensions.jl:50
[2] (::BSON.var"#33#34")(::Dict{Symbol,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\extensions.jl:53
[3] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:79
[4] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:89
[5] (::BSON.var"#43#44")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\extensions.jl:138
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:88
[7] (::BSON.var"#18#20"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:82
[8] applychildren!(::BSON.var"#18#20"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\BSON.jl:21
[9] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:82
[10] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:89
[11] raise_recursive at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:99 [inlined]
[12] load(::String) at C:\Users\millsjf.julia\packages\BSON\XAts7\src\read.jl:104
[13] top-level scope at C:\Users\millsjf.julia\packages\BSON\XAts7\src\BSON.jl:52
I have tried updating the packages. The same error occurs for different BSON files, so it's not a corrupted file.
It takes several hours to estimate the model, so I really need to be able to load the MCMC chains rather than keep running the model again. Any help would be greatly appreciated.
Thanks,
Jeff Mills
Hi,
I have a problem that on server julia v1.2
I save oos predictions into a .bson
. When I load those at this computer I successfully do. If I download them to my PC (Mac) julia v1.3.1
I get error see below. However, if I save, download, and load test.bson
containing basic Dict()
with randn(10,10)
load works.
The file contains Dict{Symbol, Any} with 4 entries: ...
and looks like this:
julia> aa = BSON.load("output/tt/asset_model_1_oos.bson")
Dict{Symbol,Any} with 4 entries:
:a_oos_res => Any[Any[[2952], 0.00335798, Float32[0.121619; 0.191023; … ;…
:a_model => Chain(Dense(11, 128, relu), Dense(128, 256, relu), Dense(25…
:asset => Any[1, :aapl, [-0.010887, 0.0324042, -0.0108924, 0.0177172,…
:a_params_dict => Dict{Symbol,Any}(:loss=>0.000304821,:pl=>0.0009687,:lr=>0.0…
But when I download it to my computer and try to load it I get the error. I don't understand this behaviour and it is really frustraiting. I do not get the point why it works on one machine and not on the other one with this particular file and it does work with different new file on both.
BSON.load("output/tt/asset_model_1_oos.bson")
ERROR: InexactError: check_top_bit(Int64, -1)
Stacktrace:
[1] throw_inexacterror(::Symbol, ::Type{Int64}, ::Int64) at ./boot.jl:560
[2] check_top_bit at ./boot.jl:574 [inlined]
[3] toUInt64 at ./boot.jl:685 [inlined]
[4] UInt64 at ./boot.jl:715 [inlined]
[5] convert(::Type{UInt64}, ::Int64) at ./number.jl:7
[6] newstruct!(::Core.CodeInfo, ::Array{Any,1}, ::Array{Int32,1}, ::Int64, ::Array{UInt8,1}, ::Nothing, ::Array{Any,1}, ::Array{Any,1}, ::Array{UInt8,1}, ::Nothing, ::Type, ::Nothing, ::Int64, ::Vararg{Any,N} where N) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:98
[7] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:125
[8] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:140
[9] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[10] (::BSON.var"#41#42"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:124
[11] collect_to!(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./generator.jl:47
[12] collect_to!(::Array{Module,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:675
[13] collect_to_with_first!(::Array{Module,1}, ::Module, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Int64) at ./array.jl:646
[14] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:640
[15] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#41#42"{IdDict{Any,Any}}}) at ./array.jl:564
[16] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:2073
[17] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:124
[18] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:140
[19] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[20] (::BSON.var"#23#24"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[21] applychildren!(::BSON.var"#23#24"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:28
[22] raise_recursive at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94 [inlined]
[23] (::BSON.var"#23#24"{IdDict{Any,Any}})(::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[24] applychildren!(::BSON.var"#23#24"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:28
[25] raise_recursive(::Array{Any,1}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[26] newstruct_raw(::IdDict{Any,Any}, ::Type{Core.TypeName}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/anonymous.jl:71
[27] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:140
[28] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[29] (::BSON.var"#19#21"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[30] applychildren!(::BSON.var"#19#21"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:21
[31] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[32] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:89
[33] (::BSON.var"#23#24"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94
[34] applychildren!(::BSON.var"#23#24"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:28
[35] raise_recursive at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:94 [inlined]
... (the last 7 lines are repeated 1 more time)
[43] (::BSON.var"#19#21"{IdDict{Any,Any}})(::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[44] applychildren!(::BSON.var"#19#21"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:21
[45] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:79
[46] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:89
[47] (::BSON.var"#45#46")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/extensions.jl:138
[48] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:88
[49] (::BSON.var"#20#22"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:82
[50] applychildren!(::BSON.var"#20#22"{IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/BSON.jl:21
[51] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:82
[52] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:89
... (the last 4 lines are repeated 1 more time)
[57] raise_recursive at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:99 [inlined]
[58] load(::String) at /Users/luboshanus/.julia/packages/BSON/lY9i8/src/read.jl:104
julia> import FileIO
julia> FileIO.save(FileIO.Stream(FileIO.format"BSON", IOBuffer()), Dict(:hey=>3))
Error encountered while saving nothing.
Fatal error:
ERROR: MethodError: no method matching bson(::Nothing, ::Dict{Symbol,Int64})
Closest candidates are:
bson(::IO, ::AbstractDict) at /Users/goretkin/.julia/dev/BSON/src/write.jl:81
bson(::String, ::AbstractDict) at /Users/goretkin/.julia/dev/BSON/src/write.jl:83
Stacktrace:
[1] handle_error(::MethodError, ::Stream{DataFormat{:BSON},Base.GenericIOBuffer{Array{UInt8,1}}}) at /Users/goretkin/.julia/dev/FileIO/src/error_handling.jl:82
[2] handle_exceptions(::Array{Any,1}, ::String) at /Users/goretkin/.julia/dev/FileIO/src/error_handling.jl:77
[3] #save#30(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(save), ::Formatted, ::Any) at /Users/goretkin/.julia/dev/FileIO/src/loadsave.jl:211
[4] save(::Formatted, ::Any) at /Users/goretkin/.julia/dev/FileIO/src/loadsave.jl:194
[5] top-level scope at none:0
In Slack, Mike mentioned that he'd be open to transferring the repository to a central location. I've given you the appropriate permissions to do so for JuliaIO, @MikeInnes.
Built own project, the current code is:
module NewProject
import Flux
import BSON
using Flux, Tracker, BSON
function read_ANN()
@BSON.load "Processmodel2.bson" m
return m
end
m = read_ANN()
end # module
Project.toml and Manifest.toml are OK, when I'm running this code directly - everything goes fine, when I'm tryin to import this project - I get the ERROR: LoadError: UndefVarError: Flux not defined caused by BSON
I am opening this issue so we can keep track of our progress in solving this bug.
Currently, BSON is unable to save an array that has one or more undefined values. Here is the minimum working example:
using BSON
x = Array{String, 1}(undef, 5)
x[1] = "a"
x[4] = "d"
println(x)
bson("test.bson", Dict(:x => x))
Expected behavior: BSON saves the data as requested.
Actual behavior: the following error is thrown:
ERROR: UndefRefError: access to undefined reference
Stacktrace:
[1] lower at /Users/dilum/.julia/packages/BSON/XPZLD/src/extensions.jl:66 [inlined]
[2] _lower_recursive(::Array{String,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:62
[3] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{String,1}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:62
[4] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/BSON.jl:21
[5] _lower_recursive(::Dict{Symbol,Array{String,1}}, ::IdDict{Any,Any}, ::Array{Any,1}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:62
[6] lower_recursive(::Dict{Symbol,Array{String,1}}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:73
[7] bson(::IOStream, ::Dict{Symbol,Array{String,1}}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:81
[8] #14 at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:83 [inlined]
[9] #open#310(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Array{String,1}}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:375
[10] open at ./iostream.jl:373 [inlined]
[11] bson(::String, ::Dict{Symbol,Array{String,1}}) at /Users/dilum/.julia/packages/BSON/XPZLD/src/write.jl:83
[12] top-level scope at none:0
Can I save a compressed BSON file?
such as using CodecZstd.jl ?
e.g. Pandas has a parameter compression='infer'
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
https://github.com/MikeInnes/BSON.jl/blob/7ed00f318f39cfa90a929927132ec252338745a6/src/write.jl#L32
Hi!
I'm using BSON
to save a DataFrame
with some custom types. For a large DataFrame
(about 3.2 GB) this fails at the line indicated because, I think, 3.2GB is larger than typemax(Int32)
(about 2.2GB).
Is BSON
unable to save larger files or is the cast to Int32
unnecessary and could also be changed to Int64
?
My exact trace is:
ERROR: LoadError: InexactError: trunc(Int32, 2397876411)
Stacktrace:
[1] throw_inexacterror(::Symbol, ::Any, ::Int64) at ./boot.jl:567
[2] checked_trunc_sint at ./boot.jl:589 [inlined]
[3] toInt32 at ./boot.jl:626 [inlined]
[4] Type at ./boot.jl:716 [inlined]
[5] bson_doc(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Pair{String,_1} where _1,1}) at [...]/.julia/ packages/BSON/kxdIr/src/write.jl:32
[6] bson_primitive(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Any,1}) at [...]/.julia/packages/BSON/kxdIr/ src/write.jl:37
[7] bson_pair(::Base.GenericIOBuffer{Array{UInt8,1}}, ::String, ::Array{Any,1}) at [...]/.julia/packages/BSON/ kxdIr/src/write.jl:22
[8] bson_doc(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Pair{String,_1} where _1,1}) at [...]/.julia/ packages/BSON/kxdIr/src/write.jl:28
[9] bson_primitive(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Any,1}) at [...]/.julia/packages/BSON/kxdIr/ src/write.jl:37
[10] bson_pair(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Symbol, ::Array{Any,1}) at [...]/.julia/packages/BSON/ kxdIr/src/write.jl:22
[11] bson_doc(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Dict{Symbol,Any}) at [...]/.julia/packages/BSON/kxdIr/src/ write.jl:28
[12] bson_primitive at [...]/.julia/packages/BSON/kxdIr/src/write.jl:36 [inlined]
[13] bson_pair(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Symbol, ::Dict{Symbol,Any}) at [...]/.julia/packages/ BSON/kxdIr/src/write.jl:22
[14] bson_doc(::IOStream, ::Dict{Symbol,Any}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:28
[15] bson_primitive(::IOStream, ::Dict{Symbol,Any}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:36
[16] bson(::IOStream, ::Dict{Symbol,DataFrame}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:81
[17] #14 at [...]/.julia/packages/BSON/kxdIr/src/write.jl:83 [inlined]
[18] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,DataFrame}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
[19] open at ./iostream.jl:367 [inlined]
[20] bson(::String, ::Dict{Symbol,DataFrame}) at [...]/.julia/packages/BSON/kxdIr/src/write.jl:83
[21] top-level scope at none:0
[22] include at ./boot.jl:317 [inlined]
[23] include_relative(::Module, ::String) at ./loading.jl:1044
[24] include(::Module, ::String) at ./sysimg.jl:29
[25] exec_options(::Base.JLOptions) at ./client.jl:231
[26] _start() at ./client.jl:425
in expression starting [...]/runheadless.jl:67
edit:
So it seems like in general all write
and read
cast the length of the object to write or read to an Int32
.
Maybe that's reasonable and I should just split up my DataFrame
instead?
Cheers
I am getting a InexactError: trunc(Int32, 8431957686)
error while saving a large BSON. I was wondering if you know what could be causing it and how to get around it. Unfortunately, I am not sure how to replicate the error.
Looks like serializing functions with keyword arguments doesn't work:
julia> using BSON
julia> f = (;x)->x^2
#7 (generic function with 1 method)
julia> BSON.@save "test.bson" f
julia> BSON.@load "test.bson" f
julia> f(x=2)
ERROR: function #7 does not accept keyword arguments
Stacktrace:
[1] kwfunc(::Any) at ./boot.jl:330
[2] top-level scope at none:0
Replacing the above with a positional argument works as expected. Would be great if this could be fixed. I would use JLD2 but while that works with keyword arguments, it can't do closures unlike BSON. I'm on Julia 1.1.0 and BSON 0.2.3.
Core.Array{UInt8,1}
not a Core.Array{UInt8,1}
?Answer: When Core.Array{UInt8,1}).uid
is not 27.
idk exactly but it looks like when BSON deserializing Dict{String, UInt8}
that was serialized
on a 64 bit julia, on to a 32 bit julia,
BSON seems to be creating a new type with the same name in the same module.
but a different uid
the true type.
This is from invenia/JLSO.jl#11
and is why the AppVeyor 32 bit tests are failing (but are not showing as I said allow failures)
but a JSLO file is just a BSON file anyway.
On 32 bit system only.
using BSON
filename = download("https://github.com/invenia/JLSO.jl/blob/4386519551912c54936ca89e61c4a352e0a2ebf8/test/specimens/v2_julia_serialize_none.jlso?raw=true")
begin
try
BSON.load(filename)
catch err
global err=err
@show typeof(err)
@show err.f
@show typeof.(err.args)
@show methods(err.f, map(typeof, err.args))
end
end
typeof(err) = MethodError
err.f = Array{UInt8,1}
typeof.(err.args) = (Array{UInt8,1},)
methods(err.f, map(typeof, err.args)) = # 0 methods for generic function "(::Type)":
# 0 methods for generic function "(::Type)":
I think a smaller working example can be created by running on a 64 bit computer
bson("mwe.bson", obj=Dict("name"=>[0x1, 0x2]))
but I have not tested.
While working at FluxML/model-zoo#209, I encountered the following problem when loading a saved closure within the function train
:
julia> import BSON
julia> function train()
model = x -> x
BSON.@save "model.bson" model
model = BSON.load("model.bson")[:model]
ŷ = model(1)
end
train (generic function with 1 method)
julia> train()
ERROR: MethodError: no method matching (::BSON.__deserialized_types__.var"#94#95")(::Int64)
The applicable method may be too new: running in world age 27203, while current world is 27204.
Closest candidates are:
#94(::Any) at REPL[73]:2 (method too new to be called from this world context.)
Stacktrace:
[1] train() at ./REPL[73]:5
[2] top-level scope at REPL[74]:1
Notice that if I just execute the body of the function in REPL, everything works fine:
julia> import BSON
julia> model = x -> x
#96 (generic function with 1 method)
julia> BSON.@save "model.bson" model
julia> model = BSON.load("model.bson")[:model]
#96 (generic function with 1 method)
julia> ŷ = model(1)
1
Saved Flux models or CuArrays while they are in GPU Memory can only be loaded again in the same julia session. Once this session is terminated and a new session is started, loading this data will either result in random values or in a CUDA error.
This can make trained and saved models useless since most of the time they will be loaded in a new julia session...
MWE (for CuArrays):
using BSON: @save, @load
using CUDAdrv
using CuArrays
using Flux
data = [1 2 3; 4 5 6]
data = data |> gpu
@show data
@save "data.bson" data
@load "data.bson" data
@show data
this gives me the correct output:
data = Float32[1.0 2.0 3.0; 4.0 5.0 6.0]
data = Float32[1.0 2.0 3.0; 4.0 5.0 6.0]
2×3 CuArray{Float32,2}:
1.0 2.0 3.0
4.0 5.0 6.0
Loading the data in a new session
using BSON: @load
using CUDAdrv
using CuArrays
using Flux
@load "data.bson" data
@show data
will result in an error:
ERROR: CUDA error: invalid argument (code #1, ERROR_INVALID_VALUE)
Stacktrace:
[1] macro expansion at /home/user/.julia/packages/CUDAdrv/WVU1H/src/base.jl:147 [inlined]
[2] #copy!#10(::Nothing, ::Bool, ::Function, ::Ptr{Float32}, ::CUDAdrv.Mem.DeviceBuffer, ::Int64) at /home/user/.julia/packages/CUDAdrv/WVU1H/src/memory.jl:344
[3] copy! at /home/user/.julia/packages/CUDAdrv/WVU1H/src/memory.jl:335 [inlined]
[4] copyto!(::Array{Float32,2}, ::Int64, ::CuArray{Float32,2}, ::Int64, ::Int64) at /home/user/.julia/packages/CuArrays/PwSdF/src/array.jl:194
[5] show(::Base.GenericIOBuffer{Array{UInt8,1}}, ::CuArray{Float32,2}) at /home/user/.julia/packages/GPUArrays/fAX0Q/src/abstractarray.jl:101
[6] #sprint#340(::Nothing, ::Int64, ::Function, ::Function, ::CuArray{Float32,2}) at ./strings/io.jl:101
[7] #sprint at ./none:0 [inlined]
[8] #repr#341 at ./strings/io.jl:208 [inlined]
[9] repr(::CuArray{Float32,2}) at ./strings/io.jl:208
[10] top-level scope at show.jl:555
The error occurs during the show command and not during loading!
I experienced the same issue when I tried to save Flux models. Saving and loading worked without errors but the loaded model had not the trained weights but random values.
The documentation of Flux only says that GPU support needs to be available when loading models which where in GPU memory when saved.
We had some discussion on slack that loading of adversarial BSON files is insecure by design and intention. Hence, this is not really a security issue, and requires no responsible disclosure. However, I think we could document this better, and maybe keep this issue open as catch-all discussion thread for security aspects of BSON.jl
.
On slack, @oxinabox asked for a PoC. Find a PoC attached (expected behavior: parse
should work without issue and allow you to inspect the file; load
runs cat /etc/passwd
before erroring out; you might consider using a VM for that).
Code execution is obtained by triggering a convert
of a SubArray
, which in turn triggers a getindex
on a Base.Broadcast.Broadcasted(run, ([
cat /etc/passwd],))
.
Loading any type of Set
seems to fail; the code
using BSON
x = Set(rand(3)) # as far as I can tell, this can be any type of Set
bson("test.bson", x=x)
BSON.load("test.bson")
throws the following error:
ERROR: DivideError: integer division error
Stacktrace:
[1] rem at ./int.jl:233 [inlined]
[2] reinterpret at ./reinterpretarray.jl:39 [inlined]
[3] reinterpret_(::Type{Nothing}, ::Array{UInt8,1}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:62
[4] (::getfield(BSON, Symbol("##41#42")))(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:71
[5] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:71
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:81
[7] (::getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[8] iterate at ./generator.jl:47 [inlined]
[9] collect_to!(::Array{Array{Float64,1},1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:656
[10] collect_to_with_first!(::Array{Array{Float64,1},1}, ::Array{Float64,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Int64) at ./array.jl:643
[11] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:637
[12] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}) at ./array.jl:561
[13] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:1987
[14] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[15] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:124
[16] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:80
[17] (::getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[18] iterate at ./generator.jl:47 [inlined]
[19] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:632
[20] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},getfield(BSON, Symbol("##43#44")){IdDict{Any,Any}}}) at ./array.jl:561
[21] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:1987
[22] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:108
[23] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/extensions.jl:124
[24] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:80
[25] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:74
[26] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[27] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:74
[28] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:81
[29] raise_recursive at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[30] load(::String) at /home/jon/.julia/packages/BSON/kxdIr/src/read.jl:96
[31] top-level scope at none:0
This is on v1.0.2 on a Linux machine, in case it's relevant:
julia> versioninfo()
Julia Version 1.0.2
Commit d789231e99 (2018-11-08 20:11 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-3610QM CPU @ 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)
Nice work, BSON.jl is very useful.
Suppose one is running simulations that generate lots of data, perhaps more than can fit in RAM. In that case one would like to be able to save the data incrementally as it is generated.
Would it be feasible to add functionality to alter the contents of a BSON file? For example,
Based on my limited understand of the BSON format, I imagine (1) might not be too hard, but (2) and (3) might be difficult since the storage needed for a variable in the middle of a file might change.
I could also imagine the answer might be, "In this case don't use BSON, use (some other storage approach)".
I use BSON in Flux to save/load my model, it works well when I save the origin model. But it failed when I save the model which load from file.
Below is the code, the last line failed.
using BSON: @save, @load
a=1
@save "a.bson" a
@load "a.bson" a
@save "a.bson" a # works well
using Flux
m = Dense(10, 5)
@save "m.bson" m # works well
@load "m.bson" m
@save "m.bson" m # TypeError: in Type, in parameter, expected Type, got Type{Vararg{Any,N} where N}
Stack trace:
Stacktrace:
[1] typeof_ at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:50 [inlined]
[2] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\kx
dIr\src\write.jl:54
[3] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.julia\p
ackages\BSON\kxdIr\src\write.jl:62
[4] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}
) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[5] _lower_recursive(::Array{UnionAll,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\pac
kages\BSON\kxdIr\src\write.jl:62
[6] (::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}})(::Array{UnionAll,1}) at D:\ju
lia\.julia\packages\BSON\kxdIr\src\write.jl:53
[7] applychildren!(::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,A
ny}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[8] (::getfield(BSON, Symbol("#_lower#8")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.juli
a\packages\BSON\kxdIr\src\write.jl:53
[9] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\kx
dIr\src\write.jl:54
[10] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.julia\
packages\BSON\kxdIr\src\write.jl:62
[11] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[12] _lower_recursive(::Array{DataType,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\pa
ckages\BSON\kxdIr\src\write.jl:62
[13] (::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}})(::Array{DataType,1}) at D:\j
ulia\.julia\packages\BSON\kxdIr\src\write.jl:53
[14] applychildren!(::getfield(BSON, Symbol("##5#9")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,
Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[15] (::getfield(BSON, Symbol("#_lower#8")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.jul
ia\packages\BSON\kxdIr\src\write.jl:53
[16] _lower_recursive(::Type, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\k
xdIr\src\write.jl:54
[17] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Type) at D:\julia\.julia\
packages\BSON\kxdIr\src\write.jl:62
[18] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[19] _lower_recursive(::Flux.Tracker.Call{Nothing,Tuple}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\
julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[20] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Flux.Tracker.Call{Nothing
,Tuple}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[21] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[22] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\package
s\BSON\kxdIr\src\write.jl:62
[23] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at D:\julia
\.julia\packages\BSON\kxdIr\src\write.jl:62
[24] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[25] _lower_recursive(::Flux.Tracker.Tracked{Array{Float64,2}}, ::IdDict{Any,Any}, ::Array{Any,1})
at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[26] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Flux.Tracker.Tracked{Arra
y{Float64,2}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[27] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[28] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\package
s\BSON\kxdIr\src\write.jl:62
[29] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at D:\julia
\.julia\packages\BSON\kxdIr\src\write.jl:62
[30] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[31] _lower_recursive(::TrackedArray{…,Array{Float64,2}}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:
\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[32] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::TrackedArray{…,Array{Flo
at64,2}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:62
[33] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1
}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:28
[34] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\package
s\BSON\kxdIr\src\write.jl:62
[35] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at D:\julia
\.julia\packages\BSON\kxdIr\src\write.jl:62
[36] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[37] _lower_recursive(::Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},TrackedArray{…,Ar
ray{Float64,1}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\kxdIr\src\writ
e.jl:62
[38] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Dense{typeof(identity),Tr
ackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}) at D:\julia\.julia\packages\BSON
\kxdIr\src\write.jl:62
[39] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol
,Any}) at D:\julia\.julia\packages\BSON\kxdIr\src\BSON.jl:21
[40] _lower_recursive(::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},Tracke
dArray{…,Array{Float64,1}}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at D:\julia\.julia\packages\BSON\k
xdIr\src\write.jl:62
[41] lower_recursive(::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},Tracked
Array{…,Array{Float64,1}}}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:73
[42] bson(::IOStream, ::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},Tracke
dArray{…,Array{Float64,1}}}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:81
[43] #14 at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:83 [inlined]
[44] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function,
::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64
,2}},TrackedArray{…,Array{Float64,1}}}}}, ::String, ::Vararg{String,N} where N) at .\iostream.jl:36
9
[45] open at .\iostream.jl:367 [inlined]
[46] bson(::String, ::Dict{Symbol,Dense{typeof(identity),TrackedArray{…,Array{Float64,2}},TrackedA
rray{…,Array{Float64,1}}}}) at D:\julia\.julia\packages\BSON\kxdIr\src\write.jl:83
[47] top-level scope at none:0
I am able to save a custom struct using BSON when one of the data members is a character but not when it is a string (it seems to hang). It is a nested struct, and the change that I am making is in the inner struct. I am not sure if this is related to padding (I did also at one point get ERROR: LoadError: Padding of type UInt8 is not compatible with type
) or another known issue. I have been able to save the struct using JSON.
e.g. a::Int64 b::Int64 c::Char d::Bool
saves but not a::Int64 b::Int64 c::String d::Bool
using BSON
data = reinterpret(UInt64, zeros(9))
img = reshape(data, (3, 3))
BSON.save("img.bson", Dict(:img => img))
julia> BSON.load("img.bson")
ERROR: TypeError: in ReshapedArray, in MI, expected MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N}, got Type{Tuple}
Stacktrace:
[1] constructtype(::Type, ::Type, ::Vararg{Any,N} where N) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/extensions.jl:46
[2] (::getfield(BSON, Symbol("##37#38")))(::Dict{Symbol,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/extensions.jl:48
[3] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:71
[4] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:81
[5] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/extensions.jl:122
[6] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:80
[7] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:74
[8] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[9] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:74
[10] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:81
[11] raise_recursive at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[12] load(::String) at /home/vchuravy/.julia/packages/BSON/kxdIr/src/read.jl:96
[13] top-level scope at none:0
Similar to x-ref: #18
Is it too optimistic to expect this package to work with 0.7? SimpleVector
was not found when I tried…
Hi,
I have saved a trained Flux model in julia 1.0 a while ago and would like to retrieve it in julia 1.1 but I am getting the following error:
julia> BSON.load("model_car_1.bson")
ERROR: BoundsError: attempt to access Core.TypeofBottom
at index [1]
Stacktrace:
[1] newstruct!(::Type, ::Array{Any,1}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/extensions.jl:91
[2] newstruct_raw(::IdDict{Any,Any}, ::Type, ::Dict{Symbol,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/extensions.jl:109
[3] (::getfield(BSON, Symbol("##47#48")))(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/extensions.jl:124
[4] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:80
[5] (::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}})(::Dict{Symbol,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:74
[6] applychildren!(::getfield(BSON, Symbol("##22#24")){IdDict{Any,Any}}, ::Dict{Symbol,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/BSON.jl:21
[7] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:74
[8] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:81
[9] raise_recursive at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:91 [inlined]
[10] load(::String) at /home/boutonm/.julia/packages/BSON/kxdIr/src/read.jl:96
[11] top-level scope at none:0
In julia 1.0 I can load the model fine. I have the same version of Flux (v0.7.2) and BSON (v0.2.1) on both installations. This is I believe the versions that were used to train and save the old model.
Do you have any suggestions on how to load the model without error in julia 1.1?
Thanks
First, thanks for this awesome package!
I was wondering if the following is expected behavior:
using BSON
z = [ones(2, 3) for i in 1:2]
typeof(z) # Array{Array{Float64,2},1}
BSON.bson("test.bson", z=z)
d = BSON.load("test.bson")
typeof(d[:z]) # Array{Any,1}
I would have expected typeof(d[:z]) == typeof(z)
. But that is not the case.
This seems to happen when typeof(z)
is an array of arrays. But it doesn't happen with arrays. i.e.,
a = ones(3)
BSON.bson("a.bson", a=a)
d2 = BSON.load("a.bson")
typeof(d2[:a]) # Array{Float64, 1}
returns Array{Float64, 1}
as expected.
I came across #26, but that seems like a different issue.
Thanks.
Hi,
I have a setup for modelling using Flux and I change activation functions. When I have my activation functions in an array and those are different:
julia> [relu, sigmoid]
2-element Array{Function,1}:
relu (generic function with 2 methods)
σ (generic function with 4 methods)
the type of the array is Function
and when saved in BSON after working with the array then the type can be Any
and this is fine for loading .bson
.
But, when the array of activation functions is only sigmoid
s then the type of the array is typeof(σ)
, such as:
julia> [sigmoid, sigmoid]
2-element Array{typeof(σ),1}:
σ (generic function with 4 methods)
σ (generic function with 4 methods)
This works fine when having the dictionary with these but it does not work well when saved and then loaded. I got this error:
(Also posted on slack: https://julialang.slack.com/archives/C7LFJTXV5/p1610360933246600 )
julia> BSON.load(file_with_sigmoids)
ERROR: MethodError: no method matching typeof(σ)()
Stacktrace:
[1] (::BSON.var"#37#38")(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:76
[2] _raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:79
[3] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:89
[4] (::BSON.var"#39#40"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[5] iterate at ./generator.jl:47 [inlined]
[6] collect_to!(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:732
[7] collect_to!(::Array{Real,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:740 (repeats 3 times)
[8] collect_to_with_first!(::Array{Int64,1}, ::Int64, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64) at ./array.jl:710
[9] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:704
[10] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}) at ./array.jl:628
[11] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:2162
[12] newstruct_raw(::IdDict{Any,Any}, ::Type{T} where T, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[13] (::BSON.var"#43#44")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:140
[14] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:88
[15] (::BSON.var"#21#22"{IdDict{Any,Any}})(::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:94
[16] applychildren!(::BSON.var"#21#22"{IdDict{Any,Any}}, ::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/BSON.jl:28
[17] raise_recursive at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:94 [inlined]
[18] (::BSON.var"#39#40"{IdDict{Any,Any}})(::Array{Any,1}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[19] iterate at ./generator.jl:47 [inlined]
[20] collect_to!(::Array{Array{Any,1},1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64, ::Int64) at ./array.jl:732
[21] collect_to_with_first!(::Array{Array{Any,1},1}, ::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Int64) at ./array.jl:710
[22] _collect(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:704
[23] collect_similar(::Array{Any,1}, ::Base.Generator{Array{Any,1},BSON.var"#39#40"{IdDict{Any,Any}}}) at ./array.jl:628
[24] map(::Function, ::Array{Any,1}) at ./abstractarray.jl:2162
[25] newstruct_raw(::IdDict{Any,Any}, ::Type{T} where T, ::Dict{Symbol,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:124
[26] (::BSON.var"#43#44")(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/extensions.jl:140
[27] raise_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:88
[28] raise_recursive at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:99 [inlined]
[29] load(::String) at /Users/luboshanus/.julia/packages/BSON/XAts7/src/read.jl:104
[30] top-level scope at REPL[20]:1
I have figured a workaround for my case and I pushed the array to be saved as Any
, but with the results I already have I don't know what to do. (Will recompute all.) Maybe I don't see my own mistake there in loading or saving, but, isn't there just an easy fix with the raise_recursive
?
Thanks.
I am trying something like the following and the error I get is ERROR: UndefVarError: Tracker not defined
ExampleModule.jl
file be the followingmodule ExampleModule
using BSON, Tracker, Flux
include("./file1.jl")
end
file1.jl
function load_weights(path) # Loading weights
c = BSON.load(path)
return c
end
using Flux; using BSON: @save;
, w = param(rand(2,2))
and then @save "wt.bson" w
.julia> include("ExampleModule.jl")
julia> ExampleModule.load_weights("wt.bson")
ERROR: UndefVarError: Tracker not defined
It throws an error. The same thing happens if w
in step 3. is replaced by Flux.Conv
as well.
Also if I do using Tracker, Flux
at the top level (i.e. in REPL session - step 4.) this starts working fine. Exporting Tracker and Flux also work fine, except for the tests.
The above example is a simplified version to recreate the error I am facing in JuliaText/TextAnalysis.jl#167
Flux v0.8.3
BSON v0.2.3
Julia 1.0.3
OS - Ubuntu 18.04.2 LTS - 64-bit although with Travis CI, doesn't work on osx.
From the looks of it, BSON searches for Tracker and Flux at the topmost level. Am I missing something here?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.