Deserialization as Vector{SubArray} breaks `push!` on DataFrame

[maleadt]

Deserialization as Vector{SubArray} breaks `push!` on DataFrame

[ericphanson]

oh I misunderstood, it's inside a nested vector. I guess copying those would do it?

df = DataFrame(Arrow.Table("/tmp/test.arrow"); copycols=true);
transform!(df, :foo => ByRow(copy) => :foo)

[ericphanson]

The workaround is to ask DataFrames to copy the columns:

DataFrame(Arrow.Table("/tmp/test.arrow")); copycols=true)

The reason for the current behavior is:

• Arrow.Table exposes an immutable view of the underlying byte-buffer (for e.g. 0-copy reads from mmap'd data)
• DataFrame accepts arbitrary vectors as columns (again to support things like 0-copy reads)
• the naive composition therefore results in immutable columns and confusing errors

(not saying it is ideal, just how/why we got here)

[Moelf]

From perspective of Arrow, a Vector{Vector{}} is stored as a content vector and an offset vector, similar to how https://github.com/JuliaArrays/ArraysOfArrays.jl works.

Now, if it actually used that, the push!() would have worked just fine, but instead Arrow.jl is doing something on its own.

Btw, if you're interested in a fully systematic way of dealing with Arrow-like schema, https://github.com/JuliaHEP/AwkwardArray.jl is something we're prototyping.

[ericphanson]

Now, if it actually used that, the push!() would have worked just fine, but instead Arrow.jl is doing something on its own.

I don't think that's really accurate, the issue isn't the layout-in-memory, it's that Arrow.Table's columns are deliberately immutable, since they are static view into the underlying bytes that back the table.

[Moelf]

When there's compression involved it won't be purely Mmaped. In general I agree, I'm saying if the resultant table uses that it would have worked. But likely out of the gate it's immutable however we implement it

[ericphanson]

Right, I'm not saying it's always mmap'd, that was an example, but I'm saying Arrow.Table always has immutable columns in the current design of this package

[maleadt]

Thanks for the quick comments!

The workaround is to ask DataFrames to copy the columns:

DataFrame(Arrow.Table("/tmp/test.arrow")); copycols=true)

Hmm, I don't see any effect of that here:

julia> typeof(df.foo)
Vector{Vector{Int64}} (alias for Array{Array{Int64, 1}, 1})

julia> Arrow.write("/tmp/test.arrow", df);
julia> df2 = DataFrame(Arrow.Table("/tmp/test.arrow"); copycols=true);

julia> typeof(df2.foo)
Vector{SubArray{Int64, 1, Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}} (alias for Array{SubArray{Int64, 1, Arrow.Primitive{Int64, Array{Int64, 1}}, Tuple{UnitRange{Int64}}, true}, 1})

The snippet you posted is a little ambiguous, but additionally calling copy or DataFrame with copycols=true (which seems like the default for copy anyway) doesn't help either:

julia> df2 = DataFrame(DataFrame(Arrow.Table("/tmp/test.arrow")); copycols=true);
julia> typeof(df2.foo)
Vector{SubArray{Int64, 1, Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}} (alias for Array{SubArray{Int64, 1, Arrow.Primitive{Int64, Array{Int64, 1}}, Tuple{UnitRange{Int64}}, true}, 1})

julia> df2 = copy(DataFrame(Arrow.Table("/tmp/test.arrow")); copycols=true);
julia> typeof(df2.foo)
Vector{SubArray{Int64, 1, Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}} (alias for Array{SubArray{Int64, 1, Arrow.Primitive{Int64, Array{Int64, 1}}, Tuple{UnitRange{Int64}}, true}, 1})