Unexpected allocations

[JoaoAparicio]

Unexpected allocations

[JoaoAparicio]

One difference that I've noticed between Vector{Int64} and the Vector{IntWrapper} cases, is on entering this function

arrow-julia/src/utils.jl

Lines 34 to 57 in 3712291

	function writearray(io::IO, ::Type{T}, col) where {T}
	if col isa Vector{T}
	n = Base.write(io, col)
	elseif isbitstype(T) && (
	col isa Vector{Union{T,Missing}} || col isa SentinelVector{T,T,Missing,Vector{T}}
	)
	# need to write the non-selector bytes of isbits Union Arrays
	n = Base.unsafe_write(io, pointer(col), sizeof(T) * length(col))
	elseif col isa ChainedVector
	n = 0
	for A in col.arrays
	n += writearray(io, T, A)
	end
	else
	n = 0
	data = Vector{UInt8}(undef, sizeof(col))
	buf = IOBuffer(data; write=true)
	for x in col
	n += Base.write(buf, coalesce(x, ArrowTypes.default(T)))
	end
	n = Base.write(io, take!(buf))
	end
	return n
	end

In the first case, col is type Vector{Int64} and matches the first if case, the in the second col is type ArrowTypes.ToArrow{Int64,Vector{IntWrapper}} and matches the last. This allocates because
data = Vector{UInt8}(undef, sizeof(col))
won't know the size to be allocated at compile time.

However at this stage something already went wrong, I believe. By inserting prints I can ask for the sizeof of col which in the first case is the whole vector, but in the second case it's just 8 which I guess is the number of bytes for a single Int64.

[baumgold]

It looks like @quinnj added the logic to write to a temporary vector prior to writing the vector to the IO. I don't understand why this is required. @quinnj - do you remember?

https://github.com/apache/arrow-julia/pull/57/files#diff-47c27891e951c8cd946b850dc2df31082624afdf57446c21cb6992f5f4b74aa2R47-R52