Segmentation Fault with Threads.@spawn + Tables.partitioner + write with compression

[apache]

Segmentation Fault with Threads.@spawn + Tables.partitioner + write with compression

[altre]

It also fails with compress=:zstd

[kobusherbst]

I have the same problem, this works without compression, but fails with. If one allows only a single thread then it doesn't fail, so perhaps compression is not thread safe?

using Random
using DataFrames
using Arrow
using Tables

function nextidrange(minId, maxId, batchsize, i)
    fromId = minId + batchsize * (i-1)
    toId = min(maxId, (minId + batchsize * i)-1)
    return fromId, toId
end

minId = 1
maxId = 1000
idrange = (maxId - minId) + 1
df = DataFrame(ID=minId:maxId, B=rand(idrange), C=randstring.(fill(5,idrange)));
batchsize = 100
numbatches = ceil(Int32, idrange / batchsize)
partitions = Array{SubDataFrame}(undef, 0)
for i = 1:numbatches 
    fromId, toId = nextidrange(minId, maxId, batchsize, i)
    push!(partitions, filter([:ID] => x -> fromId <= x <= toId, df; view = true))
end
io = IOBuffer()
Arrow.write(io, Tables.partitioner(partitions), compress=:zstd)
seekstart(io)
recordbatches = Arrow.Stream(io)
ab = Array{DataFrame}(undef,0)
for b in recordbatches 
  bt = b |> DataFrame
  println("Rows = $(nrow(bt))")
  push!(ab,bt)
end

[quinnj]

Thanks for the reports @kobusherbst and @altre; the compression machinery is indeed not threadsafe, which I've mostly resolve in my local branch, but there's also #108 which is interacting with my testing, so I'm trying to solve both issues in one go here to get threaded writing working reliably. Sorry for the slowness, but I think I'm getting close.

[kobusherbst]

Thank you @quinnj, both issues are a deal breaker for me in having to deal with huge 600 million plus row datasets.