DistributedDataFrame

[juliadata]

DistributedDataFrame

[ViralBShah]

Cc: @tanmaykm

[tanmaykm]

I was thinking of the following model for this:

• New objects: DistributedDataVector, DistributedDataFrame, DistributedGroupedDataFrame
• DistributedDataVector: simplified 1D darray
• DistributedDataFrame: collection of remote refs to DataFrames / DistributedDataVectors
• DistributedGroupedDataFrame: collection of DistributedDataFrames
• To start with we can have an interface to read from files with rows split by row numbers. Can use mmap to map different portions of a single large file.
• Implement most operations on DataFrame and DataVector as defined in operators.jl
• Have convert methods to get all parts of a DistributedDataVector / DistributedDataFrame locally as DataVector / DataFrame.
• All operations implemented using pmap / pmapreduce underneath

I can have a go at this if it sounds good. Would be glad to have any comments/discussions before I start.

[ViralBShah]

I wonder if a design is possible without using Distributed prefixes for everything. Right now, I can't think of an alternative.

[johnmyleswhite]

This would be great. I'd suggest just building the data structures up first since I think we'll see what needs to be handled there as we go on.

As you go, please let us know when you find functions that should be defined in terms of AbstractDataFrame so that DataFrame and DistributedDataFrame are handled together.

[tanmaykm]

I thought of having just the DistributedDataVector (with an abstract DataVector), but was not sure if that would be enough to handle all functionalities.

[johnmyleswhite]

I think we really want the rows of a DataFrame to be distributed, rather than the columns.

[tanmaykm]

Yes, I agree.

I was wondering if it would it be any better to implement the DistributedDataFrame as a collection of DistributedDataVectors rather than a collection of remote DataFrames.

[johnmyleswhite]

Oh, definitely. So long as you can guarantee that each of the vectors is split in the same way, I think we should follow the existing definition of DataFrame and define things in terms of columns.

[garborg]

See prototypes branch for early implementation / inspiration.