Support Apache Arrow

[cda-group]

Support Apache Arrow

[Max-Meldrum]

Just some notes on adding arrow support in arcon...

Elements in Arcon look like this:

pub struct ArconElement<A: ArconType> {
    #[prost(message, tag = "1")]
    pub data: Option<A>,
    #[prost(message, tag = "2")]
    pub timestamp: Option<u64>,
}

Let's say our data is the following:

#[arcon]
pub struct ArconStruct {
  #[prost(uint32, tag = "1")]
  pub id: u32,
  #[prost(string, tag = "2")]
  pub name: String,
}

We would like to be able to convert our ArconElement's into the following Arrow columnar data.

id: UInt32	name: Utf8	timestamp: UInt64
10	test	11535324290
15	test2	11535324299
....	....	...

This conversion would happen through a function like the one below.

fn to_arrow(elems: Vec<ArconElement>) -> arrow::RecordBatch {
   ...
}

Potential Use Cases

Serve a materialised view of "up to date" data where users may:

1. Submit SQL queries to run within Arcon
2. Efficiently retrieve Arrow data through Arrow Flight (gRPC) to other applications (e.g., Pandas, Spark).

Cons: Hard to avoid data duplication as our default storage data format is row-based and Arrow is columnar and not meant for long-term storage. However, as long as conversion is done in batches, it should be okay.