Better bounded types for TypedDatasets

[typelevel]

Better bounded types for TypedDatasets

[kanterov]

It works after #99 was rebased on latest master.

[kanterov]

Is it relevant? I guess everything works out of the box right now.

[imarios]

@kanterov I think we can close this. Let me run some thoughts by you and let me know if I am describing what we are doing correctly.

• In Spark single entries (Rows) are represented (encoded) as InternalRow. So Dataset[Vector[Int]] is represented with InternalRow(Array[Int]). That is, there is an encoder that takes you from Vector[Int] => Array[Int].
• We have a udf with this type signature Vector[Int] => String
• When we need to apply the udf to the Dataset, we cannot really apply it as is. We need to somehow adapt it to work with Array[Int] as input an potentially with UTF8String as output. Therefore the adapted udf needs to be Array[Int] => UTF8String.
• Essentially we need wrap our udf in the following: (a: Array[Int]) => UTF8Codec.encode(udf(VectorCodec.decode(a))) . Now we have a function with the correct signature Array[Int] => UTF8String.

[imarios]

Overall, if you want to be fast, if you want your UDFs to run fast, then it's better to have your case classes use types that are closer to what Spark uses internally:

that is,

case class Person(ids: Array[Int], name: UTF8String) 

would in principle perform faster than

case class Person(ids: Vector[Int], name: String)

at least it terms of UDFs/UDAFs ?

(in this way, you save yourself with a lot of encoding/decoding conversions)