provide TypedEncoder for java.time.{Instant, Duration, Period} in Spark 3.2

[chenharryhua]

provide TypedEncoder for java.time.{Instant, Duration, Period} in Spark 3.2

[jgoday]

@jgoday yep don't hesistate to rise a PR! We can discuss what to do there.

But indeed, these are all good questions. How it behaves in 3.0 and 3.1 than?

@pomadchin, I have created this PR #581.

EncoderTests and a sample with Duration, Period and Instant
seems to work fine with all versions (3.0, 3.1 and 3.2)

  case class Bar(b: String, a: Long)
  case class Foo(f: Long, o: Bar, p: Period, d: Duration, i: Instant = Instant.now())
  val fTypedDataset = TypedDataset.create(
    Foo(1, Bar("a", 2), Period.ofDays(1), Duration.ofHours(4))
    :: Foo(2, Bar("c", 30), Period.ofDays(2), Duration.ofHours(3))
    :: Foo(10, Bar("b", 20), Period.ofDays(3), Duration.ofHours(2))
    :: Nil)
  fTypedDataset.show().run()

[jgoday]

Hi, I would like to try this one if possible,
considering that it is the first time that I try frameless,
(starting to look to some typelevel/cats projects).

As I can see in the current spark docs (3.2),
catalyst types for Instant, Duration and Period are Long and Int.

Would it be enough to provide custom TypedEncoder for these types with catalystRepr (being Long and Int) ?

[pomadchin]

Hey @jgoday, thanks for looking into this issue! I think it could be enough. Spark uses these DateTime utils though (example for instant) mb we can reuse is as well.

Unless there are other obvious complains and reasons not to do it.

Keep in mind that we're cross publishing now for all 3.x versions! Hopefully it won't create any major issues though.

[jgoday]

Ok, thank you so much @pomadchin.
I was able to test it against spark 3.0.3, 3.1.2 and 3.2.0 with no problem, but there is no DayTimeIntervalType and YearMonthIntervalType sql types prior 3.2, so I have used the following ones as catalyst types for:

1. java.time.Period -> IntegerType (days)
2. java.time.Duration -> LongType (millis)

I don't know if it is the best approach.

Should I make a PR ?

[pomadchin]

@jgoday yep don't hesistate to rise a PR! We can discuss what to do there.

But indeed, these are all good questions. How it behaves in 3.0 and 3.1 than?