Bug: Filter on Vector[] fields using Vector literals does not work

[typelevel]

Bug: Filter on Vector[] fields using Vector literals does not work

[kanterov]

Hm.. does it work in vanilla Spark?

[imarios]

Column to column works even in us ... column to literal seems to have the issue.

scala> case class Foo(a: Vector[Int], b: Vector[Int])
defined class Foo

scala> val d = TypedDataset.create( Foo(Vector(), Vector(1)) :: Foo(Vector(2,3), Vector(2,3)) :: Nil )
d: frameless.TypedDataset[Foo] = [a: array<int>, b: array<int>]

scala> d.show().run
+------+------+
|     a|     b|
+------+------+
|    []|   [1]|
|[2, 3]|[2, 3]|
+------+------+


scala> d.filter( d('a) === d('b) )
res17: frameless.TypedDataset[Foo] = [a: array<int>, b: array<int>]

scala> d.filter( d('a) === d('b) ).show().run
+------+------+
|     a|     b|
+------+------+
|[2, 3]|[2, 3]|
+------+------+


scala> d.filter( d('a) =!= d('b) ).show().run
+---+---+
|  a|  b|
+---+---+
| []|[1]|
+---+---+

[imarios]

ha, accidental click on close :)

[imarios]

This seems to have been resolved with #152.

[sullivan-]

@imarios if this is fixed perhaps we could uncomment this line:
https://github.com/typelevel/frameless/blob/master/dataset/src/test/scala/frameless/FilterTests.scala#L38
also lines 58 & 77 same file.

I would just uncomment and submit PR but the "fails randomly" part makes me cautious 😀 . I could try it and run the test a few times to see if it's good. Do you have a sense of how many runs would be needed to confirm fix?