[ja] `.filter` is used instead of `.map` for non-filter methods

[eleutherai]

[ja] `.filter` is used instead of `.map` for non-filter methods

[polm-stability]

Your analysis here is correct, and this is a problem. I'm working on a fix.

I'm also trying to figure out the Spark execution model in detail, but I believe this many calls to small functions is a problem. If I understand correctly, Spark may need to send data to the JVM separately for each filter call. If that's the case we can drastically reduce overhead by wrapping all the Python functions into a single function before using filter().