Compatibility with Spark 3.2.1

[thesamet]

Compatibility with Spark 3.2.1

[pomadchin]

One thing that I can suggest is to wait for a little, mb this regression will be fixed in the future Spark releases, at this point 0.11.1 and snapshots are pretty much the same, with the Spark version difference only :/

[chris-twiner]

I'd definitely be up for a clean 3.2.1 official version, I've already got lots of workarounds in play, using official frameless again would be one less

[pomadchin]

@chris-twiner https://github.com/typelevel/frameless/releases/tag/v0.12.0 🚀

[pomadchin]

@thesamet thx for the report, yes we can definitely rollback the dep. back if that’s the case.

[thesamet]

I think it'd be impossible to have a single frameless build that works with both Spark 3.2.0 and Spark 3.2.1 without resorting to reflection or some other means to figure out which instance of Invoke is available at runtime... Moving in either direction will break it for some people :/

[chris-twiner]

Is it not possible to just put "-3.2.1" into the artefact name for a 'special' 3.2.1 build? Source code wise it's fine*, we'd just need to reflect the special status of the dataset for 3.2.1 users e.g. frameless-dataset-3.2.1_2.12 etc.

This would force users to specify which version to use only downside is dependant libraries would have to either use provided scope (and force their users to choose) or follow suite with embedding the version in the name.

If people are ok with the concept I'm happy to offer a pr.

* I've tested this by simply rebuilding and running against a large test suite with injectors, loads of type derivation and some fun array nesting etc. I also have to take this approach to provide Databricks runtime specific Expressions / HigherOrderFunctions etc.

[pomadchin]

@chris-twiner the build at this point is overly complicated: we publish 3.2 / 3.1 / 3.0, and the plan is to keep at most 3 latest (minor) versions released;

I'd like not to publish 3.2.1 specific artifacts, does it mean to make a release for all minor versions, how many than? As a workaround, we could publish the latest package with the 3.2.1 Spark version for folks who are interested.

Anyway, the right solution to this issue is to make apache/spark#35378 to the next Spark release if possible.

[pomadchin]

@chris-twiner cool, if it helps you, I can cut 0.12.0 a release rightaway (also snapshots should work for you at the moment), if you need it; I think we had a fair amount of changes and + you asked for a release. I hope no one has anything against it; cc @imarios @cchantep do you have anything else you'd like to get into 0.12.0?

[cchantep]

Fine for me

[chris-twiner]

Many thanks!!, tested on oss and Databricks 10.4 dbr - works wonderfully and lets me scrap my frameless classpath workarounds for these versions completely!

[pomadchin]

TLDR; This is internal API, won't fix (c)

See https://issues.apache.org/jira/browse/SPARK-38077 and apache/spark#35378 to follow the discussion chain.