spark-sql 3.1.2 can't work with frameless-dataset 0.11.1

[timshen24]

spark-sql 3.1.2 can't work with frameless-dataset 0.11.1

[pomadchin]

Hi @timshen24, yes we know. Try the "org.typelevel" %% "frameless-dataset-spark31" % "0.11.1" dependency.

frameless-dataset is published against Spark 3.2.x which is binary incompatible with 3.1.x; however, since frameless 0.11 we publish artifacts cross scala as well as cross Spark versioned; artifacts for previous Spark version have the -spark{major}{minor} suffix. The mainline artifact for now would be always pointing to the most recent Spark version but we will try to publish it artifacts for the previous versions as well.

Check out the readme for more details https://github.com/typelevel/frameless#versions-and-dependencies. I will try to rewrite it in a way it is more obvious (: somewhere closer to the top

[timshen24]

Thank you for the instant reply.

But setting libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.2.0" libraryDependencies += "org.typelevel" %% "frameless-dataset" % "0.11.1" still not works for me.

It will report compile error for not finding all SparkSession, DataFrame, DataSet class, etc.

---

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.1.2"
libraryDependencies += "org.typelevel" %% "frameless-dataset-spark31" % "0.11.1"

The above can work for me right now.

[pomadchin]

@timshen24 3.2.0 didn't work for you due to other libraries spark versions mismatch. If you want to use Spark 3.2.0 review your deps i.e. spark-fast-tests should be of version2.3.0_0.11.0 and spark-daria does not support Spark 3.2 at all.

I'm closing this issue for now, since it is not a frameless issue anymore, but feel free to reopen it again if needed.