Direct writemethod not working in Databricks for Spark 3.5

[jainshasha]

Direct writemethod not working in Databricks for Spark 3.5

[davidrabinowitz]

Which version of the connector are you using?

[jainshasha]

hi @davidrabinowitz thanks for replying.
Currently using Databricks version of 14.3 and with that it has spark version of 3.5
how to check the version of spark-bigquery connector version on that, can you please help me in that.

Would really appreciate your help in this

[davidrabinowitz]

If you are using the built it BigQuery connector, then the Databricks release note should have this information.

[jainshasha]

hi @davidrabinowitz
i debug into databricks i saw this jar we are using inside databricks
ls -lrt /databricks/jars/----ws_3_5--third_party--bigquery-connector--spark-bigquery-connector-hive-2.3__hadoop-3.2_2.12--118181791--fatJar-assembly-0.22.2-SNAPSHOT.jar*

[davidrabinowitz]

If I understand correctly, the jar is of version 0.22.2 which is a very old jar. For Spark 3.5 we recommend to use the spark-3.5-bigquery connector, the latest version is 0.36.1. Direct write is certainly supported there.