Unable to load sasdata set into Spark

[pathri-pk]

Unable to load sasdata set into Spark

[thesuperzapper]

@pathri-pk are you able to post that sas file? (You should be able to drag it into a reply to this issue, if its not huge)

[jene4ekjene4ek]

Hi all. I have the same problem with this file. The file is not corrupted, because I read it with pandas.
airline.zip

[mazenora]

I had the same problem, so i read the file with pandas then convert it to Spark DataFrame.

df = pd.read_sas('YOUR_FILE',format='sas7bdat')
mySchema = StructType([ ]) <-- create suitable schema structure for spark 
spark_df   = spark.createDataFrame(df,schema=mySchema)

[jene4ekjene4ek]

I've solved this issue by using jars parso and spark-sas7bdat

[mazenora]

could you please share the code?

[jene4ekjene4ek]

In my project folder I have folder jars with parso-2.0.10.jar and spark-sas7bdat-2.1.0-s_2.11.jar (I added printscreen) and run my script as spark-submit --jars jars/spark-sas7bdat-2.1.0-s_2.11.jar,jars/parso-2.0.10.jar script.py

spark = SparkSession.builder.config("spark.jars.packages", "saurfang:spark-sas7bdat:2.1.0-s_2.11") .getOrCreate()

df = spark.read.format('com.github.saurfang.sas.spark')
.load("airline.sas7bdat")