Error parsing SAS file distributed in HDFS

[saurfang]

Error parsing SAS file distributed in HDFS

[maguilarh]

I had to use parso-12.1 to can run the Spark's datasource.

If i didn't add this jar to my project i get the following execption:

Name: java.lang.NoClassDefFoundError
Message: com/ggasoftware/parso/SasFileReader
StackTrace: com.github.saurfang.sas.spark.SasRelation.inferSchema(SasRelation.scala:97)
com.github.saurfang.sas.spark.SasRelation.<init>(SasRelation.scala:31)
com.github.saurfang.sas.spark.DefaultSource.createRelation(DefaultSource.scala:34)
com.github.saurfang.sas.spark.DefaultSource.createRelation(DefaultSource.scala:23)
com.github.saurfang.sas.spark.DefaultSource.createRelation(DefaultSource.scala:11)
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:125)
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114)
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104)
$line23.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)
$line23.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
$line23.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:27)
$line23.$read$$iwC$$iwC$$iwC.<init>(<console>:29)
$line23.$read$$iwC$$iwC.<init>(<console>:31)
$line23.$read$$iwC.<init>(<console>:33)
$line23.$read.<init>(<console>:35)
$line23.$read$.<init>(<console>:39)
$line23.$read$.<clinit>(<console>)
$line23.$eval$.<init>(<console>:7)
$line23.$eval$.<clinit>(<console>)
$line23.$eval.$print(<console>)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretAddTask$1$$anonfun$apply$3.apply(ScalaInterpreter.scala:356)
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretAddTask$1$$anonfun$apply$3.apply(ScalaInterpreter.scala:351)
org.apache.toree.global.StreamState$.withStreams(StreamState.scala:81)
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretAddTask$1.apply(ScalaInterpreter.scala:350)
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretAddTask$1.apply(ScalaInterpreter.scala:350)
org.apache.toree.utils.TaskManager$$anonfun$add$2$$anon$1.run(TaskManager.scala:140)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)

[AbdealiLoKo]

I added the Parso 1.2.1 to my project too (because of the above mentioned NoClassDefFoundError) and am getting a very similar ArrayIndexOutOfBound error.

My error is:

Caused by: java.lang.ArrayIndexOutOfBoundsException: 2928
        at com.ggasoftware.parso.SasFileParser.trimBytesArray(SasFileParser.java:1312)
        at com.ggasoftware.parso.SasFileParser.processByteArrayWithData(SasFileParser.java:1111)
        at com.ggasoftware.parso.SasFileParser.access$3300(SasFileParser.java:33)
        at com.ggasoftware.parso.SasFileParser$DataSubheader.processSubheader(SasFileParser.java:857)
        at com.ggasoftware.parso.SasFileParser.readNext(SasFileParser.java:878)

Any solution ?

[saurfang]

For dataset as small as this, I would highly recommend convert it using other standalone programs. https://github.com/tidyverse/haven is an excellent choice.

[AbdealiLoKo]

I got the error on a 10gb data file...

[saurfang]

@AbdealiJK Sorry but I'm afraid I don't have the capacity to perform #10 which could solve your issue. You are more than welcomed to contribute a PR if you're interested. I'd be happy to review.

[nelson2005]

This may not be completely solved... it seems quite similar to #32