Job aborted due to stage failure

[saurfang]

Job aborted due to stage failure

[saurfang]

I could be wrong but from the limited stacktrace you were able to provide I am not sure if it is with this library. Have you been able to run any SparkSQL example in SparkR already? (i.e. a SparkR command that actually triggers SparkSQL execution)

[witwall]

it works now, but I got another issue,

Sys.setenv(SPARK_HOME="E:/jobs/spark/spark")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
library(SparkR)
packages=c("saurfang:spark-sas7bdat:1.1.4-s_2.10","com.databricks:spark-csv_2.10:1.4.0")
sc <- sparkR.init(master="local[*]",sparkPackages=packages)
sqlContext <- sparkRSQL.init(sc)
df <- loadDF(sqlContext,"Y:/of_trans.sas7bdat", "com.github.saurfang.sas.spark")
registerTempTable(df,"trans")
cnt=sql(sqlContext,"select count(1) from trans")
collect(cnt)

if sas file is some bigger, for example 152M, it is pending when I run collect(cnt)(some times, the first time is ok, but the second time, it is pending)

no matter on a 8G, 16G or 128G memeory PC or server.
spark 1.6.2, and spark-sas7bdat is the last version.

I also tried spark-shell, the same result.

[witwall]

I notice maybe it is not the problem of your spark-sas7bdat, but spark, because I tried load a big table from postgresql, got the same issue.