Cloud not get spark.eventLog.dir

[linkedin]

Cloud not get spark.eventLog.dir

[akshayrai]

@BrianGallew @canghaiyisu666, I am not sure why Spark isn't picking up 'spark.eventLog.dir'.
I have submitted a pull request to make the log directory configurable in Dr. Elephant. You can cherry pick or pull the changes once I commit it.

Then you can add your spark event log directory to the FetcherConf.xml present in app-conf directory.

[akshayrai]

@canghaiyisu666, I do not have much context on this. What is your usecase?

Can't you do, new SparkConf().get("spark.eventLog.dir", DEFAULT_LOG_DIR)?

[canghaiyisu666]

@akshayrai
new SparkConf().get("spark.eventLog.dir", DEFAULT_LOG_DIR) uses java.lang.System.getProperties() to load variable.
But in CDH cluster ,the environment variable was empty util got a spark runtime.
so, i got the DEFAULT_LOG_DIR here.

perhaps , we could use conf.addResource("PATH_TO_spark-defaults.conf"),to figure out.

[BrianGallew]

I have a possibly-related issue: spark.eventLog.dir is set in /etc/spark/conf/spark-defaults.conf, but is ignored by Dr. Elephant, resulting in failure to analyze Spark jobs.

04-29-2016 16:29:26 ERROR com.linkedin.drelephant.ElephantRunner : java.security.PrivilegedActionException: java.io.FileNotFoundException: File does not exist: /system/spark-history/application_1461865977868_0043_1.snappy

The actual logfile is at

-rwxrwx---   3 azkaban    spark          129563 2016-04-29 03:40 /var/log/spark/apps/application_1461865977868_0043

[canghaiyisu666]

@BrianGallew
we have the same issue.
spark.eventLog.dir was set in /etc/spark/conf/spark-defaults.conf, But "new SparkConf()" couldn't get this. So you got the default dir prefix "/system/spark-history"

[canghaiyisu666]

@akshayrai
Very reasonable~