Error when running with a modified docker image

[jahstreetorg]

Error when running with a modified docker image

[jahstreet]

Hi, looks familiar to what I've seen before. I'll try it out on my end and let you know on the progress.

[MBtech]

Thanks. I did get around the problem by simply providing the python dependencies using the pyfile argument in the batch create for livy but it would be useful to know why it happens in this particular case, just for the sake of knowledge.

[maciekdude]

Yo, was just struggling with this issue.
Basically, you modified the wrong image (it's livy) with wrong entrypoint.sh that just runs livy server. You need to modify image that will be used by driver/executors
It breaks here

• https://github.com/JahstreetOrg/spark-on-kubernetes-docker/blob/56cd4bac4063259af79210e0a1703a9dc622760e/livy/0.8.0-incubating-spark_3.0.1_2.12-hadoop_3.2.0_cloud/entrypoint.sh#L45

as the bash script wants to read from /opt/spark/conf which is mounted by k8s and is in essence dir with spark.properties

BR,
M

[jahstreet]

Indeed, thx @maciekdude . @MBtech, please use:

• sasnouskikh/livy:<version> - to run Livy server containers
• sasnouskikh/livy-spark:<version> - to run Spark driver and executor containers with Livy support