all-spark-notebook does not work when using spark on mesos

[jlamcanopy]

all-spark-notebook does not work when using spark on mesos

[parente]

Ah, yes. The mesos slaves need to have --no-switch-user set. Otherwise, the jovyan user must exist on all worker nodes. I meant to document this in the README and you just reminded me.

You can either have the cluster jobs run as root (dangerous), create jovyan on every worker node (painful), or set that flag (easy, if you don't mind the slaves all running as the user as which they were launched).

[jlamcanopy]

Thanks! That makes a lot of sense now.

Ideally, I would like the user to be "spark" because this is what the username we used for submitting spark jobs. Is it difficult to make it configurable (i.e. jovyan to "whatevernamewewanttouse")? :)
I think it is better to stick with --switch-user (default setting) because not everyone using mesos wants to have the same privilege as spark users. What do you think?

[parente]

I agree it would be nice, but not sure how easy to enable that. We've had lots of trouble with getting users, groups, and permissions straight. Maybe the last round of fixes in that area greased the wheels for making the user name configurable too. I'll give it a look at least.

[parente]

After a quick snoop, the answer is: not easily. The Dockerfiles have USER jovyan strewn throughout to switch back and forth between root executed commands and the notebook user commands. Last I tried, I could not use an environment variable in a Docker USER line. If that's still the case, then the only other alternative to making it work would be to su/sudo to NB_USER throughout all the Dockerfiles in place of the USER statements. The editing isn't bad: making sure every image builds properly after that change is the bulk of that work (i.e., env vars are set properly using su/sudo instead of USER).

[parente]

And to clarify my last comment, that would only allow build time changes of NB_USER. Run time is a whole other big problem involving usermod and ownership fixes.

[jerrylam]

Hey @parente, thanks for looking into it. I was hoping it is an easy sweep. I have been building images for the past few days too so I understand the pain and the complexity. Meanwhile, I think I will keep hacking the Dockerfiles!

[parente]

I documented the Mesos user requirements (create jovyan or run with --no-switch_user on slaves) in the pyspark and all-spark READMEs. I think this is the best we can do for the moment. If a more flexible option presents itself, I'll certainly revisit this.

[DmitryBe]

hey @parente,
i set MESOS_NO_SWITCH_USER=true in /etc/default/mesos-slave on agent nodes;
looks like this doesn't help
PS. also tried MESOS_SWITCH_USER=false

[DmitryBe]

ok, i think i did this wrong.
approach that works:

bash
touch /etc/mesos-slave/?no-switch_user
rm -rf /var/lib/mesos/meta