Worker jobs getting lost since they cannot download storm conf file

[mesos]

Worker jobs getting lost since they cannot download storm conf file

[echinthaka]

found the reason for this issue. Seems like when it is setting up a local file server to serv conf files to workers, it passed in the serving directory simply as conf. This might be troublesome if the startup scripts are being called from else where.

protected void setupHttpServer() throws Exception {
      _httpServer = new LocalFileServer();
      _configUrl = _httpServer.serveDir("/conf", "conf", _localFileServerPort);

      LOG.info("Started HTTP server from which config for the MesosSupervisor's may be fetched. URL: " + _configUrl);
  }

The fix here would be to have a configuration parameter to point to the correct conf directory path in these situations. I fixed this locally and will submit a PR later.

[erikdw]

@echinthaka has sent a PR with a proposed fix: #57

[erikdw]

#57 has 2 separate changes embedded in it, at least in its initial incarnation:

• the 1st change (intended to fix this "serving config file" issue (#56)), but that never got merged because of the 2nd change:
• the 2nd change is more impactful, using wget to download the executor tarball. This change is more involved and more questionable, and shouldn't have been clubbed together with the 1st change. That issue is still outstanding and deserves its own full-fledged GitHub "issue" in this project.

Notably, the "config file serving" problem is supposedly fixed by #65 per @brndnmtthws comment in #56:

This might no longer be needed since merging #65. From that PR, the config is generated at runtime and then served up to the supervisors.