Comments (12)
It's possible, but it's not wired up. As always, PRs are welcome.
from storm.
I still haven't managed to get storm to work on mesos, despite days of efforts, so first things first.. :)
from storm.
IIRC, click logviewer link from nimbus web ui will always try to connect to a constant port (8080? not for sure now), however, mesos launch supervisor in a unique and random generated directory, so the logviewer can't(maybe hack in some way) figure out where to find the correct worker's log.
from storm.
Certainly that could be fixed.
from storm.
@yaronr give me about a week to finish up and I'll post how I got it all running ontop of mesos + marathon without hardcoding the nimbus master or anything.
The gist of it is:
- A marathon application group. One for nimbus, and one for the storm ui. The app for storm ui has a dependency on nimbus, so one starts after the other.
- A run-storm.py which stores the actual host and port of the nimbus instance in zookeeper using the excellent Kazoo python lib.
- run-storm.py then writes out the
conf/storm.yaml
with the propernimbus.thrift.port
annimbus.host
settings before launching nimbus - It then writes out
ui.port
,nimbus.host
, andnimbus.thrift.port
when launching storm ui.
All of this and I actually don't run it in docker, but natively under marathon, by downloading the built storm distribution and the run-storm.py via the uris
flag in marathon json. I'm also chrooting this in a specific zookeeper path and having the framework name include that chroot path, so that multiple storm frameworks can run, potentially as different users, on a single mesos cluster.
In a perfect world, the framework would support a lot of this natively, but I'm a python/golang/c guy, and have never taken a serious stab at java.
from storm.
SEJeff - Could we use something like mesos-dns to indicate where the nimbus is located? It works well for Myriad and may work well here.
from storm.
It kind of defeats the point of mesos to require hard coding masters that might change. Mesos DNS is a reasonable workaround, as is my solution of having a python script write to ZK before starting nimbus and another to read from nimbus before starting storm ui.
The framework should do this natively.
from storm.
Makes sense that the framework should do it natively, I was just suggestion the mesos-dns as a way that the nimbus/ui could be run from any node (given access to the code to run it) and where it landed wouldn't matter to the supervisors as they are pointing to a static name that resolves to a dynamic IP.
from storm.
True. Thanks for the suggestion
from storm.
Wouldn't the most straight forward approach be to have the supervisor/executor fire up the logviewer when mesos fires it up?
If no one is working this I would like to get the logview working under mesos.
from storm.
I've raised an issue with how the PR #38 will handle multiple MesosSupervisors / topologies being scheduled onto the same mesos-slave host, please see: #38 (comment)
My proposal for the immediate future is to back out #38 , but I'm not sure what we should do long-term. I have some off-the-cuff thoughts here: #38 (comment)
Would appreciate others weighing in with their thoughts.
from storm.
I've filed a ticket against the Apache Storm project to add support for multiple container-specific logviewers: STORM-1342, thus allowing us to close PR #51.
Will require discussions with the Storm PMC type people to try to come to an acceptable design, I'm not planning on working on that in the next couple of months.
from storm.
Related Issues (20)
- Invalid host resolution (Nimbus, Docker, Marathon) HOT 11
- Tag latest not found in repository docker.io/mesos/storm HOT 9
- create templates for issues & PRs
- remove examples from release packages HOT 4
- mesos/storm docker images: Upgrading to Mesos v1.1.0 HOT 3
- Storm executors not starting with mesos+docker HOT 2
- nimbus.host is deprecated, support nimbus.seeds
- Spout is not reading/emitting data in storm cluster mode HOT 1
- Storm v1.0.3 Support - MesosSupervisor Committing Suicide HOT 10
- Storm v1.1.0+ breaks storm-mesos HOT 2
- logviewer automatic launching support with Docker
- tests are much more noisy after logviewer autolaunching change
- logviewer: check if port is available HOT 1
- Cannot submit topology in local mode on Storm 1.1.0 HOT 3
- Storm v1.0.5 breaks storm-mesos HOT 8
- Storm v1.0.4 breaks storm-mesos HOT 8
- Customizable Task IDs
- Storm Rebalance Broken HOT 1
- Storm 1.0.x Issues: Nimbus Restart and Supervisor Explosion HOT 12
- Nimbus should kill already-launched logviewers if relaunched with `sidecar.enabled=false`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from storm.