mesosphere/tachyon-mesos

A Mesos Framework for Tachyon, a memory-centric distributed file system.

Tachyon-Mesos

A Mesos Framework for Tachyon, a memory-centric distributed file system.

Word of Warning

This is the outcome of a brief hackathon, suitable only for fun and collaboration.

Prerequisites

• A Mesos cluster
• Java JDK
• SBT

Usage

$ sbt "run {tachyonUrl} {mesosMaster} {zookeeperAddress}"

Where mesosMaster is of the same form that the mesos-slave program accepts and where zookeeperAddress is of the form host:port.

Alternatively, if developing within the vagrant VM:

$ bin/run-dev

Design

Tachyon-Mesos Scheduler

The scheduler:

• registers as a framework with a Mesos master, naturally
• links against the tachyon library and starts a Tachyon Master in the same process.
• launches Tachyon worker processes on Mesos

Assumptions

• Fault tolerance for the Tachyon-Mesos scheduler can be achieved by running with a meta-framework such as Marathon

tachyon-mesos's Issues

Contributing

Currently running spark on mesos and interested in contributing to this, even though this was marked as 'just for fun.' Am I correct in saying that the best way to use tachyon with spark+mesos is to run it as a framework? If so, anything I can look at and possibly contribute to?

Start a Tachyon Master co-resident with the Scheduler

• Add Tachyon as a dependency
• Create a Tachyon Master

Launch Tachyon Worker Processes

• Set the TACHYON_JAVA_OPTS environment variable to allow the worker to find the master via Zookeeper
• Do something useful in the resourceOffers callback

First pass: run a standalone tachyon node

@adam-mesos Let's get this party started 🎈🎊

zk config value is not used, it is always 127.0.0.1:2181