daskos / epos Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 1.0 108 KB

Write DAG Workflows like epic poems via Mesos, Marathon, Chronos, Spark, Dask

Python 100.00%

chronos dask marathon mesos spark workflow-management

epos's Introduction

Apache Mesos backend for Dask scheduling library. Run distributed python dask workflows on your Mesos cluster.

Notable Features

distributively run tasks in docker container
specify resource requirements per task
bin packing for optimized resource utilization

Installation

Prerequisits: satyr, dask, toolz. All of them should be installed w/ the following commands:

pip install dask.mesos or use lensa/dask.mesos Docker image

Configuration:

MESOS_MASTER=zk://127.0.0.1:2181/mesos
ZOOKEEPER_HOST=127.0.0.1:2181

Example

from __future__ import absolute_import, division, print_function

from daskos import mesos, MesosExecutor


@mesos(cpus=0.1, mem=64)
def add(x, y):
    """Run add on mesos with specified resources"""
    return x + y


@mesos(cpus=0.3, mem=128, image='lensa/daskos')
def mul(x, y):
    """Run mul on mesos in specified docker image"""
    return x * y


with MesosExecutor(name='dask') as executor:
    """This context handles Mesos scheduler's lifecycle"""
    a, b = 23, 89
    alot = add(a, b)
    gigalot = mul(alot, add(10, 2))

    gigalot.compute(get=executor.get)  # (a + b) * (10 + 2)
    executor.compute([alot, gigalot])  # list of futures

Configuring daskos Tasks

You can configure your mesos tasks in your decorator, currently the following options are available:

cpus: The amount of cpus to use for the task.
mem: Memory in MB to reserver for the task.
disk: The amount of disk to use for the task.
image: A docker image name. If not set, mesos containerizer will be used.

Both mem and cpus are defaults to some low values set in satyr so you're encouraged to override them here. More options like constraints, other resources are on the way.

epos's People

Stargazers

Watchers

Forkers

bossjones

epos's Issues

requests dependency

chronos.py depends on requests, but it doesn't get installed with setup.py

@kszucs

Chronos tasks w/o docker

We use the $MESOS_SANDBOX variable where cloudpickle should reside but when we run our stuff cloudpickle can not be found. The variable and it's content should be checked.

MarathonGroup dask context for long running transactions

Add extras_require to setup.py

extras_require = {
    # TODO revisit
    'backends': ['odo', 'pywebhdfs', 'pymongo', 'sqlalchemy', 'paramiko',
                 'cassandra-driver'],
    # dask.mesos should be enough
    'mesos': ['dask.mesos', 'satyr', 'toolz', 'mesos.native'],
    'spark': ['pyspark', 'py4j']
}
extras_require['complete'] = sorted(set(sum(extras_require.values(), [])))

odo(spark_dataframe, 'cql://cassandrahost::cassandra_table')
odo(pandas_dataframe, 'cql://cassandrahost::cassandra_table')
odo('cql://cassandrahost::cassandra_table', sqlContext)   # spark dataframe
odo('cql://cassandrahost::cassandra_table', pd.DataFrame) # pandas dataframe

Odo Parquet Format

To support the following operations:

odo(spark_dataframe, 'hdfs://path/to/output.parquet')
~~odo(pandas_dataframe, 'hdfs://path/to/output.parquet')~~
odo('hdfs://path/to/input.parquet', sqlContext) # spark dataframe
~~odo('hdfs://path/to/input.parquet', pd.DataFrame) # pandas dataframe~~

related odo issue blaze/odo#31