mesos/elasticsearch

Elasticsearch on Mesos

Apache Mesos Repository Has Moved

Apache Mesos is now a Top-Level Apache project, and we've moved the codebase. The downloads page explains the essential information, but here's the scoop:

Please check out the source code from Apache's git repostory:

git clone https://git-wip-us.apache.org/repos/asf/mesos.git

or if you prefer GitHub, use the GitHub mirror:

git clone git://github.com/apache/mesos.git

For issue tracking and patches, we use Apache-maintained infrastructure including the JIRA issue tracker instead of the GitHub one, and review board for patches instead of pull requests.

Other information including documentation and a getting started guide are available on the Mesos website: http://mesos.apache.org

Thanks!

-- The Mesos developers

elasticsearch's Issues

Use Mesos DNS for node discovery in Elasticsearch Mesos Cloud Plugin

Currently we use the Master's state file to discover nodes in the cluster. Replace this with Mesos DNS node discovery mechanism.

• Expose services in mesos-dns
• Deploy mesosphere/mesos-dns
• Configure DNS on all nodes
• Look up ES instances through DNS in Mesos-cloud plugin
• Automate adding prepend domain-name-servers 10.11.142.163; to /etc/dhcp/dhclient.conf on all nodes

Make Docker image names configurable

The scheduler should use mesos/elasticsearch-cloud-mesos by default but you should be able to override this via a command line option.

Project status

Please advise what is current project status, is it ready for use or still in development?
Is there any roadmap?

Configure Elasticsearch port resources at the framework level

It should be possible to configure the client and transport port that is used for a framework. This way you can run multiple frameworks on a single slave, each using their own pairs of ports.

This is different than what is implemented in the https://github.com/mesos/elasticsearch/tree/docker-compose branch

Cluster deployment via Terraform

Get dataset from ElasticSearch

Gradlefy the build scripts

Add executable jar configuration for scheduler

Remove references to /etc/hosts entries in deployDcos.sh script

Upgrade Mesos to 0.22.1 in system test

System test currently fails because they do not support 0.22.1's DiscoveryInfo.

Upgrade the master and slave images to Mesos 0.22.1

Add Marathon config for deploying the scheduler

Autodetect node discovery strategy in cloud-mesos

Cloud-mesos should use Mesos-DNS if possible or fall back to simple parsing of the state file if Mesos-DNS is not available.

End to end testing

Randomize port selection

Currently the framework selects the first 2 ports from the offered port ranges. However it should select 2 random ports from the port ranges because in case of task failures there is the chance that the same port was used before by another task.

Implement ElasticsearchScheduler

Scheduler should launch Docker container with Elasticsearch + Elasticsearch Mesos cloud plugin

Document docker-compose setup

Upgrade mesos to 22.1

Required for DiscoveryInfo.

Add docker-compose config for development & testing

Replace HDFS with scheduler run webserver to distribute executors

See https://github.com/mesos/kafka/blob/master/src/scala/ly/stealth/mesos/kafka/HttpServer.scala in Kafka framework.

Docker Requirement

Does this framework require the docker containers to run? If so will there be options to use and set local docker registries so we don't have to pre distribute the containers? If not will there be directions on how to get this setup as a native framework with no Docker containers? And also, if that's the case, will there be options to build without building the docker containers.

Thanks!

Prepare demonstration

List of current issues

• Fix IndexCreationException #39 @frankscholten
• Only 2 out of 3 nodes is starting #37 @frankscholten
• Setup BigDesk @mrmrcoleman
• Add example data #38 @mwl

Connect the project to CircleCI for automated builds

Connect the project to Travis for automated builds

Implement ElasticsearchExecutor

Executor should send simple metrics from the Elasticsearch node such as number of searches back to the scheduler so the scheduler can decide whether to scale up or down.

Upgrade all cluster nodes to Java 8

Centralised logging

Fix infinite loop in cloud-mesos when state parsing throws exception

Cloud-mesos parses the state but when an exception is thrown it will be called again, creating an infinite loop.

One case where this happens is when the 'cluster' property is not found in the state.json file. This field is set by GCE but not in docker compose.

Consider changing the JSON API we use because it throws an exception if a property is not available.

Improve test coverage of executor

Add shared folder to Vagrantfile to cache gradlew

Failing build

:system-test:pmdTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':system-test:pmdTest'.
> 2 PMD rule violations were found. See the report at: file:///vagrant/system-test/build/reports/pmd/test.html

1   /vagrant/system-test/src/test/java/org/apache/mesos/elasticsearch/systemtest/DiscoverySystemTest.java   10  Avoid unused imports such as 'org.apache.mesos.elasticsearch.common.Discovery'
2   /vagrant/system-test/src/test/java/org/apache/mesos/elasticsearch/systemtest/DiscoverySystemTest.java   20  Avoid unused imports such as 'org.junit.Assert.fail'

Create vagrant configuration for the project

I prefer to work in Vagrant wherever possible so as to avoid polluting my laptop. I'm working on a centos7 box for myself right now anyway to solve this problem.

If there is demand for this, please let me know and also state your preferred distribution. I have no particular preference for centos.

Mesos does not allocate the requested FrameworkID

Despite sending the commands to set the FrameworkID, mesos does not register the framework with the requested ID.

Create docker-compose system test setup

Let's setup a way to run automated system tests with docker-compose.

The steps are

• docker-compose up - either via Gradle or within JUnit
• Perform assertions such as calling the Elasticsearch endpoints to check if node discovery works
• docker-compose rm

Prepare demo

Find out which metrics are relevant for demo'ing

Use an executor in slave docker image.

Currently the ES plugin communicates directly with mesos. It should communicate via the executor.

Remove master url parameter

The scheduler should not use the master url parameter. Instead the master url should be configured through MasterInfo when the framework is registered.

Add node discovery through framework messages

Fix IndexCreationException

Create gradle task for pushing images to Docker hub

Only 2 esdemo tasks start on a 3 node cluster

Java 7 discussion

Java 7 is almost a dead animal. Should we upgrade to Java 8?

Monitor Elasticsearch process in Executor via WrappedProcess

Mesos-DNS node discovery does not parse ports correctly

When using discovery mode 'auto' or 'mesos-dns' the port numbers are not correctly parsed.

Add a unit test & docker-compose setup with the Mesos DNS container to test this scenario.

Support elastic scaling

Currently the framework launches elasticsearch on every node.

In large clusters this is not desired, therefore the framework should also support an auto-scaling strategy. With this strategy the framework monitors resource usage and for instance when the heap size becomes 70% of total memory it launches a new elasticsearch node to scale the load.

Add Elasticsearch image based on Alpine Linux

See https://github.com/soldair/docker-alpine-elasticsearch

Add mesos/elasticsearch to CircleCI

Waiting on Ferdi to contact the Apache Foundation.

Import demo data

Improve management of port metadata in MesosStateServiceRest

Use a more elegant way to parse port ranges and reduce code complexity.

Improve build process

Even if nothing has changed, it takes lots of time to determine that nothing has changed in :scheduler:jar and :executor:jar tasks

:elasticsearch-cloud-mesos:processResources UP-TO-DATE
:elasticsearch-cloud-mesos:classes UP-TO-DATE
:elasticsearch-cloud-mesos:jar UP-TO-DATE                   <--- several minutes
:elasticsearch-cloud-mesos:assemble UP-TO-DATE

it looks the problem is caused due to shadowing grade plugin