apache/aurora

Apache Aurora - A Mesos framework for long-running services, cron jobs, and ad-hoc jobs

NOTE: The Apache Aurora project has been moved into the Apache Attic. A fork led by members of the former Project Management Committee (PMC) can be found at https://github.com/aurora-scheduler

Apache Aurora lets you use an Apache Mesos cluster as a private cloud. It supports running long-running services, cron jobs, and ad-hoc jobs. Aurora aims to make it extremely quick and easy to take a built application and run it on machines in a cluster, with an emphasis on reliability. It provides basic operations to manage services running in a cluster, such as rolling upgrades.

To very concisely describe Aurora, it is like a distributed monit or distributed supervisord that you can instruct to do things like run 100 of these, somewhere, forever.

Features

Aurora is built for users and operators.

• User-facing Features:

  • Management of long-running services
  • Cron jobs
  • Resource quotas: provide guaranteed resources for specific applications
  • Rolling job updates, with automatic rollback
  • Multi-user support
  • Sophisticated DSL: supports templating, allowing you to establish common patterns and avoid redundant configurations
  • Dedicated machines: for things like stateful services that must always run on the same machines
  • Service registration: announce services in ZooKeeper for discovery by various clients
  • Scheduling constraints to run on specific machines, or to mitigate impact of issues like machine and rack failure

• Under the hood, to help you rest easy:

  • Preemption: important services can 'steal' resources when they need it
  • High-availability: resists machine failures and disk failures
  • Scalable: proven to work in data center-sized clusters, with hundreds of users and thousands of jobs
  • Instrumented: a wealth of information makes it easy to monitor and debug

When and when not to use Aurora

Aurora can take over for most uses of software like monit and chef. Aurora can manage applications, while these tools are still useful to manage Aurora and Mesos themselves.

However, if you have very specific scheduling requirements, or are building a system that looks like a scheduler itself, you may want to explore developing your own framework.

Companies using Aurora

Are you using Aurora too? Let us know, or submit a patch to join the list!

• Amperity
• Blue Yonder
• Boxever
• Criteo
• Electronic Arts
• Fitbit
• Foursquare
• Gutefrage.net
• Houghton Mifflin Harcourt
• Kakao
• Magine TV
• Medallia
• Oscar Health
• PayPal
• Sabre Labs
• TellApart
• Twitter
• Uber

Getting Help

If you have questions that aren't answered in our documentation, you can reach out to one of our mailing lists. We're also often available in Slack: #aurora on mesos.slack.com. Invites to our slack channel may be requested via mesos-slackin.herokuapp.com

You can also file bugs/issues in our Github repo.

License

Except as otherwise noted this software is licensed under the Apache License, Version 2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

aurora's Issues

Feature request add docker/volume and volume/secret support

Feature request

I'd like support on the scheduler and client for docker/volume and volume/secret

docker/volume

http://mesos.apache.org/documentation/latest/isolators/docker-volume/

volume/secret

http://mesos.apache.org/documentation/latest/secrets/

Information Exposure

Information Exposure
Vulnerable module: org.eclipse.jetty:jetty-server
Introduced through: org.eclipse.jetty:[email protected], org.eclipse.jetty:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Overview
org.eclipse.jetty:jetty-server is a is a lightweight highly scalable java based web server and servlet engine.

Affected versions of this package are vulnerable to Information Exposure. The configuration of a Jetty server may be leaked as part of a HTTP 404 response. This is due to the DefaultHandler class producing an error page during an exception.

Directory Traversal

Directory Traversal
Vulnerable module: org.apache.shiro:shiro-web
Introduced through: org.apache.shiro:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.shiro:[email protected]
Overview
org.apache.shiro:shiro-web is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management.

Affected versions of this package are vulnerable to Directory Traversal. The requestURI : /resource/menus and resource/menus/ can both access the server resource, but the pathPattern match /resource/menus can not match resource/menus/. A user can use requestURI + "/" to simply bypass the chain filter, thereby bypassing shiro protect and gaining access to the server resources.

Directory Traversal vulnerability report

Generic exceptions within storage.write statements are not caught potentially causing inconsistent state

A finding from #31.

A user created an update to remove instances from a job. This throws a NullPointerException as mentioned in the issue above. The LoggingInterceptor actually swallows the exception. This happens because we do the initial evaluation of the update within the user calling the RPC method (follow along the start(...) method if you are not convinced).

Although the above start command throws a NullPointerException, the update is still added to the MemJobUpdateStore but not persisted to the log. We still call saveJobUpdate(...) within the ‘start(...)’ code which will add it to the memory stores. However, because a NullPointerException is thrown before the write lock is exited, these operations are never persisted to the log. The design of the storage system in the scheduler is transactional so everything is added to the log at the end of the write. Due to this, we are now in a state where the memory store does not match the log store.

I think that we should catch all unhandled exceptions within the write lock and immediately kill the scheduler. This would avoid errors leaving a potentially inconsistent state and corrupting the log preventing easy rollback.

Unsafe Dependancy Resolution

Unsafe Dependancy Resolution
Vulnerable module: com.beust:jcommander
Introduced through: com.beust:[email protected] and org.asynchttpclient:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › com.beust:[email protected]
Remediation: Upgrade to com.beust:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.asynchttpclient:[email protected] › com.typesafe.netty:[email protected] › org.testng:[email protected] › com.beust:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.asynchttpclient:[email protected] › com.typesafe.netty:[email protected] › org.reactivestreams:[email protected] › org.testng:[email protected] › com.beust:[email protected]
Overview
com.beust:jcommander is a Command line parsing framework for Java.

Affected versions of this package are vulnerable to Unsafe Dependancy Resolution due to resolving dependencies over an insecure channel (http).

If the build occurred over an insecure connection, a malicious user could have perform a Man-in-the-Middle attack during the build and alter the build artifacts that were produced. In case that any of these artifacts were compromised, any developers using these could be altered.

Note: In order to validate that this artifact was not compromised, the maintainer would need to confirm that none of the artifacts published to the registry were not altered with. Until this happens, we can not guarantee that this artifact was not compromised even though the probability that this happened is low.

We have chosen to alert on this issue when maintainers either decided to issue CVEs themselves, or in cases when maintainers decided against performing audits on there build to verify they had not been compromised.

Timing Attack

Timing Attack
Vulnerable module: org.eclipse.jetty:jetty-util
Introduced through: org.eclipse.jetty:[email protected], org.eclipse.jetty:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Overview
org.eclipse.jetty:jetty-util is a lightweight highly scalable java based web server and servlet engine.

Affected versions of this package are vulnerable to Timing Attacks. A flaw in the util/security/Password.java class makes it easier for remote attackers to obtain access by observing elapsed times before rejection of incorrect passwords.

Password/Username not forwarded by followers

How to reproduce:

Send thrift request that requires username and password to succeed to a follower.

[Discussion] Port End to End tests away from bash

Working on trying to fix the end to end test has reminded me how painful the process has become for debugging end to end tests when something goes wrong.

The current end to end test stand at 1000+ lines of bash. Running individual tests involves modifying the script and adding new tests is a very involved process.

Ideally we should rewrite these tests in a language that's more apt for the job and easier to contribute to.

I'll throw out some languages to get the discussion started in order of preference:

• Python
• Golang
• Node
• Ruby

Theoretically, we could use thrift bindings directly instead of the aurora client, which would decrease coverage for the client itself, but the client already has its own suite of tests.

I realize porting this code might be a big undertaking but it needs to be done sooner than later in my opinion.

Looking forward to hearing everyone's opinions.

Replication slots are currently not supported in Aurora

Hi,every hacker!
Before PostgreSQL used WalRecord as a log, now I need to migrate to aurora. Should I change it? Can someone help me,thanks :-)

Flag for enabling SLA Aware killing for non-prod tasks

Currently, SLA aware killing is only possible for prod tier tasks. Since the intention of SLA aware killing is for it to be used with only a limited subset of jobs in the cluster, it is understandable that it was approached in this way.

However, for existing clusters that don't use tiering this presents a significant challenge for enabling SLA aware killing. All jobs in the cluster would have to be recreated with a production tier attached to them and a quota would have to be added for every single role within the cluster. Furthermore, any task that would like to use a new role, would require setting a new role quota.

Given the issues outlined, I propose we add a flag that allows operators to enable SLA aware killing for non-production tasks. The flag would be disabled by default.

@shanmugh would be great to get your thoughts on this if you have some time.

I have a POC ready to be reviewed if no one is opposed to this idea: ridv@31bc9b4

XML External Entity (XXE) Injection

XML External Entity (XXE) Injection
Vulnerable module: c3p0:c3p0
Introduced through: org.quartz-scheduler:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.quartz-scheduler:[email protected] › c3p0:[email protected]
Overview
c3p0:c3p0 is a lIbrary for augmenting traditional (DriverManager-based) JDBC drivers with JNDI-bindable DataSources, including DataSources that implement Connection and Statement Pooling, as described by the jdbc3 spec and jdbc2 std extension. Note: This library is no longer maintained and has migrated to the artifact "com.mchange:c3p0"

Affected versions of this package are vulnerable to XML External Entity (XXE) Injection. via the extractXmlConfigFromInputStream in com/mchange/v2/c3p0/cfg/C3P0ConfigXmlUtils.java during initialization.

Denial of Service (DoS)

Denial of Service (DoS)
Vulnerable module: org.apache.zookeeper:zookeeper
Introduced through: org.apache.zookeeper:[email protected], org.apache.curator:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.zookeeper:[email protected]
Remediation: Upgrade to org.apache.zookeeper:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Overview
org.apache.zookeeper:zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Affected versions of this package are vulnerable to Denial of Service (DoS). Four letter zookeeper commands (such as wchp/wchc ) are not properly handled, which leads to the server unable to serve legitimate client requests.

Python 2 sunsetting in Jan 2020

Python 2 will officially sunset January 1st, 2020 according to the Python maintainers https://www.python.org/doc/sunset-python-2/

Since Aurora relies pretty heavily on Python to run tasks in critical spots, I think it would be quite dangerous to run with no further security patches.

At the same, I don't think we have the man power to move all our Py2 code to Py3. Any ideas on what our path forward should be?

Possible NullPointerException in InstanceActionHandler

After introducing the 'slaAware' field in this commit, there has been a possible NullPointerException in this line. This happens when an update only kills instances. There is no desiredState, so getDesiredState().getTask() will try to act on a null object. Other places in the code guard against this case by checking isSetDesiredState() before retrieval.

We should properly handle nulls in this case, and add generic end to end tests for only adding instances and only removing instances with updates.

Deserialization of Untrusted Data

Deserialization of Untrusted Data
Vulnerable module: com.fasterxml.jackson.core:jackson-databind
Introduced through: com.hubspot.jackson:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › com.hubspot.jackson:[email protected] › com.fasterxml.jackson.core:[email protected]
Remediation: Upgrade to com.hubspot.jackson:[email protected].
Overview
Affected versions of com.fasterxml.jackson.core:jackson-databind are vulnerable to Deserialization of Untrusted Data. An attacker may exploit this issue by sending a maliciously crafted input to the readValue method of the ObjectMapper.

XML External Entity (XXE) Injection

XML External Entity (XXE) Injection
Vulnerable module: org.quartz-scheduler:quartz
Introduced through: org.quartz-scheduler:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.quartz-scheduler:[email protected]
Overview
org.quartz-scheduler:quartz is a package for Enterprise Job Scheduler.

Affected versions of this package are vulnerable to XML External Entity (XXE) Injection. via the initDocumentParser method in a job description.

Count number of times partitioned tasks reenter the cluster as healthy

Currently when a task is PARTITIONED and LOST, Aurora reschedules a replacement. Later on, the task can send a message saying it was healthy and then Aurora will kill the old task. Receiving this signal is a huge indicator that you could avoid unnecessary churn in the cluster by extending timeouts.

Add a metric to monitor how often this use case happens.

Authorization Bypass

Authorization Bypass
Vulnerable module: org.eclipse.jetty:jetty-client
Introduced through: org.eclipse.jetty:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Overview
org.eclipse.jetty:jetty-client is a is an asynchronous http client module fro jetty server.

Affected versions of this package are vulnerable to Authorization Bypass. A large chunk size could be interpreted as a smaller chunk size and content sent as chunk body could be interpreted as a pipelined request. If Jetty was deployed behind an intermediary that imposed some authorization and that intermediary allowed arbitrarily large chunks to be passed on unchanged, then this flaw could be used to bypass the authorization imposed by the intermediary as the fake pipelined request would not be interpreted by the intermediary as a request.

Arbitrary Code Execution during Deserialization

Arbitrary Code Execution during Deserialization
Vulnerable module: org.beanshell:bsh
Introduced through: org.asynchttpclient:[email protected]

Detailed paths

Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.asynchttpclient:[email protected] › com.typesafe.netty:[email protected] › org.testng:[email protected] › org.beanshell:[email protected]

Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.asynchttpclient:[email protected] › com.typesafe.netty:[email protected] › org.reactivestreams:[email protected] › org.testng:[email protected] › org.beanshell:[email protected]

Overview
org.beanshell:bsh is a Java source interpreter with object scripting language features, written in Java.

Affected versions of this package are vulnerable to Arbitrary Code Execution during Deserialization. When included on the classpat by an application that uses Java serialization or XStream, A remote attacker could execute arbitrary code via crafted serialized data, related to XThis.Handler.

Move make mesos native egg to Docker containers instead of using vagrant

Using Docker containers would greatly improve the speed at which Mesos eggs could be built locally.

Access Control Bypass

Access Control Bypass
Vulnerable module: org.apache.zookeeper:zookeeper
Introduced through: org.apache.zookeeper:[email protected], org.apache.curator:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.zookeeper:[email protected]
Remediation: Upgrade to org.apache.zookeeper:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Overview
org.apache.zookeeper:zookeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Affected versions of this package are vulnerable to Access Control Bypass. ZooKeeper’s getACL() method doesn’t check any permission when retrieving the ACLs of the requested node and returns all information contained in the ACL Id field as plain text string. If Digest Authentication is in use, the unsalted hash value will be disclosed by the getACL() method for unauthenticated or unprivileged users.

Aurora pause abnormally in variable batch update strategy

When the variable batch size set to [5,5], with autopause to true, it should only pause twice, each happens after one batch completes. But in actual, aurora pause when batch updating is still in progress, and it pause 4 times in total.

How to reproduce?
variable batch size [5,5], with auto pause to true, SLA sets to 70% percent.

Move from Travis CI to Actions

Travis CI is not in good shape.

Starting Nov 13th, Github actions will be widely available and will be free to Open Source projects.

I suggest we move to Actions as soon as it is available given the happenings at Travis CI.

Authentication Bypass

Authentication Bypass
Vulnerable module: org.apache.zookeeper:zookeeper
Introduced through: org.apache.zookeeper:[email protected], org.apache.curator:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.zookeeper:[email protected]
Remediation: Upgrade to org.apache.zookeeper:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected]
…and 1 more

Overview
org.apache.zookeeper:zookeeper is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination.

Affected versions of this package are vulnerable to Authentication Bypass. No authentication/authorization is enforced when a server attempts to join a quorum, as a result an arbitrary end point could join the cluster and begin propagating counterfeit changes to the leader.

Upgrade Mesos dependency to 1.7.x

Mesos dependency should be upgrade to 1.7.x

Update custom executor example

I've been trying to run the mesos-executor as the custom executor, by following the instructions but I wasn't successful.

My custom-executor.json file looks like this

[
    {
        "executor": {
            "name": "theDefaultMesosExecutor",
            "command": {
                "value": "/usr/libexec/mesos/mesos-executor",
                "shell": "true"
            },
            "resources": [
            {
                "name": "cpus",
                "type": "SCALAR",
                "scalar": {
                    "value": 0.1
                }
            },
            {
                "name": "mem",
                "type": "SCALAR",
                "scalar": {
                    "value": 256
                }
            }
            ]
        },
        "task_prefix": "mes-exec-"
    }
]

And I get the following error

I0807 11:22:38.310045 20943 exec.cpp:162] Version: 1.6.0
I0807 11:22:38.317176 20950 exec.cpp:236] Executor registered on agent 7e84b258-a9a7-469d-994b-f5a154b665b5-S1
I0807 11:22:38.319226 20943 executor.cpp:178] Received SUBSCRIBED event
I0807 11:22:38.319492 20943 executor.cpp:182] Subscribed executor on agent-node
I0807 11:22:38.319573 20943 executor.cpp:178] Received LAUNCH event
F0807 11:22:38.325577 20943 executor.cpp:569] Expecting task 'dev-devel-hello_world2-0-fe85afac-67c5-4e07-aac8-2168942a9000' to have a command
*** Check failure stack trace: ***
    @     0x7feb93cb122d  google::LogMessage::Fail()
    @     0x7feb93cb307d  google::LogMessage::SendToLog()
    @     0x7feb93cb0e13  google::LogMessage::Flush()
    @     0x7feb93cb3a79  google::LogMessageFatal::~LogMessageFatal()
    @     0x56158be584f0  mesos::internal::CommandExecutor::launch()
    @     0x56158be5a862  mesos::internal::CommandExecutor::received()
    @     0x56158be5b1bb  _ZNSt17_Function_handlerIFvvEZZNO7process9_DeferredIZN5mesos8internal15CommandExecutor10initializeEvEUlSt5queueINS3_2v18executor5EventESt5dequeIS9_SaIS9_EEEE0_EcvSt8functionIFvT_EEIRKSD_EEvENKUlSM_E_clESM_EUlvE_E9_M_invokeERKSt9_Any_data
    @     0x7feb93bfa9d1  process::ProcessBase::consume()
    @     0x7feb93c1a4fa  process::ProcessManager::resume()
    @     0x7feb93c1aa28  process::ProcessManager::wait()
    @     0x7feb93c1b037  process::wait()
    @     0x56158be1a3b9  main
    @     0x7feb908ff3d5  __libc_start_main
    @     0x56158be1ab05  (unknown)

Thank you!

Unicode characters in docker parameter strings

Apparently some characters are replaced with unicode in .aurora files which causes the container to fail. For example I tried to pass additional parameters to docker like so:

...
container = Docker(
  image = "...",
  parameters = [
    Parameter(name = "mount", value = "type=bind,source=/mnt/foo,target=/foo/bar")
  ]
)
...

The docker container failed to launch with exit code 125. Inspecting the Aurora Struct Dump reveals that equals (=) in docker parameters were replaced by \u003.

...
"value": {
  "image": "...",
  "parameters": [ {
    "name": "mount",
    "value": "type\u003dbind,source\u003d/mnt/foo,target\u003d/foo/bar"
  }]
}
...

I also tried to pass a python raw string to the Parameter object. Unfortunately that didn't work either. I finally worked around the issue by passing a volume parameter instead of mount.

Deserialization of Untrusted Data

Deserialization of Untrusted Data
Vulnerable module: com.fasterxml.jackson.core:jackson-databind
Introduced through: com.hubspot.jackson:[email protected]
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › com.hubspot.jackson:[email protected] › com.fasterxml.jackson.core:[email protected]
Remediation: Upgrade to com.hubspot.jackson:[email protected].
Overview
com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.

Affected versions of this package are vulnerable to Deserialization of Untrusted Data. When Default Typing is enabled for an externally exposed JSON endpoint, the service has the mysql-connector-java jar in the classpath. An attacker can host a crafted MySQL server reachable by the victim and send a crafted JSON message that allows them to read arbitrary local files on the server. This occurs due to missing com.mysql.cj.jdbc.admin.MiniAdmin validation.

Deprecate Thrift backup restoration mechanism

Thrift backup restoration should be deprecated in 0.22.0 and removed in 0.23.0 given the fact that the restoration tool has existed since 0.20.0

End to end tests are broken

End to end tests are currently broken.

They need to be fixed before making the 0.22.0 release.

Upgrade dev environment to Mesos 1.6.1

Arbitrary Code Injection

Arbitrary Code Injection
Vulnerable module: jline:jline
Introduced through: org.apache.zookeeper:[email protected], org.apache.curator:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.zookeeper:[email protected] › jline:[email protected]
Remediation: Upgrade to org.apache.zookeeper:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected] › jline:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected] › jline:[email protected]
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.curator:[email protected] › org.apache.zookeeper:[email protected] › jline:[email protected]
Overview
jline:jline is a Java library for handling console input.

Affected versions of this package are vulnerable to Arbitrary Code Injection. Race condition in hawtjni-runtime/src/main/java/org/fusesource/hawtjni/runtime/Library.java in HawtJNI before 1.8, when a custom library path is not specified, allows local users to execute arbitrary Java code by overwriting a temporary JAR file with a predictable name in /tmp.

Authorization Bypass

Authorization Bypass
Vulnerable module: org.eclipse.jetty:jetty-server
Introduced through: org.eclipse.jetty:[email protected], org.eclipse.jetty:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Overview
org.eclipse.jetty:jetty-server is a lightweight highly scalable java based web server and servlet engine.

Affected versions of this package are vulnerable to Authorization Bypass. When it presented with two content-lengths headers, Jetty ignored the second. When presented with a content-length and a chunked encoding header, the content-length was ignored (as per RFC 2616). If an intermediary decided on the shorter length, but still passed on the longer body, then body content could be interpreted by Jetty as a pipelined request. If the intermediary was imposing authorization, the fake pipelined request would bypass that authorization.

Cache Poisoning

Cache Poisoning
Vulnerable module: org.eclipse.jetty:jetty-server
Introduced through: org.eclipse.jetty:[email protected], org.eclipse.jetty:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Overview
org.eclipse.jetty:jetty-server is a lightweight highly scalable java based web server and servlet engine.

Affected versions of this package are vulnerable to Cache Poisoning. An HTTP/1 style request line (i.e. method space URI space version) that declares a version of HTTP/0.9 was accepted and treated as a 0.9 request. If deployed behind an intermediary that also accepted and passed through the 0.9 version, the response sent could be interpreted by the intermediary as HTTP/1 headers. This could be used to poison the cache if the server allowed the origin client to generate arbitrary content in the response.

Cache Poisoning

Cache Poisoning
Vulnerable module: org.eclipse.jetty:jetty-http
Introduced through: org.eclipse.jetty:[email protected], org.eclipse.jetty:[email protected] and others
Detailed paths
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Introduced through: apache/aurora@apache/aurora#ea8e2f4d905e26b3b999e8b59a3cb74a08a3dbf5 › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected] › org.eclipse.jetty:[email protected]
Remediation: Upgrade to org.eclipse.jetty:[email protected].
Overview
org.eclipse.jetty:jetty-http is a is a http module for jetty server.

Affected versions of this package are vulnerable to Cache Poisoning. An HTTP/1 style request line (i.e. method space URI space version) that declares a version of HTTP/0.9 was accepted and treated as a 0.9 request. If deployed behind an intermediary that also accepted and passed through the 0.9 version, the response sent could be interpreted by the intermediary as HTTP/1 headers. This could be used to poison the cache if the server allowed the origin client to generate arbitrary content in the response.

Update Gradle version

We're currently on Gradle 4.2. Gradle recently shipped their latest version 6.0.1.

It would be great to be able to upgrade our Gradle version to a newer one.