paypal/dce-go

Docker Compose Executor to launch pod of docker containers in Apache Mesos.

dce-go's Issues

set up CI

It would be great to set up CI (e.g. Travis) so that it gets executed every time a PR is created or updated. CI should run linting as well as all unit tests.

Fix multi-line comment in makefile

Remove unsupported multi-line C-style /* */ comments in makefile.

when run example with marathon got error

I0801 02:42:11.456279 7200 logging.cpp:194] INFO level logging started!
I0801 02:42:11.457401 7200 logging.cpp:199] Logging to /var/log/mesos
I0801 02:42:11.457464 7200 fetcher.cpp:531] Fetcher Info: {"cache_directory":"/tmp/mesos/fetch/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":true,"value":"https://github.com/mesos/docker-compose-executor/releases/download/0.1.0/sample-app.tar.gz"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":false,"value":"http://192.168.1.64:8090/config.yaml"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":false,"value":"http://192.168.1.64:8090/general.yaml"}},{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":false,"value":"http://192.168.1.64:8090/executor"}}],"sandbox_directory":"/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6"}
I0801 02:42:11.464521 7200 fetcher.cpp:442] Fetching URI 'https://github.com/mesos/docker-compose-executor/releases/download/0.1.0/sample-app.tar.gz'
I0801 02:42:11.464573 7200 fetcher.cpp:283] Fetching directly into the sandbox directory
I0801 02:42:11.464634 7200 fetcher.cpp:220] Fetching URI 'https://github.com/mesos/docker-compose-executor/releases/download/0.1.0/sample-app.tar.gz'
I0801 02:42:11.464690 7200 fetcher.cpp:163] Downloading resource from 'https://github.com/mesos/docker-compose-executor/releases/download/0.1.0/sample-app.tar.gz' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/sample-app.tar.gz'
I0801 02:42:13.928701 7200 process.cpp:1246] libprocess is initialized on 172.16.1.17:40272 with 8 worker threads
I0801 02:42:14.029827 7200 fetcher.cpp:121] Extracted '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/sample-app.tar.gz' into '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6'
I0801 02:42:14.029963 7200 fetcher.cpp:580] Fetched 'https://github.com/mesos/docker-compose-executor/releases/download/0.1.0/sample-app.tar.gz' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/sample-app.tar.gz'
I0801 02:42:14.029992 7200 fetcher.cpp:442] Fetching URI 'http://192.168.1.64:8090/config.yaml'
I0801 02:42:14.030009 7200 fetcher.cpp:283] Fetching directly into the sandbox directory
I0801 02:42:14.030037 7200 fetcher.cpp:220] Fetching URI 'http://192.168.1.64:8090/config.yaml'
I0801 02:42:14.030053 7200 fetcher.cpp:163] Downloading resource from 'http://192.168.1.64:8090/config.yaml' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/config.yaml'
I0801 02:42:14.031924 7200 fetcher.cpp:580] Fetched 'http://192.168.1.64:8090/config.yaml' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/config.yaml'
I0801 02:42:14.031980 7200 fetcher.cpp:442] Fetching URI 'http://192.168.1.64:8090/general.yaml'
I0801 02:42:14.032016 7200 fetcher.cpp:283] Fetching directly into the sandbox directory
I0801 02:42:14.032052 7200 fetcher.cpp:220] Fetching URI 'http://192.168.1.64:8090/general.yaml'
I0801 02:42:14.032075 7200 fetcher.cpp:163] Downloading resource from 'http://192.168.1.64:8090/general.yaml' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/general.yaml'
I0801 02:42:14.048689 7200 fetcher.cpp:580] Fetched 'http://192.168.1.64:8090/general.yaml' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/general.yaml'
I0801 02:42:14.048729 7200 fetcher.cpp:442] Fetching URI 'http://192.168.1.64:8090/executor'
I0801 02:42:14.048740 7200 fetcher.cpp:283] Fetching directly into the sandbox directory
I0801 02:42:14.048763 7200 fetcher.cpp:220] Fetching URI 'http://192.168.1.64:8090/executor'
I0801 02:42:14.048807 7200 fetcher.cpp:163] Downloading resource from 'http://192.168.1.64:8090/executor' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/executor'
I0801 02:42:15.202059 7200 fetcher.cpp:580] Fetched 'http://192.168.1.64:8090/executor' to '/data/mesos/slaves/dc0144b9-1cbe-49a0-b5ee-096d0b2cbce7-S0/frameworks/be770577-dded-4778-83cf-b193c2716136-0000/executors/marathon-docker-compose-demo.04650ad6-7663-11e7-a1df-024262d6c900/runs/290402cb-7668-473b-b485-c49d24edf1f6/executor'
./executor: 1: ./executor: ����������: not found
./executor: 9: ./executor: Syntax error: "(" unexpected

verbose mode for docker-compose

Support verbose mode for docker-compose controlled via a flag:

docker-compose . [-f args...] --verbose

Inconsistent imports for mesos-go

mesos-go updated the directory structure and thus, any library depending on mesos-go must alter the import statements to match the update.

For example,
github.com/mesos/mesos-go/mesosproto --> github.com/mesos/mesos-go/api/v0/mesosproto
github.com/mesos/mesos-go/executor --> github.com/mesos/mesos-go/api/v0/executor

documentation gap

Address "how to use" and "how to develop" documentation gap.

Add docker dump for debugging and make it config driven

Health check should be enabled in pod monitor

Currently pod monitor only check if containers are up and running in the pod. Health check should also be part of pod monitor.

use http.Transport in http client

Create http.Transport instead of using default one to avoid unnecessary issues in future.

More context in #57

Log the reason for pod health check failure

When ever DCE is killing a pod because of health check failure, it would be helpful to log the reason for health check failure by inspecting the container.

Vagrant setup enhancement

1. create file /etc/default/marathon and put MARATHON_DECLINE_OFFER_DURATION=5000
   This changes the default from 120 secs and competes aggressively with aurora.

2. set /sys/fs/cgroup/memory/memory.user_hierarchy value to 1 from default 0

Panic on empty service in a compose file

When one of the compose file has an empty service for example, even though other compose files have the same service with details, DCE is failing. This is a valid compose file and docker compose will succeed.

#---#
#docker-compose-base.yml
 
version: '2'
services:
  ##Application container overrides
   test:
      <all the details about the service like image etc.,>

#---#
#docker-compose-extends-base.yml
 
version: '2'
services:
  ##Application container overrides
   test:

Handle docker daemon restarts for pod

Currently pod stops at docker daemon restarts. Handle docker daemon restarts via -d option.

nil pointer when failed to create mesos executor driver

If you don't give any argument when start executor, log.Errorf does not terminate process

https://github.com/paypal/dce-go/blob/develop/dce/main.go#L374-L383

	driver, err := exec.NewMesosExecutorDriver(dConfig)
	if err != nil {
		log.Errorf("Unable to create a ExecutorDriver : %v\n", err.Error())
	}

	_, err = driver.Start()

pod monitor can't detect failure of pod if container is removed

pod monitor only monitor the list of containers printed by docker-compose ps. So pod monitor can't detect the failure if container is removed instead of exit.

Pod should encapsulate variables and use receiver functions

https://github.com/paypal/dce-go/blob/develop/utils/pod/pod.go

Has a lot of global variables. This can lead to possible mutations of the variable contents that could create weird behavior.

Ideally, Pod should be an instantiated struct and all the functions which require access to the values inside of pod.

Ignore pulling images failures

Docker always pull images from repository to check if images with same tag get override.
Ignore pulling images failures helps reduce failures if images are cached already.

New Plugin Interface for custom hooks around Executor API methods

Background

DCE's current plugin mechanism is pod lifecycle centric, and allows custom extensions to be plugged in around the steps needed to launch a pod (i.e. custom plugins can be added pre/post of image pull, compose up steps)

Requirement

There are situations that demand custom logic executions outside of pod lifecycle, and aligned with the executor's API functions. For example, we have a requirement to execute some custom logic to post error related metrics whenever the the control exits LaunchTask implementation of dce-go.

This may be applicable for other executor API methods too but scope of this issue could deal with LaunchTask and allow extensibility for all API methods.

Proposed Design

Executor Hook

• ExecutorHook allows custom implementations to be plugged post execution of Docker Compose Mesos Executor's API methods
• Here is a representation of proposed ExecutorHook interface

type ExecutorHook interface {
   // PostExec is invoked  post execution of Docker Compose Mesos Executor's lifecycle function
   PostExec(taskInfo *mesos.TaskInfo) error
   // BestEffort is invoked in case a PostExec returned an error and are expected to return a bool to indicate
   // if the execution needs to continue with the next available hook or not
   BestEffort(execPhase string) bool
}

• Config for executor hook is defined based on the method of the executor API as shown below. For current requirement explained above, supporting post execution of LaunchTask is only needed. But there are possibilities to introduce "Pre" hooks and for a different API method of the executor so the config structure is designed to support that.

...
execHooks:
   LaunchTask:
      Post: ["hook1", "hook2"]
...

So, on exit from LaunchTask, most definite thing that is done in the current DCE is to send status to mesos. So to perform the post executions, we can introduce a task status channel and have the hooks executed based on various status changes.

Support bridge port map replacement when a single port is defined.

So, in compose ports section "portx:porty" is correctly handled and the host port replaced by the dynamic port assigned by mesos. However, if the compose file only has "porty" to specify the container port, the dynamic host port addition does not happen.