docker/compose-cli

Easily run your Compose application to the cloud with compose-cli

The Moby Project

Moby is an open-source project created by Docker to enable and accelerate software containerization.

It provides a "Lego set" of toolkit components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts and professionals to experiment and exchange ideas. Components include container build tools, a container registry, orchestration tools, a runtime and more, and these can be used as building blocks in conjunction with other tools and projects.

Principles

Moby is an open project guided by strong principles, aiming to be modular, flexible and without too strong an opinion on user experience. It is open to the community to help set its direction.

• Modular: the project includes lots of components that have well-defined functions and APIs that work together.
• Batteries included but swappable: Moby includes enough components to build fully featured container systems, but its modular architecture ensures that most of the components can be swapped by different implementations.
• Usable security: Moby provides secure defaults without compromising usability.
• Developer focused: The APIs are intended to be functional and useful to build powerful tools. They are not necessarily intended as end user tools but as components aimed at developers. Documentation and UX is aimed at developers not end users.

Audience

The Moby Project is intended for engineers, integrators and enthusiasts looking to modify, hack, fix, experiment, invent and build systems based on containers. It is not for people looking for a commercially supported system, but for people who want to work and learn with open source code.

Relationship with Docker

The components and tools in the Moby Project are initially the open source components that Docker and the community have built for the Docker Project. New projects can be added if they fit with the community goals. Docker is committed to using Moby as the upstream for the Docker Product. However, other projects are also encouraged to use Moby as an upstream, and to reuse the components in diverse ways, and all these uses will be treated in the same way. External maintainers and contributors are welcomed.

The Moby project is not intended as a location for support or feature requests for Docker products, but as a place for contributors to work on open source code, fix bugs, and make the code more useful. The releases are supported by the maintainers, community and users, on a best efforts basis only, and are not intended for customers who want enterprise or commercial support; Docker EE is the appropriate product for these use cases.

---

Legal

Brought to you courtesy of our legal counsel. For more context, please see the NOTICE document in this repo.

Use and transfer of Moby may be subject to certain restrictions by the United States and other governments.

It is your responsibility to ensure that your use and/or transfer does not violate applicable laws.

For more information, please see https://www.bis.doc.gov

Licensing

Moby is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

compose-cli's Issues

ACI : update Azure client ID in azure login

Currently we're using azure az CLI client ID, and calls to azure will be associated with the az command line.
MS must provide us a new client ID that will be specific to docker CLI, we'll need to update the id associated with login.

SDK : plug docker logs to grpc interfaces

Pkg : define and document packaging of the new CLI / azure integration for Ubuntu

• make bits downloadable for end users (separate download link from the GH repo to have no dependency on opensourcing the repo now)
• document how to install the new CLI.

New CLI : improve error message when trying to use unsupported commands with new backends

Currently the spike changes the list of commands based on the context. if a user tries to use old commands that are not supported with aci/ecs, we return a generic error message :

 $ docker-cli  --context aci swarm init
docker: 'swarm' is not a docker command.
See 'docker --help'

We need to change this message to make it very explicit the command is not available because of the context switch, and it only exists in the "old" non-cloud context (and non-d2).

New CLI documentation

API: events api

We need some events system similar to what moby engine has

Support Firelens Logging

Automate changelog & checksum in GH release

Follow up of https://github.com/docker/api/issues/63 :
GH release is created automatically & binaries for mac, win & unix are uploaded, we need to automate the release changelog (issues & PR for example, like https://github.com/depcheck/depcheck/releases) and checksums for binaries

Implement build and protobuf files' generation in a Dockerfile

The goal is to homogenize the build process through the devs' environments and make bumping process smoother.

Pkg : make new CLI binaries available for download, for Desktop to use

To be discussed : make binaries available through github release, automate github release on tag creation.
We'll need to have binaries for mac/linux/windows, mac/windows will be shipped through Desktop

Pkg : start `docker serve` from desktop for win

Docker Desktop must start the docker cli on a predefined named pipe. The SDK must connect by default on this npipe on win, if no specific socket is provided by the user.

• start cli serve when Desktop starts
• stop cli serve when Desktop stops

SDK will not start the daemon if not available

Linux : document packaging / install of new CLI

Since there is no Desktop on Linux, we must document how users should install the new CLI

• how do we distribute it for linux users, how do they download it along with classic cli
• where to install it, and make sure the shell out to docker-classic works fine

ACI : replace docker exec to write /etc/hosts file by a sidecar container

Currently, in order to allow service name resolution within a compose stack, once the compose containers have been started in ACI, we docker exec inside one of the containers to write service names in /etc/hosts.
This requires containers to allow /bin/sh execution, and is done after all containers are running.
We should replace this docker exec by adding a side car container to the compose stack, the side car doing nothing but writing service names to /etc/hosts.
the ps implementation will need to filter out the side car containers so that they are not visible to users

Run help shows random name

The help for --name includes the randomly generated name.

$ ./bin/docker run --help
...
      --name string           Assign a name to the container (default "crazy-allen")

The existing CLI just shows:

$ docker run --help
...
      --name string                    Assign a name to the container

Add context show command

As defined here

Context management : add docker context use

currently the new CLI only has context ls / context create command.
e2e test is using directly the old CLI in order to switch context, we must update e2e to run docker context use on the new CLI

Add volume mounting string on command line

SDK : Java SDK

SDK : provide a method to let MS know if context is default type or not

• List contexts
• get current context and context type (to know if VSCode must use dockerode or not)
  In cases where the new CLI is not installed, the SDK must be able to answer current context / type without connecting to the server (possibly getting the info from error connecting to the backend, or getting contexts from the cli cmdline directly)

Data capture in new CLI - foundation

Scope
Implement data capture for just the number of 'run' events which is stored in a way that Docker Desktop could pick up.
This should aggregate over a 24 hour period and be written to a file that Desktop can pick up.

Goal
prove data capture from CLI and 'pick up' from Desktop

Background
We cannot capture data in the Docker Desktop proxy as the proxy would need to have full knowledge of all the flags. Instead this must be done in the cli. As this will therefore be open source it will need to default to off and have a standard, user auditable way of collecting the metrics.

The current suggestion is to have a common metrics collection interface in cli that links into the cobra commands. The backend for this interface could be:

A simple file backend that the Docker Desktop service reads the data from.
Separate Windows and Mac backends that forward the command usage to the Docker Desktop APIs. What backend to use should be configurable.

Previous suggestion was to use prometheus with local file storage to gather the metrics. Docker Desktop can periodically read and clear this.

ACI : Adjust container IDs between all comands

We need to have consistent container IDs that allow to match output of docker ps with docker logs, docker exec and docker inspect.
This must work for single containers and also compose stack containers.
Single containers are fine, the container id is either generated or provided when the container is created first.
Compose stack : we need a way to identify both the compose project (mapped to a container group in ACI, also used for compose down and possibly other compose commands, logs...) and the container ID. A naming convention : container id = composeProjectName_service_name seems to work fine, it makes things easy to process technically, and also makes things easy to understand for a user when they do docker ps, like with local compose containers

Data Capture - CLI flags

Scope
Capture CLI flags by each command & by context

Background
https://docs.google.com/document/d/1lzOtnyQnHyfwc3FI8jpCdi-VcprP_IQehWN7fHf_JNI/edit?ts=5eb3e676

ACI : add azure login command

the docker context azure-login command from /docker/spiky must be moved to this repo.

Contexts : do not allow creation of context with type "foo"

 $ ./bin/docker context create test1 toto
 $ ./bin/docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENPOINT                KUBERNETES ENDPOINT                                 ORCHESTRATOR
default *           docker              Current DOCKER_HOST based configuration   unix:///var/run/docker.sock   https://kubernetes.docker.internal:6443 (default)   swarm
test1               toto                                                            

We must check the type and allow only types known by existing backends, or docker ("empty" type)

SDK : unify interfaces and .proto

example : container Create() invokes Run()

Pkg : start `docker serve` from desktop for mac

Docker Desktop must start the docker cli on a predefined unix socket. The SDK must connect by default on this socket on mac, if no specific socket is provided by the user.

• start cli serve when Desktop starts
• stop cli serve when Desktop stops

SDK will not start the daemon if not available

Data capture - context changes

Scope:
Capture the number of times the user has changed context and the contexts they have had 'in focus'.

Background
Overall background document: https://docs.google.com/document/d/1lzOtnyQnHyfwc3FI8jpCdi-VcprP_IQehWN7fHf_JNI/edit?ts=5eb3e676

ACI : provide an API/SDK for external tools (VS Code)

External tools must be able to fetch information when docker context is switched to a non-docker daemon context (there is no more daemon API)
we must provide a way for external tools to access container functionality :

• list containers
• get logs from a container
• start/stop a container
• delete a container
• inspect a container

The cli already has a serve command starting the grpc server.
We must package and ship the SDK for node js at least, and make it available for users

ACI : attach to log containers and tail logs

Currently we have an ACI command to fetch logs from a container.
ACI provides an API to specify a tail option to fetch a specified number of lines in the logs.
To follow logs, there is no streaming API, the az command line polls logs and rewrites the ouput every 2 secs.

• add --follow option to attach to logs
• add --tail option to tail logs

ACI : allow interactive run

At the moment we only allow run without specifying a command in the cmdline

Allow docker run ubuntu echo foo

ACI : azure login on windows is asking for windows firewall permissions on first execution

By default, windows firewall does not allow any .exe to open ports, which the azure login process does to interact with the browser. Once the user clicks "allow" this does not happen anymore, but we should use Desktop to configure properly the windows firwall and allow the aci backend binary to execute without warning. (Desktop already has/had some similar code to allow smb port for filesharing)

Introduce a sidecar container to inject configs from Parameters Map

Rename the cli.proto to context.proto

What's called cli.proto today is actually just an API that returns contexts, we will rename it to context.proto

Pkg : update Desktop *for mac* packaging to include the new CLI next to the moby CLI and make sure it delegates properly when using default context

Desktop packaging needs to be updated to fetch this new CLI in addition to the moby one.
Desktop docker links (or PATH on windows) must point to this new CLI, and not to the moby one.
To ensure delegation works well, we will need to adjust how the shell out is performed (2 binaries in the same folder, with different binary names, or same names in different folders... )

ACI : allow using private images from hub

Take code from /docker/spiky to fetch hub login (if available) and send it with deployment info to allow hub private images in ACI.
Same for ACR registries (supporting tokens set by az acr login).

ACI : allow CPU / mem limit when running containers, transfer limits to ACI

ACI allows CPU / mem limitations. ATM we have hardcoded these limits when sending requests to ACI, we need to set flags to docker run (--cpu, --memory, from the existing run flags) and transfer the user reqs to ACI, if specified.

new CLI SDK : documentation

If possible generate doc along with the generated nodejs SDK

Pkg : update Desktop *for win* packaging to include the new CLI next to the moby CLI and make sure it delegates properly when using default context

Desktop packaging needs to be updated to fetch this new CLI in addition to the classic one.
Desktop docker PATH must point to this new CLI, and not to the classic one.
To ensure delegation works well, we will need to adjust how the shell out is performed (2 binaries in the same folder, with different binary names, or same names in different folders... )

ACI : run / rm output

docker run / docker rm command should output the container id that is created / deleted.
That should be used in e2e tests to execute a user flow

Data Capture - CLI commands by context

Scope

Count the CLI commands used by a user by context and store them for desktop to access

Background

https://docs.google.com/document/d/1lzOtnyQnHyfwc3FI8jpCdi-VcprP_IQehWN7fHf_JNI/edit?ts=5eb3e676

Release SDK packaging

• Build full SDK (generated code and custom code for common operations : init, get contexts, ? )
• make it available for download
• ideally, publish this on a docker npm repo.
  @nebuk89 @justincormack any reason for not making the SDK public initially (before e disclose the MSFT partnership or before the new CLI is shipped with desktop)

Context management : docker context ls improvements

• show current context with *
• show default context that does not exist in store
• I had some parsing errors when using old context from /docker/spiky. we might need to check why some unexpected context fail to read and make the code more robust (./bin/docker context ls:\nCommand stdout:\nunexpected end of JSON input\n\nstderr:\n\nerror:\nexit status 1)
• show ACI resource group and location for ACI contexts

e2e tests are currently using the old cli to list context and check which is the default one, we meust update e2e to use the new CLI for this

ACI : interactive context create command

docker context create aci aci has flags for subscriptionID, resourcegroup & location. We need to also allow interactive creation like in /docker/spiky, and let the user select the resource group from a proposed list

Context management : add docker context rm

currently the new CLI only has context ls / context create command.
e2e test is using directly the old CLI in order to remove context, we must update e2e to run docker context rm on the new CLI

ACI : display exposed port/IP

when running docker run nginx -p 80:80, subsequent docker ps should display a PORTS column with the ACI public IP & port where we can connect to the deployed app.
(Code can be taken from /docker/spiky)

Add config package tests

Implement `rm` command for a container

Contexts : do not allow removing the current context

$ ./bin/docker context use test1
test1
$ ./bin/docker context rm test1
test1
$ ./bin/docker context ls
exit status 1

the classic CLI allows context rm -f, this switches to the default context before rm

ACI : package SDK for external tools (esp. node js)

define a way to share the SDK with our users : how do they download it, and use it in nodejs project.
At least nodejs, this is the target for VSCode

ACI : allow non-interactive exec with specified command

allow docker exec mycontainer "ls -al"