datastack-net/dockerized

Run popular commandline tools within docker

dockerized's Issues

Automatic port forwarding

Description

As a user, I would like to run commands like npm run serve or mkdocs serve without being required to manually map the port to the container.

This way, the commands work more like native commands, and it makes dockerized much more convenient.

---

Requirements:

1. The user does not always need to specify -p <PORT> when running dockerized.
2. Multiple commands can still run in parallel (e.g. npm run serve and npm run lint, using the same npm command).
3. User is not tied to specific ports, and can use the command as flexibly as the native command.
4. Should work equally well on each platform (dockerized should not have features that only work on some platforms)

Out of scope:

1. Commands that open multiple ports should expose all of them.

Possible solutions:

• No mapping
  • Use the host network directly
    • ❌ Not possible on Windows/Mac
• Static mapping
  Predefine ports in built-in docker-compose.yml (e.g. always expose port 8080 for npm)
  • ❌ will cause port-mapping conflicts, e.g. running npm run serve and npm run lint
  • ❌ prevents user from choosing another port
• Post-mapping
  Automatically detect that a port was opened in the container, and map it afterwards
  • ❔ Open port detection within the container:
    • https://stackoverflow.com/questions/40350456/docker-any-way-to-list-open-sockets-inside-a-running-docker-container
  • Mapping the ports after container started:
    • ❔ Exposing the port after starting the container, using native docker networking code. Seems not possible.
    • ❌ Exposing the port to a random host-port, at start. Then tunneling it on the host.
      • ❌ We wouldn't know which container port to expose
    • Map a random port to the container for communication, then map them both on the host, and within the container
      HOST:application port ⇄ HOST:$RANDOM ⇄ CONTAINER:$RANDOM ⇄ CONTAINER:application port
    • 🙁 Ugly solution (might be unreliable, slow, complex), but might work.
• Pre-mapping
  Detect which port will be needed, and then expose/map it before starting the container
  • 🤔 This will be command specific, regardless of the solution, so not as scalable.
  • ✅ Detecting port parameters, i.e. mkdocs serve --dev-addr=0.0.0.0:8000 reveals that port 8000 should be opened.
    • Possible, with predefined (regex) patterns per command. For mkdocs, this could be --dev-addr=[^:]+:(?P<port>\d+)
    • Patterns can be stored as labels in the docker-compose file: com.dockerized.port
  • Detecting ports from configuration files, e.g. dev_addr within mkdocs.yml
    • Will work for simple single-file static configuration
    • Multi-file, environment dependent configuration (e.g. npm run serve --env=production using a different port)
  • Detecting ports from environment variables, e.g. some commands may natively use $PORT to determine the port to run on.
• Two-phase mapping
  • Concept:
    1. Run the command
    2. Detect the opened ports
    3. Allow user to re-run the command with the ports mapped
  • Variations
    • ✅ Opened ports will be remembered, and automatically mapped the next run.
      • Mapping can be stored in projects dockerized.env per command
        • ❌ Prevents running the same command twice (requirement 2)
      • ❌ Can be stored in a global cache, based on path (breaks 2)
      • Cache based on all command arguments and working directory
        • 🤔 won't work for dynamic arguments, such as <command> foo $(date)
          • But, will work in many cases. Covering a bunch of them will be an improvement.
    • Opened ports are remembered, but the user needs to confirm somehow
    • Immediately kill the command, and re-run it with ports mapped
      • ❌ Not so nice. Not all opened ports will be essential to the user, and sometimes a long build-process may happen before serving.

Support for Apple Silicon (aarch64)

The script bin/dockerized isn’t set up to be able to build for Apple Silicon. Modifying the architectures should probably include a detection for aarch64 arm64, the arm64 / aarch64 target for GO.

https://github.com/datastack-net/dockerized/blob/master/bin/dockerized#L55-L63

Overriding system environment variables issue

I have noticed that on my Windows 10 laptop when I use WSL and attempt to override the Linux HOME environment variable, the overriding of the value partially works.

The reason that I am trying to do this is so my Windows home .ssh configuration is shared between the CMD and WSL environments, while using the same dockerized docker-compose.yaml file.

In Windows 10, there is no HOME environments variable, instead there is a HOMEPATH.
In WSL there is already a HOME environments variable that points to the Linux /home/<wsl user> folder. I would like to override this to point to the Windows 10 HOMEPATH environments variable instead.

NOTE: Below is a working version as I am using the HOMEPATH environment variable in the docker-compose.yml file. When I change HOMEPATH to HOME, it then fails.

Here is my setup:

# C:\Users\<user>\dockerized.env (Command Prompt)

HOME=${HOMEPATH}
COMPOSE_FILE="${COMPOSE_FILE};${HOME}/.dockerized/docker-compose.yml"

# /home/<wsl user>/dockerized.env (WSL2)

HOMEPATH=/mnt/c/Users/<user>
HOME=${HOMEPATH}
COMPOSE_FILE="${COMPOSE_FILE};${HOME}/.dockerized/docker-compose.yml"

# C:\Users\<user>\.dockerized\docker-compose.yml
# /mnt/c/Users/<user>/.dockerized/docker-compose.yml

version: "3"
services:
  dev:
    image: "my-project-build-env"
    build:
      context: ${HOMEPATH:-.}/.dockerized/apps/my-project-build-env
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    entrypoint: [ "/init.sh", "/bin/bash" ]
    volumes:
      - ${HOMEPATH:-.}/.m2:/root/.m2
      - ${HOMEPATH:-.}/.ssh:/dockerized/host/home/.ssh
      - ${HOMEPATH:-.}/.dockerized/apps/my-project-build-env/init.sh:/init.sh

Here is the successful WSL debug logs when using HOMEPATH:

> dockerized -v dev
Dockerized root: /mnt/c/Tools/dockerized
Loading: '/mnt/c/Tools/dockerized/.env'
Loading: '/home/<wsl user>/dockerized.env'
Compose files: /mnt/c/Tools/dockerized/docker-compose.yml, /mnt/c/Users/<user>/.dockerized/docker-compose.yml

# My dockerized 'dev' service works.

Here is the failing WSL debug logs when using HOME:

> dockerized -v dev
Dockerized root: /mnt/c/Tools/dockerized
Loading: '/mnt/c/Tools/dockerized/.env'
Loading: '/home/<wsl user>/dockerized.env'
Compose files: /mnt/c/Tools/dockerized/docker-compose.yml, /mnt/c/Users/<user>/.dockerized/docker-compose.yml

# My dockerized 'dev' service fails.

The above debug logs are exactly the same.

Based on the compose files specified above, I generated a docker-compose command:

> docker-compose -f /mnt/c/Tools/dockerized/docker-compose.yml -f /mnt/c/Users/<user>/.dockerized/docker-compose.yml config

ERROR: build path /home/<wsl user>/.dockerized/apps/my-project-build-env either does not exist, is not accessible, or is not a valid URL.

When I run:

> HOME=/mnt/c/Users/<user> docker-compose -f /mnt/c/Tools/dockerized/docker-compose.yml -f /mnt/c/Users/<user>/.dockerized/docker-compose.yml config

# It succeedds and shows the complete docker-compose file which was assembled from the 2 yaml files.

Based on this, I think that the system environment variables aren't being overridden, or the system variables are being applied on top of the other .env and dockerized.env variables.

Would it be possible to create an internal dockerized 'config' service so that it will show the complete docker compose file and service configuration, like the 'docker-compose config' does?

Feature: allowing docker-compose.override.yml file

Hi,

When the docker-compose command is run it will use the docker-compose.yml and docker-compose.override.yml file, and combine their configurations.
Could this be added to this projects?

This would allow me to have my own docker-compose.override.yml file which would be combined with the default configuration of dockerized, and allowing me to easily upgrade without worrying about overwriting my own local changes.

--shell tests failing for specific commands

Most mysterious bug ever.

Expected:

• dockerized --shell <command> -c env should print the current environment

Actual:

• In some specific conditions, there is no output from the container at all, and dockerized just exits.

  Run bin/dockerized -v --shell perl -c env | tee ~/shell.log
  Dockerized root: /home/runner/work/dockerized/dockerized
  Loading: '/home/runner/work/dockerized/dockerized/.env'
  Compose files: /home/runner/work/dockerized/dockerized/docker-compose.yml
  Setting up shell in container for perl...
  Entrypoint: [/bin/bash]
  Command:    [-c env]
  Preparing compose environment...
  Network dockerized_default  Creating
  Network dockerized_default  Created
  Volume "dockerized_node_modules"  Creating
  Volume "dockerized_node_modules"  Created
  Volume "dockerized_python_modules"  Creating
  Volume "dockerized_python_modules"  Created
  Volume "dockerized_pip_cache"  Creating
  Volume "dockerized_pip_cache"  Created
  Volume "dockerized_go_cache"  Creating
  Volume "dockerized_go_cache"  Created
  Volume "dockerized_home"  Creating
  Volume "dockerized_home"  Created
  Running one-off container...
  Container exited with code 0.
  

Observations:

• These shell tests worked perfectly in march, and suddenly broke.
• Many official docker images were updated under the same tags. Alpine and busybox were changed.
• Affected commands:
  • go, python, perl, rustc
• Still works perfectly locally on windows
• It only breaks the first time. Running the command twice makes it work somehow.
• Running docker-compose before dockerized also causes the --shell test to work.
• For some commands, the issue is with bash. Switching to sh for --shell fixes the bug.
• sh is not a universal fix. rustc uses sh and still doesn't work.
• Was able to reproduce once on local Ubuntu VM. Cannot reproduce after. Even docker system prune -af won't help to reproduce it, running without a tty. (true | dockerized --shell perl -c env | cat)
• docker-compose run --rm <command> always seems to work.

Hypothese (and motivation):

• Something related to setting up the networks / compose project
  • There could be small implementation differences between docker-compose run and dockerized
  • Since docker-compose seems to work without failure, there must be something different causing the bug.
• Something TTY related
  • Github Actions have no TTY whereas local running from a shell will provide a TTY, explaining reproduction difficulty
  • ❌ Then why does it work the second time?
  • Running echo helloworld the first time didn't fix perl, but disabling TTY did. So TTY is involved.
• Related to updates in Bash and Alpine in official docker images
  • This did work a month ago and suddenly broke. Re-ran an old successful action and it didn't work anymore.
  • Specifying the exact alpine version in the docker image tag has fixed the issue for some commands. Unfortunately, the old digests are not published by dockerhub, so it's not possible to restore all commands. Also, this is not a good long term solution.
  • ❌ Then why does it work the second time?
• Something specifically related to the env command

Excluded causes:

• Related to pulling the images the first time.
  • ❌ Added dockerized --pull. Running this before the shell test doesn't make a difference.
• Related to a race condition in creating the network.
  • ❌ The issue occurs consistently with a specific subset of commands and is not unpredictable

Improving support for ansible

Errors are not being output

Easier Version Selection

Instead of
NODE_VERSION=15.0.0 dockerized node
how about
dockerized node:15

Just a suggestion!

How to use dockerized Go with vim-go or JetBrains IDE?

How to use dockerized Go when I need to specify $GOROOT and $GOPATH for linting in my IDE

https://github.com/fatih/vim-go/blob/421563081bddaec1f7a66710b5c8ee305724d2a9/doc/vim-go.txt#L123-L126
https://www.jetbrains.com/help/go/configuring-goroot-and-gopath.html

[Question] Is this still maintained?

Hi,
I stumbled over this wonderful project. Really like it so far, thanks :)
Looking at the last update beeing in 2022 I was wondering, if this is still maintained / improved?

Regards,
Daniel

Extra --build arguments: --pull and --no-cache

• I'm using "dockerized --build run ..."
  As far as I understand I'm currently not able to pass some "--build" parameters to this command (?)
  https://docs.docker.com/engine/reference/commandline/compose_build/
  For example --no-cache and --pull would be quite nice for my workflow.

• As far as I understand I'm currently not able to do an "only-build" execution instead of
  "build + run" (?)
  I have a local base image as dependency for my final image.
  If I execute "dockerized --build run ..." this will (obviously) be executed, while I would only need it to be built.
  Workaround is to use "docker compose build ...." instead, which leads to a mix of "dockerized" and "docker compose" - commands in my workflow.

How do you feel about these points? :)

Regards,
Daniel

Originally posted by @danielrolfes2307 in #50 (comment)

Add more features

Currently there are almost no features, and they are implemented twice, once in Bash and once in Powershell.
This is important to avoid any dependencies beyond Docker, and to make it cross-platform.

In the long term, implementing everything twice in languages like Bash and Powershell may not be sustainable.

How should Dockerized be implemented, to support more features, keep it cross-platform, and make it easy to contribute and maintain?

Here are all the options I've thought of, happy to hear your thoughts.

• Platform supported scripts (Bash + Powershell)
  • ✔️ Natively supported by platform without dependencies
  • 👎 Requires maintaining 2 implementations
  • 👎 Difficult to maintain as scripts become more complex
  • 👎 Not accessible for contributors when scripts turn into full programs
  • 👎 Hard to test (no unit tests, debugging)
• Scripts: e.g. Python / Javascript
  • ✔️ Easy to use for developers.
  • ✔️ Great free IDE (PyCharm)
  • ❌ Requires an interpreter to be installed. This is a deal-breaker, as dockerized should run without dependencies (other than docker+compose).
• Scripts packaged into executables:
  • ⁉ Is there any packager that can cross-compile?
    • ❌ Github actions does not host all platforms (e.g. arm64), so this would mean dockerized cannot be used everywhere.
  • Python + PyInstaller
    • ❌ Cannot cross-compile. You need the target platform to compile to it.
  • ...?
• Compiled languages:
  • ✔️ Single executable without dependencies.
  • 👎 Requires managing releases and versions, compiling to multiple target platforms.
    • Semantic release bot and Github actions can automate all of this.
  • Go:
    • ✔️ Small binaries.
    • ✔️ Cross-platform compilation possible.
    • ✔️ Docker-compose is written in Go, potentially able to reuse libraries.
    • ✔️ For users who have go installed, can easily install dockerized with go get.
    • 👎 Limited IDE options (Goland is not free, VsCode limited compared to full IDE)
    • 👎 I've never written any application in Go, and have no clue about best practices.
    • 👎 Somewhat low-level language. Fewer potential collaborators compared to python or node.
  • C#:
    • ✔️ Easy to pick up OOP language
    • ✔️ Cross-platform compilation possible.
    • ✔️ Free IDE available (Visual Studio)
    • ✔️ My main language at the moment
    • 👎 Recommended IDE (Rider) not free
    • 👎 Not as mainstream as Python or Node.
    • 👎 Big binaries as they need to include the dotnet sdk (58mb)
    • 👎 Not very mainstream.
• Script language executed within docker (e.g. dockerized python bin/dockerized.py)
  • ✔️ Any language can be chosen. It will run in docker so all dependencies can be there.
  • 👎 The script will need to call docker-compose from within the container. I'm afraid this will cause problems.
  • ❌ This means 2 calls to docker start. On my super fast Windows system docker start takes 1.5 seconds. Dockerized would become unbearably slow.
• Compiled Language (Go, C#), pre-compiled within docker, and executed on host

  ℹ️ E.g. Include dockerized.go, launch a golang container that compiles an .exe for Windows, extract it, and run it on the host. This can be done for all platforms

  • ✔️ Same benefits as all other compiled language options
  • ✔️ No releases or versioning needed
  • 👎 Additional logic for the compilation process in bash and powershell
  • ❔ Not sure if it's possible to compile
• Any language, and auto-download a standalone interpreter onto the host

  Let dockerized figure out the platform, download the executable into a local directory and then continue in the other language

  • ❌ Python - Does not publish prebuilt binaries for unix platforms (only packages in repositories)
    • A project that attempts to package python https://python-build-standalone.readthedocs.io/en/latest/ warns that the Windows builds are 'extremely unstable'
  • ✔️ Node - Zips/Tars for all platforms and architectures, including windows. Small size (20-50mb)
    • npm install would need to run on the host, as in some cases precompiled binaries are included.
    • ☹️ 32 bit linux not supported anymore
  • ✔️ Go - Available for all platforms as Zip, but quite big (~100Mb)

First run of `dockerized node:16.14 --version` results in an error

❯ bin/dockerized node:16.14 --version
[+] Running 6/0
 ⠿ Network dockerized_default          Created                                                                         0.0s
 ⠿ Volume "dockerized_pip_cache"       Created                                                                         0.0s
 ⠿ Volume "dockerized_go_cache"        Created                                                                         0.0s
 ⠿ Volume "dockerized_home"            Created                                                                         0.0s
 ⠿ Volume "dockerized_node_modules"    Created                                                                         0.0s
 ⠿ Volume "dockerized_python_modules"  Created                                                                         0.0s
[+] Running 10/10
 ⠿ node Pulled                                                                                                        69.2s
   ⠿ c732c99090fe Pull complete                                                                                       42.9s
   ⠿ 26cef8e19215 Pull complete                                                                                       43.1s
   ⠿ 741a7b3bab3c Pull complete                                                                                       43.3s
   ⠿ 66bff08cf973 Pull complete                                                                                       44.9s
   ⠿ 3d35ac8f6f1a Pull complete                                                                                       66.9s
   ⠿ a25eb52b374a Pull complete                                                                                       67.0s
   ⠿ bc98bafd2f98 Pull complete                                                                                       67.9s
   ⠿ 82c51e978386 Pull complete                                                                                       68.0s
   ⠿ 24dba3274ff0 Pull complete                                                                                       68.0s
Error response from daemon: error while creating mount source path '/Users/austin/.dockerized/apps/node': chown /Users/austin/.dockerized/apps/node: permission denied

Second run outputs the expected version (v16.14.2).

Trying with dockerized python:3 --version results in a similar error on the first run:

❯ bin/dockerized python:3 --version
[+] Running 10/10
 ⠿ python Pulled                                                                                                       7.5s
   ⠿ fa223d8c149d Already exists                                                                                       0.0s
   ⠿ 17ffbd2ab159 Already exists                                                                                       0.0s
   ⠿ a3953e76cee2 Already exists                                                                                       0.0s
   ⠿ 6f906570592f Already exists                                                                                       0.0s
   ⠿ c478f6a18a39 Already exists                                                                                       0.0s
   ⠿ 965f10264bda Pull complete                                                                                        3.1s
   ⠿ fae4e8c73329 Pull complete                                                                                        6.1s
   ⠿ 2ac2661d4813 Pull complete                                                                                        6.2s
   ⠿ 993092bb086d Pull complete                                                                                        6.3s
Error response from daemon: error while creating mount source path '/Users/austin/.dockerized/apps/python': chown /Users/austin/.dockerized/apps/python: permission denied

Mismatching User/Group cause file permission issues on Unix

Description

Files created by dockerized commands are not owned by the host's current user, but for example root.
This makes dockerized hard to use on unix systems, as generated files cannot be modified by the user.

It is also inconsistent with the design goal that Dockerized commands should behave the same as native commands.

Reproduction

> dockerized bash -c 'touch foo'
ls -l foo
-rw-r--r-- 1 root root 0 Apr 22 10:57 foo

Possible Solutions

These are all ideas I could come up with so far. Your thoughts and suggestions are welcome.
The goal is to find a solution with no blockers (:x:), and remove ❔ by doing more research.

• Run the docker image with the uid/gid of the host system
• ❔ Ensure container images still work
  • ❌ The host's user id will not exist in the container. This leads to several problems:
    • Container files owned by root (which is the default on most images) may become inaccessible, e.g. /root
    • The user will not have a username
    • The user will not have a home directory (~ will point to /)
  • ❔ Create a user inside the docker container with the host uid/gid, upon start (i.e. with an entrypoint)
    • ❔ Might clash if the specific container already uses that user-id.
  • ❔ Create a dockerfile for each app, that creates a host user, baked into the image
    • ❔ How to ensure this works for multiple host users? E.g. if the host switches to user2, dockerized should still work, and create files as user2.
• ❔ Map host ids to container ids
  • userns-remap
    • Requires re-configuring docker, and is global.
      • ❔ Will this impact the user's other containers? If so, this is not a great option.
  • ID translation
    • ❔ Does this exist somehow? Docker doesn't seem to allow ad-hoc uid translation.
  • Resources
    • https://linuxnatives.net/2019/align-user-ids-inside-and-outside-docker-with-subuser-mapping
    • https://echorand.me/posts/docker-user-namespacing-remap-system-user/
• ❔ ID translation at the filesystem level
  • ❔ Change mounting settings so that id's are immediately mapped by docker
  • ❔ Re-mount the working directory within Unix, and somehow do id translation there, then mount that directory into docker
    • https://github.com/hanwen/go-fuse/
• ❔ ID translation within the container
  • Somehow wrap the executed command and translate id's at the user level
    • ❌ This is pretty much the same as running the container with --user, causing permission problems, e.g. reading /root
  • Somehow remount the mounted host directory within the container, somehow translating the ids
    • https://www.joyfulbikeshedding.com/blog/2021-03-15-docker-and-the-host-filesystem-owner-matching-problem.html
    • ❔ how
• ✔/:x: Changing the owners/permissions of the generated files after each command is run
  • ❌ This won't work for long running commands, such as npm run which continuously updates files. Permissions won't be fixed until the program finishes.
  • Detecting which files need permission fixes:
    • ❌ Cannot use file stats, as some commands output files with specific mtimes (e.g. tar)
    • ❌ Cannot rely on just detecting new files, as some files may have been re-created with different owners
    • ✔️ Remembering all files + permissions before the command, and detecting new files and files with different owners/permissions.
      • ❔☹ Performance. Although filesystem access can be insanely fast: For example https://github.com/boyter/scc
    • ❔ Another way to detect generated / changed files?
  • Implementation: Let Dockerized fix the permissions
    • ❌ requires running dockerized as root
  • ✔️ Implementation: Let docker fix the permissions. I.e. run dockerized alpine chown $(id -u):$(id -g) $FILES (this works)
    • ☹ Performance (running another container adds 450ms, on my local virtual ubuntu)
    • If docker can generate the files as root on the host system, it can also run chmod on those files
    • If docker cannot generate files as root, then the problem didn't occur either, although the files may be owner by docker or another user.
  • Implementation: Run chown within the container, after the user's command. E.g. with an entrypoint script.
    • ❔ Do all unix containers have chown?
    • 😄 Performance will be better (no extra container overhead)
    • Can combine the two approaches (within container if possible, otherwise in separate container)

Does not compile in docker under Apple Silicon

❯ bin/dockerized --compile=docker
Compiling dockerized...
go: downloading github.com/docker/compose/v2 v2.3.3
go: downloading github.com/hashicorp/go-version v1.3.0
go: downloading github.com/mitchellh/mapstructure v1.4.3
go: downloading github.com/opencontainers/go-digest v1.0.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/imdario/mergo v0.3.12
go: downloading github.com/docker/docker v20.10.3-0.20220121014307-40bb9831756f+incompatible
go: downloading github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/theupdateframework/notary v0.6.1
go: downloading github.com/buger/goterm v1.0.4
go: downloading github.com/containerd/containerd v1.6.1
go: downloading github.com/docker/buildx v0.7.1
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/moby/buildkit v0.9.1-0.20211019185819-8778943ac3da
go: downloading github.com/opencontainers/image-spec v1.0.2
go: downloading github.com/sanathkr/go-yaml v0.0.0-20170819195128-ed9d249f429b
go: downloading golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: downloading github.com/docker/go v1.5.1-1.0.20160303222718-d30aec9fd63c
go: downloading github.com/morikuni/aec v1.0.0
go: downloading github.com/containerd/console v1.0.3
go: downloading github.com/AlecAivazis/survey/v2 v2.3.2
go: downloading github.com/golang/mock v1.6.0
go: downloading google.golang.org/grpc v1.45.0
go: downloading github.com/klauspost/compress v1.13.5
go: downloading github.com/opencontainers/runc v1.1.0
go: downloading golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
go: downloading github.com/moby/sys/signal v0.6.0
go: downloading github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415
go: downloading github.com/agl/ed25519 v0.0.0-20170116200512-5312a6153412
go: downloading golang.org/x/net v0.0.0-20211216030914-fe4d6282115f
go: downloading github.com/moby/sys/symlink v0.2.0
go: downloading golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
go: downloading github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/docker/go-metrics v0.0.1
go: downloading github.com/prometheus/client_golang v1.11.0
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/prompt/prompt.go:20:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/progress/writer.go:24:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:26:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build_buildkit.go:26:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:27:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:28:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:29:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/prompt/prompt_mock.go:10:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build_classic.go:40:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/api/labels.go:22:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]+incompatible/pkg/archive/archive.go:26:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:32:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]+incompatible/cli/command/image/build/dockerignore.go:8:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:33:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/compose/[email protected]/pkg/compose/build.go:34:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]+incompatible/pkg/jsonmessage/jsonmessage.go:11:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]+incompatible/pkg/jsonmessage/jsonmessage.go:12:2: zip: not a valid zip file
/go/pkg/mod/github.com/distribution/distribution/[email protected]/digestset/set.go:9:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]+incompatible/api/types/registry/registry.go:7:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]+incompatible/pkg/idtools/idtools_unix.go:17:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]/counter.go:3:8: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]/handler.go:7:2: zip: not a valid zip file
/go/pkg/mod/github.com/docker/[email protected]/sockets/proxy.go:9:2: zip: not a valid zip file
/go/pkg/mod/golang.org/x/[email protected]/ssh/terminal/terminal.go:14:2: zip: not a valid zip file
/go/pkg/mod/github.com/containerd/[email protected]/errdefs/grpc.go:24:2: zip: not a valid zip file
/go/pkg/mod/github.com/containerd/[email protected]/errdefs/grpc.go:25:2: zip: not a valid zip file
Failed to compile dockerized

Compiling on the host works just fine.

Release v2.15.0 doesn't contain build artifacts, missing /bin folder.

The latest release v2.15.0 doesn't contain build artifacts, missing /bin folder.

Docker Image locking

Lock default versions to specific image hashes, to make command upgrades explicit.

Why:

Last week all major official dockerhub images were updated and existing tags replaced. This has broken the --shell integration tests for several commands.

For example, golang:1.16 now uses alpine3.15, whereas before, it may have been 3.14 or even a different linux distro (it's not actually possible to know what was used before).

This reveals that specifying versions (or even tags) is not enough to guarantee reproducibility, i.e. dockerized may work well one day, and be broken another. Differences between systems may occur, depending on which image was present locally for a specific tag.

Functional requirements

• Commands that are included with dockerized are locked to specific image digests
• The .env file is still readable and intuitive, i.e. GO_VERSION=1.17.8 and not GO_VERSION=f837w389875t0438r
• The versions are locked to the same digest for all users (so not determined upon first use, but included in the release)
• A way for maintainers to upgrade the digest of a version

Technical implementation

Todo.

• A lock file will probably make the most sense

Out of scope

These features can be considered for a future iteration:

• Per user and per project locked versions.
• Version locking of ad-hoch specified versions (e.g. dockerized go:1.16 will then lock go 1.16 to a particular digest, 'forever')