compose-spec/compose-go

Reference library for parsing and loading Compose YAML files

compose-go

Go reference library for parsing and loading Compose files as specified by the Compose specification.

Usage

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/compose-spec/compose-go/v2/cli"
)

func main() {
	composeFilePath := "docker-compose.yml"
	projectName := "my_project"
	ctx := context.Background()

	options, err := cli.NewProjectOptions(
		[]string{composeFilePath},
		cli.WithOsEnv,
		cli.WithDotEnv,
		cli.WithName(projectName),
	)
	if err != nil {
		log.Fatal(err)
	}

	project, err := cli.ProjectFromOptions(ctx, options)
	if err != nil {
		log.Fatal(err)
	}

	// Use the MarshalYAML method to get YAML representation
	projectYAML, err := project.MarshalYAML()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(string(projectYAML))
}

Build the library

To build the library, you could either use the makefile

make build

or use the go build command

go build ./...

Run the tests

You can run the tests with the makefile

make test

or with the go test command

gotestsum ./...

Other helpful make commands

Run the linter

make lint

Check the license headers

make check_license

Check the compose-spec.json file is sync with the compose-spec repository

make check_compose_spec

Used by

• compose
• containerd/nerdctl
• compose-cli
• tilt.dev
• kompose
• kurtosis
• testcontainers-go's Compose module

compose-go's Issues

character '-' dash no longer valid environment variable

got this error unexpected character "-" in variable name near "AIRFLOW_CONN_COMPOSER-WORKER-SAMPLE..."

afaik, specs for env is IEEE Std 1003.1-2001 - any printable character but = and NUL is valid

i'm not sure if the problems come from this lib or docker/compose

please close this issue if it's not relevant in this repository.

thank you

Service called "extensions" throws exception in loader

Apparently calling a service in a docker-compose file "extensions" is not allowed. I didn't find any mention about it being a keyword or anything else in the docker-compose file reference so I think it's a bug.

How to reproduce: Create a docker-compose file with a service called "extensions" try to up the service (or any service else in the docker-compose for that matter).

Following exception will be thrown:

panic: interface conversion: interface {} is string, not map[string]interface {}

goroutine 1 [running]:
github.com/compose-spec/compose-go/loader.loadServiceWithExtends({0xc0001df8c0, 0x3c}, {0xc0000388ca, 0x5}, 0xc0005bdf20, {0xc0000c8090, 0x29}, 0x23dce26def9, 0xc000047020, 0xc00061e350)
        github.com/compose-spec/[email protected]/loader/loader.go:491 +0x6ab
github.com/compose-spec/compose-go/loader.LoadServices({0xc0001df8c0, 0x3c}, 0x190d525, {0xc0000c8090, 0x29}, 0x0, 0x6f439bc26e497e4)
        github.com/compose-spec/[email protected]/loader/loader.go:470 +0x1d1
github.com/compose-spec/compose-go/loader.loadSections({0xc0001df8c0, 0x3c}, 0x0, {{0x0, 0x0}, {0xc0000c8090, 0x29}, {0xc0005981e0, 0x2, 0x2}, ...}, ...)
        github.com/compose-spec/[email protected]/loader/loader.go:295 +0x2f4
github.com/compose-spec/compose-go/loader.Load({{0x0, 0x0}, {0xc0000c8090, 0x29}, {0xc0005981e0, 0x2, 0x2}, 0xc000046a50}, {0xc0005861f0, 0x2, ...})
        github.com/compose-spec/[email protected]/loader/loader.go:183 +0x912
github.com/compose-spec/compose-go/cli.ProjectFromOptions(0xc000403730)
        github.com/compose-spec/[email protected]/cli/options.go:337 +0x29e
github.com/docker/compose/v2/cmd/compose.(*projectOptions).toProject(0xc0000d7780, {0xc00041bdd0, 0x10000c00083fab8, 0x3}, {0xc00083fa78, 0x18, 0x23da8f62588})
        github.com/docker/compose/v2/cmd/compose/compose.go:158 +0x76
github.com/docker/compose/v2/cmd/compose.(*projectOptions).WithServices.func1({0x1ae9340, 0xc0002b6040}, {0xc00041bdd0, 0x1, 0x3})
        github.com/docker/compose/v2/cmd/compose/compose.go:120 +0x98
github.com/docker/compose/v2/cmd/compose.Adapt.func1({0x1ae9340, 0xc0002b6040}, 0x2, {0xc00041bdd0, 0x1, 0xc00083fb20})
        github.com/docker/compose/v2/cmd/compose/compose.go:87 +0x36
github.com/docker/compose/v2/cmd/compose.AdaptCmd.func1(0xc00033ef00, {0xc00041bdd0, 0x1, 0x3})
        github.com/docker/compose/v2/cmd/compose/compose.go:66 +0x223
github.com/spf13/cobra.(*Command).execute(0xc00033ef00, {0xc00007e7f0, 0x3, 0x3})
        github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc00045b180)
        github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:902
github.com/docker/cli/cli-plugins/plugin.RunPlugin(0xc000073e28, 0xc00033ec80, {{0x19086a4, 0x5}, {0x190fedf, 0xb}, {0x1aafba0, 0x6}, {0x0, 0x0}, ...})
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:51 +0x130
github.com/docker/cli/cli-plugins/plugin.Run(0x198c118, {{0x19086a4, 0x5}, {0x190fedf, 0xb}, {0x1aafba0, 0x6}, {0x0, 0x0}, {0x0, ...}, ...})
        github.com/docker/[email protected]+incompatible/cli-plugins/plugin/plugin.go:64 +0xee
main.pluginMain()
        github.com/docker/compose/v2/cmd/main.go:41 +0xdf
main.main()
        github.com/docker/compose/v2/cmd/main.go:74 +0x1de

My setup:
Docker Desktop for Windows Version: 4.6.1
Engine: 20.10.13
Compose: 2.3.3
Credential Helper: 0.6.4
Kubernetes: 1.22.5
Snyk: 1.827.0

I hope this is the correct repo to post this issue because the exception is being thrown by compose-go/loader.loadServiceWithExtends routine.
If you need anything else, just let me know.

offer simple way to express (un)supported keys

We need to replace #11 with a pluggable API for implementations to express keys they don't support.

v3 broke the v2 approach to define services options as a list of fields, that can be easily white|blacklisted as []string. We need a way to filter an actual yaml tree.

Proposal:
introduce option for implementation to declare a whitelist (or blacklist) of yaml paths. Check configDict map[string]interface{} to only/not include elements matching such paths

It doesn't seem there's a standard syntax to express paths in yaml tree, nothing comparable to xpath. Assuming we can use a comparable syntax, we could define a blacklist like this:

• services/*/deploy/placement
• services/*/volumesFrom

service.pull_policy is ignored

service.pull_policy seems ignored.

The following test (base: e358952) does not work:

diff --git a/loader/loader_test.go b/loader/loader_test.go
index 49fa36e..5448520 100644
--- a/loader/loader_test.go
+++ b/loader/loader_test.go
@@ -1730,3 +1730,16 @@ services:
 `)
        assert.ErrorContains(t, err, "invalid string value for 'count' (the only value allowed is 'all')")
 }
+
+func TestServicePullPolicy(t *testing.T) {
+       actual, err := loadYAML(`
+services:
+  hello-world:
+    image: redis:alpine
+    pull_policy: always
+`)
+       assert.NilError(t, err)
+       svc, err := actual.GetService("hello-world")
+       assert.NilError(t, err)
+       assert.Equal(t, "always", svc.PullPolicy)
+}

$ go test -run TestServicePullPolicy ./loader
--- FAIL: TestServicePullPolicy (0.01s)
    loader_test.go:1744: assertion failed: always (string) !=  (svc.PullPolicy string)
FAIL
FAIL    github.com/compose-spec/compose-go/loader       0.015s
FAIL

Export loadOptions?

When using cli.ProjectFromOptions, the functional options pattern allows you to specify behavior for every exported field of cli.ProjectOptions.

However, there is one field, loadOptions, which is not exported. This is problematic for me because I would like to use cli.ProjectFromOptions, but I want to specify that the loader should skip validation.

Would a PR where either
(1) loadOptions is exported so it can be controlled via functional options like the rest of the fields

or

(2) the loader is specified to cli.ProjectFromOptions via dependency injection

be accepted?

Otherwise, I would have to use loader.Load and repeat logic that already exists via cli.ProjectFromOptions.

Thanks!

Add support for volume bind option SELinux :z :Z

The docker-compose CLI v2 uses the docker/compose-go package for
parsing Docker Compose YAML specification files and currently it misses
support for the SELinux relabeling with :z or :Z.

It is a regression compared to docker-compose CLI v1 that written in Python.

I have created a Pull Request that allows to use the SELinux field to set the :z or :Z bind option for relabeling SELinux label.

References:
-Configure the selinux label

`UlimitsConfig.MarshalYAML` is incorrect

See https://github.com/compose-spec/compose-go/blob/master/types/types.go#L790-L796

When used with yaml.Marshal and "gopkg.in/yaml.v3", this causes an infinite loop.

To reproduce:

yaml.Marshal(&types.UlimitsConfig{Soft: 123})

Support YAML 1.2 and later

Initially reported as docker/compose-cli#1621

compose-go can't load a yaml file using Yaml 1.2 directive

%YAML 1.2
---
...

require go-yaml/yaml#298
expected to be fixed in gopkg.in/yaml.v3

Use a reference to compose-spec

We use a copy of the compose specification, would better use a reference to canonical repo docker/compose-spec

Implement "profiles"

see compose-spec/compose-spec#110

Feature request: Check for cyclic dependencies

Hi,
I have a feature request for you: At the moment the loader accepts a compose file with cyclic dependencies, even when the validation step is enabled.

With cyclic dependencies, I mean something like this:

services:
   service1:
      depends_on:
         - service2
   service2:
      depends_on:
         - service3
   service3
      depends_on:
         - service1

Serialize Go model back to file

Hi,
is it possible to serialize the Go models back to compose files? If yes, how can this be achieved?

Thanks in advance

Request: tag a release

For ease of dependency management, could you consider creating a release with git tag?

LinkLocalIPs missing from ServiceNetworkConfig struct

The compose spec supports link_local_ips for a service network config. In looking at why this field is not honored in compose v2 I discovered that compose-go is missing support for this in ServiceNetworkConfig. Related issue for compose is here: docker/compose#9659

I believe that adding support for this field is a prerequisite to being able to fix this in compose as it has a dependency on compose-go.

Check `foo` implementation do support all `x-foo-*` extension fields

With extension field support, it's easy to add extension but also hard to report errors to user. Typically, we can't catch a typo.

I'd like we introduce a mechanism so that Compose implementation can declare extension it allows, and we can catch:

x-foo-unknown: bar

Would be awesome if such a mechanism can be used to define types, so that we don't just use interface{} for extensions.

Project name constraints only applied to `-p`, `name:` config property

The project name constraints articulated in compose-spec/compose-spec#314 are only enforced when using docker compose -p or the name: config file property (once #362 lands in docker compose). Non-normalized project names from COMPOSE_PROJECT_NAME, the project dir, or the working dir are still allowed, which docker compose will then normalize.

I'm happy to implement the necessary changes to enforce the project name constraints across all mechanisms. I'm also happy to update the documentation PR to reflect the current behavior instead, if it's desirable to leave the current behavior as is for some reason.

Background

I noticed this while preparing the following documentation change after having compose-spec/compose-spec#314 and #362 merged today:

• mbland/docs@62d5fcd

I experimented locally using all the mechanisms described in that change to confirm the current behavior.

It seems the JSON schema change from #362 will catch invalid name: values in the config (per the loader/loader_test.go failure in that PR) once it lands in docker compose. However, the COMPOSE_PROJECT_NAME, project dir, and current dir mechanisms would need an update.

Maybe users are less likely to encounter issues when the project name is defined via the other mechanisms. Still, it would seem more correct and less surprising to enforce the project name constraints across all the available mechanisms.

Related issues

• docker/compose#9741
• docker/compose#9434
• docker/compose#9429
• docker/compose#8655
• docker/compose#229

ConfigFileEnv and DefaultConfigPath should be used in conjunction

in nerdctl, we encounter a non-compatibility between docker-compose and nerdctl compose (compose-spec) :

suppose we have 3 compose files under /home/wordpress-compose/ :
docker-compose.yml

services:
  hello1:
    image: alpine:3.13

docker-compose.test.yml

services:
  hello2:
    image: alpine:3.14

docker-compose.override.yml

services:
  hello1:
    image: alpine:3.14

in nerdctl :

COMPOSE_FILE=/home/wordpress-compose/docker-compose.yml:/home/wordpress-compose/docker-compose.test.yaml nerdctl compose  config --services

hello1
hello2

but with docker-compose

COMPOSE_FILE=/home/wordpress-compose/docker-compose.yml:/home/wordpress-compose/docker-compose.test.yaml docker-compose config --services

hello2
hello1

###How to Fix this ?

compose-go/cli/options.go

Line 86 in 8c0547c

func WithConfigFileEnv(o *ProjectOptions) error {

and

compose-go/cli/options.go

Line 104 in 8c0547c

func WithDefaultConfigPath(o *ProjectOptions) error {

should be used in conjunction to support COMPOSE_FILE env and docker-compose.override.yaml

CI fails on Windows nodes due to path formats

Example: https://github.com/compose-spec/compose-go/runs/417001328?check_suite_focus=true

remove UnsupportedProperties / DeprecatedProperties / ForbiddenProperties

compose-go do define types.UnsupportedProperties, types.DeprecatedProperties and types.ForbiddenProperties which are actually only used by https://github.com/docker/cli and should not be part of reference library as "supported properties" is implementation specific

Investigate incompatibility with mergo 0.3.9

see #37

Throttle device parsing problem

Hi,
Here is the problem:
For blkio_config we receive parsing error when using string for rate like in following:
device_write_bps:

• path: /dev/sdb
  rate: '1024k'

Error message states that only integer is expected, while spec states that for device_write_bps and device_read_bps both an integer or a string is allowed - see below:

device_read_bps, device_write_bps
Set a limit in bytes per second for read / write operations on a given device. Each item in the list must have two keys:

• path, defining the symbolic path to the affected device
• rate, either as an integer value representing the number of bytes or as a string expressing a byte value

Support env var resolution before substitution inside .env

Please consider the following project:

# .env
MAIN_DOMAIN=foo.bar
SUB_DOMAIN=a.${MAIN_DOMAIN}

# docker-compose.yaml
services:
  alpine:
    image: alpine
    environment:
      - SUB_DOMAIN

Here I get (as expected):

$ docker compose run --rm alpine env | grep SUB_DOMAIN
SUB_DOMAIN=a.foo.bar

But when I override MAIN_DOMAIN env I still get the same:

$ MAIN_DOMAIN=another.domain docker compose run --rm alpine env | grep SUB_DOMAIN
SUB_DOMAIN=a.foo.bar

I'd like the defined env variable to take precedence before substitution so I would get:

$ MAIN_DOMAIN=another.domain docker compose run --rm alpine env | grep SUB_DOMAIN
SUB_DOMAIN=a.another.domain

I don't know if original repository joho/godotenv supports this but it would be a nice feature for huge projects based on env vars :)

compose-go should resolve all relative paths

For all attribute which support relative paths by Compose-specification, compose-go should offer a way to resolve relative paths so that Compose implementations get a complete model. Can at same time detect missing files.

reminder:

• relative paths MUST be resolved based on (first) compose file working directory
• path starting by . MUST be resolved based on process working directory (os.Getwd) compose project working directory, i.e. compose.yaml parent folder or path explicitly set by user as project directory.
• path starting by ~/ MUST be resolved based on user's home directory (os.UserHomeDir)

Reject a Compose file with unexpected fields

As Compose specification evolves and introduce new fields/features, a Compose implementation MAY just reject a compose file using a higher level version than the one it has been built on.

On the other hand, even if a compose file uses a newer version it doesn't mean it also use recent fields, and implementation would not be able to use it.

Last but not least, as we made version: "3" and alias for "latest" there's no way to guarantee a Compose file do not target a more recent specification than the one being implemented.

So, a Compose implementation need to be able to reject a compose file which uses fields introduced by a newer version of the specification. Schema validation can be used here, but we also can make the code stricter regarding this scenario: mapstructure used internally to map yaml sections into go structs can track unmapped attributes. We can rely on this to offer informative diagnostic to the end-user.

Remove dependency on docker/cli

Should recognize inexplicit defined `default` network

My compose file is like this, which is not considered valid by this package yet.

version: "3"
services:
  test:
    build: .
    networks:
    - default

The default network here is technically valid, even though it's not defined in a networks section for the compose file , since

1. It is the default network created for the user when using docker compose.
2. Running docker-compose up on the same file doesn't show an error, proving that it is a valid compose file.

I think this is a bug and I've scrambled together a sufficient test, but due to my unfamiliarity with golang I'm not sure where/how would I implement a fix.
Is it as straightforward as just ensure there's a default network available in loader/validate.go:checkConsistency ?

Corresponding test:

func TestValidateInexplicitDefaultNetwork(t *testing.T) {
    project := &types.Project{
        Services: types.Services([]types.ServiceConfig{
            {
                Name:  "myservice",
                Image: "scratch",
                Networks: map[string]*types.ServiceNetworkConfig{
                    "default": {},
                },
            },
        }),
    }
    err := checkConsistency(project)
    assert.NilError(t, err)
}

Invalid type of Published attribute in ServicePortConfig

Description

In the ServicePortConfig struct, the Published attribute is of type string. However, such implementation led to some problems when in docker-compose. In fact, considering this docker-compose YAML file:

version: "3.9"
services:
  nginx:
    image: nginx
    ports:
      - "80:80"

The output of the command docker compose convert is the following

name: compose
services:
  nginx:
    image: nginx
    networks:
      default: null
    ports:
    - mode: ingress
      target: 80
      published: "80"
      protocol: tcp
networks:
  default:
    name: compose_default

As you can see, the published attribute is of string type, which lead to this error while executing docker stack deploy:

services.nginx.ports.0.published must be a integer

This issue is mainly related to an issue raised in docker compose repository.

Unix-like absolute file secrets are transformed as relative on Windows

Environment:

• Windows
• minikube
• github.com docker.exe and docker-compose.exe from Git master built with docker buildx bake
• DOCKER_HOST configured to use docker daemon from minikube (minikube docker-env | Invoke-Expression)
• minikube mount ./:/minikube-docker mount call from C:\Users\my_user\Testing to allow bind mounts from docker-compose.yaml as /minikube-docker/<something>

When docker-compose.yaml contains the following snippet:

secrets:
  my_password:
    file: /minikube-docker/config/MY_PASSWORD

than the file is wrongly converted as relative to C:\Users\my_user\Testing\minikube-docker\config\MY_PASSWORD and that is also what docker-compose sees and passes to docker daemon. This leads to error Error response from daemon: invalid mount config for type "bind": invalid mount path: 'C:/Users/my_user/Testing/minikube-docker/config/MY_PASSWORD'.

The same works fine on volumes, though.

I added code to docker-compose into cmd/compose/compose.go immediately after cli.ProjectFromOptions and it clearly shows that the secret's file is already absolute, so the problem is somewhere in the compose-go project. I tried to check the root cause by reading the code, but I am confused by the fact that the filepath.IsAbs should work and absPath function in loader.go checks filepath.IsAbs before making the path absolute. But because I am not a Go expert, I am curently not able to compile this on Windows and run some tests.

Working with v2 compose files

Hi! I'm investigating how I can consume v2 compose files in my application since there's still a lot of them in the wild. (Compose-go has worked great for the v3 Compose files I've tested it on so far!)

I know that the spec is only for v3, but do you all have any advice on how I should approach working with v2 files? I haven't dug too deep yet, but it looks like there's some initial error handling for extends and volumes_from already that I'd need to disable.

Environment Variables Not Replaced

When using the loader from compose-go, environment variables in the docker-compose file do not get replaced (variables defined with $). However, when using the loader from the docker cli, environment variables do get replaced.

The code in compose-go is quite similar to that in the docker cli, so this feels like a bug.

Here is a repo that reproduces this issue.

There are 2 branches - master and old-loader

go run main.go in master shows that compose-go does not replace environment variables.

go run main.go in old-loader shows that the docker-cli loader does replace the environment variables.

The code in main.go is the same in both branches. The only difference is the import in go.mod.

I would like to replace the docker cli loader in docker-lock with compose-go's loader, now that docker supports docker compose.

Does not load from file with no content bytes.

The godoc for types.ConfigFile indicates that the Content bytes will be loaded from Filename if they are not set:

type ConfigFile struct {
	// Filename is the name of the yaml configuration file
	Filename [string](https://pkg.go.dev/builtin#string)
	// Content is the raw yaml content. Will be loaded from Filename if not set
	Content [][byte](https://pkg.go.dev/builtin#byte)
	// Config if the yaml tree for this config file. Will be parsed from Content if not set
	Config map[[string](https://pkg.go.dev/builtin#string)]interface{}
}

This is not the case: https://github.com/compose-spec/compose-go/blob/v1.6.0/loader/loader.go#L169-L173

		if configDict == nil {
                         // should load here if len(file.Content) == 0
			dict, err := parseConfig(file.Content, opts)
			if err != nil {
				return nil, err
			}
			configDict = dict
			file.Config = dict
			configDetails.ConfigFiles[i] = file
		}

`transformMappingOrList` makes wrong assumption that `value` is of `string` type and panics when it is not.

I have a compose yaml where environment is defined using array syntax according to https://github.com/compose-spec/compose-spec/blob/master/spec.md#environment

services:
  emby:
    image: linuxserver/emby:4.6.7
    environment:
      - host: "1000"
        container: PGID
      - host: "1000"
        container: PUID
      - host: America/Los_Angeles
        container: TZ
    ports:
      - target: 8096
        published: "8096"
        protocol: tcp
      - target: 8920
        published: "8920"
        protocol: tcp
    volumes:
      - type: bind
        source: /DATA/AppData/emby/config
        target: /config
      - type: bind
        source: /DATA/Media/TV Shows
        target: /data/tvshows
      - type: bind
        source: /DATA/Media/Movies
        target: /data/movies
    devices: []
    network_mode: _default
    privileged: true
    cap_add: []
    restart: unless-stopped
    command: []
    cpu_shares: 90
    deploy:
      resources:
        reservations:
          memory: 3922

When loading using following code:

	project, err := loader.Load(
		types.ConfigDetails{
			ConfigFiles: []types.ConfigFile{
				{
					Content: []byte(yaml),
				},
			},
			Environment: map[string]string{},
		},
        )

it panics at

full callstack when it panics:

2023/04/28 15:49:16 http: panic serving 127.0.0.1:58254: interface conversion: interface {} is map[string]interface {}, not string
goroutine 63 [running]:
net/http.(*conn).serve.func1()
        /home/wxh/.local/go/src/net/http/server.go:1854 +0x148
panic({0x32de6e0, 0xc00096ab40})
        /home/wxh/.local/go/src/runtime/panic.go:890 +0x262
github.com/compose-spec/compose-go/loader.transformMappingOrList({0x3134ec0, 0xc000011ad0}, {0x3b5b9e0, 0x1}, 0x1)
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:1104 +0x51e
github.com/compose-spec/compose-go/loader.transformMappingOrListFunc.func1({0x3134ec0, 0xc000011ad0})
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:1093 +0x8c
github.com/compose-spec/compose-go/loader.createTransformHook.func1({0x3134ec0, 0xc000011ad0}, {0x3b9f420, 0x33c6c80}, {0x3134ec0, 0xc000011ad0})
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:447 +0xe9
github.com/mitchellh/mapstructure.DecodeHookExec({0x33071c0, 0xc000cbd960}, {0x3134ec0, 0xc000011ad0, 0x97}, {0x33c6c80, 0xc000cc66e0, 0x195})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/decode_hooks.go:47 +0x4f2
github.com/mitchellh/mapstructure.ComposeDecodeHookFunc.func1({0x3134ec0, 0xc000011ad0, 0x97}, {0x33c6c80, 0xc000cc66e0, 0x195})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/decode_hooks.go:69 +0x1c5
github.com/mitchellh/mapstructure.DecodeHookExec({0x3276620, 0xc0008949a0}, {0x3134ec0, 0xc000011ad0, 0x97}, {0x33c6c80, 0xc000cc66e0, 0x195})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/decode_hooks.go:51 +0x305
github.com/mitchellh/mapstructure.(*Decoder).decode(0xc0000132a0, {0x3147430, 0xb}, {0x3134ec0, 0xc000011ad0}, {0x33c6c80, 0xc000cc66e0, 0x195})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/mapstructure.go:459 +0x1e8
github.com/mitchellh/mapstructure.(*Decoder).decodeStructFromMap(0xc0000132a0, {0x0, 0x0}, {0x328c100, 0xc00096a030, 0x15}, {0x36477a0, 0xc000cc6500, 0x199})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/mapstructure.go:1411 +0x1cca
github.com/mitchellh/mapstructure.(*Decoder).decodeStruct(0xc0000132a0, {0x0, 0x0}, {0x328c100, 0xc00096a030}, {0x36477a0, 0xc000cc6500, 0x199})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/mapstructure.go:1235 +0x209
github.com/mitchellh/mapstructure.(*Decoder).decode(0xc0000132a0, {0x0, 0x0}, {0x328c100, 0xc00096a030}, {0x36477a0, 0xc000cc6500, 0x199})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/mapstructure.go:482 +0x994
github.com/mitchellh/mapstructure.(*Decoder).Decode(0xc0000132a0, {0x328c100, 0xc00096a030})
        /home/wxh/go/pkg/mod/github.com/mitchellh/[email protected]/mapstructure.go:417 +0xc6
github.com/compose-spec/compose-go/loader.Transform({0x328c100, 0xc00096a030}, {0x3318e00, 0xc000cc6500}, {0x0, 0x0, 0x0})
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:398 +0x34b
github.com/compose-spec/compose-go/loader.LoadService({0xc000923dc8, 0x4}, 0xc00096a030, {0x0, 0x0}, 0xc0010afa68, 0x0, 0x0)
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:616 +0x105
github.com/compose-spec/compose-go/loader.loadServiceWithExtends({0x0, 0x0}, {0xc000923dc8, 0x4}, 0xc000953fb0, {0x0, 0x0}, 0xc0010afa68, 0xc0001e2ff0, 0xc0010af2f8)
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:538 +0x279
github.com/compose-spec/compose-go/loader.LoadServices({0x0, 0x0}, 0xc000953fb0, {0x0, 0x0}, 0xc0010afa68, 0xc0001e2ff0)
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:513 +0x2c5
github.com/compose-spec/compose-go/loader.loadSections({0x0, 0x0}, 0xc000953e00, {{0x0, 0x0}, {0x0, 0x0}, {0xc0009470e0, 0x1, 0x1}, ...}, ...)
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:337 +0x43e
github.com/compose-spec/compose-go/loader.Load({{0x0, 0x0}, {0x0, 0x0}, {0xc0009470e0, 0x1, 0x1}, 0xc000947110}, {0xc000a9c530, 0x1, ...})
        /home/wxh/go/pkg/mod/github.com/compose-spec/[email protected]/loader/loader.go:203 +0x9aa```

Improve usability of Project & Service objects

Description

The Project and Service objects returned by this library often
confuse users because certain fields are not always initialized.

For example, CustomLabels (a map[string]string) will be nil
on each Service.

Potential Solutions

• Add NewProject and NewService constructors that initialize
  all fields consistently
• Add helper methods, e.g. AddCustomLabel that internally handle
  initialization lazily

Related Issues

docker/compose#9808

Is there a way to marshal a golang compose project back into yaml?

I'd like to use this project to pre-process a compose file - basically inject some labels, verify configuration, etc. - before marshalling it back to yaml for use by docker compose. Is there a way to do that with compose-go?

Example usage documentation for getting started?

It would be great to have some few lines in the README or wherever showing how to parse/load a compose file.

I am not quite sure how to get started using this library 🤔

Cannot marshal and unmarshal `types.Project`

I need to pass types.Project object via API by JSON marshal/unmarshal.

However, a field of ths struct which is supposed to be array, is always serialized into map.

To demonstrate the issue:

package main

import (
	"encoding/json"

	types "github.com/compose-spec/compose-go/types"
)

func main() {
	out := types.Project{
		Name: "foo",
		Services: []types.ServiceConfig{ // <- notice this field is an array
			{
				Name:  "bar",
				Image: "hello-world",
			},
		},
	}

	buf, err := json.Marshal(out.Services)
	if err != nil {
		panic(err)
	}

	println(string(buf)) // <- notice the Services field now a map, which is incorrect!

	var in types.Project
	if err := json.Unmarshal(buf, &in); err != nil {
		panic(err)
	}
}

The code fails unexpectly

{"name":"foo","services":{"bar":{"command":null,"entrypoint":null,"image":"hello-world"}}}
panic: json: cannot unmarshal object into Go struct field Project.services of type types.Services

goroutine 1 [running]:
main.main()
	/tmp/sandbox1081064727/prog.go:29 +0x168

in the code, the out object is serialized as

{
    "name": "foo",
    "services": {
        "bar": {
            "command": null,
            "entrypoint": null,
            "image": "hello-world"
        }
    }
}

which should really be something like

{
    "name": "foo",
    "services": [
        {
            "name": "bar",
            "command": null,
            "entrypoint": null,
            "image": "hello-world"
        }
    ]
}

UPDATE

I wanted to implement UnmarshalYAML() and UnmarshalJSON{} for the services type, then I found there are many other types having same issues - only custom marshalling, no corresponding custom unmarshalling. Implementing for all these types requires dedicated time effort which I currently don't have. Would be nice for some full-time teammeber from Docker company to work on this.

Cannot Marshal a `command` as a single string.

The command field of a compose service can be either a list of strings, or a single string.

This project represents it as

// ShellCommand is a string or list of string args
type ShellCommand []string

However, there doesn't seem to be a way to marshal it as a single string --it always comes out as a list. Does this mean that only the list form is supported?

Expose COMPOSE_PROJECT_NAME for interpolation

compose-spec/compose-spec#206 introduced a top-level project name property to the compose file spec.

#231 implemented support for this property, but I believe the implementation does not yet comply with the second sentence in the spec:

Whenever project name is defined by top-level name or by some custom mechanism, it MUST be exposed for
interpolation and environment variable resolution as COMPOSE_PROJECT_NAME

From what I can see in loader/loader.go, determining the name happens after parsing/interpolating, and indeed: With compose version 2.3.3, ${COMPOSE_PROJECT_NAME} is only replaced when the project name is set via the COMPOSE_PROJECT_NAME env variable.

If the name is set via command line parameter, the top-level name property, or left as the folder name default (all of which should work according to my interpretation of the spec text), ${COMPOSE_PROJECT_NAME} is not replaced.

Fixing this would also address the long-standing and much requested docker/compose#2294

(ping @ndeloof @ulyssessouza since you two authored the two PRs mentioned above)

Support for non-ASCII characters in environment variable names

Why not allow all characters in variable names when env_file is used? Why not let the system shell handle this if Docker allows it?
Related issue: docker/compose#8862

Docker version 20.10.17, build 100c701

$ docker run --rm -e árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép ubuntu env

output:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=18f81a2d4cf5
árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép
HOME=/root

docker-compose version 1.29.2, build 5becea4c

docker-compose.yml (environment):

services:
  cli:
    image: ubuntu
    environment:
      árvíztűrő-TÜKÖRFÚRÓGÉP: ÁRVÍZTŰRŐ-tükörfúrógép
    command: env

output:

$ docker-compose up
Creating network "ramdisk_default" with the default driver
Creating ramdisk_cli_1 ... done
Attaching to ramdisk_cli_1
cli_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
cli_1  | HOSTNAME=73fcdcfb0656
cli_1  | árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép
cli_1  | HOME=/root
ramdisk_cli_1 exited with code 0

docker-compose.yml (env_file):

services:
  cli:
    image: ubuntu
    env_file:
      - env.test.txt
    command: env

env.test.txt:

árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép

output:

$ docker-compose up
Starting ramdisk_cli_1 ... done
Attaching to ramdisk_cli_1
cli_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
cli_1  | HOSTNAME=73fcdcfb0656
cli_1  | árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép
cli_1  | HOME=/root
ramdisk_cli_1 exited with code 0

Docker Compose version v2.6.1

docker-compose.yml (environment):

services:
  cli:
    image: ubuntu
    environment:
      árvíztűrő-TÜKÖRFÚRÓGÉP: ÁRVÍZTŰRŐ-tükörfúrógép
    command: env

output:

$ docker-compose up
[+] Running 1/0
 ⠿ Container ramdisk-cli-1  Created                                                                                                                                                      0.1s
Attaching to ramdisk-cli-1
ramdisk-cli-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ramdisk-cli-1  | HOSTNAME=53828dcd3646
ramdisk-cli-1  | árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép
ramdisk-cli-1  | HOME=/root
ramdisk-cli-1 exited with code 0

docker-compose.yml (env_file):

services:
  cli:
    image: ubuntu
    env_file:
      - env.test.txt
    command: env

env.test.txt:

árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép

output:

$ docker-compose up
unexpected character "¡" in variable name near "árvíztűrő-TÜKÖRFÚRÓGÉP=ÁRVÍZTŰRŐ-tükörfúrógép\n"

characters '[' and ']' can't be used in environment variable name

I get following error if I call docker-compose V2 for my project:
unexpected character "[" in variable name near "kinesis.property[1].propertyId=1"
This property is being loaded from the .env file.
This used to work with docker-compose V1, with V2 it's not any more.
Please allow the usage of characters '[' and ']' in environment variable names which are located in .env files.

Allow interpolation in label/env var names when using dict syntax

Currently, interpolation in label names only works when using the array of strings syntax, i.e. this works:

...
labels:
  - "traefik.http.services.${COMPOSE_PROJECT_NAME}.loadbalancer.server.port=8080"

But this doesn't:

...
labels:
  traefik.http.services.${COMPOSE_PROJECT_NAME}.loadbalancer.server.port: 8080

The spec is kind of vague about whether the second variation should work. It says "Values in a Compose file can be set by variables, and interpolated at runtime". Taken literally, keys to a dictionary aren't quite values. On the other hand, the keys in labels and env variables are used-defined content and not like the "normal" static keys for compose properties either.

Either way, since it is clearly useful to allow this (for labels and env variables at least) and one variation works already, I think it would be good for consistency if the other variation worked as well.

Investigate adoption of go mod

we got issues in the past adopting go modules with docker dependencies, but with a minimal depencies list compose-go sounds a good candidate to start migration effort

Compose handles key-only .env file differently than Docker and Docker-Compose

Hi! I have been testing docker compose with environment variables and the result differs from Docker and Docker-Compose.

Test

Define a .env file with the following:

ENV_A
ENV_B=
ENV_C=env_c

And a docker-compose.yml file:

services:
  alpine:
    image: alpine
    env_file: .env

Docker

$ docker run --rm --env-file=.env alpine env
ENV_B=
ENV_C=env_c

Here ENV_A has not been set and ENV_B is set with an empty value.

Docker-Compose

$ docker-compose run --rm alpine env
ENV_B=
ENV_C=env_c

With Docker-Compose, we get the same outcome as Docker

Compose

$ docker compose run --rm alpine env
ENV_C=env_c
ENV_A=ENV_B=

Here's where the outcome is quite unexpected: ENV_A is set with the value ENV_B=.

Compose seems to have a problem with key-only items in .env file.

Missing support for `mode` parameter of `tmpfs` type volumes

TL;DR: Docker Compose (at least version 2.x) does not support the mode parameter for tmpfs volume definitions, even though it should.

---

In the Compose Specification, we find that one should be able to define a filesystem mode for the tmpfs mount. The specification itself contains this parameter at line 409.

However, the compose-go repository's corresponding file does not contain the parameter in the corresponding specification file (it only has the size parameter), see line 401. Nor does the struct support mode, only size.

All this results in that Docker compose version 2.6.0 (which I happen to be running) does not support specifying mode for a tmpfs volume definition, even though https://docs.docker.com/compose/compose-file/ states that "The Compose spec merges the legacy 2.x and 3.x versions, aggregating properties across these formats and is implemented by Compose 1.27.0+."

For example, the following service definition (snippet) errors with services.postgres.volumes.2.tmpfs Additional property mode is not allowed when running docker compose config:

services:
  postgres:
    build: ./postgres/docker
    restart: unless-stopped
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./postgres/docker/init-app-db.sh:/docker-entrypoint-initdb.d/init-app-db.sh:ro
      - type: tmpfs
        target: /run
        tmpfs:
          size: 128k
          mode: 777

(PS: Don't ask me why I "want" to set mode 777 for this tmpfs mount - it's just an evil necessity due to various other limitations.)

Allow `loader.ParseYAML` and `loader.Load` to use strict YAML unmarshalling

loader.ParseYAML (and correspondingly loader.Load) use yaml.Unmarshal, which performs non-strict YAML unmarshalling (in yaml.v2 package), while in some cases a caller would like to perform strict YAML unmarshalling (e.g. to avoid duplicate keys in a map).

Would it be possible to add a strict boolean parameter to loader.ParseYAML and add a corresponding option to loader.Load and loader.parseConfig? I'm willing to do this myself, if possible.

Add a contribution guide

Add a CONTRIBUTION.md detailing the process for contributing to this repo

Services broken that start with x-

I hit something strange today that seems to be caused by:

compose-go/loader/loader.go

Line 209 in b5ef325

func groupXFieldsIntoExtensions(dict map[string]interface{}) map[string]interface{} {

I have a simple compose file:

version: "3"
services:
  x-kiosk:
    image: busybox:latest
    command: "sleep 10m"

When I call proj.WithServices, I get 2 two services. "x-kiosk" and one named "extensions". I'm not sure I fully understand this portion of the spec. However, I think that groupXFieldsIntoExtensions needs to skip service keys while recursing through the dict.

Environment fallback not working in .env file.

Given system variable SYS_VAR=the_value when set:
and an .env file:
Some__env__var=${SYS_VAR:-fallback}

Using docker-compose v1.29.2, this results in either:
Some__env__var=the_value when SYS_VAR has a value or
Some__env__var=fallback when SYS_VAR is not set or is empty.

In docker compose v2.2.3 this results in
Some__env__var=the_value:-fallback when SYS_VAR has a value or
Some__env__var=:-fallback when SYS_VAR is not set or is empty.

To me, the behaviour in v2.2.3 is unexpected.

docker-compose with volume paths relative to respective config file paths

The issue is that all relative volume paths are resolved to the location of the first given config file instead of the config file they are specified in. It would make sense to introduce a flag or an environment variable that changes this behavior.

See this issue in docker/compose

Include reference implementation of "convergence"

Define API and reference implementation for convergence (a.k.a "reconciliation") between observed state of application on platform vs Compose application desired state.
Both observed state retrieval and reconciliation actions MUST be abstracted and not depend on docker API, so this reference implementation can be used for arbitrary platform.