[BUG] v2.24.7 multiple changes in `config` output

[Re4zOon]

[BUG] v2.24.7 multiple changes in `config` output

[stasadev]

We see the same in DDEV (all our tests are failing)

• ddev/ddev#5939

extra_hosts:
    - host.docker.internal=host-gateway

became

extra_hosts:
    - host.docker.internal:host-gateway

I tracked it down to:

• #11556

[Re4zOon]

Here you go:
test.zip

What I found:

• a new default network is created, even tho nothing asked for it
• if you add an "override", the config becomes a list, instead of a dictionary
• labels are using equal sign instead of colon
• extra_hosts are using colon instead of equal sign

[luciangabor]

There's this:

a new default network is created, even tho nothing asked for it

[luciangabor]

There's this:

a new default network is created, even tho nothing asked for it

@ndeloof did you notice that?

$ docker compose -f - config <<-'EOF'
name: oh-no
services:
  service:
    image: image
    networks:
      net:
networks:
  net:
EOF
name: oh-no
networks:
  default:
    name: oh-no_default
  net:
    name: oh-no_net
services:
  service:
    image: image
    networks:
      net: null

(up doesn't create it)

[Re4zOon]

I'll try to gather and share some simple outputs from both versions illustrating the issues (our stack yaml is about 6000 lines now).

[rfay]

@Re4zOon could you please retitle this to something like "[BUG] v2.24.7 extra_hosts is output with colon instead of equals sign"

[luciangabor]

Well, today is a weekend day after all: https://github.com/dwmkerr/hacker-laws?tab=readme-ov-file#hyrums-law-the-law-of-implicit-interfaces

[Re4zOon]

Well, today is a weekend day after all: https://github.com/dwmkerr/hacker-laws?tab=readme-ov-file#hyrums-law-the-law-of-implicit-interfaces

Yes, thats true. Just like: https://xkcd.com/1172/
However breaking compatibility still shouldn't result in a "patch" version according to semver.

[luciangabor]

Ok, but what compatibility is broken? If I pipe its output through itself (with varius 2.2*.* versions that I keep around) again everything seems alright: docker compose config | docker compose -f - config -q (using the docker-compose.yml from test.zip I get a deprecation warning, but that's all) ...

[ndeloof]

This change indeed took place to follow docker engine preferred format after compose-spec/compose-go#499

services environment variables were switched to a list
this was required to allows resurrection of the --no-interpolate support.

in both case, this is not "breaking compatibility": config render a compose-spec compliant output, the fact you wrote your own parser without considering the many aspects of the spec is unfortunately an issue you have to manage on your own.

I'm closing this issue as "won't fix"

[Re4zOon]

I can understand the other point, but:

environment variables were switched to a list

Its either a list, or a dictionary and its dependent on other things.
Why do we have a spec, where it can be either with no clear setting as to which to use?
config just decides it wants to use a list for only ONE service out of 70+, even tho the compose version did not change in our config (still on 3.5).

This means I get random behaviour and my parser will have to check if its a list, or a dictionary, as I can't anticipate which one it will be. Fun.

[Re4zOon]

There's this:

a new default network is created, even tho nothing asked for it

As I said, I can understand all other points. My main two problems:

• schema changed without any notification from one patch version to the next (even tho we still use version 3.5 )
• env variable is either a list or a dictionary, based on... something.

My parser gets a 70+ service yaml with 6000+ lines. ONE of those services has a list, all other is dictionary due to a patch.
Its the inconsistency, that bothers me.

[luciangabor]

even tho we still use version 3.5

maybe I'm wrong and this is not about swarm/stack, but if that's the case, why don't you just use docker stack config ..
(I keep seeing this and I don't get it - why pipe compose's config output, which is like an "ng" variant, to tools that expect the "classic" variant ... and expect it to keep working)

[Re4zOon]

oh. docker stack config didn't cross my mind...

[luciangabor]

(there's also docker buildx bake to build images without compose)
((neither load .env nor set COMPOSE_* variables, but ...))

[ndeloof]

Why do we have a spec, where it can be either with no clear setting as to which to use?

compose file format initially evolved organically, and offered flexibility to users. The spec defines the many ways things can be set, but there's no "single way", compared to kube.yaml which just reflect the internal API structs.

config just decides it wants to use a list for only ONE service out of 70+, even tho the compose version did not change in our config (still on 3.5).

version attribute is obsolete and should be removed, it doesn't guarantee anything and is just a legacy from the time compose was implemented in python.

This means I get random behaviour and my parser will have to check if its a list, or a dictionary, as I can't anticipate which one it will be. Fun.

Indeed, this is the price to pay with the flexibility we offer in the compose file format. You should not try to parse such a file with a shell script or naive parser expecting a fixed format. If you need to transform the config output, you should better rely on compose-go library and write your own transformation as actual code

[ndeloof]

indeed, we miss removal for unused resources - but that's a distinct issue then

[Re4zOon]

I've switched to docker stack config, so from my side this issue can be closed.
We hope we can dismantle swarm in the "not-so-near" future anyways.

[luciangabor]

indeed, we miss removal for unused resources - but that's a distinct issue then

and to take into account profiles: #11587

$ docker compose -f - config <<-'EOF'
name: hidden
services:
  service:
    image: image
    profiles:
      - hidden
EOF
name: hidden
networks:
  default:
    name: hidden_default
services:
  service:
    image: image
    networks:
      default: null
    profiles:
      - hidden

# although --services works
$ docker compose -f - config --services <<-'EOF'
name: hidden
services:
  service:
    image: image
    profiles:
      - hidden
EOF

[luciangabor]

I've switched to docker stack config, so from my side this issue can be closed. We hope we can dismantle swarm in the "not-so-near" future anyways.

https://github.com/search?q=repo%3Adocker%2Fcompose+mirantis&type=issues
https://lwn.net/Articles/905164/

[rfay]

Should have been noted here that the problem behavior here was reverted in v2.25.0.

DDEV's tests can now pass with v2.25.0

• ddev/ddev#5939

[luciangabor]

Well, but why rely on that?

The relevant part of the change was to restore a case where config did "more" - to inline the content of env files (when found) ... and this change is already subject to ... change: #11627

Who knows ... maybe config will convert all lists to maps where possible ... that's kinda' more aligned with doing "more" (plain lists are mostly for "short" forms and "long" forms, where they exist, are ... kinda' preferred and that's what I meant by doing "more" - to have config "show" you what you meant writing those files).

[ndeloof]

actually, the docker engine API uses key=value to set environment variables, so the mapping syntax for those only exists in compose for legacy reasons

[luciangabor]

maps also "just work":

services:
  s1:
    image: image
    environment:
      - VAR=foo

  s2:
    extends: s1
    environment:
      - VAR=bar

But, being a weekend day after all, when it comes to extra hosts I find it funny that the long syntax is actually shorter that the short one :D