Building native image with spring-boot:build-image throws DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 400 "Bad Request"

[danparisi]

Building native image with spring-boot:build-image throws DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 400 "Bad Request"

[scottfrederick]

Thanks for the report and all the detail. It's hard to tell what the Docker daemon doesn't like about the call to /v1.24/containers/create. Can you get more detail from the daemon logs when the error occurs?

One thing that I'm not sure about in your configuration is the <bindings>/var/run/docker.sock:/var/run/docker.sock</bindings>. Typically that would require a valid path /var/run/docker.sock in the local environment so that it could bind the same path in the container, but since you're setting <host>tcp://localhost:2375</host> I assume you can't use /var/run/docker.sock to talk to the daemon locally. If that binding source path doesn't exist locally that might cause the daemon to throw an error.

If the tcp:// address is also available to containers with dind, you might try setting <bindHostToBuilder>true</bindHostToBuilder> in the <docker></docker> section instead. Alternatively, you can set a DOCKER_HOST environment variable, which will be used by the local Spring Boot plugin code and by the CNB lifecycle code running in the container.

[danparisi]

Hi @scottfrederick, thanks for your quick reply.
I did many other tests, I'm listing them below. Keep in mind that in all of them the DOCKER_HOST environment property is available:

- name: DOCKER_HOST
  value: "tcp://localhost:2375"

Test 1:

1. No <host> tag
2. No <bindings> tag
3. No <bindHostToBuilder> tag

Result:

[INFO]     [creator]     ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: connection refused
[...]
Caused by: org.springframework.boot.buildpack.platform.build.BuilderException: Builder lifecycle 'creator' failed with status code 1

---

Test 2:

1. No <host> tag
2. No <bindings> tag
3. Added <bindHostToBuilder>true</bindHostToBuilder> tag

Result:

ERROR: failed to initialize analyzer: getting previous image: Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?

---

Test 3:

1. No <bindings> tag
2. No <bindHostToBuilder> tag
3. Added <host>tcp://localhost:2375</host> tag

Result:

ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: connection refused

---

Test 4:

1. No <host> tag
2. No <bindHostToBuilder> tag
3. Added <bindings>/var/run/docker.sock:/var/run/docker.sock</bindings> tag

Result:

DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 400 "Bad Request"

---

I also tried to downgrade docker daemon version and increase CPU and memory limits for the containers running the build and the docker daemon. But same result. Unluckily, I cannot find any other docker logs other the ones I listed above, it just returns 400 it seems. no debug / error log at all.

Please note that for all test cases the image is correctly pulled by the docker daemon, so communication fully work in that case (DOCKER_HOST env var is enough I assume), but the creator always fails for the reasons you can see above. Not sure if it can help. Any idea?

[scottfrederick]

Unfortunately this is difficult to diagnose without access to a running environment.

Please note that for all test cases the image is correctly pulled by the docker daemon, so communication fully work in that case

This means that the Spring Boot code running locally on the host machine can talk to the Docker daemon. This is usually pretty easy to get running, as you can test local docker connectivity with the docker CLI and other tools.

but the creator always fails...

This is the hard part, as it is requires a container running in the Docker daemon to have access to the Docker host somehow (via a socket or a remote host address), and this is entirely dependent on how things like docker-in-docker are set up by a CI platform. Users have had had other issues with environments like this.

Any idea?

Logs that show what's going on when the processes running inside the container try to access the Docker host would be helpful. Otherwise, you can see if there's a way to shell into a CI container and try docker commands to see what works (for example, what sockets are available, what permissions are on the socket, if a remote host address works).

[danparisi]

In the end I decided to go for another direction in order to let the pipeline works, I'll explain it below.
For anyone who got the same issue, It's worth to mention that I didn't test the dind-rootless image in the setup you see above, maybe it would have fixed the issue, or maybe not: I don't know.

What I did in the end is to rely on the host machine docker daemon instead of starting up a new docker container in the Jenkins agent pod. To let it work, I just need to mount the docker socket:

apiVersion: v1
kind: Pod
spec:
  volumes:
    ...
    - hostPath:
        type: Socket
        path: /run/docker.sock
      name: docker-sock
    - hostPath:
        type: Directory
        path: /var/lib/docker
      name: docker-lib
containers:
  ...
  - name: maven
    image: maven:3.9.6-eclipse-temurin-21
    tty: true
    command: [ "sleep" ]
    args: [ "infinity" ]
    volumeMounts:
    ...
    - mountPath: /var/run/docker.sock
      name: docker-sock
      readOnly: false
    - mountPath: /var/lib/docker
      name: docker-lib
      readOnly: false

I also got rid of DOCKER_HOST env variable and of docker host, url and bindings tags in the spring-boot-maven-plugin configuration. Everything works fine now.

[scottfrederick]

@danparisi Thanks for letting us know that you found a way to get this working.