Docker swarm incompatability

[concourse]

Docker swarm incompatability

[4n70w4]

THe same issue:

worker_1  | {"timestamp":"2021-01-18T13:19:06.540640000Z","level":"error","source":"baggageclaim","message":"baggageclaim.fs.run-command.failed","data":{"args":["bash","-e","-x","-c","\n\t\tif [ ! -e $IMAGE_PATH ] || [ \"$(stat --printf=\"%s\" $IMAGE_PATH)\" != \"$SIZE_IN_BYTES\" ]; then\n\t\t\ttouch $IMAGE_PATH\n\t\t\ttruncate -s ${SIZE_IN_BYTES} $IMAGE_PATH\n\t\tfi\n\n\t\tlo=\"$(losetup -j $IMAGE_PATH | cut -d':' -f1)\"\n\t\tif [ -z \"$lo\" ]; then\n\t\t\tlo=\"$(losetup -f --show $IMAGE_PATH)\"\n\t\tfi\n\n\t\tif ! file $IMAGE_PATH | grep BTRFS; then\n\t\t\tmkfs.btrfs --nodiscard $IMAGE_PATH\n\t\tfi\n\n\t\tmkdir -p $MOUNT_PATH\n\n\t\tif ! mountpoint -q $MOUNT_PATH; then\n\t\t\tmount -t btrfs -o discard $lo $MOUNT_PATH\n\t\tfi\n\t"],"command":"/bin/bash","env":["PATH=/usr/local/concourse/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","MOUNT_PATH=/worker-state/volumes","IMAGE_PATH=/worker-state/volumes.img","SIZE_IN_BYTES=258752974848"],"error":"exit status 1","session":"3.1","stderr":"+ '[' '!' -e /worker-state/volumes.img ']'\n+ touch /worker-state/volumes.img\n+ truncate -s 258752974848 /worker-state/volumes.img\n++ losetup -j /worker-state/volumes.img\n++ cut -d: -f1\n+ lo=\n+ '[' -z '' ']'\n++ losetup -f --show /worker-state/volumes.img\nlosetup: cannot find an unused loop device\n+ lo=\n","stdout":""}}
worker_1  | {"timestamp":"2021-01-18T13:19:06.540741000Z","level":"error","source":"baggageclaim","message":"baggageclaim.failed-to-set-up-driver","data":{"error":"failed to create btrfs filesystem: exit status 1"}}
worker_1  | error: failed to create btrfs filesystem: exit status 1
concourse-docker_worker_1 exited with code 1

[FallingSnow]

I'm running into this issue on #70 but I'm not using docker swarm, just docker.

[Eeems]

moby/moby#24862
Looks like this wont be solved anytime soon.

[Eeems]

I've managed to get a little further by replacing privileged: true with cap_add: [NET_ADMIN] and setting CONCOURSE_RUNTIME to containerd

I'm now stuck on the following error:

{"timestamp":"2022-01-28T19:11:04.686241153Z","level":"error","source":"baggageclaim","message":"baggageclaim.api.volume-server.create-volume-async.failed-to-create","data":{"error":"operation not permitted","handle":"cbd0b4dd-84f8-4a9d-4b01-8ac8c27a968e","privileged":true,"session":"4.1.10","strategy":{"type":"import","path":"/usr/local/concourse/resource-types/docker-image/rootfs.tgz","follow_symlinks":false}}}

Which shows up as run check: find or create container on worker 3272415d73a2: failed to create volume on web ui.

It might be because I'm trying to create a privileged docker-image container.

[balthild]

I've just brought up worker service in docker swarm successfully with sysbox-runc. But it requires me to set the default runtime of nodes that will run worker containers because docker stack does not suppport the runtime prop on docker-compose.yml:

# cat /etc/docker/daemon.json

{
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "default-runtime": "sysbox-runc"
}

When I try running the hello-world example pipeline from the doc, I get a similar error:

run check: find or create container on worker dc72cdcf8d3d: failed to create volume

However, the reason showed in logs is strange:

concourse_worker.0.j6y38t6ei8o7@swarm-2    | {"timestamp":"2022-06-24T15:24:51.529209021Z","level":"error","source":"baggageclaim","message":"baggageclaim.api.volume-server.create-volume-async.failed-to-create","data":{"error":"invalid argument","handle":"7c77c360-c3ed-46aa-62bc-dae9695f43b6","privileged":false,"session":"4.1.87","strategy":{"type":"cow","volume":"df398c77-1fcc-42d2-5987-77b450071893"}}}

It's not something about permissions, but "invalid argument".

Here's my docker-compose.yml:

version: '3.9'

services:
  web:
    image: concourse/concourse
    command: web
    ports:
      - published: 8084
        target: 8080
        mode: host
    networks:
      - concourse
    deploy:
      mode: global
      placement:
        constraints:
          - "node.role == manager"
    secrets:
      - authorized_worker_keys
      - session_signing_key
      - tsa_host_key
      - tsa_host_key.pub
    environment:
      CONCOURSE_EXTERNAL_URL: https://concourse.xxxxxxxxxxxx.com
      CONCOURSE_POSTGRES_HOST: xxxxxxxxxxxx
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: xxxxxxxxxxxx
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_ADD_LOCAL_USER: balthild:xxxxxxxxxxxx
      CONCOURSE_MAIN_TEAM_LOCAL_USER: balthild

      CONCOURSE_SESSION_SIGNING_KEY: /run/secrets/session_signing_key
      CONCOURSE_TSA_AUTHORIZED_KEYS: /run/secrets/authorized_worker_keys
      CONCOURSE_TSA_HOST_KEY: /run/secrets/tsa_host_key
      CONCOURSE_TSA_PUBLIC_KEY: /run/secrets/tsa_host_key.pub
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "10m"

  worker:
    image: concourse/concourse
    command: worker
    networks:
      - concourse
    #privileged: true
    #runtime: sysbox-runc
    depends_on: [web]
    stop_signal: SIGUSR2
    deploy:
      mode: global
      placement:
        constraints:
          - "node.role != manager"
    secrets:
      - tsa_host_key.pub
      - worker_key
      - worker_key.pub
    environment:
      CONCOURSE_TSA_PUBLIC_KEY: /run/secrets/tsa_host_key.pub
      CONCOURSE_TSA_WORKER_PRIVATE_KEY: /run/secrets/worker_key
      CONCOURSE_TSA_HOST: web:2222
      CONCOURSE_RUNTIME: containerd
      CONCOURSE_BIND_IP: 0.0.0.0
      CONCOURSE_BAGGAGECLAIM_BIND_IP: 0.0.0.0
      # avoid using loopbacks
      CONCOURSE_BAGGAGECLAIM_DRIVER: overlay
      # work with docker-compose's dns
      CONCOURSE_CONTAINERD_DNS_PROXY_ENABLE: "true"
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "10m"

secrets:
  session_signing_key:
    file: ./keys/web/session_signing_key
  authorized_worker_keys:
    file: ./keys/web/authorized_worker_keys
  tsa_host_key:
    file: ./keys/web/tsa_host_key
  tsa_host_key.pub:
    file: ./keys/web/tsa_host_key.pub
  worker_key:
    file: ./keys/worker/worker_key
  worker_key.pub:
    file: ./keys/worker/worker_key.pub

networks:
  concourse:
    driver: overlay

[balthild]

It seems that the invalid argument error is related to #42. But the workaround mentioned there (mount a volume to /worker-state) does not work for me.

Update: The real message describes the actual error is produced by kernel, and it can be viewed with journalctl -f.

Jun 24 16:28:48 swarm-2 kernel: overlayfs: idmapped layers are currently not supported

It's said that the support for idmapped layers in overlayfs will be available in Linux 5.19 (current mainline kernel is 5.18).