willfarrell / docker-autoheal Goto Github PK

Monitor and restart unhealthy docker containers.

License: MIT License

Shell 91.46% Dockerfile 8.54%

docker

docker-autoheal's Introduction

Docker Autoheal

Monitor and restart unhealthy docker containers. This functionality was proposed to be included with the addition of HEALTHCHECK, however didn't make the cut. This container is a stand-in till there is native support for --exit-on-unhealthy moby/moby#22719.

Supported tags and Dockerfile links

How to use

1. Docker CLI

UNIX socket passthrough

docker run -d \
    --name autoheal \
    --restart=always \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -v /var/run/docker.sock:/var/run/docker.sock \
    willfarrell/autoheal

TCP socket

docker run -d \
    --name autoheal \
    --restart=always \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -e DOCKER_SOCK=tcp://$HOST:$PORT \
    -v /path/to/certs/:/certs/:ro \
    willfarrell/autoheal

TCP with mTLS (HTTPS)

docker run -d \
    --name autoheal \
    --restart=always \
    --tlscacert=/certs/ca.pem \
    --tlscert=/certs/client-cert.pem \
    --tlskey=/certs/client-key.pem \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -e DOCKER_HOST=tcp://$HOST:2376 \
    -e DOCKER_SOCK=tcps://$HOST:2376 \
    -e DOCKER_TLS_VERIFY=1 \
    -v /path/to/certs/:/certs/:ro \
    willfarrell/autoheal

The certificates and keys need these names and resides under /certs inside the container:

ca.pem
client-cert.pem
client-key.pem

See https://docs.docker.com/engine/security/https/ for how to configure TCP with mTLS

Change Timezone

If you need the timezone to match the local machine, you can map the /etc/localtime into the container.

docker run ... -v /etc/localtime:/etc/localtime:ro

2. Use in your container image

Choose one of the three alternatives:

a) Apply the label autoheal=true to your container to have it watched;
b) Set ENV AUTOHEAL_CONTAINER_LABEL=all to watch all running containers;
c) Set ENV AUTOHEAL_CONTAINER_LABEL to existing container label that has the value true;

Note: You must apply HEALTHCHECK to your docker images first.
See https://docs.docker.com/engine/reference/builder/#healthcheck for details.

Docker Compose (example)

services:
  app:
    extends:
      file: ${PWD}/services.yml
      service: app
    labels:
      autoheal-app: true

  autoheal:
    deploy:
      replicas: 1
    environment:
      AUTOHEAL_CONTAINER_LABEL: autoheal-app
    image: willfarrell/autoheal:latest
    network_mode: none
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock

Optional Container Labels

`autoheal.stop.timeout=20`	Per containers override for stop timeout seconds during restart

Environment Defaults

Variable	Description
`AUTOHEAL_CONTAINER_LABEL=autoheal`	set to existing label name that has the value `true`
`AUTOHEAL_INTERVAL=5`	check every 5 seconds
`AUTOHEAL_START_PERIOD=0`	wait 0 seconds before first health check
`AUTOHEAL_DEFAULT_STOP_TIMEOUT=10`	Docker waits max 10 seconds (the Docker default) for a container to stop before killing during restarts (container overridable via label, see below)
`AUTOHEAL_ONLY_MONITOR_RUNNING=false`	All containers monitored by default. Set this to true to only monitor running containers. This will result in Paused contaners being ignored.
`DOCKER_SOCK=/var/run/docker.sock`	Unix socket for curl requests to Docker API
`CURL_TIMEOUT=30`	--max-time seconds for curl requests to Docker API
`WEBHOOK_URL=""`	post message to the webhook if a container was restarted (or restart failed)

Testing (building locally)

docker buildx build -t autoheal .

docker run -d \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -v /var/run/docker.sock:/var/run/docker.sock \
    autoheal

docker-autoheal's People

Contributors

$fracai avatar$

Stargazers

Watchers

Forkers

rpawel swift-nav etsangsplk krassif funkwerk dcharbonnier djerihovs govindkailas mak2099 luckydonald-backup luckydonald-forks fabriziocucci gamlerhart mtsr apm963 lucasfo brokergenius hasnat kd7lxl dottgonzo shidenggui trebolcinco biskyt ldaume eddyhub santana1053 magi1053 adaouda agpopikov cyber1000 dimidroll nyurik omritoptix naaooj aaugustinas adithya321 normanrz dsemenov87 cryi johnnymarnell asteny clover5 lbouriez opencollective grad-michal 4406arthur bigbes riotkit-org stalinkay realab adbrvn kitter redlink-gmbh mason-splunk mlenkeit applechaning jasooonko tkrisztian95 tempestnano constructor-io lmamakos hello-anmol deboj coreshift thebiggerguy d-d-up ssddn fractalriver nenyiarkaah yellowdog internetofwater pschmitt pjknkda bengreenier freekers nblazincic vovakaplenko thomdietrich 2403905 nihiloid arbal llamaqcom tjlcast c-why rc014 firstvenue swipswaps ssavva05 johncharrington likewhatevs colemamd phynias ahmedsajid silveraid hjaske ap-wtioit offr3al remogloor danieltadresu 01000001-01001110

docker-autoheal's Issues

Feature Request: Container blacklist

I would like to see a container blacklist rather than the opt-in whitelist methods already available. Where I might set the environment variables to "all" and "blacklist" a container or two that don't respond well to being reset. Especially those with dependencies or dependant containers.

Deadlocks

Running a docker compose of services, with custom VPN network providing outbound service, I am using health check to restart dependent sevices when the parent VPN service restarts. The health check is a simple curl command to fetch a google.com homepage, when the VPN has restarted this request fails.

The health check monitor, however locks up and can not be restarted at all, the processes have to be manually killed.

Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:11:14 2017
OS/Arch: linux/amd64

Server:
Engine:
Version: 17.12.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:09:47 2017
OS/Arch: linux/amd64
Experimental: false

Daily builds?

My watchtower constantly updates autoheal because of the daily builds, which is kind of bothersome since I always get notifications for that.

Now I saw that the Dockerfile uses alpine:3.13.5 hardcoded, so I don't really get why it is built daily when it is always the same EOL alpine image.

I think it would make more sense to either use alpine:latest and just build if alpine:latest is updated or just disable daily builds as they don't bring any benefits in the current state.

I wrote a workflow for keepalived that checks if the keepalived package in the alpine repo has updates https://github.com/shawly/docker-keepalived/blob/main/.github/workflows/update-keepalivd-version.yml, you could use this as a starting point for building a workflow that only builds when packages or the main image get updated.

Update to alpine 3.13

Hello,

Could you please update your image to alpine 3.13 ?

Thank you

Dependent containers restart

Is it possible to restart containers that are dependent on the container that failed the health check?
Say I have containers A and B that have C as a dependency. In simple terms A and B need a healthy C to function properly.
Is it possible to restart A, B if C's health check is negative?

If not please consider it as a potential enhancement.

Use DOCKER_HOST instead of mounting /var/run/docker.sock

Hi,

thanks for that great tool.

I think it would be better to communicate with the docker daemon via the DOCKER_HOST variable instead of /var/run/docker.sock - you then do not have to mount a volume and health checking on other hosts would also be possible.

Autoheal statuses

When running docker ps what does mean status "Restarting (7) 3 weeks ago"?
Is autoheal service itself restaring, or did it restart something else 3 weeks ago?

Could these status texts be more descriptive?

Referencing environment variables from .env file make system crash

This is my docker-compose file:

services:
  autoheal:
  # https://github.com/willfarrell/docker-autoheal
  # monitor and restart unhealthy docker containers
  # - no health check available (if it fails, it just restarts by default)  
    image: willfarrell/autoheal
    container_name: autoheal
    environment: # referecing environment tags from .env file give errors and make the system stop
      - AUTOHEAL_CONTAINER_LABEL=${AH_CONTAINER_LABEL}
      - AUTOHEAL_INTERVAL=${AH_INTERVAL}
      - AUTOHEAL_START_PERIOD=${AH_START_PERIOD}
      - AUTOHEAL_DEFAULT_STOP_TIMEOUT=${AH_DEFAULT_STOP_TIMEOUT}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
    labels: 
      - autoheal=true
    restart: always

and these are the environment variables within the .env file:

AH_RESTART=true
AH_CONTAINER_LABEL=all
AH_INTERVAL=5
AH_START_PERIOD=15
AH_DEFAULT_STOP_TIMEOUT=30

When I run the image as it is, i get these errors:

sleep: invalid number '5   # check every 5 seconds'
Container /autoheal (f2ecb832259a) found to be unhealthy - Restarting container now with 10s timeout

and

jq: 1 compile error
sh: 15   # wait 15 seconds before first health check: bad number
jq: error: syntax error, unexpected $end (Unix shell quoting issues?) at <top-level>, line 1:
foreach .[] as $CONTAINER([];[]; $CONTAINER | .Id, .Names[0], .Labels["autoheal.stop.timeout"] // 30   # Docker waits max 10 seconds (the Docker default) for a container to stop before killing during restarts) 
Container stopped

But when I run the env vars plainly like this:

services:
  autoheal:
  # https://github.com/willfarrell/docker-autoheal
  # monitor and restart unhealthy docker containers
  # - no health check available (if it fails, it just restarts by default)  
    image: willfarrell/autoheal
    container_name: autoheal
    environment: # referecing environment tags from .env file give errors and make the system stop
      - AUTOHEAL_CONTAINER_LABEL=all
      - AUTOHEAL_INTERVAL=5
      - AUTOHEAL_START_PERIOD=15
      - AUTOHEAL_DEFAULT_STOP_TIMEOUT=30
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
    labels: 
      - autoheal=true
    restart: always

Then the container runs normally... This only happens with autoheal, as I have all other containers pulling env vars from the .env file without issues.

Also noticed that the issue just happens with env vars, as labels, volumes and any other block do take the referenced tag just fine.

Expected behavior:
To take the value from the referenced variable and run without errors or stopping.

Thanks for the hard work!

Autoheal container itself unhealthy

What I can do in this case? I was completely stuck.
I deployed it with other containers using docker-compose. Rest of the containers use host network while autoheal uses bridge network.

Logs of autoheal container

$ sudo docker logs d778c7a8ca96
Monitoring containers for unhealthy status in  second(s)
18-01-2019 02:36:10 Container /pritunl-alpine (3f096d7cb6d8) found to be unhealthy - Restarting container now
18-01-2019 02:36:10 Restarting container 3f096d7cb6d8 failed
18-01-2019 02:36:45 Container /xxx-alpine (3f096d7cb6d8) found to be unhealthy - Restarting container now
18-01-2019 02:37:10 Container /xxx-alpine (3f096d7cb6d8) found to be unhealthy - Restarting container now
18-01-2019 02:37:36 Container /xxx-alpine (3f096d7cb6d8) found to be unhealthy - Restarting container now
18-01-2019 02:38:01 Container /xxx-alpine (3f096d7cb6d8) found to be unhealthy - Restarting container now
.......

When I did sudo docker-compose down to shutdown all the running containers and start again I got following error

ERROR: for deploy_autoheal_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=70)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.

I was unable to kill the container or stop it. It was stuck there with unhealthy tag.

Inspect command:

sudo docker inspect -f '{{json .State}}' d778c7a8ca96
{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":8235,"ExitCode":0,"Error":"","StartedAt":"2019-01-17T04:37:13.934233217Z","FinishedAt":"0001-01-01T00:00:00Z","Health":{"Status":"unhealthy","FailingStreak":358,"Log":[{"Start":"2019-01-17T21:02:44.852668621-06:00","End":"2019-01-17T21:02:44.877638-06:00","ExitCode":-1,"Output":"containerd: container not found"},{"Start":"2019-01-17T21:02:49.888834605-06:00","End":"2019-01-17T21:02:49.903345313-06:00","ExitCode":-1,"Output":"containerd: container not found"},{"Start":"2019-01-17T21:02:54.915125297-06:00","End":"2019-01-17T21:02:54.927540155-06:00","ExitCode":-1,"Output":"containerd: container not found"},{"Start":"2019-01-17T21:02:59.938747565-06:00","End":"2019-01-17T21:02:59.952247723-06:00","ExitCode":-1,"Output":"containerd: container not found"},{"Start":"2019-01-17T21:03:05.371827569-06:00","End":"2019-01-17T21:03:05.389847054-06:00","ExitCode":-1,"Output":"containerd: container not found"}]}}

At the end I restarted my docker service and it was working fine again.

ENHANCEMENT - Notifications

If there can be a pushover (and any other notifier platform) included in case of an unhealthy docker

When using docker-compose container is not found when autoheal label set to true

When I add the autoheal=true label to a container the value is set to the string "True" with a capital 'T' instead of true. This causes the container not to be found when scanning for unhealthy containers.

docker-compose.yml:

    labels:
      autoheal: "true"

also tried

   labels:
      - "autoheal=true"

with the same result

docker inspect:

ARM support (multi-arch manifest) not working?

I see code and a couple closed issues (#19, #14) about ARM support, but pulling willfarrell/autoheal on an ARM device like a Raspberry Pi results in the x86_64 image being downloaded:

Inspecting the manifest doesn't seem to include manifest platform info as expected (compared to MongoDB):

Not sure if I'm overlooking something, but the the muti-arch portion of the build seems to be broken?

Thanks for reading...

EDIT: I found another tool for checking the manifest:

Append "Last Output" from Healthcheck to WEBHOOK_URL message

Would it be possible to add the last output from the Healthcheck to the existing message

eg "Container XXXXX (949155d6fe15) found to be unhealthy. Successfully restarted the container! The last output was ABCDEF"

Why is there a daily job for building the docker image?

I noticed that https://containrrr.dev/watchtower/ updates autoheal every day.

The github CI workflow contains a cron schedule to trigger a docker build every day, addtionally to every push to master and every tag.

This definitely seems like overkill. Why do you need to build daily if the container content doesn't change every day? Wouldn't the build on every push to master be enough?

Issue with restarting VPN container

Hello,

I have autoheal set to restart my VPN container when it goes down. It is also configured to restart the containers linked to it when they go down.

The issue is when the VPN goes down, it gets restart, that is fine and expected. It does not however restart the containers depending to it. This is an issue because the web UI ports of those containers are mapped to the VPN container so I can access them like normal.

So for those web UI ports to be available back those containers need to be started AFTER the VPN container starts. With this configuration, only the VPN container is restarted since the other ones are still healthy anyway.

Is there a solution for that?

Thanks!

how to restart a container by removing the orphan container

How can I restart a container by removing the orphan container by autoheal?

Do you still own the docker hub account?

I've noticed the docker image :latest label is updated very frequently. Since the image is manually pushed (instead of using autobuild https://docs.docker.com/docker-hub/builds/link-source/) I have no idea whether it's actually from you.

Can you set up auto build so that we know the image is indeed built from this repo? I see the last auto build was from 2 years ago: https://hub.docker.com/r/willfarrell/autoheal/builds

autoheal constantly restarts on linux/arm/v7

Tried deploying a container with this image on Armbian (an OrangePi PC board), but it constantly restarts.

First, tried running it with the following DockerCompose (I previously deployed the exact same Compose on a linux/amd64 host, and there works fine):

version: '3'

services:
  autoheal:
    # https://github.com/willfarrell/docker-autoheal
    container_name: autoheah
    image: willfarrell/autoheal:latest
    network_mode: none
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
      - AUTOHEAL_INTERVAL=10
      - AUTOHEAL_START_PERIOD=60
      - AUTOHEAL_DEFAULT_STOP_TIMEOUT=25
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    restart: always

The container is constantly restarting, producing the following output (restarts about once each ~+60 seconds because I set the AUTOHEAL_START_PERIOD to 60):

2021-08-28T11:13:04.616302674Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:14:07.059787538Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:15:09.648184993Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:16:11.946453731Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:17:14.241354914Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:18:16.564788868Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:19:18.855314561Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:20:21.163773499Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:21:23.453673455Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:22:25.853702825Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:23:28.171957971Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:24:30.817528380Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:25:33.123744990Z Monitoring containers for unhealthy status in 60 second(s)
2021-08-28T11:26:35.328544128Z Monitoring containers for unhealthy status in 60 second(s)

Running the proposed docker run command:

docker run -d \
    --name autoheal \
    --restart=always \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -v /var/run/docker.sock:/var/run/docker.sock \
    willfarrell/autoheal

makes the container to be constantly restarting, and no output is given. If I set the restart policy to none, the container exists with code 28 in both cases.

Host system info:

 OS: Debian 10 buster
 Kernel: armv7l Linux 5.10.43-sunxi
 Uptime: 4d 21h 58m
 Packages: 479
 Shell: 17285
 CPU: ARMv7 rev 5 (v7l) @ 4x 1.368GHz [53.0°C]
 GPU: 
 RAM: 362MiB / 999MiB

/docker-entrypoint: exec: line 101: autoheal: not found

I'm using docker-autoheal in a docker-compose file but with a socket-proxy service (fluencelabs/docker-socket-proxy)

autoheal:
  image: willfarrell/autoheal
  container_name: autoheal
  restart: unless-stopped
  networks:
    - t2_proxy
    - socket_proxy
  depends_on:
    - socket-proxy
  environment:
    - AUTOHEAL_CONTAINER_LABEL=all

Because of this I don't use the volume /var/run/docker.sock but a network socket_proxy

When I run docket-autoheal I see in the logs file only this line repeating

/docker-entrypoint: exec: line 101: autoheal: not found.

Does this affect at how the container is working?

Could not do a head request for "willfarrell/autoheal:latest", ...

Got this notification after watchtower tried updating my container....anyone else seeing this? Any way to resolve?

"Could not do a head request for "willfarrell/autoheal:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present" "

Container exits instead of restarting

Hi there,

I could get some help please because my container exits instead of the restarting behavior I would expect.

The use case is:

A service named gateway has access to some distant host 10.0.0.42 by setting up an authenticated VPN connection
A service named proxy makes use of gateway's network stack to get access to this distant host

gateway would restart from time to time because the VPN connection can break. If gateway restarts, I also have to restart proxy. If the VPN server does not respond, gateway would keep restarting until the connection is established again.

Here is a representative configuration one can try easily:

version: '3.5'
services:

  proxy:
    image: marcnuri/port-forward
    restart: unless-stopped
    environment:
      - REMOTE_HOST=10.0.0.42
      - REMOTE_PORT=8080
    labels:
      autoheal: "true"
    healthcheck:
      test: ["CMD", "ping", "-c", "1", "10.0.0.42"]
      interval: 20s
      timeout: 15s
      retries: 1
      start_period: 10s
    depends_on:
      - gateway
    network_mode: service:gateway

  gateway:
    image: alpine
    restart: unless-stopped
    command: sleep 10
    labels:
      autoheal: "true"
    healthcheck:
      test: ["CMD", "ping", "-c", "1", "10.0.0.42"]
      interval: 20s
      timeout: 15s
      retries: 1
      start_period: 10s

  autoheal:
    image: willfarrell/autoheal:1.1.0
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

In this sample configuration, gateway always restarts, which is the required behavior simulation. I would expect for proxy to also restart, but instead it exits. Am I missing something?

Thanks guys

Question: Restart if unhealthy for more than x mins?

Is it possible to configure autoheal to restart only if container is unhealthy for more than x mins? I have a container which has a db optimize routine which can sometimes make the container unresponsive and trigger an "unhealthy" state for a few minutes. As soon as it's done, the container goes healthy. Is there a way to tell autoheal "only restart if unhealthy for greater than x mins?" Thanks in advance.
Ed

DOCKER_SOCK not changeable

Heya.
Currently the socket is hardcoded to be /tmp/docker.sock.

I'd like to change it to the default /var/run/docker.sock.

So I propose replacing the line with

DOCKER_SOCK=${DOCKER_SOCK:-/var/run/docker.sock}

That way it is settable via environment variables,
and the original /tmp/docker.sock is still possible.

autoheal not working :(

I am using docker-compose:

version: '3.3'
services:
  autoheal:
    container_name: autoheal
    restart: always
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
      - '/etc/localtime:/etc/localtime:ro'
    image: willfarrell/autoheal

Creating container:

[root@docker-fedora-nuc autoheal]# docker-compose up
Recreating autoheal ... done
Attaching to autoheal

That is all. No log no nothing. What am I missing?

autoheal stop restarting unhealthy container after some time

Hello, thanks for this tool,

I might be facing a bug, however.

Basically I have a container that has to be restarted every two hours, more or less (I won't explain why). The need for restart is detected through a docker health check.

Docker-autoheal does the job successfully until a few days of uptime, then I have to restart both containers.

This is what is found in the logs :

18-03-2020 16:33:14 Restarting container 3fc39d98c331 failed
18-03-2020 18:35:21 Container /******** (3fc39d98c331) found to be unhealthy - Restarting container now with 10s timeout
18-03-2020 20:36:50 Container //******** (3fc39d98c331) found to be unhealthy - Restarting container now with 10s timeout
18-03-2020 22:38:20 Container /******** (3fc39d98c331) found to be unhealthy - Restarting container now with 10s timeout
sh: you need to specify whom to kill
Monitoring containers for unhealthy status in 0 second(s)
sh: you need to specify whom to kill
Monitoring containers for unhealthy status in 0 second(s)

Personally I got other options for my use-case, but I thought you might like to know that docker-autoheal is maybe unstable when having to restart containers on an hourly basis.

Best regards,

Check container status before restart_container

docker inspect [ID] |jq '[.[].State] | [.[].Status,.[].Health.Status]'
[
"restarting",
"unhealthy"
]

Container health status can be unhealthy while status is all ready restarting.

Check container status before issuing restart and skip for containers all ready in restarting state?

Timezone for logs

Is there any way to set the timezone for the logs?
It looks like it is defaulted to UTC

Respect manual stopped containers

First of all: thanks for this great little helper.

I love the fact that autoheal restarts unhealthy containers and died containers which where previously unhealthy if it has the propper label. And i love the fact that it do not touch other stopped containers.

In many situations a restart is a good thing to heal a service, but sometimes it will end-up in a crashloop forever. In that situations a admin may want to stop (temp.) the container and keep the container for further investigation. In that case you do not want to let autoheal to start the container again. Autoheal should respect manual stopped containers. A admin may want to stop the crashloop to prevent flooding the central log server with all the restarts logs. There are valid reasons to keep the container (but stopped), e.g. to investigate via docker logs / docker container diff / docker inspect . The investigation will be maybe postponed to a later point in time.

If i catch the crashlooping container in "health: starting" or "healthy" phase then it will stay stopped. But if i am a little late in a crashloop (unhealthy) then the container will be restarted, due to the unhealthy status, even if i stopped it manually via docker container stop. Another admin is maybe not aware of autoheal and may wonder that the container comes back. Least surprise principle.

I am not sure if it is possible for autoheal to distinguish between a died container and a manual stopped container - if not, then maybe a crashloop counter, crashloop threshold and a crashloop threshold period (stored in autoheal) could help to detect crashloops and then stop restarting. Alternative: Track the last 5 periods between restarts and if they are equal +/- 5 seconds stop restarting. All these parameters should be configurable as ENV for autoheal container (general setting) and as ENV for labeled / tracked containers.

Only restart containers with ENV autoheal=true not working

I start my autoheal container with this parameters:
sudo docker run -d --name autoheal --restart always -v /var/run/docker.sock:/var/run/docker.sock docker-registry/autoheal:latest

and my test container:
sudo docker run -d --name autoheal-tst -e autoheal=true docker-registry/autoheal-tst:latest
this will start health however is unhealthy after 2 minutes

when i start the autoheal with parameter: -e AUTOHEAL_CONTAINER_LABEL=all
autoheal is working fine, and i have logging in docker, however its restarting all unhealthy containers, without this parameter, i dont have docker logging and nothing is restarting

Support for windows containers

Been looking into something like this, for running on AWS ECS with Windows containers

But unsure if this socket thing is possible with Windows?
Anyone have any experience with something similar?

Is there a way to test notifications?

I have the notification webhook doing to a discord webhook, but it doesn't seem to be working. Is there. Way to test webhook notifications on demand? Thanks in advance.

unfortunately not working on ARM architechture / RaspberryPi's

Any chance we can get a build that works on a raspberry pi (3?)

DOCKER_SOCK not working as expected

when I set DOCKER_SOCK on container start, I get the following error from the logs:

/docker-entrypoint: exec: line 128: autoheal: not found
/docker-entrypoint: exec: line 128: autoheal: not found
/docker-entrypoint: exec: line 128: autoheal: not found
/docker-entrypoint: exec: line 128: autoheal: not found
/docker-entrypoint: exec: line 128: autoheal: not found
/docker-entrypoint: exec: line 128: autoheal: not found

Thats my container run:
docker run -d --name autoheal --restart=always -e AUTOHEAL_CONTAINER_LABEL=all -e DOCKER_SOCK=tcp://XXX:2376 -v /home/XXX/test/:/certs/:ro willfarrell/autoheal

Add sendmail feature

Hi,

First of all: thanks for this great little helper.

I forked the repo for personal use a while ago and added a little extra feature to get email notifications about when autoheal restart containers for me. I'm not using it anymore but I would be glad to contribute it back and create a PR containing the changes.

What do you think, would it make sense to add configurable sendmail or slack message notifications?

Thanks!

The docker-autoheal container always becomes unhealthy after some time on my system (on EC2)

I have a very standard EC2 instance on which I run docker-autoheal to watch over about 8 other containers. It works absolutely perfectly and always restarts these containers when they become unhealthy, so super thanks for this awesome awesome tool! However, it always becomes unhealthy after running for some time (few weeks), and then it stops restarting the other containers. Any ideas why this might happen? This is the state from docker inspect

"State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4600,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-06-02T05:18:01.345930625Z",
            "FinishedAt": "2022-06-02T05:17:59.341327446Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 97892,
                "Log": [
                    {
                        "Start": "2022-06-08T14:07:59.829228335Z",
                        "End": "2022-06-08T14:07:59.832537865Z",
                        "ExitCode": -1,
                        "Output": "cannot exec in a stopped state: unknown"
                    },
                    {
                        "Start": "2022-06-08T14:08:04.835753734Z",
                        "End": "2022-06-08T14:08:04.838610024Z",
                        "ExitCode": -1,
                        "Output": "cannot exec in a stopped state: unknown"
                    },
                    {
                        "Start": "2022-06-08T14:08:09.841808883Z",
                        "End": "2022-06-08T14:08:09.844731715Z",
                        "ExitCode": -1,
                        "Output": "cannot exec in a stopped state: unknown"
                    },
                    {
                        "Start": "2022-06-08T14:08:14.847771049Z",
                        "End": "2022-06-08T14:08:14.850751022Z",
                        "ExitCode": -1,
                        "Output": "cannot exec in a stopped state: unknown"
                    },
                    {
                        "Start": "2022-06-08T14:08:19.854019851Z",
                        "End": "2022-06-08T14:08:19.86835772Z",
                        "ExitCode": -1,
                        "Output": "cannot exec in a stopped state: unknown"
                    }
                ]
            }
        },

Unrecognized input header error in 1.1.0

Updated to latest v1.1.0 and container exits immediately with "Unrecognized input header: 99" error. rolling back to 1.0.0 works fine.

Server Version: 18.09.8

Critical CVE detected on latest tag

Hi !

When scanning this image with trivy (vulnerability scanner), it found 4 CRITICAL CVE.
Here is the complete result for the latest tag :

$ trivy image --scanners vuln willfarrell/autoheal:latest

willfarrell/autoheal:latest (alpine 3.13.5)
===========================================
Total: 37 (UNKNOWN: 0, LOW: 0, MEDIUM: 6, HIGH: 27, CRITICAL: 4)

┌──────────────┬────────────────┬──────────┬───────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│   Library    │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                            Title                            │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ apk-tools    │ CVE-2021-36159 │ CRITICAL │ 2.12.5-r0         │ 2.12.6-r0     │ libfetch before 2021-07-26, as used in apk-tools, xbps, and │
│              │                │          │                   │               │ other products, mishandles...                               │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-36159                  │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ busybox      │ CVE-2021-42378 │ HIGH     │ 1.32.1-r6         │ 1.32.1-r7     │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42378                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42379 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42379                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42380 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42380                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42381 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42381                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42382 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42382                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42383 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42383                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42384 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42384                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42385 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42385                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42386 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42386                  │
│              ├────────────────┤          │                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-28391 │          │                   │ 1.32.1-r8     │ busybox: remote attackers may execute arbitrary code if     │
│              │                │          │                   │               │ netstat is used                                             │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-28391                  │
│              ├────────────────┤          │                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-30065 │          │                   │ 1.32.1-r9     │ busybox: A use-after-free in Busybox's awk applet leads to  │
│              │                │          │                   │               │ denial of service...                                        │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-30065                  │
│              ├────────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42374 │ MEDIUM   │                   │ 1.32.1-r7     │ busybox: out-of-bounds read in unlzma applet leads to       │
│              │                │          │                   │               │ information leak and denial...                              │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42374                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42375 │          │                   │               │ busybox: incorrect handling of a special element in ash     │
│              │                │          │                   │               │ applet leads to...                                          │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42375                  │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ libcrypto1.1 │ CVE-2021-3711  │ CRITICAL │ 1.1.1k-r0         │ 1.1.1l-r0     │ openssl: SM2 Decryption Buffer Overflow                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-3711                   │
│              ├────────────────┼──────────┤                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-3712  │ HIGH     │                   │               │ openssl: Read buffer overruns processing ASN.1 strings      │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-3712                   │
│              ├────────────────┤          │                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-0778  │          │                   │ 1.1.1n-r0     │ openssl: Infinite loop in BN_mod_sqrt() reachable when      │
│              │                │          │                   │               │ parsing certificates                                        │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-0778                   │
│              ├────────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-2097  │ MEDIUM   │                   │ 1.1.1q-r0     │ openssl: AES OCB fails to encrypt some bytes                │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-2097                   │
├──────────────┼────────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│ libssl1.1    │ CVE-2021-3711  │ CRITICAL │                   │ 1.1.1l-r0     │ openssl: SM2 Decryption Buffer Overflow                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-3711                   │
│              ├────────────────┼──────────┤                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-3712  │ HIGH     │                   │               │ openssl: Read buffer overruns processing ASN.1 strings      │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-3712                   │
│              ├────────────────┤          │                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-0778  │          │                   │ 1.1.1n-r0     │ openssl: Infinite loop in BN_mod_sqrt() reachable when      │
│              │                │          │                   │               │ parsing certificates                                        │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-0778                   │
│              ├────────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-2097  │ MEDIUM   │                   │ 1.1.1q-r0     │ openssl: AES OCB fails to encrypt some bytes                │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-2097                   │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ ssl_client   │ CVE-2021-42378 │ HIGH     │ 1.32.1-r6         │ 1.32.1-r7     │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42378                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42379 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42379                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42380 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42380                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42381 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42381                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42382 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42382                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42383 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42383                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42384 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42384                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42385 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42385                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42386 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42386                  │
│              ├────────────────┤          │                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-28391 │          │                   │ 1.32.1-r8     │ busybox: remote attackers may execute arbitrary code if     │
│              │                │          │                   │               │ netstat is used                                             │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-28391                  │
│              ├────────────────┤          │                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2022-30065 │          │                   │ 1.32.1-r9     │ busybox: A use-after-free in Busybox's awk applet leads to  │
│              │                │          │                   │               │ denial of service...                                        │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-30065                  │
│              ├────────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42374 │ MEDIUM   │                   │ 1.32.1-r7     │ busybox: out-of-bounds read in unlzma applet leads to       │
│              │                │          │                   │               │ information leak and denial...                              │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42374                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42375 │          │                   │               │ busybox: incorrect handling of a special element in ash     │
│              │                │          │                   │               │ applet leads to...                                          │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42375                  │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ zlib         │ CVE-2022-37434 │ CRITICAL │ 1.2.11-r3         │ 1.2.12-r2     │ zlib: heap-based buffer over-read and overflow in inflate() │
│              │                │          │                   │               │ in inflate.c via a...                                       │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-37434                  │
│              ├────────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│              │ CVE-2018-25032 │ HIGH     │                   │ 1.2.12-r0     │ zlib: A flaw found in zlib when compressing (not            │
│              │                │          │                   │               │ decompressing) certain inputs...                            │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2018-25032                  │
└──────────────┴────────────────┴──────────┴───────────────────┴───────────────┴─────────────────────────────────────────────────────────────┘

Could this be addressed somehow ?

Thanks a lot !

Add Pushbullet support

Add Pushbullet support.

Assignees:
Labels: feature

jq: error (at <stdin>:1): Cannot index string with string "Id" `

Anyone else see this?

root@yetibot:~# docker run -d \
>     --name autoheal \
>     --restart=always \
>     -e AUTOHEAL_CONTAINER_LABEL=all \
>     -v /var/run/docker.sock:/var/run/docker.sock \
>     willfarrell/autoheal
a0208d9e3768b019fc88945d96b2ab286132e4272a4df3212ba978b8f1677ec1
root@yetibot:~# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                            POR
TS                                            NAMES
a0208d9e3768        willfarrell/autoheal     "/docker-entrypoint a"   3 seconds ago       Up 1 seconds (health: starting)
                                              autoheal
f7ffc783f1a4        yetibot/yetibot:0.5.43   "lein with-profile +l"   22 hours ago        Up 14 minutes (healthy)           0.0
.0.0:65432->65432/tcp, 0.0.0.0:80->3003/tcp   yetibot
9d0a8c8f5131        postgres:latest          "docker-entrypoint.sh"   13 months ago       Up 13 months                      0.0
.0.0:5432->5432/tcp                           postgres
root@yetibot:~# docker logs -f autoheal
Monitoring containers for unhealthy status in 0 second(s)
jq: error (at <stdin>:1): Cannot index string with string "Id"

docker-compose throwing error "No command specified"

version: '3'

services:
  myservice:
    privileged: true
    image: myimage:latest
    container_name: mysensor
    volumes:
      - "/dev/bus/usb:/dev/bus/usb"
      - "/home/pi/healthcheck:/root/healthcheck"
    networks:
      myvlan:
        ipv4_address: 2.0.0.16
    tty: true
    labels:
       - "autoheal=true"
    healthcheck:
       test: ["CMD-SHELL", "/root/healthcheck/check_ERROR.sh" ]
       interval: 5s
       retries: 1
       start_period: 5s
       timeout: 19s
  autoheal:
    image: autoheal:latest
    tty: true
    container_name: autoheal
    network_mode: none
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
      # check every 5 seconds
      - AUTOHEAL_INTERVAL=5
      # wait 0 seconds before first health check
      - AUTOHEAL_START_PERIOD=0
      # Docker waits max 10 seconds (the Docker default) for a container to stop before killing during restarts
      - AUTOHEAL_DEFAULT_STOP_TIMEOUT=10
      # Unix socket for curl requests to Docker API
      - DOCKER_SOCK=/var/run/docker.sock
      # --max-time seconds for curl requests to Docker API
      - CURL_TIMEOUT=30
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always

Gives me this error when i try to start the containers:


pi@raspberrypi:~$ docker-compose -f docker-compose-autoheal-test.yml up -d
myservice is up-to-date
Creating autoheal ... error

ERROR: for autoheal  Cannot create container for service autoheal: No command specified

ERROR: for autoheal  Cannot create container for service autoheal: No command specified
ERROR: Encountered errors while bringing up the project.

What command is it expecting autoheal service to start with?

container stopper

If the restart does not work the script should try until it works, example :

Monitoring containers for unhealthy status
14-10-2017 09:25:45 Container f9c87ee22e2d found to be unhealthy. Restarting container ...
09-11-2017 07:09:02 Container ba92a3723643 found to be unhealthy. Restarting container ...
09-11-2017 07:09:28 Container ba92a3723643 found to be unhealthy. Restarting container ...
09-11-2017 07:09:53 Container ba92a3723643 found to be unhealthy. Restarting container ...
{"message":"Cannot restart container ba92a372364331477175d9a9c24b9e168e9f30b94a688f5c0096ef7f2d641b62: failed to initialize logging driver: gelf: cannot connect to GELF endpoint: XXXXXXXXXX dial udp: lookup XXXXXXX on 127.0.0.1:53: read udp 127.0.0.1:52979->127.0.0.1:53: i/o timeout"}

License

Hi
I really like the project! :-)
However, I cannot find a license file. In order for me to reuse your project, it must have a license file.
Best suitable for me would be an MIT, BSD or Apache 2.0 license. ;-)
What do you think? Is there any plan to add a license?
Cheers
Michael

autoheal container constantly restarts. "autoheal exited with code 7"

When trying to add autoheal to my container stack, it seems to fail and restart over and over. "autoheal exited with code 7" is emitted to the log each time. Google hasn't been particularly helpful; in particular, there are no results at all for this error message, which doesn't bode well for me.

This is how I have autoheal set up in my compose file:

  autoheal:
    image: willfarrell/autoheal:latest
    tty: true
    container_name: autoheal
    restart: always
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I've tried it on several stacks with the same result. The system is a Pi4 4GB on Bullseye. Docker version 20.10.19, build d85ef84.

I had it running without issue on an identical stack on this Pi once before, but after reimaging and starting over the issue began. It has never run successfully in the system's current incarnation.

Any ideas?

Paused containers are restarted

Autoheal seems to restart paused containers.
We use docker compose pause $service to pause certain containers for e.g. backing up their volumes w/o risking concurrent access to it.
It would be great if autoheal would leave paused containers alone instead of restarting them.

improve documentation

I can understand how to "autoheal" all containers (option b), but I'm unclear how to auto heal a specific container or a specific list of containers (option a or c).

Can you please provide examples.

Would this be correct for docker-compose to monitor the jackett container, but not the radarr container.

version: '3.4'
services:
  autoheal:
    restart: always
    image: willfarrell/autoheal
    environment:
      - AUTOHEAL_CONTAINER_LABEL=jackett
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  jackett:
    image: linuxserver/jackett
    container_name: jackett
    restart: unless-stopped
    environment:
      - TZ=${TZ}
    ports:
      - 9117:9117
    healthcheck:
      test: curl --fail -s http://localhost:9117/UI/Dashboard || exit 1
      interval: 90s
      timeout: 3s
      retries: 3
    labels:
      - autoheal=true
  radarr:
    image: linuxserver/radarr
    container_name: radarr
    restart: unless-stopped
    environment:
      - TZ=${TZ}
    ports:
      - 8787:8787

Does it work with `--restart=unless-stopped`?

Or it always needs to be always?

Release tag for webhook support

This is more of a question, when could we expect a tagged release (that is not Latest) ex. 1.3.0 with the webhook support?

Really excited to see this feature added to allow us to monitor when the autoheal kicks in.

Thanks so much!

Integration with docker-compose

Should I only run one instance per node or should I run one instance per network? Im trying to understand whether I should add it to all my docker-compose.yml or not

Container stuck in "mkdir $TMP_DIR"

Can't really say why, but today I had an issue with a docker-autoheal container that got stuck on an infinite loop because it couldn't go past "mkdir $TMP_DIR" in docker-entrypoint saying it already existed.

I am running it with restart: always using docker-compose.

I am tempted to change that line to mkdir -p $TMP_DIR

Is there any reason why it doesn't check if the directory exists already, or why it shouldn't just continue in case it does?

Error : context deadline exceeded

Since some weeks, docker produce this kind of logs multiple times per day :

# /var/log/syslog

dockerd[663]: time="2018-10-06T22:13:02.836190692+02:00" level=warning msg="Health check for container 105a28172561cf5963995cf346d7597b16c36736f2ebe39fc92a94d183199717 error: context deadline exceeded"

Do you have any idea which could causes this issue ? Can it be safely ignored ?

docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:18 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:22:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

docker info

Containers: 13
 Running: 13
 Paused: 0
 Stopped: 0
Images: 11
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-8-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.631GiB
Name: mondedie-web
ID: GG5P:ADAC:MZJU:BLUI:5I6D:CAMO:QUPJ:UT37:XVHS:IPW7:CK44:VFEO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

autoheal:
    restart: always
    image: willfarrell/autoheal
    container_name: autoheal
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    healthcheck:
      disable: true

free -m
              total        used        free      shared  buff/cache   available
Mem:           3718         786         769         106        2162        2634
Swap:           951          94         857