Implement HEALTHCHECK instruction

[iBobik]

Implement HEALTHCHECK instruction

[SalathielGenese]

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

[Ajedi32]

FYI, it looks like there's now a healthcheck script built into the image, it's just not called in the Dockerfile for some reason. You can enable it like so:

services:
  mariadb:
     environment:
       MARIADB_MYSQL_LOCALHOST_USER: true
     healthcheck:
       test: ["CMD", "/usr/local/bin/healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
  other_service:
    depends_on:
      mariadb:
        condition: service_healthy

For more details, and additional tests that can be enabled, see https://github.com/MariaDB/mariadb-docker/blob/master/healthcheck.sh

IMO this (or something similar) should be enabled by default. I'm a firm believer in setting reasonable defaults, and I don't think external services failing to start because docker-compose has no way of checking whether the service is healthy qualifies as "reasonable".

users will have their own idea of what is "healthy"

Users can customize the health check command if they want to. That's not a good reason for the default behavior to be "no health checks at all; just fail silently and don't give the container engine any indication that the service is unhealthy".

it does not actually test that the service is listening to connections outside of localhost

Okay. Again, that's not an argument for not having a basic healthcheck by default. If users want something more comprehensive than what Docker's internal health check mechanism can provide then they can always write it themselves.

some of the Official Images even purposely start in a localhost only mode for database initialization and then kill and start the main service with full network availability

All the more reason for those official images to have a proper health check predefined, so users don't try to write their own health check and discover that behavior the hard way. In images where it's applicable the default health check should detect "localhost only mode" and react appropriately. If programing that is too much effort for some reason, then leave out the health check on the images where that's a problem, and include a note about why. That'd be disappointing, but no worse than the current status quo.

after upgrading their images, current users will have extra unexpected load on their systems for healthchecks they don't necessarily need/want and may be unaware of

It's a healthcheck, not a stress test. A simple healthcheck script running twice a minute should not impose any significant additional load on a system. If for whatever reason the user wants something more complicated that is resource intensive, then let them enable that themselves so they're aware of the trade-off.

[Luc45]

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

This is wrong, please edit your reply to not induce others into error.

If you do this, it will do the ping with root user with an empty password, which can fail (silently) with this message:

root@b8ab90e765c2:/# mysqladmin ping       
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: NO)'
root@b8ab90e765c2:/# echo $?
0

As you can see, the access was denied, but the exit status code was still 0!

You need to specify the root password to use that command. For instance, if your root password is 123:

healthcheck:
    test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p123 | grep 'mysqld is alive' || exit 1"]
    interval: 2s
    retries: 15

[jerdoe]

FYI, it looks like there's now a healthcheck script built into the image, it's just not called in the Dockerfile for some reason. You can enable it like so:

services:
  mariadb:
     environment:
       MARIADB_MYSQL_LOCALHOST_USER: true
     healthcheck:
       test: ["CMD", "/usr/local/bin/healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
  other_service:
    depends_on:
      mariadb:
        condition: service_healthy

For more details, and additional tests that can be enabled, see https://github.com/MariaDB/mariadb-docker/blob/master/healthcheck.sh

When I use these settings, the healthcheck test result for option --connect is correctly reported but I get a warning in my logs every time the healthcheck.sh script is run :

root@0fe582a68cc7:/# healthcheck.sh --su-mysql --connect
root@0fe582a68cc7:/# echo $?
0

$ sudo docker compose logs -f mariadb
proj_mariadb-1  | 2022-05-08 21:10:02 1453 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:10:32 1461 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:11:03 1463 [Warning] Access denied for user 'mysql'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:11:05 1443 [Warning] Access denied for user 'mysql'@'127.0.0.1' (using password: NO)

Samely, if I enter the following command, the same warning is thrown in my docker compose logs and in my console :

root@0fe582a68cc7:/# gosu mysql mysql -h localhost --protocol tcp -e 'select 1'
ERROR 1045 (28000): Access denied for user 'mysql'@'127.0.0.1' (using password: NO)

but this would not throw the warning:

root@0fe582a68cc7:/# gosu mysql mysql -h localhost -e 'select 1'
+---+
| 1 |
+---+
| 1 |
+---+

Is that something which is expected or is there something wrong with my setup ?

[ebuildy]

Pay attention, if you use the /docker-entrypoint-initdb.d folder to run some SQL scripts at runtime, it may be useful to test if data is ready instead of just the server.

[grooverdan]

MARIADB_MYSQL_LOCALHOST_USER creates mysql@localhost but the --connect is using a tcp connection so [email protected] is the expected user. Tested locally and mysql@::1 was used.

Its possible to hide these with log_warnings=1.

I maybe think I'll just expand the MARIADB_MYSQL_LOCALHOST_USER to create the [email protected]/ mysql@::1 user. I hope it's not too confusing to have MARIADB_MYSQL_LOCALHOST_GRANTS not give the same grants on those users.

Thanks for your comment @jerdoe. Nothing is really wrong with your setup. Creating new issues is ok too.

[Luc45]

For whoever it may concern, I ended up doing it like this:

services:
  db:
    healthcheck:
      test: ["CMD-SHELL", "mysql dbname -udbuser -pdbpass -e 'SELECT 1;'  || exit 1"]
      interval: 2s
      retries: 120
  php:
    depends_on:
      db:
        condition: service_healthy

The healthcheck will only pass when a SELECT has successfully ran against the DB, which solved all racing conditions for me of the DB not being ready, even when the ping was successful.

[monkeyhybrid]

@grooverdan Thanks for that previous comment, suggesting the use of a custom healthcheck.cnf with the relevant auth credentials and using that instead of the mysql@localhost user (which as well as access denied warnings, also failed to ever pass the --innodb_initialized test for me).

I can now use the following healthcheck command and it works perfectly for both tests with no warnings.

healthcheck:
      test: ["CMD", "/usr/local/bin/healthcheck.sh", "--defaults-file=/etc/mysql/healthcheck.cnf-notdefault", "--connect", "--innodb_initialized"]

With this, there is also no need to set the MARIADB_MYSQL_LOCALHOST_USER environment variable.

Thanks!

[PrynsTag]

If you have secrets and environment variables, you could also do this.

services:
  db:
    environment:
        FILE__MYSQL_ROOT_PASSWORD: /run/secrets/mysql_root_password
    secrets:
        - mysql_root_password
    healthcheck:
      test: test: ["CMD-SHELL", "mysqladmin ping -P $$(echo $$MARIADB_PORT) -p$$(cat $$FILE__MYSQL_ROOT_PASSWORD) | grep 'mysqld is alive' || exit 1"]
      interval: 2s
      retries: 120
  php:
    depends_on:
      db:
        condition: service_healthy

[Yszty]

Yesterday I rebuilt my containers and health check also stopped work. The problem was that on latest mariadb containers command mysql does not work,(read here #512) after change to mariadb, everything is fine in my case.

Doesnt work:
health-check.sh

#!/bin/sh

mysql --user=root --password=$(cat $MARIADB_ROOT_PASSWORD_FILE) --silent --execute "SELECT 1"
exit $?;

Fix:
health-check.sh

#!/bin/sh

mariadb --user=root --password=$(cat $MARIADB_ROOT_PASSWORD_FILE) --silent --execute "SELECT 1"
exit $?;

docker-compose.yml

  db:
    platform: linux/amd64
    build:
      context: ./mariadb
    ports:
      - '3306:3306'
    restart: always
    healthcheck:
      test: health-check.sh
      timeout: 30s
      retries: 10
      interval: 2s

Dockerfile

FROM mariadb
COPY health-check.sh /bin/health-check.sh
RUN chmod +x /bin/health-check.sh

[orefalo]

Yes, saw it - thank you.
I can live with the Warnings, now that I know it's a non issue ;-)

[grooverdan]

@Luc45 watch out, the initialization starts a temporary server on initialization and upgrades. Its possible this healthcheck will succeed when its in a getting ready stage. --protocol tcp will save you however there is....

While I haven't implemented user and pass on a command line, there are configuration files that can be passed that can contain:

[mariadb-client]
user=
password=
database=

And then:
test: ["CMD-SHELL", "healthcheck.sh --defaults-file=/etc/mysql/healthcheck.cnf-notdefault" --connect]

[GagMirz]

If you have secrets and environment variables, you could also do this.

services:
  db:
    environment:
        FILE__MYSQL_ROOT_PASSWORD: /run/secrets/mysql_root_password
    secrets:
        - mysql_root_password
    healthcheck:
      test: test: ["CMD-SHELL", "mysqladmin ping -P $$(echo $$MARIADB_PORT) -p$$(cat $$FILE__MYSQL_ROOT_PASSWORD) | grep 'mysqld is alive' || exit 1"]
      interval: 2s
      retries: 120
  php:
    depends_on:
      db:
        condition: service_healthy

Thanks, this one was the only working option on my side.

[grooverdan]

see #512 and https://mariadb.org/mariadb-server-docker-official-images-healthcheck-without-mysqladmin/

[iBobik]

Maybe to make it working also if MYSQL_USER or MYSQL_PASSWORD is not filled → use root. Then if MYSQL_ALLOW_EMPTY_PASSWORD and other related settings.

Any ideas?

[tianon]

docker-library/cassandra#76 (comment) has some good points regarding adding default health checks to official images 👍

(https://github.com/docker-library/healthcheck also has copies of several images with reasonably-generic HEALTHCHECK instructions added)

[yosifkit]

Current stance is still the same as linked above in cassandra.

I do not feel that generalized healthchecks on the official images are really that useful.

• users will have their own idea of what is "healthy"
• it does not actually test that the service is listening to connections outside of localhost (see https://github.com/docker-library/healthcheck for some examples that do more than what's proposed here, including attempting to check whether the service is listening remotely)
• some of the Official Images even purposely start in a localhost only mode for database initialization and then kill and start the main service with full network availability
• after upgrading their images, current users will have extra unexpected load on their systems for healthchecks they don't necessarily need/want and may be unaware of

Closing old issue.

[katsar0v]

Pay attention, if you use the /docker-entrypoint-initdb.d folder to run some SQL scripts at runtime, it may be useful to test if data is ready instead of just the server.

That's an issue. Can we reopen this issue? If you have entrypoint sql or sh, the healthcheck from @SalathielGenese does not work

[grooverdan]

ideas welcome - https://jira.mariadb.org/browse/MDEV-25434

[Ajedi32]

@Luc45 Ping just checks the connection, not whether your credentials are correct. It will exit with 1 if the server is not running, which might be good enough depending on your goal. If you need something more complicated than that, see the example in my previous comment.

[orefalo]

I too see

│ 2022-06-01 15:02:47 215 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:02:57 216 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:02:57 217 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:07 218 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:07 219 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:17 220 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:17 221 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:27 222 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:27 223 [Warning] Access denied for user 'root'@'localhost' (using password: NO)                                                                      │
│ 2022-06-01 15:03:37 224 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

on the logs, and ended up on this thread.

[grooverdan]

yep, started working on it in #430, I haven't overcome all cases yet, for the moment --log-warnings=1 can hide them.

[grooverdan]

ack. Yes I do need to do something better.

[Yszty]

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

This is wrong, please edit your reply to not induce others into error.

If you do this, it will do the ping with root user with an empty password, which can fail (silently) with this message:

root@b8ab90e765c2:/# mysqladmin ping       
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: NO)'
root@b8ab90e765c2:/# echo $?
0

As you can see, the access was denied, but the exit status code was still 0!

You need to specify the root password to use that command. For instance, if your root password is 123:

healthcheck:
    test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p123 | grep 'mysqld is alive' || exit 1"]
    interval: 2s
    retries: 15

MySQL Documentation says about mysqladmin ping

Check whether the server is available. The return status from mysqladmin is 0 if the server is running, 1 if it is not. This is 0 even in case of an error such as Access denied, because this means that the server is running but refused the connection, which is different from the server not running.

[gremo]

@grooverdan @monkeyhybrid can you elaborate your answer a bit more? Same problem here, lot of warnings due to the healthcheck script.

[Kamikaze01git]

If i use

healthcheck:
      test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p MyRootPW | grep 'mysqld is alive' || exit 1"]
      interval: 2s
      retries: 15

just to ping my db, i got status unhealthy with log
Segmentation fault (core dumped)

any help?

[saschabrockel]

Using MariaDB 11.0.2 in Docker I am very curious that nothing mentioned here works. If I try to execute healthcheck.sh --su-mysql --connect --innodb_initialized in the container I get the following error: failed switching to "mysql": operation not permitted

I have set MARIADB_MYSQL_LOCALHOST_USER to true and also tried to set it to 1. Does not change anything. In the container the commands like mysql or mysqladmin are not found and unknown.

And last but not least:
healthcheck.sh --defaults-file=/etc/mysql/healthcheck.cnf-notdefault --connect --innodb_initialized Could not open required defaults file: /etc/mysql/healthcheck.cnf-notdefault
Fatal error in defaults handling. Program aborted
healthcheck innodb_initialized failed

Is something wrong with my permissions?

[grooverdan]

Like the SO, I don't know what causes "failed switching to "mysql": operation not permitted". More information on the OS / Docker version might be applicable here.

MARIADB_MYSQL_LOCALHOST_USER doesn't create mysql/mysqladmin links.

"Could not open required defaults file: /etc/mysql/healthcheck.cnf-notdefault" - looks like you haven't passed this file into the container as a volume.

I was updating https://mariadb.com/kb/en/using-healthcheck-sh-script/ today and drafting a blog about the removal of mysql/mysqladmin that was in the release notes.

Please create a new issue for bugs. Keeping track of old threads where the situation has changed significantly is difficult.

[buggsi]

Set the MYSQL_ROOT_PASSWORD (or MARIADB_ROOT_PASSWORD) variable in a .env file and add this to the mariadb service in docker-compose.yml:

    healthcheck:
      test: mariadb-admin ping -h 127.0.0.1 --password=$MYSQL_ROOT_PASSWORD
      start_period: 20s
      start_interval: 1s
      interval: 30s
      timeout: 5s
      retries: 30