peter-evans / docker-compose-healthcheck Goto Github PK

How to wait for container X before starting Y using docker-compose healthcheck

License: MIT License

docker-compose healthcheck postgresql kong docker health-check

docker-compose-healthcheck's Introduction

docker-compose-healthcheck

The healthcheck property was originally introduced in the 2.1 Compose file format and is now part of the Compose Specification used by recent versions of Docker Compose. This allows a check to be configured in order to determine whether or not containers for a service are "healthy."

How can I wait for container X before starting Y?

This is a common problem and in earlier versions of docker-compose requires the use of additional tools and scripts such as wait-for-it and dockerize. Using the healthcheck parameter the use of these additional tools and scripts is often no longer necessary.

Waiting for PostgreSQL to be "healthy"

A particularly common use case is a service that depends on a database, such as PostgreSQL. We can configure docker-compose to wait for the PostgreSQL container to startup and be ready to accept requests before continuing.

The following healthcheck has been configured to periodically check if PostgreSQL is ready using the pg_isready command. See the documentation for the pg_isready command here.

healthcheck:
  test: ["CMD-SHELL", "pg_isready"]
  interval: 10s
  timeout: 5s
  retries: 5

If the check is successful the container will be marked as healthy. Until then it will remain in an unhealthy state. For more details about the healthcheck parameters interval, timeout and retries see the documentation here.

Services that depend on PostgreSQL can then be configured with the depends_on parameter as follows:

depends_on:
  postgres-database:
    condition: service_healthy

Waiting for PostgreSQL before starting Kong

In this complete example docker-compose waits for the PostgreSQL service to be "healthy" before starting Kong, an open-source API gateway. It also waits for an additional ephemeral container to complete Kong's database migration process.

Test it out with:

docker-compose up -d

Wait until all services are running:

Test by querying Kong's admin endpoint:

curl http://localhost:8001/

License

MIT License - see the LICENSE file for details

docker-compose-healthcheck's People

Contributors

Stargazers

Watchers

Forkers

andersrehn deevodavis71 sbelalex nuaays danielleone trajectnz webster722 eharrow junneyang michaelpak ziplex mahendrashelke taihptvng onuryaz vedhagiri christiangutierrezgrability veiii beingzeroin ozanteoman kczan docdocker simundhe22 sijokappen youngkingtaco mib-dotnet tullo elbeshenrique arun9theja krbcmt alonsoir l3th2nh iamsethi dolanor-galaxy thoddle miblue119 dmdv nogsantos patsevanton mister2d hefx sotasato-stst sergey-erofeev despernal vijaykrishnaa614 stungkit phiponacci motlogelwan jsdotx3 nmatsui prakash-pun ferrarijefferson aytuar z-sector liviob74 alexrogalskiy egorsalenko

docker-compose-healthcheck's Issues

FATAL: role "root" does not exist

I notice the following error in my logs appearing every 5 minutes or so:

FATAL:  role "root" does not exist

Whilst not harmful, and whilst the healthcheck is indeed working as intended (a docker ps displays a (healthy) status) is there any way to suppress this message? I assume this happens because the official PostgreSQL Docker images do not have a USER postgres declaration at the end of their Dockerfiles?

Version 3 no longer supports the condition form of depends_on.

Take care, this won't work with compose file version 3.

As per documentation:

Version 3 no longer supports the condition form of depends_on.

pg_ready ignores provided arguments.

pg_ready ignores the user and database arguments. A more reliable check is psql -U postgres -c "\q"

I've tested this by having an initdb script that sleeps and then create a new database and user. pg_ready always returns 0 even when the database and user don't yet exist.

healthcheck condition is not supported in docker compose file v3

Thanks for providing this resource. While attempting to get it working on my system, I think I have identified that it is not supported in docker compose v3. I found the first hint here: https://stackoverflow.com/a/59072727. I looked through the docs for v3 and cannot find "condition" or "service_healthy" (https://docs.docker.com/compose/compose-file/compose-file-v3/#depends_on).
Is it still working for you in v3? If not, maybe you could add a comment that it is supported by v2?

Here's their reasoning (from https://docs.docker.com/compose/startup-order/):

The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.
To handle this, design your application to attempt to re-establish a connection to the database after a failure. If the application retries the connection, it can eventually connect to the database.

Thanks for directing me to pg_isready

Get error

kong            | 2022/02/15 04:18:17 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: New migrations available; run 'kong migrations up' to proceed
kong            | stack traceback:
kong            |       [C]: in function 'error'
kong            |       /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: in function 'check_state'
kong            |       /usr/local/share/lua/5.1/kong/init.lua:506: in function 'init'
kong            |       init_by_lua:3: in main chunk
kong            | nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: New migrations available; run 'kong migrations up' 
to proceed
kong            | stack traceback:
kong            |       [C]: in function 'error'
kong            |       /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: in function 'check_state'
kong            |       /usr/local/share/lua/5.1/kong/init.lua:506: in function 'init'
kong            |       init_by_lua:3: in main chunk

Full log

docker-compose up
[+] Running 4/3
 - Network docker-compose-healthcheck_default  Created                                                                                                       0.1s 
 - Container kong-postgres                     Created                                                                                                       2.8s 
 - Container kong-migration                    Created                                                                                                       0.1s
 - Container kong                              Created                                                                                                       0.1s
Attaching to kong, kong-migration, kong-postgres
kong-postgres   | ********************************************************************************
kong-postgres   | WARNING: POSTGRES_HOST_AUTH_METHOD has been set to "trust". This will allow
kong-postgres   |          anyone with access to the Postgres port to access your database without
kong-postgres   |          a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
kong-postgres   |          documentation about "trust":
kong-postgres   |          https://www.postgresql.org/docs/current/auth-trust.html
kong-postgres   |          In Docker's default configuration, this is effectively any other
kong-postgres   |          container on the same system.
kong-postgres   |
kong-postgres   |          It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
kong-postgres   |          it with "-e POSTGRES_PASSWORD=password" instead to set a password in
kong-postgres   |          "docker run".
kong-postgres   | ********************************************************************************
kong-postgres   | The files belonging to this database system will be owned by user "postgres".
kong-postgres   | This user must also own the server process.
kong-postgres   |
kong-postgres   | The database cluster will be initialized with locale "en_US.utf8".
kong-postgres   | The default database encoding has accordingly been set to "UTF8".
kong-postgres   | The default text search configuration will be set to "english".
kong-postgres   |
kong-postgres   | Data page checksums are disabled.
kong-postgres   |
kong-postgres   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
kong-postgres   | creating subdirectories ... ok
kong-postgres   | selecting default max_connections ... 100
kong-postgres   | selecting default shared_buffers ... 128MB
kong-postgres   | selecting default timezone ... Etc/UTC
kong-postgres   | selecting dynamic shared memory implementation ... posix
kong-postgres   | creating configuration files ... ok
kong-postgres   | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
kong-postgres   | initializing pg_authid ... ok
kong-postgres   | setting password ... ok
kong-postgres   | initializing dependencies ... ok
kong-postgres   | creating system views ... ok
kong-postgres   | loading system objects' descriptions ... ok
kong-postgres   | creating collations ... ok
kong-postgres   | creating conversions ... ok
kong-postgres   | creating dictionaries ... ok
kong-postgres   | setting privileges on built-in objects ... ok
kong-postgres   | creating information schema ... ok
kong-postgres   | loading PL/pgSQL server-side language ... ok
kong-postgres   | vacuuming database template1 ... ok
kong-postgres   | copying template1 to template0 ... ok
kong-postgres   | copying template1 to postgres ... ok
kong-postgres   | syncing data to disk ... 
kong-postgres   | WARNING: enabling "trust" authentication for local connections
kong-postgres   | You can change this by editing pg_hba.conf or using the option -A, or
kong-postgres   | --auth-local and --auth-host, the next time you run initdb.
kong-postgres   | ok
kong-postgres   |
kong-postgres   | Success. You can now start the database server using:
kong-postgres   |
kong-postgres   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
kong-postgres   |
kong-postgres   | waiting for server to start....LOG:  database system was shut down at 2022-02-15 04:18:03 UTC
kong-postgres   | LOG:  MultiXact member wraparound protections are now enabled
kong-postgres   | LOG:  autovacuum launcher started
kong-postgres   | LOG:  database system is ready to accept connections
kong-postgres   |  done
kong-postgres   | server started
kong-postgres   | CREATE DATABASE
kong-postgres   | 
kong-postgres   |
kong-postgres   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
kong-postgres   |
kong-postgres   | waiting for server to shut down....LOG:  received fast shutdown request
kong-postgres   | LOG:  aborting any active transactions
kong-postgres   | LOG:  autovacuum launcher shutting down
kong-postgres   | LOG:  shutting down
kong-postgres   | LOG:  database system is shut down
kong-postgres   |  done
kong-postgres   | server stopped
kong-postgres   |
kong-postgres   | PostgreSQL init process complete; ready for start up.
kong-postgres   |
kong-postgres   | LOG:  database system was shut down at 2022-02-15 04:18:05 UTC
kong-postgres   | LOG:  MultiXact member wraparound protections are now enabled
kong-postgres   | LOG:  autovacuum launcher started
kong-postgres   | LOG:  database system is ready to accept connections
kong-migration  | Bootstrapping database...
kong-migration  | migrating core on database 'kong'...
kong-migration  | core migrated up to: 000_base (executed)
kong-migration  | core migrated up to: 003_100_to_110 (executed)
kong-migration  | core migrated up to: 004_110_to_120 (executed)
kong-migration  | core migrated up to: 005_120_to_130 (executed)
kong-migration  | core migrated up to: 006_130_to_140 (executed)
kong-migration  | core migrated up to: 007_140_to_150 (executed)
kong-migration  | core migrated up to: 008_150_to_200 (executed)
kong-migration  | core migrated up to: 009_200_to_210 (executed)
kong-migration  | core migrated up to: 010_210_to_211 (executed)
kong-migration  | core migrated up to: 011_212_to_213 (executed)
kong-migration  | core migrated up to: 012_213_to_220 (executed)
kong-migration  | core migrated up to: 013_220_to_230 (executed)
kong-migration  | core migrated up to: 014_230_to_270 (executed)
kong-migration  | migrating acl on database 'kong'...
kong-migration  | acl migrated up to: 000_base_acl (executed)
kong-migration  | acl migrated up to: 002_130_to_140 (executed)
kong-migration  | acl migrated up to: 003_200_to_210 (executed)
kong-migration  | acl migrated up to: 004_212_to_213 (executed)
kong-migration  | migrating acme on database 'kong'...
kong-migration  | acme migrated up to: 000_base_acme (executed)
kong-migration  | migrating basic-auth on database 'kong'...
kong            | 2022/02/15 04:18:17 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
kong            | nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
kong            | 2022/02/15 04:18:17 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: New migrations available; run 'kong migrations up' to proceed
kong            | stack traceback:
kong            |       [C]: in function 'error'
kong            |       /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: in function 'check_state'
kong            |       /usr/local/share/lua/5.1/kong/init.lua:506: in function 'init'
kong            |       init_by_lua:3: in main chunk
kong            | nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: New migrations available; run 'kong migrations up' 
to proceed
kong            | stack traceback:
kong            |       [C]: in function 'error'
kong            |       /usr/local/share/lua/5.1/kong/cmd/utils/migrations.lua:20: in function 'check_state'
kong            |       /usr/local/share/lua/5.1/kong/init.lua:506: in function 'init'
kong            |       init_by_lua:3: in main chunk
kong-migration  | basic-auth migrated up to: 000_base_basic_auth (executed)
kong-migration  | basic-auth migrated up to: 002_130_to_140 (executed)
kong-migration  | basic-auth migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating bot-detection on database 'kong'...
kong-migration  | bot-detection migrated up to: 001_200_to_210 (executed)
kong-migration  | migrating hmac-auth on database 'kong'...
kong-migration  | hmac-auth migrated up to: 000_base_hmac_auth (executed)
kong-migration  | hmac-auth migrated up to: 002_130_to_140 (executed)
kong-migration  | hmac-auth migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating ip-restriction on database 'kong'...
kong-migration  | ip-restriction migrated up to: 001_200_to_210 (executed)
kong-migration  | migrating jwt on database 'kong'...
kong-migration  | jwt migrated up to: 000_base_jwt (executed)
kong-migration  | jwt migrated up to: 002_130_to_140 (executed)
kong-migration  | jwt migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating key-auth on database 'kong'...
kong-migration  | key-auth migrated up to: 000_base_key_auth (executed)
kong-migration  | key-auth migrated up to: 002_130_to_140 (executed)
kong-migration  | key-auth migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating oauth2 on database 'kong'...
kong-migration  | oauth2 migrated up to: 000_base_oauth2 (executed)
kong-migration  | oauth2 migrated up to: 003_130_to_140 (executed)
kong-migration  | oauth2 migrated up to: 004_200_to_210 (executed)
kong-migration  | oauth2 migrated up to: 005_210_to_211 (executed)
kong-migration  | migrating rate-limiting on database 'kong'...
kong-migration  | rate-limiting migrated up to: 000_base_rate_limiting (executed)
kong-migration  | rate-limiting migrated up to: 003_10_to_112 (executed)
kong-migration  | rate-limiting migrated up to: 004_200_to_210 (executed)
kong-migration  | migrating response-ratelimiting on database 'kong'...
kong-migration  | response-ratelimiting migrated up to: 000_base_response_rate_limiting (executed)
kong-migration  | migrating session on database 'kong'...
kong-migration  | session migrated up to: 000_base_session (executed)
kong-migration  | session migrated up to: 001_add_ttl_index (executed)
kong-migration  | 42 migrations processed
kong-migration  | 42 executed
kong-migration  | Database is up-to-date
kong exited with code 1
kong-migration exited with code 0
kong            | 2022/02/15 04:18:18 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
kong            | nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
kong            | 2022/02/15 04:18:18 [notice] 1#0: using the "epoll" event method
kong            | 2022/02/15 04:18:18 [notice] 1#0: openresty/1.19.9.1
kong            | 2022/02/15 04:18:18 [notice] 1#0: built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
kong            | 2022/02/15 04:18:18 [notice] 1#0: OS: Linux 5.10.60.1-microsoft-standard-WSL2
kong            | 2022/02/15 04:18:18 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker processes
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1098
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1099
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1100
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1101
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1102
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1103
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1104
kong            | 2022/02/15 04:18:18 [notice] 1#0: start worker process 1105
kong            | 2022/02/15 04:18:18 [notice] 1098#0: *2 [lua] warmup.lua:92: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by
context: init_worker_by_lua*
kong            | 2022/02/15 04:18:18 [notice] 1098#0: *2 [lua] warmup.lua:129: single_dao(): finished preloading 'services' into the core_cache (in 0ms), contextcache (in 0ms), context: init_worker_by_lua*
kong            | 2022/02/15 04:18:18 [notice] 1099#0: *1 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:18 [notice] 1104#0: *5 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:18 [notice] 1101#0: *6 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:18 [notice] 1105#0: *7 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:18 [notice] 1103#0: *4 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:18 [notice] 1100#0: *3 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:18 [notice] 1102#0: *8 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*      
kong            | 2022/02/15 04:18:24 [crit] 1098#0: *12 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:18:24 [crit] 1101#0: *15 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:18:24 [crit] 1104#0: *14 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:18:24 [crit] 1103#0: *16 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:18:24 [crit] 1100#0: *17 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:18:24 [crit] 1105#0: *13 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:18:24 [crit] 1102#0: *18 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
Gracefully stopping... (press Ctrl+C again to force)