garywiz/chaperone

Lightweight process-tree manager for Docker-like containers

chaperone's Issues

pid changes?

haproxy supports lightweight restarts which generate a new pid and take over the work of the previous process.

In this case, the active PID changes, but I've seen chaperone not updating the active pid when given a pidfile.

(sometimes it seems to honor the change, othertimes not)

Any clue on a timer what chaperone looks at the pid file?

Trailing commas at end of block

The chaperone documentation examples use trailing commas for entries that come last in a block:

nginx.service: {
  [...]
  env_set: {
    NGINX_PID_FILE: "$(NGINX_PID_FILE:-/run/nginx.pid)"
  },
  service_group: server,
}

However, this results in an error when chaperone starts:

Configuration Error: extra keys not allowed @ data['nginx.service']['service_group']

Removing the traililng comma for the last entry in block resolves the issue and chaperone starts correctly:

[...]
  service_group: server
}

Transfer ownership

@garywiz would you be willing to transfer ownership of this repo to me, or create an organization for it and add some more people as maintainers? Would be nice to maintain it without having a separate fork.

Project status

Is this project dead?

Allow to specify udp port in syslog_host

Hi,
i want to use chaperone to relay syslog messages to logstash (that not running as root, so can't bind to port < 1024).
So can you allow to specify the port where the syslog messages are sent in the syslog_hostoption.
Ex:

syslog_host: "logstash:1514"

Thanks

How should I call a runtime startup script?

It's common practice in docker containers to have some script that gets run at run time to do things like take values from environment values and construct the odd config file with them before the main daemon process(es) get run. This commonly gets shoe-horned into the CMD or ENTRYPOINT directive.

It seems in keeping with the philosophy of chaperone that this should probably hang off the chaperone config, and also I suspect not doing it that way would mean that the wrong process would be PID 1, and responsible for zombies. I'm still just finding my way here, but I'm not seeing a really elegant way to do it with chaperone. What am I missing?

Maybe I could create a one-shot service, and set up some before/after rules. Is there a better way?

Python 3.7 compatibility: asyncio.async and async keyword

Chaperone version 0.3.9 does not work with Python 3.7+ because it uses asyncio.async and async is now a keyword.

type forking and exit_fills true

Hi,

i want to start a service that fork and want to kill the system if the service (which have a pid) die.
For example for my postfix service i use:

postfix.service: {
  type: forking,
  service_groups: IDLE,
  command: "postfix start",
  pidfile: /var/spool/postfix/pid/master.pid,
  after: "postgrey.service",
  exit_kills: true,
}

The problem is that the postfix start command start the postfix master process but return when it has fork the master process. So chaperone see that the postfix start end and kill the whole system (because of the exit_kills with value true) even if the pid is valid and the forking process is running.

EDIT the logs i have:

May 16 19:39:07 7f454733d6e4 chaperone[1]: system will be killed when '/usr/sbin/postfix' exits
May 16 19:39:07 7f454733d6e4 postfix-script[321]: starting the Postfix mail system
May 16 19:39:07 7f454733d6e4 master[323]: daemon started -- version 3.0.4, configuration /etc/postfix
May 16 19:39:07 7f454733d6e4 chaperone[1]: REAP pid=257,status=0
May 16 19:39:07 7f454733d6e4 chaperone[1]: REAP pid=0,status=0
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service exit status for pid=257 is '<ProcStatus exit_status=0>'
May 16 19:39:07 7f454733d6e4 chaperone[1]: Request made to kill system.
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service waiting for PID file: /var/spool/postfix/pid/master.pid
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service successfully started
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service notified waiters upon completion
May 16 19:39:15 7f454733d6e4 master[323]: terminating on signal 15
May 16 19:39:15 7f454733d6e4 chaperone[1]: no child processes present
May 16 19:39:15 7f454733d6e4 chaperone[1]: Final termination phase.

Task was destroyed but it is pending!

After sending mysqld a SIGTERM singal to shut it down,
chaperone detects this but an error message occurs.

mysqld.conf for mariadb/mysql official Dockerhub image:

# mysqld.conf

settings: {
  env_set: {
    "MYSQL_PID_FILE":  "/run/mysqld/mysqld.pid"
  },
  env_inherit: [
    "MYSQL_ROOT_PASSWORD",
    "MYSQL_DATABASE",
    "MYSQL_USER",
    "MYSQL_PASSWORD",
    "MYSQL_ALLOW_EMPTY_PASSWORD",
    "MYSQL_RANDOM_ROOT_PASSWORD",
    "MYSQL_ONETIME_PASSWORD"
  ]
}

mysql1.service: {
  type: forking,
  command: "mysqld",
  enabled: false,
  uid: mysql,
  gid: mysql,
  service_groups: database,
  pidfile: "$(MYSQL_PID_FILE)",
}

# setup script starts also mysqld
mysql.service: {
  type: simple,
  command: "/docker-entrypoint.sh mysqld",
    # Dockerfile entrypoint + default cmd
  uid: root,
  gid: root,
  pidfile: "$(MYSQL_PID_FILE)",
  process_timeout: 60,
  restart: true,
  enabled: true,
  stderr: "log",
  service_groups: database,
}

$ docker exec test killall -s SIGTERM mysqld

Docker log:

Dec  6 12:33:13 e1588f1e7257 mysql[152]: 2016-12-06 12:33:13 140467036891072 [Note] mysqld (mysqld 10.1.19-MariaDB-1~jessie) starting as process 9 ...
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: REAP pid=9,status=0
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: no child processes present
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: mysql.service exit status for pid=9 is '<ProcStatus exit_status=0>'
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: Final termination phase.
Task was destroyed but it is pending!
task: <Task pending coro=<_monitor_service() running at /usr/local/lib/python3.4/dist-packages/chaperone/cproc/pt/simple.py:29> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SubProcess.add_pending.<locals>.<lambda>() at /usr/local/lib/python3.4/dist-packages/chaperone/cproc/subproc.py:631]>

The mysql process should have been restarted by chaperone,
but it terminated completely and the Docker container stopped.

Incompatibility with PyYAML 6.0

Hello

chaperone fails with

Configuration Error: load() missing 1 required positional argument: 'Loader'

To fix, I forced the version PyYAML == 5.2

Regards

Python 2 support

Would you be interested in accepting PR with Python 2 support?

Allow to add the PRI field in front of logging message

Hi,

actually we can use the extendedoption to prefixes every output syslog line with the facility and priority.
The problem is that it can be difficult and time consuming to parse this fields to convert them to numeric levels.
So can you add an option to prefix every output syslog line with the "standard" PRI field (surround by <> like defined in RFC3164)
Ex:

<13> Jun 15 02:09:33 su [27]: pam_unix(su:session): session opened for user root by (uid=1000)

Thanks

add docs for telchap

telchat is hinted at, but no detailed docs on telchap that I could find.

(outside telchap help)

Clarification: console, syslog

Does console encompass all logs, including from syslog?

console.logging: {
  file: '/var/log/chaperone-syslog.log'
}

Would it be also possible to enable or disable logging to stdout globally?
For debugging, it is helpful but in production it would unnecessarily strain the server.

Edit: Is the stdout/stderr of cron type services logged, too?
Or will this be logged to a default cron log file (/var/log/cron)?

Shutdown/cleanup script

Is there a way to make chaperone run a script with cleanup stuff before it quits? My use case is: I have a vpn service running in a docker container that sets some firewall rules and I want to make sure those get cleared when the container/chaperone stops (for whatever reason), something like the functionality provided by SysV run levels... Thanks!

exitkills doesn't pass the error code

It would be useful if the error code of the process is passed as the error code of chaperone if exitkills is enabled for a process.

chaperone/chaperone/cproc/subproc.py

Lines 560 to 572 in 9ff2c3a

	def process_exit(self, code):
	self.returncode = code
	if self._exit_event:
	self._exit_event.set()
	self._exit_event = None
	if self.exit_kills:
	self.logwarn("{0} terminated with exit_kills enabled", self.service.name);
	# Since we're dead, and the system is going away, disable any process management
	self._proc = None
	self.pid = None
	self._kill_system();

build failure due to undocumented dependency

On both ubuntu:14.04 and debian:jessie, installing chaperone requires both python3-pip and also the build-essential and python3-dev packages which python3-pip recommends, but does not require.

As a result, this fails:

from ubuntu:14.04

ENV DEBIAN_FRONTEND noninteractive
ENV DEBIAN_PRIORITY critical
ENV DEBCONF_NOWARNINGS yes

RUN apt-get update \
    && apt-get upgrade -y

RUN apt-get install --no-install-recommends -y \
    python3-pip

RUN pip3 install chaperone

In the interests of building small containers, using apt-get --no-install-recommends -y is typically a good idea, but here it means that the other two packages need to be explicitly installed (or apt-get install -y python3-pip needs to be in a separate RUN statement to other packages that will be installed, which uses an extra filesystem layer for little advantage).

Reasonable people could argue, but consider stating the dependency in the README instructions. And if not, then at least this issue might point out the problem for someone.

chaperone python3.2 issues :(

hi,

benn trying to install chaperone with python3.2 (debian7), it clearly fails :(
error logs:

    no previously-included directories found matching 'documentation/_build'
Installing collected packages: chaperone, docopt, PyYAML, voluptuous, aiocron, setproctitle, croniter, trollius, python-dateutil, six
  Running setup.py install for chaperone
      File "/usr/local/lib/python3.2/dist-packages/chaperone/cutil/servers.py", line 46
        self.server = yield from self._create_server()
                               ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cutil/syslog_handlers.py", line 130
        (transport, protocol) = yield from connect
                                         ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cutil/notify.py", line 49
        yield from self.run()
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/commands.py", line 45
        result = yield from self.do_exec(opts, protocol.owner.controller)
                          ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/subproc.py", line 24
        data = yield from stream.readline()
                        ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/process_manager.py", line 182
        yield from asyncio.sleep(0.1)
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/pt/forking.py", line 11
        result = yield from self.timed_wait(self.process_timeout, self._exit_timeout)
                          ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/pt/cron.py", line 74
        yield from super().start()
                 ^
    SyntaxError: invalid syntax

this seems to be related to chaned appeared in python3.3:
https://docs.python.org/3/whatsnew/3.3.html

by any chance, is there any way to make chaperone compatible with python3.2 ?

thanks anyway for this awesome piece of work :)

Chaperone with cron job got crazy on DST change

If cron job is configured and DST change occurs Chaperone starts executing job in loop regardless cron configuration.

Log:

Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: Switching all chaperone logging to /dev/log
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: chaperone version 0.3.9, ready.
start
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service scheduled using interval spec '* * * * *'
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: periodic_long_running.service attempting start '/bin/echo EXECUTION'...
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: REAP pid=8,status=0
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: no child processes present
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request

Test case:
chaperone-dst-bug.tar.gz

To run test case. Execute attached archive and run ./run_test.sh. Script is using sudo command to change system datetime.

Disabling time sync may be required. (Note that Virtual Box Guest additions synchronizes your time with host clock)

Env:
Ubuntu 14.04

One failed cron job execution stops any further execution

If only one cron fail than all further execution are ignored:

2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: REAP pid=49,status=256
2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: REAP pid=0,c=0
2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: executor.service exit status for pid=49 is '<ProcStatus exit_status=1>'
2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: executor.service terminated abnormally with <ProcStatus exit_status=1>
2016-12-30 16:31:00 | Dec 30 16:31:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:31:00 | Dec 30 16:31:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.
2016-12-30 16:32:00 | Dec 30 16:32:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:32:00 | Dec 30 16:32:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.
2016-12-30 16:33:00 | Dec 30 16:33:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:33:00 | Dec 30 16:33:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.
2016-12-30 16:34:00 | Dec 30 16:34:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:34:00 | Dec 30 16:34:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.

Syslog implementation does remove utf-8 characters

Scenario:

• syslog functionality enabled by default
• service writes utf-8 characters to stdout/stderr, for example "request processed in 100 µs"
• chaperone/Python does remove the utf-8 charcater => "request processed in 100 s"

Config:

settings: {
  env_set: {
    'LANG': 'en_US.UTF-8',
    'LC_CTYPE': '$(LANG)'
  }
}

test.service: {
  command: '/display-utf8.sh',
}

console.logging: {
  enabled: true,
  stdout: true
}

display-utf8.sh:

#!/bin/bash

echo "++++TEST++++"
echo "request processed in 100 µs"
echo "++++/TEST++++"

Config validation support?

Hi!
does chaperone support validation of config and exit accordingly? My use case would be to test chaperone config at container build time and fail container build if something is wrong, without actually starting chaperone.

thanks!

[Question] Is this project actively maintained?

I'm looking at supervisord alternatives and stumbled upon Chaperone. It looks great and is exactly what I need.

However, there has been no activity after 2016. Not to sound rude, but I plan to use this on production, and would like to know if the community is interested in actively maintaining this project.