clear-code/lua-collectd-monitor

lua-collectd-monitor

collectd plugins which provides fault recovery feature written in Lua. Following 2 plugins are included:

• collectd/monitor/remote.lua
  • Receives pre-defined recovery commands from a remote host via MQTT and execute them. In addition it can replace collectd's config file with a new config received via MQTT.
  • This plugin itself doesn't have the feature to detect system faults.
  • It aims to detect system faults by another host which receives metrics data via collectd's network plugin, and send recovery commands from the host to this plugin.
• collectd/monitor/local.lua
  • Detects system faults according to metrics data collected by local collectd daemon and executes recovery commands. Trigger conditions are written in Lua code.

Prerequisites

• Lua or LuaJIT
  • LuaJIT 2.1.0-beta3 is verified
• LuaRocks
• Forked version of collectd
  • You need to install customized version of collectd: https://github.com/clear-code/collectd/releases/tag/5.12.0.16.g6e9604f
  • Additional required callback functions are implemented in this branch.
• MQTT Broker
  • VerneMQ is verified
  • At least 2 topics should be accessible
    • For sending commands from a server
    • For replying command results from collectd

Install

• Download and install lua-collectd-monitor:

$ git clone https://github.com/clear-code/lua-collectd-monitor
$ sudo luarocks make

• Add settings like the following example to your collectd.conf (see conf/collectd/collectd.conf.monitor-remote-example for more options of remote monitoring feature):

<LoadPlugin lua>
  Globals true
</LoadPlugin>

<Plugin lua>
  BasePath "/usr/local/share/lua/5.1"

  # Use remote monitoring feature.
  Script "collectd/monitor/remote.lua"
  <Module "collectd/monitor/remote.lua">
    MonitorConfigPath "/etc/collectd/monitor/config.json"
  </Module>

  # Use local monitoring feature.
  Script "collectd/monitor/local.lua"
  <Module "collectd/monitor/local">
    MonitorConfigPath "/etc/collectd/monitor/config.json"
    LocalMonitorConfigDir "/etc/collectd/monitor/local/"
  </Module>
</Plugin>

• Configure available recovery commands and connection settings at /etc/collectd/monitor/config.json
  • Copy conf/collectd/monitor/config.json to /etc/collectd/monitor/config.json
  • Set permission of the file because it may include credentials chmod 600 /etc/collectd/monitor/config.json
  • Edit it to define available recovery commands and to connect to MQTT broker (if you use remote monitoring feature)
• If you use local monitoring feature, put additional config files written in Lua to /etc/collectd/monitor/local/ with the extension ".lua". See conf/collectd/monitor/local/example.lua for examples.

Remote command feature

Steps to test remote command feature

• Enable collectd/monitor/remote.lua in your collectd.conf
• Start collectd daemon
• Execute send-command.lua like the following example. It will send a command and receive a result:

$ luajit /usr/local/share/lua/5.1/collectd/monitor/send-command.lua \
  hello \
  exec \
  --host 192.168.xxx.xxx \
  --user test-sender \
  --password test-sender \
  --topic test-topic \
  --result-topic test-result-topic
Send command: {"timestamp":"2020-11-26T00:41:19Z","service":"hello","task_id":3126260400,"command":"exec"}
{ -- PUBREC{type=5, packet_id=2}
  packet_id = 2,
  type = 5,
  <metatable> = {
    __tostring = <function 1>
  }
}
Received a result: { -- PUBLISH{qos=2, packet_id=1, dup=false, type=3, payload="{\\"timestamp\\":\\"2020-11-26T00:41:19Z\\",\\"message\\":\\"Hello World!\\",\\"task_id\\":3126260400,\\"code\\":0}", topic="test-result-topic", retain=false}
  dup = false,
  packet_id = 1,
  payload = '{"timestamp":"2020-11-26T00:41:19Z","message":"Hello World!","task_id":3126260400,"code":0}',
  qos = 2,
  retain = false,
  topic = "test-result-topic",
  type = 3,
  <metatable> = {
    __tostring = <function 1>
  }
}

• See luajit ./collectd/monitor/send-command.lua --help and its source code for more details

Setting items of config.json

See conf/collectd/monitor/config.json for an example.

Key	Type	Contents
Host	string	Host name and port of MQTT broker (e.g. host or host:1883)
User	string	Username for authorization on MQTT broker
Password	string	Password for authorization on MQTT broker
Secure	boolean	Use TLS or not to connect to MQTT broker
CleanSession	boolean	MQTT's Clean Session flag
QoS	number	MQTT's QoS level (0, 1, 2)
CommandTopic	string	MQTT topic name for sending commands
CommandResultTopic	string	MQTT topic name for sending command results
Services	object	List of services to define recovery commands (See below)
LogDevice	string	Device to outout log (stdout or syslog)
LogLevel	string	Log level (fatal, error, warn, info, debug)

Each key of Services is a service name and value is a following object:

Key	Type	Contents
Commands	Object	List of recovery commands: Each key is a command name and value is a recovery command

The message format of a remote command

Remote command & command result messages are formated in JSON. Here is an example and member definitions of these messages:

Command message:

An example:

{
  "task_id": 3126260400,
  "timestamp": "2020-11-26T00:41:19Z",
  "service": "hello",
  "command": "exec"
}

Members:

Field	Type	Content
task_id	number	An unique task ID assigned by a command sender
timestamp	string	Timestamp of a command (ISO8601 UTC)
service	string	A service name defined in the config file specified by MonitorConfigPath
command	string	A command name defined in the config file specified by MonitorConfigPath

Command result message:

An example:

{
  "task_id":3126260400,
  "timestamp": "2020-11-26T00:41:19Z",
  "message": "Hello World!",
  "code": 0
}

Members:

Field	Type	Content
task_id	number	An unique task ID assigned by a command sender
timestamp	string	Timestamp of a command result (ISO8601 UTC)
message	string	Message of a command (STDOUT)
code	number	Exit status of a command

Sending collectd.conf feature

Steps to test sending collectd.conf feature

• Enable collectd/monitor/remote.lua in your collectd.conf
• Start collectd daemon
• Execute send-config.lua like the following example. It will send a config and receive a result:

$ ./test-send-config.sh \
  path/to/new/collectd.conf \
  --host 192.168.xxx.xxx \
  --user test-sender \
  --password test-sender \
  --topic test-topic \
  --result-topic test-result-topic
Send config: {"timestamp":"2021-01-03T04:45:09Z","task_id":3689797623,"config":"LoadPlugin cpu\n<LoadPlugin lua>\n\tGlobals true\n</LoadPlugin>\n..."}
{ -- PUBREC{type=5, packet_id=2}
  packet_id = 2,
  type = 5,
  <metatable> = {
    __tostring = <function 1>
  }
}
Received a result: { -- PUBLISH{qos=2, packet_id=1, dup=false, type=3, payload="{\\"timestamp\\":\\"2021-01-03T04:45:09Z\\",\\"message\\":\\"Succeeded to replace config.\\",\\"task_id\\":3689797623,\\"code\\":0}", topic="test-result-topic", retain=false}
  dup = false,
  packet_id = 1,
  payload = '{"timestamp":"2021-01-03T04:45:09Z","message":"Succeeded to replace config.","task_id":3689797623,"code":0}',
  qos = 2,
  retain = false,
  topic = "test-result-topic",
  type = 3,
  <metatable> = {
    __tostring = <function 1>
  }
}

• See luajit ./collectd/monitor/send-config.lua --help and its source code for more details

Configuring collectd.conf path

The path of collectd.conf can be configured by adding like the following setting to your config.json. If this setting isn't exist, lua-collectd-monitor tryies to find it automatically, but it may fail depending environment.

  "Services": {
    ...
    "collectd": {
      "ConfigPath" : "/path/to/collectd.conf"
    },
    ...
  }

The message format of sending collectd.conf

Message to sending collectd.conf & results are formated in JSON. Here is an example and member definitions of these messages:

A message to send collectd.conf

An example:

{
  "task_id": 3126260401,
  "timestamp": "2020-12-26T00:41:19Z",
  "config": "<Plugin>\ncpu</Plugin>..."
}

Members:

Field	Type	Content
task_id	number	An unique task ID assigned by a sender
timestamp	string	Timestamp of a message (ISO8601 UTC)
config	string	Content of new collectd.conf

Result message of sending collectd.conf:

An example:

{
  "task_id":3126260401,
  "timestamp": "2020-12-26T00:41:19Z",
  "message": "Succeeded to replace config.",
  "code": 0
}

Members:

Field	Type	Content
task_id	number	An unique task ID assigned by a sender
timestamp	string	Timestamp of a result (ISO8601 UTC)
message	string	A result message
code	number	A result code (See below)

Here is the defined result codes:

Code	Content
0	Succeeded
8192 (0x2000)	Another task is already running
8193 (0x2001)	Failed to write new config
8194 (0x2002)	New config is broken
8195 (0x2003)	Cannot stop collectd
8196 (0x2004)	pid file of collectd isn't removed
8197 (0x2005)	Failed to backup old collectd.conf
8198 (0x2006)	Failed to replace collectd.conf
8199 (0x2007)	Recovered by the old collectd.conf due to failing restart
8200 (0x2008)	Failed to restart and failed to recover by the old collect.conf
8201 (0x2009)	Cannot get new pid

Local monitoring feature

Steps to test local monitoring feature

• Use conf/collectd/collectd.conf.monitor-local-example as your collectd.conf
• Copy conf/collectd/monitor/config.json to /etc/collectd/monitor/config.json
  • Check the file to confirm pre-defined recovery commands and edit it if needed
• Copy conf/collectd/monitor/local/example.lua to /etc/collectd/monitor/local/
  • Check the file to confirm recovery conditions and edit it if needed
• Start collectd daemon
• See your syslog to confirm that a collectd's notification is emitted like this:

Notification: severity = OKAY, host = localhost, plugin = lua-collectd-monitor-local, type = /etc/collectd/monitor/local/example.lua::write::memory_free_is_under_10GB, message = {"message":"Hello World!","task_id":244078840,"code":0}

Configuration of recovery commands and trigger conditions

• Recovery commands are pre-defined in config.json as same as remote.lua plugin. Please see conf/collectd/monitor/config.json for an example.
• Trigger conditions are written in Lua. Please see conf/collectd/monitor/local/example.lua for an example.

Notification of command resutls

Results of commands are notified by collectd's Notification feature. You can receive them at remote hosts too by using collectd's network plugin.

The values of each fields in Notification is following:

Field	Content
severity	4 (NOTIF_OKAY): Success, 1 (NOTIF_FAILURE): Fail
host	Host name
plugin	lua-collectd-moitor-local (Fixed)
plugin_instance	None (Empty string)
type	A call back name which executed the command
type_instance	None (Empty string)
time	Timestamp (UNIX time)
message	Detail of the result (JSON format)

The contents of the message:

Field	Type	Content
task_id	number	An unique task ID
message	string	Message of a command (STDOUT)
code	number	Exit status of a command

Note that message will be omitted if it's too long because max length of message field of Notificatoin is 128.