This repository contains code used to collect monitoring metrics for Ganglia:
Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. […] It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. cf.
Send monitoring information to Ganglia with gmetric:
Program | Description |
---|---|
ganglia-monitor-infiniband | Collect Infiniband channel adapter metrics with perfquery |
ganglia-monitor-slurm | Collect jobs, node and scheduler statistics from Slurm |
Ganglia can be extended by Python and C/C++ modules. Modules are executed (in intervals) by gmond in contrast to data collected with gmetric. The Debian package ganglia-monitor-python provides the required environment to enable Python modules.
Module | Configuration | Description |
---|---|---|
infiniband.py | infiniband.pyconf | Read Infiniband host channel performance metrics from perfquery |
ipmi.py | ipmi.pyconf | Read the BMC sensors with ipmitool |
cvmfs_nioerr.py | cvmfs_nioerr.pyconf | Count I/O errors on CernVM-FS mounts |
Path | Description |
---|---|
/usr/lib/ganglia/python_modules |
Default directory for Python modules |
/etc/ganglia/conf.d/*.pyconf |
Module configuration files |
The module section of the configuration file requires following attributes:
- The name value matches the file name of the Python module’s
.py
file. - The language value for a Python module must be "python".
- Each param block must include a name and a value, which are passed to the
metric_init()
function of the module. The name of the param block represents the key that corresponds to the parameter value in the Python dictionary object. The value directive specifies the parameter value (always a string!).
The collection_group section must contain:
- The name_match uses PCRE regex matching to configure metrics.
- The title is displayed in the web-interface as name of the graph.
Following examples illustrates the configuration of a module called "infiniband":
modules {
module {
name = "infiniband"
language = "python"
param "interval" { value = 20 }
param "error_metrics" { value = yes }
}
}
collection_group {
collect_every = 15
time_threshold = 45
metric {
name_match = "infiniband_xmtdata_port"
title = "Infiniband Send Bytes"
}
[…]
}
Start developing new Python modules from ↴lib/python_modules/example.py
Check if the metric description is correctly used by gmond:
» gmond -c /etc/ganglia/gmond.conf -m | grep infini
infiniband_rcvconstrainterrors_port1 (module python_module)
infiniband_rcvswrelayerrors_port1 (module python_module)
infiniband_vl15dropped_port1 (module python_module)
infiniband_symbolerrors_port1 (module python_module)
infiniband_rcverrors_port1 (module python_module)
[…]
» gmond -c /etc/ganglia/gmond.conf -d 2 -f
[…]
Run gmond in foreground and debugging mode the see if everything works as expected.
Extension code for the Ganglia Web Frontend
File | Description |
---|---|
infiniband_report.json | Incoming and outgoing traffic on an Infiniband adapter in bytes per second |
slurm_job_states_report.json | Number of running/pending jobs in Slurm |
The file name must end with _report.json
. Graph configuration attributes are:
Attribute | Description |
---|---|
report_name | Name of the report that web UI uses |
title | Title of the report to show on a graph |
vertical_label | Y-axis description (optional) |
series | An array of metrics to use to compose a graph |
The attributes inside the series key contain:
Attribute | Description |
---|---|
metric | Name of a metric, such as load_one and cpu_system. If the metric doesn’t exist it will be skipped. |
color | A 6 hex-decimal color code, such as 000000 for black. |
label | Metric label, such as Load 1. |
type | Item type. It can be either line or stack. |
line_width | If type is set to line, this value will be used as a line width. If this value is not specified, it defaults to 2. If type is stack, it’s ignored even if set. |
Deploy custom composite graphs:
- Copy the JSON configuration file into the directory
/usr/share/ganglia-webfrontend/graph.d
on the server hosting the web frontend. - Enable a graph using the "Edit Optional Graphs" button in the user interface of the web frontend.
Create Debian package with the configuration in debian/
apt -y install build-essential debhelper devscripts dh-systemd
dch -i # adjust changelog if required
dpkg-buildpackage -b -us -uc -tc # build package
dpkg -c ../ganglia-monitor-slurm_*_all.deb # list package content
Copyright 2014-2016 Victor Penso
This is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.