nrel/tracknodes

Tracknodes keeps a history of node state and comment changes. It allows system administrators of HPC systems to determine when nodes were down and discover trends such as recurring issues. Supports Torque, PBSpro and SLURM.

tracknodes

Description

Tracknodes keeps a history of node state and comment changes. It allows system administrators of HPC systems to determine when nodes were down and discover trends such as recurring issues. Supports Torque, PBSpro and has limited support for SLURM.

Installation

$ pip install tracknodes

or

$ easy_install tracknodes

Usage

Setup a cronjob on an admin node. This step is required for node state changes to be tracked.

$ crontab -u root -e
# Track Node State Every Minute
* * * * * (/usr/bin/tracknodes --update >/dev/null 2>&1)

Use the below command to see the history of node changes.

$ tracknodes
History of Nodes
=========
n101 | 2016-11-28 21:30:01 | online | ''
n101 | 2016-11-28 20:30:01 | offline,down | 'Hardware issue bad DIMM'
n092 | 2016-11-27 19:30:01 | online | ''
n092 | 2016-11-27 12:00:01 | offline | 'Hardware issue failed disk'
n021 | 2016-11-27 09:00:01 | online | ''
n021 | 2016-11-26 19:00:01 | offline,down | 'DIMM Configuration Error'
-- --

You can setup the configuration file for tracknodes to change the database location or the command to get node status. Use the below as an example.

$ cat /etc/tracknodes.conf
---
dbfile: "/opt/tracknodes.db"
cmd: "/opt/pbsnodes"

Tracknodes uses a sqlite database to store the node history, you can determine what database its using with the -v argument.

$ tracknodes -v
Resource Manager Detected as torque
cmd: /opt/pbsnodes
dbfile: ~/.tracknodes.db
...

For usage information you can use --help.

$ tracknodes --help
Usage: tracknodes [options]

Options:
  -h, --help            show this help message and exit
  -U, --update          Update Database From Current Node States
  -f DBFILE, --dbfile=DBFILE
                        Database File
  -c CMD, --cmd=CMD
                        Location of command to show node state, example: /opt/pbsnodes, /opt/sinfo
  -v, --verbose         Verbose Output

License

tracknodes is released under the GPLv3 License.

tracknodes's Issues

Add Support for PBSpro

The pbsnodes command for PBSpro does not have the -n option, it defaults with the -n option.

To detect PBSpro:
$ /opt/pbs/bin/pbsnodes —version
pbs_version =

If its PBSpro:
Use pbsnodes -l, instead of pbsnodes -nl.

Down nodes without a note are not being tracked correctly

A down node without a note does not show as failed in the database.

Add Support for SLURM

$ sinfo -dR
REASON USER TIMESTAMP NODELIST
broken ram root 2017-01-02T09:09:82 n010
DIMM failed root 2017-01-03T06:03:16 n020

pbspro is not being detected correctly

The tests passed with mock up output, but on a test cluster it fails to detect the resource manager as pbspro.

-David

Comment updates are not being commited to sqlite db afte run

$ tracknodes -U
History of Nodes

c3 | 2016-12-06 22:07:38 | offline,down | 'offline for testing'
c1 | 2016-12-06 21:56:20 | down | 'node down: communication closed'
c2 | 2016-12-06 21:56:20 | down | 'node down: communication closed'
c3 | 2016-12-06 21:56:20 | down | 'node down: communication closed'
c4 | 2016-12-06 21:56:20 | down | 'node down: communication closed'

$ tracknodes
History of Nodes

c1 | 2016-12-06 21:56:20 | down | 'node down: communication closed'
c2 | 2016-12-06 21:56:20 | down | 'node down: communication closed'
c3 | 2016-12-06 21:56:20 | down | 'node down: communication closed'
c4 | 2016-12-06 21:56:20 | down | 'node down: communication closed'

Make lib/tracknodes/tracknodes.py runnable and independent

For some environments it would be easier to just copy the python file rather than installing with pip.

if name == "main":
...
.....

nodes_cmd option does not work unless its in one of the predefined paths

tracknodes -c /opt/pbsnodes
Exception: Cannot find pbsnodes or sinfo in PATH.

python3 support?

Any chance of python3 support? If you need help with that I might be able to assit.

Improve unit test coverage for sqlite