widhalmt/check_test

A monitoring plugin strictily for testing

check_test

A monitoring plugin strictily for testing

While check_dummy fine for simple tests I needed a more sophisticated testing plugin. This works for Icinga (2) as well as every other tool using monitoring plugins like Shinken, Naemon, Nagios, etc.

Things which can not be done with check_dummy or check_random :

• Get several times an OK state and suddenly switching to Critical without restarting / reloading the monitoring software or injecting passive check results. Either by changing the configfile by hand or have it automatically change after a set amount of checks.
• Switch randomly between a predefined set of statuses
• Change state at a certain time (useful for testing planned downtimes)
• Change the state under certain controlled conditions and change it back later

Disclaimer

This plugin is very simple at the moment. I will expand it's possibilities as I need more functionality while running tests. If you need special cases: Patches are welcome.

Monitoring plugins should not use external config files or temporary files but since this is not to be used in production environments it should be ok to use them anyway.

Usage

Since the whole reason why I wanted this plugin was to be able to change the output whithout changing Icinga 2, it has only one commandline argument: Where to find a configuration file. This is what you change while Icinga 2 is running.

sudo -u icinga ./check_test -f [configfile]

Keep in mind that you can set the file to use on a per check basis. This way you can create several test cases in one setup.

Configuration file

The configuration file by now only supports setting the return code and the output of the plugin. More options are to come.

Example file:

STATUS=2
OUTPUT="Custom output"

Acknowledgements

• This plugin was created for running some tests for the upcoming Icinga 2 2.6 release
• I created this plugin as part of my work for Netways

check_test's Issues

Make configuration of flapping mode more clear and documentation

Right now the derivation of when to switch from one state to the other is not clearly documented.

Make config file a real commandline option

Right now, the script just takes the second option given, no matter if you use -f or not. This should be changed to a more sane way of defining the configuration file.

Add mode `flapping`

Idea: Set starttime and endtime and within flapping period return random return value. Maybe not completly random but alternating between two states.

I think it's better to use random instead of strictily alternating because this likely to be more like a real world example.

Allow setting of options via commandline for use on satellites

While a configuration file seems to be a good way to handle most cases, commandline arguments would be useful when using it on remote instances.

Check for status- and log directory

check if the directories are available. Creating them might not be a good idea.

Add a `scheduled` mode

Takes 4 configuration items:

• state A
• state B
• When to switch from state A to B
• When to switch back from state B to A

It would be nice if we could use Icinga timeperiod objects but maybe it would be easier to implement with "real" time.

Add `interval` mode

Uses 4 configuration items:

• State A
• State B
• After how many checks to switch to state A
• After how many checks to switch to state B

Decide if a common logfile or seperate logfiles would be better

Or even an option to choose between them.

Create a more meaningful output for `flapping` mode

Add --help as commandline option

The subject says it all.

Introduce `manual` as default mode

This mode just reads its exit code and output from the configuration file. If you want to change something, manually change the configuration - it will be read during the next run.

Provide a Plugin that reads the logfile

While there are more sophisticated solutions like Logstash, Graylog or check_logfiles this should be a very specific solution just for test cases.

I'm not quite sure how I'll handle more complicated tests but the goal is to end up with several Icinga services running different test cases and one (ore one for each test-service) service that shows whether the tests were successful. This could be a very powerful help in testing new versions.

Provide more default configuration for Icinga 2

At least a Service object, an User object and a Notification object.

Introduce temporary files

For saving check counts, etc.

Their name could be derived from the name of the used configuration file or from a "testname". The later could be used via commandline, too.

Use `STATUS` as status for `flapping` mode

Write a common logfile for all tests

Maybe into the default Icinga 2 log directory. Maybe make configurable if syslog should be used.

Use an easy to parse syntax for parsing with Logstash.

Provide notification script

This should write into a common logfile and provide when which notification was triggered.

Provide a commandline tool to generate configuration files

I'm still not sure if this is really helpful.

Find paths for configuration file and logfile matching Icinga 2's style