gitter-badger/openwisp-monitoring

Work in progress

openwisp-monitoring

---

OpenWISP 2 monitoring module (Work in progress).

---

Table of Contents:

• Install Dependencies
• Setup (integrate in an existing Django project)
• Device Health Status
  • UNKNOWN
  • OK
  • PROBLEM
  • CRITICAL
• Settings
  • OPENWISP_MONITORING_SHORT_RETENTION_POLICY
  • OPENWISP_MONITORING_AUTO_PING
  • OPENWISP_MONITORING_AUTO_GRAPHS
  • OPENWISP_MONITORING_CRITICAL_DEVICE_METRICS
  • OPENWISP_MONITORING_HEALTH_STATUS_LABELS
  • OPENWISP_MONITORING_MANAGEMENT_IP_ONLY
  • OPENWISP_MONITORING_DEVICE_RECOVERY_DETECTION
• Signals
  • device_metrics_received
• Installing for development

---

Install Dependencies

We use InfluxDB to store metrics and Redis as celery broker (you can use a different broker if you want). The recommended way for development is running them using Docker so you will need to install docker and docker-compose beforehand.

In case you prefer not to use Docker you can install InfluxDB and Redis from your repositories, but keep in mind that the version packaged by your distribution may be different.

Install spatialite and sqlite:

sudo apt-get install sqlite3 libsqlite3-dev openssl libssl-dev
sudo apt-get install gdal-bin libproj-dev libgeos-dev libspatialite-dev

Optionally, install fping if you need to use the ping active check:

sudo apt install -y fping

Setup (integrate in an existing Django project)

Follow the setup instructions of openwisp-controller, then add the settings described below.

INSTALLED_APPS = [
    # django apps
    # openwisp2 admin theme (must be loaded here)
    'openwisp_utils.admin_theme',
    # all-auth
    'django.contrib.sites',
    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    'django_extensions',
    # openwisp2 modules
    'openwisp_users',
    'openwisp_controller.pki',
    'openwisp_controller.config',
    # monitoring
    'notifications',
    'openwisp_monitoring.monitoring',
    'openwisp_monitoring.device',
    'openwisp_monitoring.check',
    # admin
    'django.contrib.admin',
    'django.forms',
    # other dependencies ...
]

# Make sure you change them in production
INFLUXDB_USER = 'openwisp'
INFLUXDB_PASSWORD = 'openwisp'
INFLUXDB_DATABASE = 'openwisp2'

urls.py:

from django.conf import settings
from django.conf.urls import include, url
from django.contrib.staticfiles.urls import staticfiles_urlpatterns

from openwisp_utils.admin_theme.admin import admin, openwisp_admin

openwisp_admin()

urlpatterns = [
    url(r'^admin/', include(admin.site.urls)),
    url(r'', include('openwisp_controller.urls')),
    url(r'', include('openwisp_monitoring.urls')),
]

urlpatterns += staticfiles_urlpatterns()

Add apptemplates.Loader to template loaders:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(os.path.dirname(BASE_DIR), 'templates')],
        'OPTIONS': {
            'loaders': [
                'apptemplates.Loader',
                'django.template.loaders.filesystem.Loader',
                'django.template.loaders.app_directories.Loader',
                'openwisp_utils.loaders.DependencyLoader',
            ],
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    }
]

Configure caching (you may use a different cache storage if you want):

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://localhost/0',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        }
    }
}

SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
SESSION_CACHE_ALIAS = 'default'

Configure celery (you may use a different broker if you want):

# here we show how to configure celery with redis but you can
# use other brokers if you want, consult the celery docs
CELERY_BROKER_URL = 'redis://localhost/1'
CELERY_BEAT_SCHEDULE = {
    'run_checks': {
        'task': 'openwisp_monitoring.check.tasks.run_check',
        'schedule': timedelta(minutes=5),
    },
}

INSTALLED_APPS.append('djcelery_email')
EMAIL_BACKEND = 'djcelery_email.backends.CeleryEmailBackend'

If you decide to use redis (as shown in these examples), install the requierd python packages:

pip install redis django-redis

Device Health Status

The possible values for the health status field (DeviceMonitoring.status) are explained below.

UNKNOWN

Whenever a new device is created it will have UNKNOWN as it's default Heath Status.

It implies that the system doesn't know whether the device is reachable yet.

OK

Everything is working normally.

PROBLEM

One of the metrics has a value which is not in the expected range (threshold value crossed).

Example: CPU usage should be less than 90% but current value is at 95%.

CRITICAL

One of the metrics defined in OPENWISP_MONITORING_CRITICAL_DEVICE_METRICS has a value which is not in the expected range (threshold value crossed).

Example: ping is by default a critical metric which is expected to be always 1 (reachable).

Settings

OPENWISP_MONITORING_SHORT_RETENTION_POLICY

type:	str
default:	24h0m0s

The default retention policy used to store raw device data.

This data is only used to assess the recent status of devices, keeping it for a long time would not add much benefit and would cost a lot more in terms of disk space.

OPENWISP_MONITORING_AUTO_PING

type:	bool
default:	True

Whether ping checks are created automatically for devices.

OPENWISP_MONITORING_AUTO_GRAPHS

type:	list
default:	('traffic', 'wifi_clients', 'uptime', 'packet_loss', 'rtt')

Automatically created graphs.

OPENWISP_MONITORING_CRITICAL_DEVICE_METRICS

type:	list of dict objects
default:	[{'key': 'ping', 'field_name': 'reachable'}]

Device metrics that are considered critical: when a threshold related to one of this type of metric is crossed, the health status of the device related to the metric moves into CRITICAL.

By default, if devices are not reachable by ping they are flagged as CRITICAL.

OPENWISP_MONITORING_HEALTH_STATUS_LABELS

type:	dict
default:	{'unknown': 'unknown', 'ok': 'ok', 'problem': 'problem', 'critical': 'critical'}

This setting allows to change the health status labels, for example, if we want to use online instead of ok and offline instead of critical, you can use the following configuration:

OPENWISP_MONITORING_HEALTH_STATUS_LABELS = {
    'ok': 'online',
    'problem': 'problem',
    'critical': 'offline'
}

OPENWISP_MONITORING_MANAGEMENT_IP_ONLY

type:	bool
default:	True

By default, only the management IP will be used to perform active checks to the devices.

If the devices are connecting to your OpenWISP instance using a shared layer2 network, hence the OpenWSP server can reach the devices using the last_ip field, you can set this to False.

OPENWISP_MONITORING_DEVICE_RECOVERY_DETECTION

type:	bool
default:	True

The setting allows to detect recoveries as soon as they happen thus making the monitoring system faster.

Signals

device_metrics_received

Path: openwisp_monitoring.device.signals.device_metrics_received

Arguments:

• instance: instance of Device whose metrics have been received
• request: the HTTP request object

This signal is emitted when device metrics are received to the DeviceMetric view (only when using HTTP POST).

The signal is emitted just before a successful response is returned, it is not sent if the response was not successful.

Installing for development

Install your forked repo:

git clone git://github.com/<your_fork>/openwisp-monitoring
cd openwisp-monitoring/
python setup.py develop

Install test requirements:

pip install -r requirements-test.txt

Start Redis and InfluxDB using docker-compose:

docker-compose up -d

Create the Django database:

cd tests/
./manage.py migrate
./manage.py createsuperuser

Launch development server:

./manage.py runserver 0.0.0.0:8000

You can access the admin interface at http://127.0.0.1:8000/admin/.

Run celery and celery-beat with the following commands (separate terminal windows are needed):

# (cd tests)
celery -A openwisp2 worker -l info
celery -A openwisp2 beat -l info

Run tests with:

./runtests.py