valeriansaliou/vigil

🚦 Microservices Status Page. Monitors a distributed infrastructure and sends alerts (Slack, SMS, etc.).

Vigil

Microservices Status Page. Monitors a distributed infrastructure and sends alerts (Slack, SMS, etc.).

Vigil is an open-source Status Page you can host on your infrastructure, used to monitor all your servers and apps, and visible to your users (on a domain of your choice, eg. status.example.com).

It is useful in microservices contexts to monitor both apps and backends. If a node goes down in your infrastructure, you receive a status change notification in a Slack channel, Email, Twilio SMS or/and XMPP.

Tested at Rust version: rustc 1.71.1 (eb26296b5 2023-08-03)

🇭🇺 Crafted in Budapest, Hungary.

👉 See a live demo of Vigil on Crisp Status Page.

📰 The Vigil project was announced in a post on my personal journal.

Who uses it?

Crisp	Meilisearch	miragespace	Redsmin	Image-Charts	Pikomit	Notice	Bareconnect

👋 You use Vigil and you want to be listed there? Contact me.

Features

• Monitors your infrastructure services automatically
• Notifies you when a service gets down or gets back up via a configured channel:
  • Email
  • Twilio (SMS)
  • Slack
  • Zulip
  • Telegram
  • Pushover
  • Gotify
  • XMPP
  • Matrix
  • Cisco Webex
  • Webhook
• Generates a status page, that you can host on your domain for your public users (eg. https://status.example.com)
• Allows publishing announcements, eg. let your users know that a planned maintenance is upcoming

How does it work?

Vigil monitors all your infrastructure services. You first need to configure target services to be monitored, and then Vigil does the rest for you.

There are three kinds of services Vigil can monitor:

• HTTP / TCP / ICMP services: Vigil frequently probes an HTTP, TCP or ICMP target and checks for reachability
• Application services: Install the Vigil Reporter library eg. on your NodeJS app and get reports when your app gets down, as well as when the host server system is overloaded
• Local services: Install a slave Vigil Local daemon to monitor services that cannot be reached by the Vigil master server (eg. services that are on a different LAN)

It is recommended to configure Vigil, Vigil Reporter or Vigil Local to send frequent probe checks, as to ensure you are quickly notified when a service gets down (thus to reduce unexpected downtime on your services).

Hosted alternative to Vigil

Vigil needs to be hosted on your own systems, and maintained on your end. If you do not feel like managing yet another service, you may use Crisp Status instead.

Crisp Status is a direct port of Vigil to the Crisp customer support platform.

Crisp Status hosts your status page on Crisp systems, and is able to do what Vigil does (and even more!). Crisp Status is integrated to other Crisp products (eg. Crisp Chatbox & Crisp Helpdesk). It warns your users over chatbox and helpdesk if your status page reports as dead for an extended period of time.

As an example of a status page running Crisp Status, check out Enrich Status Page.

How to use it?

Installation

Vigil is built in Rust. To install it, either download a version from the Vigil releases page, use cargo install or pull the source code from master.

👉 Each release binary comes with an .asc signature file, which can be verified using @valeriansaliou GPG public key: 🔑valeriansaliou.gpg.pub.asc.

Install from packages:

Vigil provides pre-built packages for Debian-based systems (Debian, Ubuntu, etc.).

Important: Vigil only provides 64 bits packages targeting Debian 10, 11 & 12 for now (codenames: buster, bullseye & bookworm). You will still be able to use them on other Debian versions, as well as Ubuntu.

First, add the Vigil APT repository (eg. for Debian bookworm):

echo "deb [signed-by=/usr/share/keyrings/valeriansaliou_vigil.gpg] https://packagecloud.io/valeriansaliou/vigil/debian/ bookworm main" > /etc/apt/sources.list.d/valeriansaliou_vigil.list

curl -fsSL https://packagecloud.io/valeriansaliou/vigil/gpgkey | gpg --dearmor -o /usr/share/keyrings/valeriansaliou_vigil.gpg

apt-get update

Then, install the Vigil package:

apt-get install vigil

Then, edit the pre-filled Vigil configuration file:

nano /etc/vigil/vigil.cfg

Finally, restart Vigil:

service vigil restart

Install from Cargo:

If you prefer managing vigil via Rust's Cargo, install it directly via cargo install:

cargo install vigil-server

Ensure that your $PATH is properly configured to source the Crates binaries, and then run Vigil using the vigil command.

Install from source:

The last option is to pull the source code from Git and compile Vigil via cargo:

cargo build --release

You can find the built binaries in the ./target/release directory.

Install libssl-dev (ie. OpenSSL headers) and libstrophe-dev (ie. XMPP library headers; only if you need the XMPP notifier) before you compile Vigil. SSL dependencies are required for the HTTPS probes and email notifications.

Install from Docker Hub:

You might find it convenient to run Vigil via Docker. You can find the pre-built Vigil image on Docker Hub as valeriansaliou/vigil.

Pre-built Docker version may not be the latest version of Vigil available.

First, pull the valeriansaliou/vigil image:

docker pull valeriansaliou/vigil:v1.26.3

Then, seed it a configuration file and run it (replace /path/to/your/vigil/config.cfg with the path to your configuration file):

docker run -p 8080:8080 -v /path/to/your/vigil/config.cfg:/etc/vigil.cfg valeriansaliou/vigil:v1.26.3

In the configuration file, ensure that:

• server.inet is set to 0.0.0.0:8080 (this lets Vigil be reached from outside the container)
• assets.path is set to ./res/assets/ (this refers to an internal path in the container, as the assets are contained there)

Vigil will be reachable from http://localhost:8080.

Configuration

Use the sample config.cfg configuration file and adjust it to your own environment.

You can also use environment variables with string interpolation in your configuration file, eg. manager_token = ${VIGIL_MANAGER_TOKEN}.

Available configuration options are commented below, with allowed values:

[server]

• log_level (type: string, allowed: debug, info, warn, error, default: error) — Verbosity of logging, set it to error in production
• inet (type: string, allowed: IPv4 / IPv6 + port, default: [::1]:8080) — Host and TCP port the Vigil public status page should listen on
• workers (type: integer, allowed: any number, default: 4) — Number of workers for the Vigil public status page to run on
• manager_token (type: string, allowed: secret token, default: no default) — Manager secret token (ie. secret password)
• reporter_token (type: string, allowed: secret token, default: no default) — Reporter secret token (ie. secret password)

[assets]

• path (type: string, allowed: UNIX path, default: ./res/assets/) — Path to Vigil assets directory

[branding]

• page_title (type: string, allowed: any string, default: Status Page) — Status page title
• page_url (type: string, allowed: URL, no default) — Status page URL
• company_name (type: string, allowed: any string, no default) — Company name (ie. your company)
• icon_color (type: string, allowed: hexadecimal color code, no default) — Icon color (ie. your icon background color)
• icon_url (type: string, allowed: URL, no default) — Icon URL, the icon should be your squared logo, used as status page favicon (PNG format recommended)
• logo_color (type: string, allowed: hexadecimal color code, no default) — Logo color (ie. your logo primary color)
• logo_url (type: string, allowed: URL, no default) — Logo URL, the logo should be your full-width logo, used as status page header logo (SVG format recommended)
• website_url (type: string, allowed: URL, no default) — Website URL to be used in status page header
• support_url (type: string, allowed: URL, no default) — Support URL to be used in status page header (ie. where users can contact you if something is wrong)
• custom_html (type: string, allowed: HTML, default: empty) — Custom HTML to include in status page head (optional)

[metrics]

• poll_interval (type: integer, allowed: seconds, default: 120) — Interval for which to probe nodes in poll mode
• poll_retry (type: integer, allowed: seconds, default: 2) — Interval after which to try probe for a second time nodes in poll mode (only when the first check fails)
• poll_http_status_healthy_above (type: integer, allowed: HTTP status code, default: 200) — HTTP status above which poll checks to HTTP replicas reports as healthy
• poll_http_status_healthy_below (type: integer, allowed: HTTP status code, default: 400) — HTTP status under which poll checks to HTTP replicas reports as healthy
• poll_delay_dead (type: integer, allowed: seconds, default: 10) — Delay after which a node in poll mode is to be considered dead (ie. check response delay)
• poll_delay_sick (type: integer, allowed: seconds, default: 5) — Delay after which a node in poll mode is to be considered sick (ie. check response delay)
• poll_parallelism (type: integer, allowed: any number, default: 4) — Maximum number of poll threads to be ran simultaneously (in case you are monitoring a lot of nodes and/or slow-replying nodes, increasing parallelism will help)
• push_delay_dead (type: integer, allowed: seconds, default: 20) — Delay after which a node in push mode is to be considered dead (ie. time after which the node did not report)
• push_system_cpu_sick_above (type: float, allowed: system CPU loads, default: 0.90) — System load indice for CPU above which to consider a node in push mode sick (ie. UNIX system load)
• push_system_ram_sick_above (type: float, allowed: system RAM loads, default: 0.90) — System load indice for RAM above which to consider a node in push mode sick (ie. percent RAM used)
• script_interval (type: integer, allowed: seconds, default: 300) — Interval for which to probe nodes in script mode
• script_parallelism (type: integer, allowed: any number, default: 2) — Maximum number of script executor threads to be ran simultaneously (in case you are running a lot of scripts and/or long-running scripts, increasing parallelism will help)
• local_delay_dead (type: integer, allowed: seconds, default: 40) — Delay after which a node in local mode is to be considered dead (ie. time after which the node did not report)

[plugins]

[plugins.rabbitmq]

• api_url (type: string, allowed: URL, no default) — RabbitMQ API URL (ie. http://127.0.0.1:15672)
• auth_username (type: string, allowed: username, no default) — RabbitMQ API authentication username
• auth_password (type: string, allowed: password, no default) — RabbitMQ API authentication password
• virtualhost (type: string, allowed: virtual host, no default) — RabbitMQ virtual host hosting the queues to be monitored
• queue_ready_healthy_below (type: integer, allowed: any number, no default) — Maximum number of payloads in RabbitMQ queue with status ready to consider node healthy.
• queue_nack_healthy_below (type: integer, allowed: any number, no default) — Maximum number of payloads in RabbitMQ queue with status nack to consider node healthy.
• queue_ready_dead_above (type: integer, allowed: any number, no default) — Threshold on the number of payloads in RabbitMQ queue with status ready above which node should be considered dead (stalled queue)
• queue_nack_dead_above (type: integer, allowed: any number, no default) — Threshold on the number of payloads in RabbitMQ queue with status nack above which node should be considered dead (stalled queue)
• queue_loaded_retry_delay (type: integer, allowed: milliseconds, no default) — Re-check queue if it reports as loaded after delay; this avoids false-positives if your systems usually take a bit of time to process pending queue payloads (if any)

[notify]

• startup_notification (type: boolean, allowed: true, false, default: true) — Whether to send startup notification or not (stating that systems are healthy)
• reminder_interval (type: integer, allowed: seconds, no default) — Interval at which downtime reminder notifications should be sent (if any)
• reminder_backoff_function (type string, allowed: none, linear, square, cubic, default: none) — If enabled, the downtime reminder interval will get larger as reminders are sent. The value will be reminder_interval × pow(N, x) with N being the number of reminders sent since the service went down, and x being the specified growth factor.
• reminder_backoff_limit (type: integer, allowed: any number, default: 3) — Maximum value for the downtime reminder backoff counter (if a backoff function is enabled).

[notify.email]

• to (type: string, allowed: email address, no default) — Email address to which to send emails
• from (type: string, allowed: email address, no default) — Email address from which to send emails
• smtp_host (type: string, allowed: hostname, IPv4, IPv6, default: localhost) — SMTP host to connect to
• smtp_port (type: integer, allowed: TCP port, default: 587) — SMTP TCP port to connect to
• smtp_username (type: string, allowed: any string, no default) — SMTP username to use for authentication (if any)
• smtp_password (type: string, allowed: any string, no default) — SMTP password to use for authentication (if any)
• smtp_encrypt (type: boolean, allowed: true, false, default: true) — Whether to encrypt SMTP connection with STARTTLS or not
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send emails only for downtime reminders or everytime

[notify.twilio]

• to (type: array[string], allowed: phone numbers, no default) — List of phone numbers to which to send text messages
• service_sid (type: string, allowed: any string, no default) — Twilio service identifier (ie. Service Sid)
• account_sid (type: string, allowed: any string, no default) — Twilio account identifier (ie. Account Sid)
• auth_token (type: string, allowed: any string, no default) — Twilio authentication token (ie. Auth Token)
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send text messages only for downtime reminders or everytime

[notify.slack]

• hook_url (type: string, allowed: URL, no default) — Slack hook URL (ie. https://hooks.slack.com/[..])
• mention_channel (type: boolean, allowed: true, false, default: false) — Whether to mention channel when sending Slack messages (using @channel, which is handy to receive a high-priority notification)
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send Slack messages only for downtime reminders or everytime

[notify.zulip]

• bot_email (type: string, allowed: any string, no default) — The bot mail address as given by the Zulip interface
• bot_api_key (type: string, allowed: any string, no default) — The bot API key as given by the Zulip interface
• channel (type: string, allowed: any string, no default) — The name of the channel to send notifications to
• api_url (type: string, allowed: URL, no default) — The API endpoint url (eg. https://domain.zulipchat.com/api/v1/)
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send messages only for downtime reminders or everytime

[notify.telegram]

• bot_token (type: string, allowed: any strings, no default) — Telegram bot token
• chat_id (type: string, allowed: any strings, no default) — Chat identifier where you want Vigil to send messages. Can be group chat identifier (eg. "@foo") or user chat identifier (eg. "123456789")

[notify.pushover]

• app_token (type: string, allowed: any string, no default) — Pushover application token (you need to create a dedicated Pushover application to get one)
• user_keys (type: array[string], allowed: any strings, no default) — List of Pushover user keys (ie. the keys of your Pushover target users for notifications)
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send Pushover notifications only for downtime reminders or everytime

[notify.gotify]

• app_url (type: string, allowed: URL, no default) - Gotify endpoint without trailing slash (eg. https://push.gotify.net)
• app_token (type: string, allowed: any string, no default) — Gotify application token
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send Gotify notifications only for downtime reminders or everytime

[notify.xmpp]

Notice: the XMPP notifier requires libstrophe (libstrophe-dev package on Debian) to be available when compiling Vigil, with the feature notifier-xmpp enabled upon Cargo build.

• to (type: string, allowed: Jabber ID, no default) — Jabber ID (JID) to which to send messages
• from (type: string, allowed: Jabber ID, no default) — Jabber ID (JID) from which to send messages
• xmpp_password (type: string, allowed: any string, no default) — XMPP account password to use for authentication
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send messages only for downtime reminders or everytime

[notify.matrix]

• homeserver_url (type: string, allowed: URL, no default) — Matrix server where the account has been created (eg. https://matrix.org)
• access_token (type: string, allowed: any string, no default) — Matrix access token from a previously created session (eg. Element Web access token)
• room_id (type: string, allowed: any string, no default) — Matrix room ID to which to send messages (eg. !abc123:matrix.org)
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send messages only for downtime reminders or everytime

[notify.webex]

• endpoint_url (type: string, allowed: URL, no default) — Webex endpoint URL (eg. https://webexapis.com/v1/messages)
• token (type: string, allowed: any string, no default) - Webex access token
• room_id (type: string, allowed: any string, no default) - Webex room ID to which to send messages (eg. Y2lzY29zcGFyazovL3VzL1JPT00vMmJmOD)
• reminders_only (type: boolean, allowed: true, false, default: false) — Whether to send messages only for downtime reminders or everytime

[notify.webhook]

• hook_url (type: string, allowed: URL, no default) — Web Hook URL (eg. https://domain.com/webhooks/[..])

[probe]

[[probe.service]]

• id (type: string, allowed: any unique lowercase string, no default) — Unique identifier of the probed service (not visible on the status page)
• label (type: string, allowed: any string, no default) — Name of the probed service (visible on the status page)

[[probe.service.node]]

• id (type: string, allowed: any unique lowercase string, no default) — Unique identifier of the probed service node (not visible on the status page)
• label (type: string, allowed: any string, no default) — Name of the probed service node (visible on the status page)
• mode (type: string, allowed: poll, push, script, local, no default) — Probe mode for this node (ie. poll is direct HTTP, TCP or ICMP poll to the URLs set in replicas, while push is for Vigil Reporter nodes, script is used to execute a shell script and local is for Vigil Local nodes)
• replicas (type: array[string], allowed: TCP, ICMP or HTTP URLs, default: empty) — Node replica URLs to be probed (only used if mode is poll)
• scripts (type: array[string], allowed: shell scripts as source code, default: empty) — Shell scripts to be executed on the system as a Vigil sub-process; they are handy to build custom probes (only used if mode is script)
• http_headers (type: map[string, string], allowed: any valid header name and value, default: empty) — HTTP headers to add to HTTP requests (eg. http_headers = { "Authorization" = "Bearer xxxx" })
• http_method (type string, allowed: GET, HEAD, POST, PUT, PATCH, no default) — HTTP method to use when polling the endpoint (omitting this will default to using HEAD or GET depending on the http_body_healthy_match configuration value)
• http_body (type string, allowed: any string, no default) — Body to send in the HTTP request when polling an endpoint (this only works if http_method is set to POST, PUT or PATCH)
• http_body_healthy_match (type: string, allowed: regular expressions, no default) — HTTP response body for which to report node replica as healthy (if the body does not match, the replica will be reported as dead, even if the status code check passes; the check uses a GET rather than the usual HEAD if this option is set)
• reveal_replica_name (type: boolean, allowed: true, false, default: false) — Whether to reveal replica name on public status page or not (this can be a security risk if a replica URL is to be kept secret)
• rabbitmq_queue (type: string, allowed: RabbitMQ queue names, no default) — RabbitMQ queue associated to node, which to check against for pending payloads via RabbitMQ API (this helps monitor unacked payloads accumulating in the queue)
• rabbitmq_queue_nack_healthy_below (type: integer, allowed: any number, no default) — Maximum number of payloads in RabbitMQ queue associated to node, with status nack to consider node healthy (this overrides the global plugins.rabbitmq.queue_nack_healthy_below)
• rabbitmq_queue_nack_dead_above (type: integer, allowed: any number, no default) — Threshold on the number of payloads in RabbitMQ queue associated to node, with status nack above which node should be considered dead (stalled queue, this overrides the global plugins.rabbitmq.queue_nack_dead_above)

Run Vigil

Vigil can be run as such:

./vigil -c /path/to/config.cfg

Usage recommendations

Consider the following recommendations when using Vigil:

• Vigil should be hosted on a safe, separate server. This server should run on a different physical machine and network than your monitored infrastructure servers.
• Make sure to whitelist the Vigil server public IP (both IPv4 and IPv6) on your monitored HTTP services; this applies if you use a bot protection service that challenges bot IPs, eg. Distil Networks or Cloudflare. Vigil will see the HTTP service as down if a bot challenge is raised.

What status variants look like?

Vigil has 3 status variants, either healthy (no issue ongoing), sick (services under high load) or dead (outage):

Healthy status variant

Sick status variant

Dead status variant

What do announcements look like?

Announcements can be published to let your users know about any planned maintenance, as well as your progress on resolving a downtime:

What do alerts look like?

When a monitored backend or app goes down in your infrastructure, Vigil can let you know by Slack, Twilio SMS, Email and XMPP:

You can also get nice realtime down and up alerts on your eg. iPhone and Apple Watch:

What do Webhook payloads look like?

If you are using the Webhook notifier in Vigil, you will receive a JSON-formatted payload with alert details upon any status change; plus reminders if notify.reminder_interval is configured.

Here is an example of a Webhook payload:

{
  "type": "changed",
  "status": "dead",
  "time": "08:58:28 UTC+0200",

  "replicas": [
    "web:core:tcp://edge-3.pool.net.crisp.chat:80"
  ],

  "page": {
    "title": "Crisp Status",
    "url": "https://status.crisp.chat/"
  }
}

Webhook notifications can be tested with eg. Webhook.site, before you integrate them to your custom endpoint.

You can use those Webhook payloads to create custom notifiers to anywhere. For instance, if you are using Microsoft Teams but not Slack, you may write a tiny PHP script that receives Webhooks from Vigil and forwards a notification to Microsoft Teams. This can be handy; while Vigil only implements convenience notifiers for some selected channels, the Webhook notifier allows you to extend beyond that.

How can I create script probes?

Vigil lets you create custom probes written as shell scripts, passed in the Vigil configuration as a list of scripts to be executed for a given node.

Those scripts can be used by advanced Vigil users when their monitoring use case requires scripting, ie. when push and poll probes are not enough.

The replica health should be returned by the script shell as return codes, where:

• rc=0: healthy
• rc=1: sick
• rc=2 and higher: dead

As scripts are usually multi-line, script contents can be passed as a literal string, enclosed between '''.

As an example, the following script configuration always return as sick:

scripts = [
  '''
  # Do some work...
  exit 1
  '''
]

Note that scripts are executed in a system shell ran by a Vigil-owned sub-process. Make sure that Vigil runs on an UNIX user with limited privileges. Running Vigil as root would let any configured script perform root-level actions on the machine, which is not recommended.

How can I integrate Vigil Reporter in my code?

Vigil Reporter is used to actively submit health information to Vigil from your apps. Apps are best monitored via application probes, which are able to report detailed system information such as CPU and RAM load. This lets Vigil show if an application host system is under high load.

Vigil Reporter Libraries

• NodeJS: node-vigil-reporter
• TypeScript: ts-vigil-reporter
• Python: py-vigil-reporter
• Golang: go-vigil-reporter
• Rust: rs-vigil-reporter
• Dart: dart-vigil-reporter
• C#: cs-vigil-reporter

👉 Cannot find the library for your programming language? Build your own and be referenced here! (contact me)

Vigil Reporter HTTP API

In case you need to manually report node metrics to the Vigil endpoint, use the following HTTP configuration (adjust it to yours).

👉 Read the Vigil Reporter HTTP API protocol specifications.

How can I administrate Vigil through Vigil Manager?

Vigil Manager can be used to perform administrative actions on a running Vigil instance. For instance, it can be used to publish public announcements.

Vigil Manager HTTP API

Vigil Manager can be interacted with over its dedicated HTTP API.

👉 Read the Vigil Manager HTTP API protocol specifications.

How can I monitor services on a different LAN using Vigil Local?

Vigil Local is an (optional) slave daemon that you can use to report internal service health to your Vigil-powered status page master server. It is designed to be used behind a firewall, and to monitor hosts bound to a local loop or LAN network, that are not available to your main Vigil status page.

Vigil Local monitors local poll and script replicas, and reports their status to Vigil on a periodic basis.

You can read more on Vigil Local on its repository, and follow the setup instructions.

🚸 Troubleshoot Issues

ICMP replicas always report as dead

On Linux systems, non-priviledge users cannot create raw sockets, which Vigil ICMP probing system requires. It means that, by default, all ICMP probe attempts will fail silently, as if the host being probed was always down.

This can easily be fixed by allowing Vigil to create raw sockets:

setcap 'cap_net_raw+ep' /bin/vigil

Note that HTTP and TCP probes do not require those raw socket capabilities.

🔥 Report A Vulnerability

If you find a vulnerability in Vigil, you are more than welcome to report it directly to @valeriansaliou by sending an encrypted email to [email protected]. Do not report vulnerabilities in public GitHub issues, as they may be exploited by malicious people to target production servers running an unpatched Vigil server.

⚠️ You must encrypt your email using @valeriansaliou GPG public key: 🔑valeriansaliou.gpg.pub.asc.

vigil's Issues

Service Discovery

Would be great to have support for Service Discovery (DNS SRV records) such as used by consul, Eureka, etc.

replicas (type: array[string], allowed: TCP or HTTP URLs, default: empty) — Node replica URLs to be probed (only used if mode is poll)

At present I can only list containerized services I have exposed on the load balancer which is only a small subset of services that are running as the rest talk to each other directly. Also because they are proxied there is only one replica for Vigil even though there might be a half dozen instances running so having count as 1 is not ideal.

Auto-throttle reminders for some channels (eg. Twilio SMS) after a certain downtime time

SMS can be quite "spammy" and cost a lot of in terms of API charges for long-duration downtimes (eg. 1h+).

Add a way to configure an auto-throttle function that would increase the time between each SMS notification.

gotify notify

Good platform for reports, push notification and etc...
https://gotify.net/

Deploying in ZEIT Now V2

I'm not able to deploy vigil on ZEIT Now. The now.json is:

{ "version": 2, "name": "vigil", "builds": [ { "src": "Cargo.toml", "use": "@now/rust", "config": { "rust": "nightly" } } ] }

and fails with code 101:
https://pastebin.com/yZMrBJxE

Any suggestions? Thanks!

RabbitMQ connexion

Hi,

I tried to use rabbitmq plugins, but the probe.service.node not show any replica

[[probe.service.node]]

id = "ai.ocr"
label = "OCR Queued Messages"
mode = "push"
rabbitmq_queue = "ocr_process_dev"

Thx

Error trying to do manual reporting

I get this error:

(INFO) - POST /reporter/quivr_web/api application/x-www-form-urlencoded:
(ERROR) - No matching routes for POST /reporter/quivr_web/api application/x-www-form-urlencoded.

When i try running this command:

wget --post-file=./api_request localhost:8080/reporter/quivr_web/api/

where the api_request file contains:

 {
  "replica": "www.quivr.be",
  "interval": 30,
  "load": {
    "cpu": 0.30,
    "ram": 0.80
  }
}

The probe "quivr_web" and it's node "api" are both defined in the vigil config.

[probe]

[[probe.service]]

id = "quivr_web"
label = "Quivr website"

[[probe.service.node]]

id = "api"
label = "quivr api"
mode = "push"

What am I doing wrong/how can I fix this?

Do i need the rabbitmq plugin to use manual reporting?

[POLL] What should I build next for Vigil?

Hello!

If anyone has quick feature suggestions or lacks a feature in Vigil, please use this thread to submit your ideas.

Getting errors when trying to build

Hi, I am having 2 errors when trying to build with cargo, I am using rust nightly-2018-12-14 as instructed.

The errors are:

- imports can only refer to extern crate names passed with `--extern` on stable channel (see issue #53130)

- use of unstable library feature 'duration_as_u128' (see issue #50202)

Am I doing something wrong? (I have never used cargo so I don't really know how to troubleshoot, but if you need any more info, let me know)

If RabbitMQ queue is full, check again 1-2 seconds later

Check if it's not a false positive.

Vigil reporting incorrect outage

I'm trying this out locally to replace our limited home-spun solution.

I have it working fine for a system which returns a JSON body, but another one which simply returns "ok" is not behaving.

The config that fails is:

[[probe.service.node]]
id = "block.veoo.com"
label = "block.veoo.com access"
mode = "poll"
replicas = ["https://block.veoo.com/health_check"]

This URL returns with GET:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Content-Length: 2
Date: Tue, 29 Jan 2019 23:21:46 GMT
Connection: keep-alive

ok

So I'm not sure why it's reporting as failed. Any ideas?

Using Docker build, prober not receiving correct response

Running vigil using

docker run --rm -p 6080:8080 -v $(pwd)/vigil.cfg:/etc/vigil.cfg valeriansaliou/vigil:v1.9.0

gets following result

(DEBUG) - will probe replica: HTTPS("https://status.crisp.chat/robots.txt") with retry count: 2
(DEBUG) - prober poll will fire for http target: https://status.crisp.chat/robots.txt?1560398131
(DEBUG) - resolving host="status.crisp.chat"
(DEBUG) - connecting to 178.62.224.146:443
(DEBUG) - adding I/O source: 4194304
(DEBUG) - scheduling Write for: 0
(DEBUG) - connected to Some(V4(178.62.224.146:443))
(DEBUG) - scheduling Read for: 0
(DEBUG) - scheduling Read for: 0
(DEBUG) - dropping I/O source: 0
(DEBUG) - prober poll result was not received for url: https://status.crisp.chat/robots.txt?1560398131

this is my vigil.cfg

# Vigil
# Microservices Status Page
# Configuration file
# Example: https://github.com/valeriansaliou/vigil/blob/master/config.cfg


[server]

log_level = "debug"
inet = "0.0.0.0:8080"
workers = 4
reporter_token = "REPLACE_THIS_WITH_A_SECRET_KEY"

[assets]

path = "./res/assets/"

[branding]

page_title = "Crisp Status"
page_url = "https://status.crisp.chat/"
company_name = "Crisp IM SARL"
icon_color = "#3C82E7"
icon_url = "https://valeriansaliou.github.io/vigil/images/crisp-icon.png"
logo_color = "#3C82E7"
logo_url = "https://valeriansaliou.github.io/vigil/images/crisp-logo.svg"
website_url = "https://crisp.chat/"
support_url = "mailto:[email protected]"
custom_html = ""

[metrics]

poll_interval = 30
poll_retry = 2

poll_http_status_healthy_above = 200
poll_http_status_healthy_below = 400

poll_delay_dead = 20
poll_delay_sick = 10

push_delay_dead = 20

push_system_cpu_sick_above = 0.90
push_system_ram_sick_above = 0.90

[plugins]

[plugins.rabbitmq]

api_url = "http://127.0.0.1:15672"
auth_username = "rabbitmq-administrator"
auth_password = "RABBITMQ_ADMIN_PASSWORD"
virtualhost = "crisp"

queue_ready_healthy_below = 500
queue_nack_healthy_below = 100
queue_loaded_retry_delay = 500

[notify]

reminder_interval = 300

[notify.email]

from = "[email protected]"
to = "[email protected]"

smtp_host = "localhost"
smtp_port = 587
smtp_username = "user-access"
smtp_password = "user-password"
smtp_encrypt = false

[notify.twilio]

to = [
  "+336xxxxxxx",
  "+337xxxxxxx"
]

service_sid = "service-sid"
account_sid = "account-sid"
auth_token = "auth-token"

reminders_only = true

[notify.slack]

hook_url = "https://hooks.slack.com/services/xxxx"

[notify.xmpp]

from = "[email protected]"
to = "[email protected]"

xmpp_password = "xmpp-password"

[probe]

[[probe.service]]

id = "web"
label = "Web nodes"

[[probe.service.node]]

id = "status"
label = "Access to status page"
mode = "poll"
replicas = ["https://status.crisp.chat/robots.txt"]
http_body_healthy_match = "User-agent:.*"

RSS Feed

I usually use the FEED (as a user) to monitor 3rd party servers on Slack!

Install errors

getting this error when i try to install

[root@status ~]# cargo install vigil-server
    Updating registry `https://github.com/rust-lang/crates.io-index`
  Installing vigil-server v1.1.2
   Compiling serde v1.0.27
error: the optimizations s or z are only accepted on the nightly compiler

error: failed to compile `vigil-server v1.1.2`, intermediate artifacts can be found at `/tmp/cargo-install.P2pvoDcsgJnE`

Caused by:
  Could not compile `serde`.

To learn more, run the command again with --verbose.
[root@status ~]#

Disk usage reporting

Disk space usage seems to be a really basic metric to probe.

Some of services are often victims of full disk and that risk could be mitigated by allowing Vigil to report such situations as unhealthy above some percentage of disk usage.

Probably worth handling that?

Auto-Refresh when general status changes

Fetch & update the page DOM via JS w/o performing a full page reload; automatically when the status changes.

Poll the current status every 30s from Vigil (get the status code only), and request a full page refresh if the status changed relative to current status.

Issue with simple https endpoint

Hello !

The endpoint below yield a 200 status code but I still get an error in vigil.

[[probe.service.node]]
id = "redsmin-api"
label = "API"
mode = "poll"
replicas = ["https://api.redsmin.com"]

Any idea on what is going wrong? :)

Customize request URL

fn proceed_replica_probe_http(url: &str, body_match: &Option<Regex>) -> bool {
    let url_bang = format!("{}?{}", url, time::now().to_timespec().sec);

Nice project, I have a couple of backend services that don't like having extraneous query params so I can't use them currently. I can understand you want to cache break the requests but this implementation prevents using requests with query params as well

IPv6 ICMP probing support

Due to the replacement of fastping-rs with ping, ICMPv6 support might have been dropped on some platforms.

A fork of the unmaintained ping library, and a full rework into a cleaner library would be much needed, adding support for ICMP IPv6 on all platforms.

Update year in copyright notice

I'm running the latest version of Vigil in Docker and the copyright notice at the bottom of my status page says 2018 instead of 2019. Is this something you could fix in the next version?

Show replica name in tooltip when hovering replica item (opt-in option)

First of all, thanks so much for vigil. I have tried so many alternatives and vigil is the only one that works consistently and works best.

I do have a feature request however. It would be nice to be able to mark or show somehow which replica's is which on the web page. Doesn't have to give away sensitive info, could be a label or anything really.

If this is by design let me know or already possible then just ignore me :D I think it may be useful sometimes to know in automated workflows what got added/removed or perhaps you don't want to alert but curious which replica has unusual latency (which I'm not sure why some replica's show latency and others none in that popup).

Prometheus metrics

The project looks really nice but my main alerting and monitoring solution is Prometheus. I was thinking about introducing Vigil as a monitoring component due to the fact that blackbox_exporter (Prometheus own exporter to monitor HTTP / TCP services is hard to setup, needs a lot of relabelling etc). So if vigil would be scraping and collecting info about which service is down or not and would expose that in Prometheus metrics format it would be great.

So I`m just curious if you have something like that in mind?

Installation Failed with Regex-Syntax

Hi there,

Operating System: Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux
Rustc Version: rustc 1.36.0-nightly (37ff5d388 2019-05-22)
Cargo Version: cargo 1.36.0-nightly (c4fcfb725 2019-05-15)

I'm trying to install Vigil using cargo install vigil-server, but the installation process fails with the following error:

   Compiling regex-syntax v0.6.6
error: failed to compile `vigil-server v1.9.0`, intermediate artifacts can be found at `/tmp/cargo-installWlu03T`

Caused by:
  Could not compile `regex-syntax`.

Caused by:
  process didn't exit successfully: `rustc --crate-name regex_syntax /root/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.6/src/lib.rs --color always --crate-type lib --emit=dep-info,metadata,link -C opt-level=s -C metadata=ccda77ecd63c6781 -C extra-filename=-ccda77ecd63c6781 --out-dir /tmp/cargo-installWlu03T/release/deps -L dependency=/tmp/cargo-installWlu03T/release/deps --extern ucd_util=/tmp/cargo-installWlu03T/release/deps/libucd_util-f229009622233513.rlib --cap-lints allow` (signal: 9, SIGKILL: kill)

Many thanks!

Kind regards,
Casper.

right: `true`: assets directory not found: "./res/assets/"', src/main.rs:143:5

I am getting this error when runing vigil.exe.

I built it with cargo using the latest nightly build of rust. There were no errors when building so I am not sure what is missing.

Show node CPU load, RAM used or latency on status page

When a node is shown as under high load (orange color), we don't show why it was reported as being loaded or slow.

If the node is a pull node, show in the hover tooltip:

1. TCP or HTTP latency (milliseconds)

If the node is a push node, show in the hover tooltip:

1. CPU load (percent)
2. RAM used (percent)
3. RabbitMQ queued + rejected packets (count) [if RabbitMQ plugin is active]

Vigil fails on compile when using Dockerfile

Hello @valeriansaliou ,

I'm trying to deploy vigil using Dockerfile copied from this repo but whatever I do it fails on openssl, for example:

Compiling aho-corasick v0.7.6                                                                                                                                                             
error: failed to run custom build command for `openssl v0.9.24`                                                                                                                              
process didn't exit successfully: `/app/target/release/build/openssl-e0c9d0620fbd0538/build-script-build` (exit code: 101)                                                                   
--- stderr                                                                                                                                                                                   
thread 'main' panicked at 'Unable to detect OpenSSL version', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.9.24/build.rs:16:14                                        
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.                                                                                                               
                                                                                                                                                                                             
warning: build failed, waiting for other jobs to finish...                                                                                                                                   
error: build failed 

The Dockerfile, Cargo.toml and Cargo.lock have been copied from this repo.

Do you have any idea how to fix that?

Thank you in advance.

Best regards,
Denis

Problem start with docker

Hello,
I have a small problem with start vigil with docker

I have this error: thread 'vigil-responder' panicked at 'Cannot assign requested address (os error 99)', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/rocket-0.4.2/src/error.rs:192:17

Thanks you

Timezone support

While I do try to stick to UTC as a timezone, I think Vigil would do a better job if it had the possibility of changing timezones.

As this isn't in the documentation, I assumed there's no possibility to change timezones.

A few ideas here:

• Timezone should change depending on the visitor (not sure if there's a standard way to do this, but perhaps https://stackoverflow.com/questions/6939685/get-client-time-zone-from-browser that should help).
• There should be a way to set up a different timezone for notifications (if the administrator lives in Beijing, they probably don't care about UTC notifications).
• Lastly, if timezone cannot be detected, it should resort to UTC (as opposed to resorting to the administrator's timezone).

Docker location

i just tried to install vigil on docker and i cannot find the location of where it intalled vigil too.

Support Typetalk?

Hi, how about adding notifier for Typetalk ?
If the suggestion is acceptable, I'd love to send PR. 🚀

Be able to call an endpoint to retrieve node information/health

It would be a nice to have an endpoint per probe/node that you can call and retrieve its health in a standard form.

Do you have something like this in mind? What do you think about this?

Local reporter to Vigil for LAN hosts behind a NAT

Provide LAN host-behind-NAT TCP probing capabilities; fork https://github.com/crisp-im/crisp-status-local and rebrand to Vigil + implement improvements from Vigil.

Persistent states across restarts

Hi 👋

Great project!

It would be great if the last state of a probe/replica is stored somewhere persistent to make sure it doesn't trigger an obsolete notification (which was already sent when the service actually went down) again upon next start.

High Availability / Clustered options

Is there any way to install this in a clustered way or have a High Availability option? (like running 2 different nodes)

where do I buy you a cup of coffee? ☕

Awesome project! What if the community wants to donate a cup of coffee (or beer), is there a link for that?

Custom probes (shell scripting)

Add a custom probe type, where a Vigil user can program a check script in Lua shell.

The script can be used to do custom network check, like POST data to an HTTP API and compare response body with a reference template. It can also be used to build a sequence of checks that must all pass for the node to be healthy.

The Lua interpreter is lightweight and can be embedded in Vigil with a low overhead on the total Vigil binary size.
(One alternative is to embed V8 and allow for JS scripts/workers, but it’s super heavy to embed; where Lua would be good-enough for custom probes)

Reason for change: the Lua runtime does not provide any HTTP library by default, and even if V8 does, it adds a ~35MB compiled overhead to the Vigil binary, which is unacceptable.

Support for discord webhook

Hi,

it would be awesome to have a notifier for discord.

Regards,
Alessandro

Disabled IPv6 on System, now geting panic on vigil-prober

I disabled IPv6 on my server, and after rebooting vigil does not want to start, giving the following error...

thread 'vigil-prober' panicked at 'failed to create icmp pinger: "Address family not supported by protocol (os error 97)"', src/libcore/result.rs:1192:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

My vigil config file:

# Vigil
# Microservices Status Page
# Configuration file
# Example: https://github.com/valeriansaliou/vigil/blob/master/config.cfg


[server]

log_level = "info"
inet = "0.0.0.0:8080"
workers = 4
reporter_token = "REPLACE_THIS_WITH_A_SECRET_KEY"

[assets]

path = "./res/assets/"

[branding]
...
[metrics]

poll_interval = 120
poll_retry = 2

poll_http_status_healthy_above = 200
poll_http_status_healthy_below = 400

poll_delay_dead = 30
poll_delay_sick = 10

push_delay_dead = 20

push_system_cpu_sick_above = 0.90
push_system_ram_sick_above = 0.90

[plugins]
...

[probe]

[[probe.service]]

id = "network"
label = "Network"

[[probe.service.node]]

id = "router"
label = "Router"
mode = "poll"

replicas = [
  "icmp://router.mydomain.com"
]

As far as I can tell there is no way in the configuration to specify only using IPv4 ICMP polling, so I am stuck at this point. Any help would be appreciated.

Monitor other RabbitMQ queue status parameters (eg. if queue is stopped, then report node as DEAD)

Caddy reverse proxy

I'm not very experienced with Caddy, because for now, I've always resorted to simply using Nginx, however I'm finding some difficulties when setting up Caddy and Vigil.

To check that it wasn't Caddy's fault, I have also tried to reverse proxy other webpages, and everything seemed to work perfectly fine.

And to be perfectly clear, serving Vigil over port 8080 (HTTP only) also works perfectly fine, both dialing the IP as well as using the hostname (although most web browsers refuse to connect because HSTS is enforced, I checked this using cURL which doesn't observe HSTS).

I use the following settings in vigil.cfg to serve on port 8080:

inet = "0.0.0.0:8080"

Initially, my Caddyfile looked like this:

status.hostname.com

reverse_proxy / {
	to http://localhost:8080
}

(I replaced my actual domain with hostname.com).

According to Caddy's documentation, it passes on all original headers unmodified to the upstream server. Since I suspected there may be an issue with the Host header, I modified the Caddyfile as so:

status.hostname.com

reverse_proxy / {
	to http://localhost:8080
	header_up Host localhost
	header_up -Upgrade-Insecure-Requests
	header_up -Pragma
	header_up -Cache-Control
}

However, it unfortunately still does not work, and I'm not sure at this point whether or not this is my fault, Caddy's, or Vigil's.

The relevant fragment of logs from Caddy looks like this:

(DEBUG) - Incoming stream
(DEBUG) - Request Line: Get AbsolutePath("/status/text/") Http11
(DEBUG) - Headers { Host: aa.bb.cc.dd:8080
, Connection: keep-alive
, User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
, DNT: 1
, Accept: */*
, Referer: http://aa.bb.cc.dd:8080/
, Accept-Encoding: gzip, deflate
, Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
, }
(INFO) - GET /status/text/:
(INFO) - Matched: GET /status/text (status_text)
(INFO) - Outcome: Success
(DEBUG) - writing head: Http11 Ok
(DEBUG) - headers [
Headers { Content-Type: text/plain; charset=utf-8
, Server: Rocket
, Content-Length: 7
, Date: Mon, 06 Apr 2020 17:57:41 GMT
, }]
(DEBUG) - write 7 bytes
(INFO) - Response succeeded.
(DEBUG) - keep_alive = true for xx.yy.zz.ww:58157
(DEBUG) - ioerror in keepalive loop = Custom { kind: UnexpectedEof, error: "end of stream before headers finished" }
(DEBUG) - keep_alive loop ending for xx.yy.zz.ww:58157

Install failed - Seems to be issue with lettre_email package.

I'm using the rust nightly:

rustc 1.39.0-nightly (521d78407 2019-08-25)

Get the following error during install (using cargo install vigil-server):

error[E0283]: type annotations required: cannot resolve `std::string::String: std::convert::AsRef<_>`
   --> /home/webadmin/.cargo/registry/src/github.com-1ecc6299db9ec823/lettre_email-0.8.3/src/lib.rs:387:70
    |
387 |         self.add_header(("Content-Type", format!("{}", content_type).as_ref()));
    |                                                                      ^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0283`.
error: Could not compile `lettre_email`.
warning: build failed, waiting for other jobs to finish...

cargo configuration breaks non-nightly builds

Using cargo 0.25.0 with both rustc 1.25.0 and rustc 1.24.1, the documented install instructions fail, in whatever deep dependency is tried first, because:

error: the optimizations s or z are only accepted on the nightly compiler

My rust knowledge is almost non-existent, but as far as I can tell, this is the vigil project's Cargo.toml having, in [profile.release], opt-level = "s".

Error installing from Docker

I have followed the README.me instructions:

sudo docker pull valeriansaliou/vigil:v1.9.0

The path to config.cfg is valid, but this error comes when executing the run command:

sudo docker run -p 8080:8080 -v /.../config.cfg -e RUST_BACKTRACE=1 valeriansaliou/vigil:v1.9.0

thread 'main' panicked at 'cannot find config file: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:70
             at src/libstd/sys_common/backtrace.rs:58
             at src/libstd/panicking.rs:200
   2: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:215
             at src/libstd/panicking.rs:478
   3: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:385
   4: rust_begin_unwind
             at src/libstd/panicking.rs:312
   5: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   6: core::result::unwrap_failed
   7: std::sync::once::Once::call_once::{{closure}}
   8: std::sync::once::Once::call_inner
             at src/libstd/sync/once.rs:387
   9: <vigil::APP_CONF as core::ops::deref::Deref>::deref
  10: vigil::main
  11: std::rt::lang_start::{{closure}}
  12: main
  13: __libc_start_main
  14: _start

Any idea what can be wrong?

Execute actions based on health changes

What do you think about having an option to execute actions based on health changes? It could be just as simple as executing a power shell script (let's say, to restart a service).

I guess for now there are ways to do this using slack integrations but could be nice to have it directly in vigil.

Auto refresh of the status page.

I moved to Vigil from another amazing monitorwall Monitoror as wanted to keep track of a lot of servers not only visually but with alerts too as those VMs having acceptance testing backed into those and exposed through a http endpoint. All seems great in case of Vigil except its status page doesn't refresh automatically like it happens in case of the Monitoror. Could the auto refresh be added here as well? Thanks!

DNS probe

Feature request, add ability to check availability and validity of responses of a DNS server.
Send predefined query to specified endpoint and validate that response contains expected record.

In my use case that would be SOA dn42 query sent to recursive resolver with the expectation to see one SOA record in response.

Vigil causes error messages to show up in Caddy's logs

I'm currently hosting my sites behind the reverse proxy Caddy and I'm using Vigil to poll the status of these sites as HTTP targets. Now, the problem is that whenever Vigil polls a site, the following error message shows up in Caddy's logs:

2018/12/19 23:14:51 Unsolicited response received on idle HTTP channel starting with "0\r\n\r\n"; err=<nil>

The error message appears to be harmless in the sense that both Vigil and Caddy work fine in spite of it, but I still think that it's something that should be looked into because something seems to be wrong with the HTTP request that Vigil sends out when it polls a site.

I found some information about this error message on this Golang issue.

Add ability to check probe output for arbitrary string value

One of the greatest features I see implemented by comparable services is the ability to check the output of the probe sent to an HTTP endpoint for arbitrary strings instead of just HTTP response codes. I would love have this feature implemented for Vigil.

Move from reqwest to http_req (lighter)

Move away from reqwest to http_req.

This has already been done in a Vigil alternative: https://github.com/crisp-im/crisp-status-local

Redesign default Status Page template

Modernize the template.

Golang reporter package

It would be important have go package (as it was made from JavaScript)