iloire/watchmen

A simple node.js service monitor

watchmen: a service monitor for node.js

• What is watchmen?
• Screenshots
• Installation
• Running and stopping watchmen
• Development workflow
• Managing your node processes with pm2
• Managing processes with node-foreman
• Configuration
• Ping services
• Monitor plugins
• Storage providers
• Using fake data for development
• Running with docker
• Terraform scripts for digital-ocean
• Tests
• Debugging
• Contributing
• Style guide
• History
• Donations
• License

What is watchmen?

• watchmen monitors health (outages, uptime, response time warnings, avg. response time, etc) for your servers.
• ping types are pluggable through npm modules. At this time, http-head and http-contains are available. Read more about ping services and how to create one below.
• watchmen provides custom actions through plugins (console outpug, email notifications, etc).
• the code base aims to be simple and easy to understand and modify.

Screenshots

Installation

Requirements

Get redis from redis.io and install it.

Installing watchmen

Clone the repo by using

$ git clone [email protected]:iloire/watchmen.git

or

$ git clone https://github.com/iloire/watchmen.git

Then install the required dependencies using npm

$ cd watchmen
$ npm install

Running and stopping watchmen

Make sure you have redis-server in your PATH. Then you can run watchmen services:

$ redis-server redis.conf
$ node run-monitor-server.js
$ node run-web-server.js

Development workflow

Fetching bower dependencies and building static assets

$ npm run build

Dev build watch

$ npm run build:watch

Running tests

See below.

Managing your node processes with pm2

Install pm2:

$ npm install -g pm2

Configure env variables:

$ export WATCHMEN_WEB_PORT=8080

Run servers:

$ pm2 start run-monitor-server.js
$ pm2 start run-web-server.js

Server list:

$ pm2 list

Managing processes with node-foreman

node-foreman can be used to run the monitor and web server as an Upstart service. On Ubuntu systems, this allows the usage of service watchmen start.

Watchmen already include a Procfile so you can also manage with nf.

$ npm install -g foreman
$ nf start

To export as an Upstart script using the environment variables in a .env file:

$ PATH="/home/user/.nvm/versions/v5.1.0/bin:$PATH" nf export -o /etc/init -a watchmen

You can run this without the -o /etc/init flag and move the files to this directory (or the appropriate Upstart) directory yourself. Make sure you have the correct path to the node bin, you can find out with which node.

More documentation on node-foreman:

https://github.com/strongloop/node-foreman

Configuration

Config is set through env variables.

Have a look at the /config folder for more details, but the general parameters are:

export WATCHMEN_BASE_URL='http://watchmen.letsnode.com'
export WATCHMEN_WEB_PORT='8080'
export WATCHMEN_ADMINS='[email protected]'
export WATCHMEN_GOOGLE_ANALYTICS_ID='your-GA-ID'

Authorization settings (since 2.2.0)

Watchmen uses Google Auth through passportjs for authentication. If your google email is present in WATCHMEN_ADMINS env variable, you will be able to manage services.

Make sure you set the right hostname so the OAuth dance can be negociated correctly:

export WATCHMEN_BASE_URL='http://watchmen.letsnode.com/'

You will also need to set the Google client ID and secret using env variables accordingly. (Login into https://console.developers.google.com/ to create them first)

export WATCHMEN_GOOGLE_CLIENT_ID='<your key>'
export WATCHMEN_GOOGLE_CLIENT_SECRET='<your secret>'

Disabling authentication

If you want to disable authentication access and let anyone access and edit your services use:

export WATCHMEN_WEB_NO_AUTH='true'

Ping services

Embedded ping services

HTTP-HEAD

https://www.npmjs.com/package/watchmen-ping-http-head

HTTP-CONTAINS

https://www.npmjs.com/package/watchmen-ping-http-contains

Third party contributions

NightmareJS Plugin for Watchmen

Allows Nightmare scripts to be executed by a Watchmen instance https://www.npmjs.com/package/watchmen-ping-nightmare

Creating your own ping service

Ping services are npm modules with the 'watchmen-ping' prefix.

For example, if you want to create a smtp ping service:

a) create a watchmen-ping-smtp module and publish it. This is how a simple HTTP ping service looks like:

var request = require('request');

function PingService(){}

exports = module.exports = PingService;

PingService.prototype.ping = function(service, callback){
  var startTime = +new Date();
  request.get({ method: 'HEAD', uri: service.url }, function(error, response, body){
    callback(error, body, response, +new Date() - startTime);
  });
};

PingService.prototype.getDefaultOptions = function(){
  return {}; // there is not need for UI confi options for this ping service
}

b) npm install it in watchmen:

     npm install watchmen-ping-smtp

c) create a service that uses that ping service

Monitor plugins

AWS SES Notifications plugin (provided)

https://github.com/iloire/watchmen-plugin-aws-ses

Settings

export WATCHMEN_AWS_FROM='your@email'
export WATCHMEN_AWS_REGION='your AWS region'
export WATCHMEN_AWS_KEY='your AWS Key'
export WATCHMEN_AWS_SECRET='your AWS secret'

Nodemailer Notifications plugin (third party contribution)

https://www.npmjs.com/package/watchmen-plugin-nodemailer

Slack Notifications plugin (third party contribution)

https://www.npmjs.com/package/watchmen-plugin-slack

Console output plugin (provided)

https://github.com/iloire/watchmen-plugin-console

Creating your own custom plugin

A watchmen instance will be injected through your plugin constructor. Then you can subscribe to the desired events. Best is to show it through an example.

This what the console plugin looks like:

var colors = require('colors');
var moment = require('moment');

var eventHandlers = {

  /**
   * On a new outage
   * @param {Object} service
   * @param {Object} outage
   * @param {Object} outage.error check error
   * @param {number} outage.timestamp outage timestamp
   */

  onNewOutage: function (service, outage) {
    var errorMsg = service.name + ' down!'.red + '. Error: ' + JSON.stringify(outage.error).red;
    console.log(errorMsg);
  },

  /**
   * Failed ping on an existing outage
   * @param {Object} service
   * @param {Object} outage
   * @param {Object} outage.error check error
   * @param {number} outage.timestamp outage timestamp
   */

  onCurrentOutage: function (service, outage) {
    var errorMsg = service.name + ' is still down!'.red + '. Error: ' + JSON.stringify(outage.error).red;
    console.log(errorMsg);
  },

  /**
   * Failed check (it will be an outage or not according to service.failuresToBeOutage
   * @param {Object} service
   * @param {Object} data
   * @param {Object} data.error check error
   * @param {number} data.currentFailureCount number of consecutive check failures
   */

  onFailedCheck: function (service, data) {
    var errorMsg = service.name + ' check failed!'.red + '. Error: ' + JSON.stringify(data.error).red;
    console.log(errorMsg);
  },

  /**
   * Warning alert
   * @param {Object} service
   * @param {Object} data
   * @param {number} data.elapsedTime (ms)
   */

  onLatencyWarning: function (service, data) {
    var msg = service.name + ' latency warning'.yellow + '. Took: ' + (data.elapsedTime + ' ms.').yellow;
    console.log(msg);
  },

  /**
   * Service is back online
   * @param {Object} service
   * @param {Object} lastOutage
   * @param {Object} lastOutage.error
   * @param {number} lastOutage.timestamp (ms)
   */

  onServiceBack: function (service, lastOutage) {
    var duration = moment.duration(+new Date() - lastOutage.timestamp, 'seconds');
    console.log(service.name.white + ' is back'.green + '. Down for '.gray + duration.humanize().white);
  },

  /**
   * Service is responding correctly
   * @param {Object} service
   * @param {Object} data
   * @param {number} data.elapsedTime (ms)
   */

  onServiceOk: function (service, data) {
    var serviceOkMsg = service.name + ' responded ' + 'OK!'.green;
    var responseTimeMsg = data.elapsedTime + ' ms.';
    console.log(serviceOkMsg, responseTimeMsg.gray);
  }
};

function ConsolePlugin(watchmen) {
  watchmen.on('new-outage', eventHandlers.onNewOutage);
  watchmen.on('current-outage', eventHandlers.onCurrentOutage);
  watchmen.on('service-error', eventHandlers.onFailedCheck);

  watchmen.on('latency-warning', eventHandlers.onLatencyWarning);
  watchmen.on('service-back', eventHandlers.onServiceBack);
  watchmen.on('service-ok', eventHandlers.onServiceOk);
}

exports = module.exports = ConsolePlugin;

Storage providers

Redis

Data schema

service - set with service id's
service:latestOutages - latest outages for all services
service:<serviceId> - hashMap with service details
service:<serviceId>:outages:current - current outage for a service (if any)
service:<serviceId>:outages - sorted set with outages info
service:<serviceId>:latency - sorted set with latency info
service:<serviceId>:failurecount - number of consecutive pings failures (to determine if it is an outage)

Configuration

export WATCHMEN_REDIS_PORT_PRODUCTION=1216
export WATCHMEN_REDIS_DB_PRODUCTION=1

export WATCHMEN_REDIS_PORT_DEVELOPMENT=1216
export WATCHMEN_REDIS_DB_DEVELOPMENT=2

Using fake data for development

cd scripts
sh populate-dummy-data-120days.sh # will populate data for a 120 day period

or

sh populate-dummy-data-30days.sh

etc..

Running with docker

To run with docker, make sure you have docker-compose installed: https://docs.docker.com/compose/install/

Also, have a docker host running. A good way is using docker-machine with a local VM: https://docs.docker.com/machine/get-started/

Edit docker-compose.env and set up the configuration.

Then, to build and run Watchmen, run the following:

docker-compose build
docker-compose up

After this Watchmen webserver will be running and exposed in the port 3000 of the docker host.

To configure, please look docker-compose.yml and docker-compose.env.

Terraform scripts for digital-ocean

You can deploy watchmen in a Digital Ocean droplet in a matter of minutes.

1. Install terraform from terraform.io.
2. If you don't have one yet, create an account in Digital Ocean. Use this link to get an initial $10 credit so you can play around with it for free.
3. Open a terminal in terraform/digital-ocean
4. Fill up user-data.yml with your configuration. Don't forget to also setup DIGITALOCEAN_TOKEN and SSH_FINGERPRINT env variables (see apply.sh):
5. Run sh apply.sh to create your droplet.

Terraform will create a droplet based on Ubuntu with the necessary packages, compile nodejs from source, install a redis server, install and configure an nginx in front of watchmen, and setup and run watchmen from the latest master using pm2.

Tests

$ npm test

Test coverage

$ npm run coverage

Then check the coverage reports:

$ open coverage/lcov-report/lib/index.html

Debugging

watchmen uses debug

set DEBUG=*

Contributing

You can contribute by:

• Addressing one if the items on the TODO list or one of the open issues.
• Creating monitor plugins.
• Creating ping services.
• Reporting bugs.

Contributors

• Iván Loire
• Oden
• Tom Atkinson
• Martin Bučko
• Eric Elliott
• Dan Abramov
• Vaios Karampinis
• Emily Horsman
• Luís Bianchin
• Marcus Kida
• Adriano Emerick Cola

Style guide

Please use semantic commit messages

History

3.4.0

• Add non authenticated access
• Add redis-store for session persistence

3.3.1

• Fixed redis sorted set member uniqueness (zadd). Thanks Adriano Emerick Cola!
• Add terraform for digital ocean

3.3.0

• Add docker-compose

3.2.0

• Add collapsed nav-bar in mobile.
• Mobile friendly admin buttons.
• Move services menu to sidebar.
• Add support for node foreman.
• Request avatar with the current URL scheme.
• Fix package.json loading when using node >= 5.x
• Other minor fixes and optimisations.

3.1.0

• Only notify on sustained outages in-progress (new service.failureThreshold property).
• Introduce a new "service-error" event that gets triggered when a ping fails.

3.0.0

• Watchmen monitor has been refactored. There is not backwards compatibility with previous watchmen databases.
• Pluggable ping services as separate npm modules (http-head and http contains included).
• Plugins for watchmen monitor as separate npm modules (console and AWS SES Notifications included).
• Services are persisted in the database.
• UI panel to add/edit/delete/reset services.
• Store latency points. Latency charts.
• Restricted services to users (by email). Login with Google Auth to access those.

2.5.0

• Rewrite notification system (support for postmark and AWS-SES - it is easy to add new ones).
• Add 'alwaysAlertTo' to notifications.
• Refactor configuration files. IMPORTANT: Please update your configuration files if you are upgrading (host/service config is still the same)!
• Use postmark module instead of custom code for talking to postmark service.
• Add istanbul for test coverage.
• Fix: Cancel timeout to avoid hammering the server when the controller gets called multiple times.
• Add colors to server console output.

2.4.0

• Frontend revamp using angularjs.
• Client side pagination using ngTable.
• Client dependencies now managed by bower.
• Extract analytics ID to config.

2.3.0

• Use passport-google-oauth2 as Google authentication strategy.

2.2.0

• Added service/host authorization with passportjs and GoogleStrategy.

2.1.0

• Fix issue #7. Make load_services async so eventually services can be fetched form a database or remote server.

2.0.0

• Upgrade to Express 4 (requires Node 0.10 or later), hence bumping to 2.x.
• Bump ejs
• Remove dynamic helpers

1.1.1

• Persist table sorting options in localStorage.
• Improve styling and responsiveness.
• General code cleanup.
• Display date of oldest event stored in the database in details view.
• Bump redis, moment and ejs.
• Some other minor changes.

1.1.0

• Delete events older than a certain threshold (configurable in a per-host basis)
• Bump jQuery to 1.11.1
• Bump Handlebars to 2.0.0
• Bump bootstrap to 3.2.0
• Responsive design based on bootstrap

1.0.alpha1 Major changes and improvements

• Storages are now pluggable. redis storage is used by default but you can create your own : couchdb, mongodb, text file, etc (see lib/storage).
• Ping services are also pluggable now. So far you can use http and smtp (smtp is just checking tcp connection right now). You can create your own or improve the existent ones easily.
• Watchmen daemon now inherits from events.EventEmitter, so you can instanciate it and subscribe to the events of your choice (service_error, service_back, etc) to implement your custom logic (see server.js).
• Knockout.js has been removed. Watchmen uses handlebars now instead. Faster, simpler code, and avoids some client side memory leacks.
• Client side is using moment.js for rendering dates.
• Express.js routes now are handled on /routes
• Mocha is used for unit testing. Mocked storages and ping services are used.
• Configuration is now spread in separate files, under the /config directory
• Better reporting web interface. Uptime statistics. Outages count, warnings count.

0.9

• Major refactor, improved performance.
• Added tests and mocked objects for testing.
• Separate files for request, utils and watchmen library.

0.8

• Removed logging to file.
• Bug fixing when storing event. Needed to add port to redis key to make it unique.
• Added callback when sending email to registered problems in delivery.

0.7

• Targets node 0.6.x
• Added knockoutjs for view model binding.
• Auto async refresh main page.
• Filter by name in main page.
• Added counter (hosts up and down).
• UI Improvements.
• Tablesorter sorts status and time tags.
• Added Google Analytics.

0.6

• Added current status info (site is up or down) to database.
• Added icons to display status (disable, error or ok).
• TableSorter jQuery plugin orders by status by default.

0.5

• Added expiration time to event records.
• Stores avg response time for each url.
• Warns if response time > limit.
• Multiple recipients in notifications.
• Removed "retry_in" option. Watchmen works in a smarter way now.
• REDIS backend.
• Web UI to display reports (express.js app using REDIS backend).

0.4

• Be able to disable entries in config file at url level
• When site is back, displays and logs information about how long the site has been down.

0.3

• Logs "site down" and "site back up" messages to a file (logs in a different file per host)
• Fix bug when reading url_conf.attempts on site back.

0.2

• Allow POST method (for testing forms).
• Added Marak/colors.js to output success and error messages.
• Displays request duration time.
• Refactoring.

0.1

• First release.

TODO

• Use a beautiful template, like the Gentella Admin Bootstrap Theme.
• Regular expressions support in the http-contains plugin.
• Define a data expiration period (per service)
• Handle auth in ping services.

Third party attribution

• Bootstrap - http://getbootstrap.com/
• "Font Awesome by Dave Gandy - http://fortawesome.github.com/Font-Awesome"
• C3 charts - http://c3js.org/
• ngTable - http://ng-table.com/

(see package.json and bower.json for a complete list of libraries and dependencies)

Donations

You can buy me a beer by sending BTC at 1DQRybRW7cQtRq65qCxkpGNLX9BVjmfJaY.

Yeah, it turns out researchers have uncovered that beer release endorphins in our brain that we use as a reward and motivation for coding on open source projects.

License

Copyright (c) 2012 - 2015 Iván Loire

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

watchmen's Issues

Can't chdir to '/home/ec2-user/watchmen': No such file or directory

It's a fresh install and I am getting this while running
redis-server redis.conf

MemoryStore warning in production environment

run-web-server-23 (err): Warning: connect.session() MemoryStore is not
run-web-server-23 (err): designed for a production environment, as it will leak
run-web-server-23 (err): memory, and will not scale past a single process.

I am getting this warning when I use production environment. I hope things are ok?

scripts.js:18 Failed to parse x '1463399700000' to Date object

After setting up a site to monitor - viewing charts results in
scripts.js:18 Failed to parse x '1463399700000' to Date object

Is there lib update or any format settings for dates returned by service monitor?

docker-compose fails to build with getaddrinfo ENOTFOUND bower.herokuapp.com

Docker on CentOS 7:
docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch
docker-ce-17.03.2.ce-1.el7.centos.x86_64

Running docker-compose fails with:

bower ng-table#^0.5.4 invalid-meta for:/tmp/3b8e7371c202d3e4200f1b59b81eed3c/bower/e5920f7ab843ab19710708196914eb0d-48-s20uH7/bower.json
bower ng-table#^0.5.4 invalid-meta The "main" field cannot contain minified files
bower ng-table#^0.5.4 invalid-meta The "main" field cannot contain minified files
bower angular#~1 resolved https://github.com/angular/bower-angular.git#1.6.5
bower bootstrap#3.3.x extract archive.tar.gz
bower angular#>=1.0.0 <1.5.0 resolved https://github.com/angular/bower-angular.git#1.4.14
bower ng-table#^0.5.4 resolved https://github.com/esvit/ng-table.git#0.5.5
bower lodash#~3.8.0 resolved https://github.com/lodash/lodash.git#3.8.0
bower c3#~0.4.10 resolved https://github.com/masayuki0812/c3.git#0.4.14
bower moment#2.9.x resolved https://github.com/moment/moment.git#2.9.0
bower bootstrap#3.3.x resolved https://github.com/twbs/bootstrap.git#3.3.7
bower spin.js#~2.0.0 retry Download of https://github.com/fgnass/spin.js/archive/v2.0.2.tar.gz failed with ENOTFOUND, retrying in 2.0s
bower spin.js#~2.0.0 extract archive.tar.gz
bower spin.js#~2.0.0 mismatch Version declared in the json (2.0.1) is different than the resolved one (2.0.2)
bower d3#~3.5.0 ENOTFOUND Request to https://bower.herokuapp.com/packages/d3 failed: getaddrinfo ENOTFOUND bower.herokuapp.com bower.herokuapp.com:443
bower spin.js#~2.0.0 resolved https://github.com/fgnass/spin.js.git#2.0.2
ERROR: Service 'watchmenweb' failed to build: The command '/bin/sh -c set -x && npm install -g bower && npm install && bower install --allow-root' returned a non-zero code: 1

In another shell I exec into the build container while it is running and am able to manually download the files via wget and ping github.com & bower.herokuapp.com.

Question: Example of integration with hosted storage service

Hi

Is it possible to see an example of how to use a hosted service, like RedisToGo, CouchIris, or MongoHQ with WatchMen?

thanks

Add services using the UI

This may involve:

• Persist services in the database.
• Creating an UI to manage services. An opportunity to introduce a frontend framework? angularjs.
• Handle authorization.
• Restart/shutdown services accordingly after changes are made

Which OAuth Credentials in Google?

The docs say:

You will also need to set the Google client ID and secret using env variables accordingly. (Login into https://console.developers.google.com/ to create them first)

Can you give some more details on this? On Google there are several types of OAuth credentials (web browser / web server || access your data / access users' data) and a bunch of different parameters (origin URL || redirect URL).

I read in another issue that you need to enable the Google+ API, so I did that; then generated a couple of different credentials for OAuth. They all result in an error.

Store other metrics (e.g. latency)

It would be nice it WatchMen could store other basic metrics in additional from just a binary up/down state - e.g. latency/response time.

E.g. Uptime has something similar:

http://fzaninotto.github.io/uptime/

Add elapsedTime and timestamp to notitication callbacks

Hi

It would be very useful to have timestamp and elapsedTime in all notification event callbacks for more precise calculations.

Thanks

Is it possible to work without oauth ?

Hello.
watchmen looks like a greate solutions.

but i think you have a lack of documentation.

Is it possible to work without google oauth ?
Is it possible to login as an admin without using oauth ?
is it possible to add domain without web interface at all ?

thanks !

Add REST API to return latest outages

It could be something like

api-service-route.js

/services/outages

Parameters:

• since
• max-items

Considerations:

• Filter restricted services according to the request credentials.
• Make it efficient for large number of services.
• Check #37

Downtime: NaN seconds

I get this message in logs and emails instead of the real downtime.

Better outage notification emails

An UI to add hosts and services dynamically

With an UI, you can allow non technical person to add more tests (whiteout restart server)

Feature request: gimme generic OAuth please

Hi there 😄 great tool!

I do have one request through: is it possible to add support for generic OAuth providers please?

Webserver Restart to see updated stats?

I started the server and the web server. I then added some new hosts to the hosts.js file, shutdown and restarted the server.js, but the web server wasn't reporting any of the new statistics until I shutdown and restarted the web server. I didn't see anything obvious. Is there some delay in the webserver getting the data from the redis store? Or is it caching the service list?

Ability to comment on outages

Considerations

• Admin can comment on all outages.
• A logged user can comment on outages he has specific access to (restrictedTo property)
• It would be nice if comments would display on chart outages (on mouse hover)
• If #36 is implemented, add a parameter to opt in for comments to the API method.

SMTP not working at all?

If i configure a Host to be monitored over SMTP it says it is "down" but i am able to connect via telnet (port25)

my configuration is correct..i even see WatchMen connect at my SMTP-Server.

Pause monitoring

Thanks for the great app. It would be nice to have a Pause option in the Settings pull-down menu to stop monitoring a service

Feature request: Ability for services and plugins to add settings.

It would be great if services and plugins could add their own settings (API keys, default service settings, etc.), instead of having to rely on a config file.

What, ideally, should this look like?

Right now I am thinking a "Settings" button that shows up next to "Add New Service" only if there are any plugins that define settings.

On that page would either be tabs or sections for each service/plugin that defines any settings.

This could probably be very similar in implementation to #68 to allow for plugins to define new fields for the service page.

I'm thinking of taking this (and #68) on in the next few weeks if it would be a welcome feature. Thoughts?

Add authentication/authorisation using passportjs or similar.

Documentation improvements

Firstly, thanks for creating watchmen!

As a node noobie, I would have liked instructions on how to install node in the same way you link to redis. First I tried sudo apt-get install nodejs, but got silent errors when running "node run-monitor-server.js". Then I followed the 5.x instructions here and they worked great: https://github.com/nodesource/distributions#debinstall

Running this on Ubuntu 15.10: git clone [email protected]:iloire/watchmen.git
yields Cloning into 'watchmen'... Permission denied (publickey). I'm kind of a github-noobie as well, so maybe im supposed to connect my github-account key to my git installation or replace it with some username. Still.

The pm2 addition is great, I didn't know of that either =) But: "npm install -g pm2" required sudo for me at least. That's because of the way I installed nodejs 5.x, ending up in /usr/bin/ which is root area.

Lastly I have a question: I just wanted to demo this myself, fiddle around, but I miss some documentation on how to disable auth @ development. I'm using 8080 locally and I don't open that up for the web, so a no-auth-mode would be great! And does other auth-methods like basic auth work? I'm eventually planning to use this on a closed network, without access to the Internet.

Ps. Crossing my fingers for an updated docker-image =)

Any plans or possibility to use basic smtp to send emails?

Async Storage Provider Creation

I started creating a mongodb storage provider. The problem I'm having is that I can't do a clean setup since the creation of the storage happens sync for redis. For mongo I have to create a connection async, I can catch this later on but that means a lot of 'hacky' code to handle the async setup.

So what do you think, changing getStorageInstance to an async method?

New Ping Service

@iloire created a new ping service to allow for synthetic website monitoring.
https://www.npmjs.com/package/watchmen-ping-nightmare

It will allow to interact with a browser to login or get more data from a site

Nicer bootstrap template

Any takers? :)

Enhance http ping service

Ideas :

• http request headers
• basic authentification

Only notify on sustained outages

I recently setup WatchMen to monitor ~150 websites and web applications. With the default settings it is very chatty. 50-100 emails per hour. I modified the intervals and timeout to be more forgiving, but I am still getting about 3 outage emails per hour. When I go to the sites that are reported as being down, the sites are not down. I believe that my definition of an outage, is different than what WatchMen considers an outage.

I only want to know about sustained outages, and it seems that WatchMen does not have this capability at present. 1 timeout or connection reset is not indicative of a real outage, but multiple failed pings in sequence would be.

I've looked into the codebase, and it seems like 1 way to do this would be to create a new event called sustained-outage, with a matching callback onSustainedOutage. Then create a storage method which could tell us how many failed pings have occurred in a row. sustained-outage would be emitted when a configured number of pings failed. Then I'd modify the SES notification plugin to only send its new outage email during the onSustainedOutage callback.

Is there a better way? Am I missing the existence of this feature in WatchMen as it is today?

Etherpad Usage

We're considering using WatchMen to monitor Etherpad instances.

One thing I can't see from the docs is how we'd go about building out our own client tests.

We use a client library (etherpad-cli-client) to test pad connectivity. This goes way beyond a simple HTTP or TCP test. https://www.npmjs.com/package/etherpad-cli-client

I could hack support into WatchMen specifically for our use case but it appears that having sort sort of support for this would be a globally required thing? I was thinking a config such as this would be ideal..

{
  name:'Etherpad Beta Test Instance',
  host: 'beta.etherpad.org',
  port:443,
  ping_interval: one_minute,
  ping_service_name: 'https', 
  failed_ping_interval: one_minute,
  enabled: true, //enables/disables this host
  alert_to: ['[email protected]'], 
  warning_if_takes_more_than: 2000,
  remove_events_older_than_seconds : 60 * 60 * 24 * 10, 
  services : [
    {
      name : 'Pad Contents',
      method: 'getEtherpad',
      url : '/p/lskdjflksjdflkjsdlkfj',
      expected: {contains: 'some pad contents'}
    } ,
    {
      name : 'home page',
      method: 'get',
      url : '/',
      expected: {statuscode: 200, contains: 'Contact page'}
    }
  ]
} 

So I'm either missing something in the docs or this isn't possible yet with WatchMen? Nagios has this support by leveraging scripts, it works pretty well. I'd possibly suggest a npm powered plugin approach though..

Custom plugin requiring to alter views (service-edit)

Hey there,

Firstly, nice job so far with this project. I've been using it for few months now and, with except of few random crashes, it performed pretty decently.

Anyhow, for more prompt interventions while traveling, I needed these notifications via SMS as well.
So I've written a plugin to send SMS notifications via Twilio.
All's fine and dandy, except I cannot find an elegant way to extend the webserver/views/service-edit.html view from within the plugin. The plugin should be able to insert a new field per service (ex. "Alert to phone numbers") to be used by the new plugin.

Do you have any suggestions on how could I accomplish this elegantly? I'd like to publish the plugin afterwards, maybe there are others interested in it, and I can't do it without this step.

Thanks,
Cheers!

Notify with email question

I'm using version 3.3.1. Have to code a little to support email notification and disable login.

Seems disable login is done now. Then how is going with the email part?

Thanks, :)

Can you explain how to include external service lib

How to include external service library for notifications?

Disabling Auth in Docker files not working

I've recently worked on getting this stood up using the included Docker files leveraging docker-compose. In the docker-compose.env I added the following:

WATCHMEN_WEB_NO_AUTH='true'

However, when everything comes up, I'm still being asked to "Sign in with Google"

Edit: What is interesting is i can visit /services/add, but I can't hit the "add" button.

Combine server.js and app.js

It would be great to have these combined. This way it can run on the heroku free tier for example.

I've done a quick proof of concept, which is quite a hack. It does work, but there is a circular reference in 'host' in watchmen.get_hosts(redis, config.hosts, function (err, hosts){. I'm just deleting the reference.

Thanks, great app.

node 0.8.x support

it can't support in node 0.8.x..

Non HTTPS gravatar

I have put watchmen on https and it shows me warning that my connection is not fully encrypted. gravatar is comging from http.

Who watches the watchmen?

Pagination

With the amount of servers I tested your code with, I got safari to his knees (1.65GB ram, 100% CPU)

is it possible to ping ips thats not running any services?

Watchmen Down.

I'm using the docker version of watchmen. Love it! But every few months, watchmen seems to have a problem. The service begins marking almost all sites down (currently have 12) when they are not. They are marked active again after about a minute (interval check) only to be then remarked as down shortly thereafter. I generally just reboot the server but today, that didn't do it. Can I get you a log or anything else that might help both of us?! Issue is active as we speak.

Feature request: Add tagging, filter view by tag(s), up/downtime by tag(s)

Hi there,

We'd find being able to tag really useful for when monitoring different code platforms or data centre providers to identify any patterns where a particular platform/provider has more outages than the others.

We've previously looked at the node/uptime project which did this in a very nice way but seems to be all but abandoned.

Regards,
Richard

Make services async

The way you coded this you need to specify resources in a file. It would be useful to do http calls to get a dynamic list of servers to ping.

Google auth internal server error

Hi, I set up the credentials for GA but get a 500 when performing the authentication.

Redirect URLs setup correctly.

InternalOAuthError: failed to fetch user profile
at /var/www/watchmen/node_modules/passport-google-oauth2/lib/oauth2.js:92:28
at passBackControl (/var/www/watchmen/node_modules/passport-google-oauth2/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:123:9)
at IncomingMessage. (/var/www/watchmen/node_modules/passport-google-oauth2/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:143:7)
at IncomingMessage.emit (events.js:129:20)
at _stream_readable.js:908:16
at process._tickCallback (node.js:355:11)

Seen this before?

Admin settings dropdown menu automatically closes down

If we click on the admin settings icon the drop-down menu appears and disappears.

Noticed that this happens only when the DOM is rerendered after every service call (/api/report/services).

Whenever there are new services It would cause ngRepeat to remove all elements of existing services and create them again, which might be expensive. That means a lot of DOM operations.

Track by to the rescue in Angular 1.2 a new addition was made to the syntax of ngRepeat: the amazingly awesome track by clause. It allows you to specify your own key for ngRepeat to identify objects by, instead of just generating unique IDs.

Required change
ng-repeat="row in $data | filter:serviceFilter track by row.service.id"

Is there any way to develop without gulp-building?

I find that I need to build the javascript files after I modified them. And then add a script resource location "app.js" ,in the 'index.html' file. like this:

  <script src="<%=baseUrl%>build/vendor.js"></script>
  <script src="<%=baseUrl%>build/app.js"></script>

So it can finally works. I wonder know, if there are some ways to allow me to develop without building.
Seems include all raw js file is not a good idea.

Decouple front end code from backend code

This would be way nicer with a API and front end that feeds from that API

Plugin for Staytus

Hey there!

I created a plugin for updating services on a Staytus installation!
https://github.com/lookapanda/watchmen-plugin-staytus

More tests in service.expected

It can be useful to add :

• multiple contains
• multiple checks in HTTP response header

How use http-contain

I need use string: 502 Bad Gateway

This error in nginx.

How monitoring this?

InternalOAuthError

Running docker-composer filled in the google dev stuff tested and gave me 500 internal error

watchmenweb_1     | InternalOAuthError: failed to fetch user profile
watchmenweb_1     |     at /watchmen/node_modules/passport-google-oauth2/lib/oauth2.js:92:28
watchmenweb_1     |     at passBackControl (/watchmen/node_modules/passport-google-oauth2/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:123:9)
watchmenweb_1     |     at IncomingMessage.<anonymous> (/watchmen/node_modules/passport-google-oauth2/node_modules/passport-oauth2/node_modules/oauth/lib/oauth2.js:143:7)
watchmenweb_1     |     at emitNone (events.js:72:20)
watchmenweb_1     |     at IncomingMessage.emit (events.js:166:7)
watchmenweb_1     |     at endReadableNT (_stream_readable.js:913:12)
watchmenweb_1     |     at nextTickCallbackWith2Args (node.js:442:9)
watchmenweb_1     |     at process._tickCallback (node.js:356:17)