Giter Site home page Giter Site logo

sensu-docker's Introduction

Sensu

Dockerfiles for each component and Docker Compose file for setting up a Sensu server.

Installing Docker

Follow the instructions on the docker site for your platform.

I recommend using the Docker maintained repos for Ubuntu and do/will not support boot2docker on OSX

Installing Docker Compose

curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Clone and Run the Sensu Server

Clone the repo git clone https://github.com/jbrien/sensu-docker.git

Generate the environment variables and ssl certs for RabbitMQ and Sensu. The ssl certificates and generated in /usr/local/etc/sensu-docker on the docker host. Take note of this to be able and distribute the client folder and RabbitMQ password from the sensu.env file for new sensu clients

cd sensu-docker
sudo ./setup_sensu_docker.sh generate
sudo docker-compose -f base.yml build
sudo docker-compose -f base.yml up

it's normal to see a rc or console errors and they can safely be ignored. once the download and build process is done you can run sudo docker-compose -f base.yml up to start the cluster. Once up browse to http://[your-server-ip]:3000/ to see the sensu dashboard.

This setup also monitors itself so you should be able to see three docker containers in the client list (the Docker container HOSTNAME is reflected in the IP Address column). Each Redis, RabbitMQ, and Sensu's components being monitored respectively. It's up to you at this point to secure the dashboard if you are going to use this in production.

There is a volume shared from the Sensu server container in /etc/sensu/conf.d. Installing new checks to this location can restarting the sensu-server service will load them.

sudo docker exec -t sensudocker_sensu_1 supervisorctl restart sensu-server

Load Sensu with Metrics

There is a metric configuration that adds InfluxDB and Grafana to the base stack. The Grafana dashboard is available at http://[your-server-ip]:4000 and InfluxDB's dashboard is available at http://[your-server-ip]:8083.

sudo docker-compose -f metrics.yml build
sudo docker-compose -f metrics.yml up

There is an environment variable for Grafana's dashboard to be able to connect to InfluxDB, INFLUXDB_ENV_IP, that defaults to localhost. Grafana needs a public IP or URL (without the http://) for InfluxDB to make queries with it's API.

INFLUXDB_EXT_IP=www.example.com sudo docker-compose -f metrics.yml up

The root password for InfluxDB is available in the /usr/local/etc/sensu-docker/sensu.env as is the sensu user used by Sensu to push its time series to InfluxDB.

Exploring the Metric Series

After a few minutes you can log into InfluxDB with the sensu username and password and using sensu as the database (The sensu user is restricted to just seeing the data in the sensu database and can not log in otherwise). The data is InfluxDB is disposable; so each restart of the InfluxDB container will destroy any historical data.

  • Select Explore Data in the header
  • putting list series in the Query fields will show all the metric series being collected
  • putting select * from /.*/ limit 5 will show & graph the last five results for each of the time series

Other Queries working out of the box

select * from cpu_total_user where host =~ /sensu-server/
select mean(value) from cpu_total_idle group by time(30s) where time > now() - 1d and host =~ /sensu-server/
select mean(value) from load_avg_five

Development

There is a Vagrant file provided in this repository that will install both Docker and Docker Compose. All necessary ports are setup for development and defaults will work as expected. The contents of this repository are located in /vagrant when the VM come up.

Connecting a new Ubuntu Client

Copy the /usr/local/etc/sensu-docker/client folder from the Docker host (generated when setting up the server) to the client along with the install_client.sh script. If you no longer have this folder you will need to replace the certs on the server after running the script again.

make sure you have wget installed and run install_client.sh

sudo ./install_client.sh

Copy the client cert.pem and key.pem from the /usr/local/etc/sensu-docker/client folder on the Docker host. Modify the client config /etc/sensu/config.json with the necessary information.

Replace %RABBITMQ_ADDR_OR_IP% with the address for RabbitMQ from the docker-compose launch. Replace %RABBITMQ_PASSWD% with the password for RabbitMQ found in the /usr/local/etc/sensu-docker/sensu.env file. Replace %NODE_NAME% with a unique name to identify this client. Replace %HOSTNAME% with the hostname or IP of the client.

Adjust subscriptions to meet your needs

{
  "rabbitmq": {
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
    "port": 5671,
    "host": "%RABBITMQ_ADDR_OR_IP%",
    "user": "sensu",
    "password": "%RABBITMQ_PASSWD%",
    "vhost": "/sensu"
  },
  "client": {
    "name": "%NODE_NAME%",
    "address": "%HOSTNAME%",
    "subscriptions": [ "default" ]
  }
}

In Progress & Todo

  • Data persistance where it makes sense
  • Client install script for Yum based distros
  • Documentation for running in production
  • Documentation to scale InfluxDB / RabbitMQ (Docker Swarm?, Kubernetes?, CoreOS?)

sensu-docker's People

Contributors

hybridindie avatar petecheslock avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

sensu-docker's Issues

setup-sensu-docker could use user-defined path

Hi:

I don't really like the fact that the setup-sensu-docker.sh uses /usr/local to store its things.
I would like to be able to leave all that stuff in a directory inside the working folder, or somewhere else.

So this issue is a feature request to allow the user to define a custom path.

I'll probably work on the feature myself and submit you a pull request, but I would like to have your feedback about it.

Problems with the setup_sensu-docker.sh script

I hit some problems running the setup_sensu-docker.sh script - specifically in the "generate_ssl" function.

I had to make some directories in order for the script to run successfully.

Added this 4 lines after the "cd cd /usr/local/etc/sensu-docker" statement:

rm -rf sensu_ca
mkdir sensu_ca
mkdir -p sensu_ca/private
mkdir -p sensu_ca/certs

Generated sensu client /etc/default/sensu configuration paramters are bad

Getting errors when I start the sensu client. The install script needs to remove the double quotes around the 2 configuration parameters in /etc/default/sensu and make sure they both start in column 1.

/etc/init.d/sensu-client restart

/etc/default/sensu: line 1: EMBEDDED_RUBY=true: command not found
/etc/default/sensu: line 2: LOG_LEVEL=info: command not found

Sensu server unable to connect to rabbitmq after the changes made to create the certs and passwords

Been in communication with JD on this problem, so he is aware of it.

rs":["default"]}],"load_metrics":[null,{"type":"metric","handlers":["influxdb"],"command":"/etc/sensu/plugins/system/load-metrics.rb","interval":30,"subscribers":["default"]}]}}}
{"timestamp":"2015-04-08T12:18:08.399547+0000","level":"warn","message":"loading extension files from directory","directory":"/etc/sensu/extensions"}
{"timestamp":"2015-04-08T12:18:08.399651+0000","level":"warn","message":"loaded extension","type":"mutator","name":"json","description":"returns JSON formatted event data"}
{"timestamp":"2015-04-08T12:18:08.399731+0000","level":"warn","message":"loaded extension","type":"mutator","name":"ruby_hash","description":"returns ruby hash event data"}
{"timestamp":"2015-04-08T12:18:08.399866+0000","level":"warn","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns check output"}
{"timestamp":"2015-04-08T12:18:08.399914+0000","level":"warn","message":"loaded extension","type":"handler","name":"debug","description":"returns raw event data"}
{"timestamp":"2015-04-08T12:18:08.400267+0000","level":"debug","message":"connecting to redis","settings":{"host":"172.17.0.15","port":6379}}
{"timestamp":"2015-04-08T12:18:08.400799+0000","level":"debug","message":"connecting to transport","name":"rabbitmq","settings":{"ssl":{"cert_chain_file":"/usr/local/etc/sensu-docker/client/cert.pem","private_key_file":"/usr/local/etc/sensu-docker/client/key.pem"},"port":5671,"host":"172.17.0.14","user":"sensu","password":"OVVJUitNQUg1aEZYeEtia0hL","vhost":"/sensu"}}
{"timestamp":"2015-04-08T12:18:08.452256+0000","level":"debug","message":"subscribing to keepalives"}
{"timestamp":"2015-04-08T12:18:08.452482+0000","level":"debug","message":"subscribing to results"}
{"timestamp":"2015-04-08T12:18:08.452920+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:18:08.452984+0000","level":"warn","message":"reconnecting to transport"}
{"timestamp":"2015-04-08T12:18:08.453027+0000","level":"warn","message":"unsubscribing from keepalive and result queues"}
{"timestamp":"2015-04-08T12:18:08.453066+0000","level":"debug","message":"not currently master"}
{"timestamp":"2015-04-08T12:19:14.439086+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:19:19.517910+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:19:29.674635+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:19:34.748976+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:19:49.973974+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:19:55.056374+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:20:00.130148+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:20:20.433346+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:24:49.422984+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:25:19.897492+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:25:55.443051+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:26:15.744679+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
{"timestamp":"2015-04-08T12:26:56.334475+0000","level":"error","message":"[amqp] Detected TCP connection failure"}
(END)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.