sjeanpierre/passenger-datadog-monitor

Golang application for reporting Phusion Passenger stats to DataDog via statsD

passenger-datadog-monitor

Send health metrics from Phusion Passenger to Datadog using the StatsD interface on the server agent.

Purpose

Graph and track Passenger threads and possibly detect misbehaving threads before they become a problem.

Tracked Metrics

Aggregated

• Processed requests: min,max,average,total
• Memory usage: min,max,average,total
• Thread uptime: min, max, average
• Request queue depth
• Threads in use vs Max thread configured & Threads used between runs

Discrete

• Process memory usage per Passenger Process PID
• OS thread count per Passenger Process PID
• Requests processed per Passenger Process PID
• Process Idle time per Passenger Process PID

Installation

Downloading from Github

The passenger-datadog-monitor binary can be downloaded from the releases area of this repository for Linux

Building the binary

You will first need to build the passenger-datadog-monitor executable using Go. You can download the source and dependencies, and build the binary by running:

go get -v github.com/Sjeanpierre/passenger-datadog-monitor

Once it completes, you should find your new passenger-datadog-monitor executable in your $GOROOT/bin directory.

Note that if you are building in a different environment from where you plan to deploy, you should configure your target operating system and architecture.

The Makefile in this repository will cross compile for Linux.

Installing the binary

After you've built the executable, you should install it on your server (e.g. in /usr/bin/).

Usage

passenger-datadog-monitor runs as a daemon with a 10 second sampling interval. Monit, God, SupervisorD, or any other daemon management tool should be used to manage the process.

Sample Monit config

check process passenger-datadog-monitor with pidfile /var/run/passenger-datadog-monitor.pid
start program = "/etc/init.d/passenger-datadog-monitor start"
stop  program = "/etc/init.d/passenger-datadog-monitor stop"

You should run passenger-datadog-monitor as root, since access to passenger-status requires root.

Flags

flag	type	description	example
-host	string	StatsD collector IP - useful when running with a Kubernetes DaemonSet or other external collector	-host=100.124.102.21
-port	string	StatsD collector UDP Port - useful when running in Docker or other custom environments	-port=81333
-print	bool	Enable debug and stats printing	-print

Full example:

passenger-datadog-monitor -host=$STATSD_IP -port=$STATSD_PORT

Testing

udp.rb can be run locally when you want to see what is being received on the server side.

Alternatively you can listen using netcat: nc -kulvw 0 8125

passenger-datadog-monitor's Issues

Send the app root also

It would be good to have the app root and environment sent also - the PIDs don't tell you which app they are serving

go-charset not available as dependancy anymore

Hello,

Today I tried to build the tool with:
go get -v github.com/Sjeanpierre/passenger-datadog-monitor
but get:

package code.google.com/p/go-charset/charset: unable to detect version control system for code.google.com/ path
package code.google.com/p/go-charset/data: unable to detect version control system for code.google.com/ path

I think to use this package as a dependancy from google code we need some kind of permissions:
https://code.google.com/archive/p/go-charset/downloads

This brakes the build of the project. Can you have a look?

Implement parser for passenger-status --show=requests

https://www.phusionpassenger.com/library/admin/apache/overall_status_report.html

Panic when passenger is restarted

panic: runtime error: index out of range

goroutine 1 [running]:
main.summerizeStats(0xc20801e6e0, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/jstevenson/src/misc/go/src/github.com/sjeanpierre/passenger-datadog-monitor/main.go:55 +0x187
main.processed(0xc2080a3f38, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/jstevenson/src/misc/go/src/github.com/sjeanpierre/passenger-datadog-monitor/main.go:93 +0x206
main.chartProcessed(0xc2080a3f38, 0xc208056b00)
/Users/jstevenson/src/misc/go/src/github.com/sjeanpierre/passenger-datadog-monitor/main.go:132 +0x33
main.main()
/Users

automated release process

While I was working on #10 - I also realize I need a standard way to download and use a trusted binary in multiple places.

I've been using goreleaser for this, which is kind of amazing. I have a branch with a config. With the appropriate setup (github token and docker privs) can publish your binaries, build docker images, etc.

(I'm building a really small docker image here, mostly so that one can use COPY --from=remote/image binary /usr/local/binary style installs rather than needing to do a curl or something.)

I'm writing an issue to see if you're interested in adding that to your repo (since there are obvious implications like potentially adding CI builds, which would be based on your credentials, name, etc)