etherlabsio/healthcheck

An simple, easily extensible and concurrent health-check library for Go services

Healthcheck

A simple and extensible RESTful Healthcheck API implementation for Go services.

Health provides an http.Handlefunc for use as a healthcheck endpoint used by external services or load balancers. The function is used to determine the health of the application and to remove unhealthy application hosts or containers from rotation.

Instead of blindly returning a 200 HTTP status code, a healthcheck endpoint should test all the mandatory dependencies that are essential for proper functioning of a web service.

Implementing the Checker interface and passing it on to healthcheck allows you to test the the dependencies such as a database connection, caches, files and even external services you rely on. You may choose to not fail the healthcheck on failure of certain dependencies such as external services that you are not always dependent on.

Example

package main

import (
    "context"
    "database/sql"
    "net/http"
    "time"

    "github.com/etherlabsio/healthcheck/v2"
    "github.com/etherlabsio/healthcheck/v2/checkers"
    _ "github.com/go-sql-driver/mysql"
    "github.com/gorilla/mux"
)

func main() {
    // For brevity, error check is being omitted here.
    db, _ := sql.Open("mysql", "user:password@/dbname")
    defer db.Close()

    r := mux.NewRouter()
    r.Handle("/healthcheck", healthcheck.Handler(

        // WithTimeout allows you to set a max overall timeout.
        healthcheck.WithTimeout(5*time.Second),

        // Checkers fail the status in case of any error.
        healthcheck.WithChecker(
            "heartbeat", checkers.Heartbeat("$PROJECT_PATH/heartbeat"),
        ),

        healthcheck.WithChecker(
            "database", healthcheck.CheckerFunc(
                func(ctx context.Context) error {
                    return db.PingContext(ctx)
                },
            ),
        ),

        // Observers do not fail the status in case of error.
        healthcheck.WithObserver(
            "diskspace", checkers.DiskSpace("/var/log", 90),
        ),
    ))

    http.ListenAndServe(":8080", r)
}

Based on the example provided above, curl localhost:8080/healthcheck | jq should yield on error a response with an HTTP statusCode of 503.

{
  "status": "Service Unavailable",
  "errors": {
    "database": "dial tcp 127.0.0.1:3306: getsockopt: connection refused",
    "heartbeat": "heartbeat not found. application should be out of rotation"
  }
}

License

This project is licensed under the terms of the MIT license. See the LICENSE file.

healthcheck's Issues

use request context in http handler

The handler uses context.Background() in the http handler. If the client (calling service) drop the connection, the processing will continue till an error occur when the handler tries to write a response.

Maintain checker specific timeouts

In addition to a common and overall timeout value defined for all health checkers, it'll be useful to override with a specific timeout value for a checker.

Concern:
My assumption is that it will be useful for smaller timeout values when compared to an overall timeout across all checkers, when we want to guarantee strict response times of the dependency.

Does it even make sense to allow a higher value than the overall checker limit ? How would the implementation change.

go-routine leak when check times out

If there is a time out before the checker is able to complete, then the goroutine started by the timeout checker forever waits to write on a channel.

type timeoutChecker struct {
	checker Checker
}

func (t *timeoutChecker) Check(ctx context.Context) error {
	checkerChan := make(chan error)
	go func() {
		checkerChan <- t.checker.Check(ctx) // cannot write here to the channel, no reader
	}()
	select {
	case err := <-checkerChan:
		return err
	case <-ctx.Done():
		return errors.New("max check time exceeded")
	}
}

Declaring the channel with size of 1 fixes the bug.

Add disk space checker

Add a checker that checks the available disk space and raises error if the expected max threshold is exceeded.

Proposal: Please start using Semantic Versioning

I found that this project already supports Go modules. But sadly, the tags doesn't follow Semantic Versioning, which means that all tags of this project will be ignored by Go modules and replaced by pseudo-versions, go get acts weirdly when tags are not in that form. It would be great to have the tagged release be named in the format vX.X.X format so that go mod can read it.

	github.com/etherlabsio/healthcheck v0.0.0-20191224061800-dd3d2fd8c3f6

Else the mod file shows something like github.com/etherlabsio/healthcheck v0.0.0-20191224061800-dd3d2fd8c3f6 which is not very readable and difficult to upgrade. It’s hard to verify which version is in use. This is not conducive to version control.

So, I propose this project to follow Semantic Versioning in future versions. For example, v1.0.1, v2.0.0, v3.1.0-alpha, v3.1.0-beta.2etc, so that other project can use tag in go.mod.

syscall package is deprecated

syscall is deprecated and it doesn't work on mac os.

Add code coverage badge

Add the code coverage report badge to the Readme section after integrating with circleCi to generate the code coverage report.

Error undefined: syscall.Statfs_t

got error below when running service with windows

# github.com/etherlabsio/healthcheck/v2/checkers
..\..\..\go\pkg\mod\github.com\etherlabsio\healthcheck\[email protected]\checkers\diskspace.go:16:34: undefined: syscall.Statfs_t
..\..\..\go\pkg\mod\github.com\etherlabsio\healthcheck\[email protected]\checkers\diskspace.go:25:16: undefined: syscall.Statfs_t
..\..\..\go\pkg\mod\github.com\etherlabsio\healthcheck\[email protected]\checkers\diskspace.go:46:22: undefined: syscall.Statfs

Make the call to checkers concurrent.

Since Healthcheck usually checks dependencies present over the network there is a possibility of making multiple IO calls. Right now we are calling the check() function serially, so we'll wait till the function is returned. By using goroutines and waitgroup, we can make the calls more concurrent and hopefully reduce the overall latency of the healthcheck API.

Since healthcheck API's are called frequently, lower latecy response time is imperative.

PS: We'll also need to maintain an overall TTL for the healthcheck request. But that's a different issue to be addressed.

Add option to not fail the healthcheck response for certain dependencies

By default, any error occurring in the checker responses leads to a 503 HTTP status code. But we may not want to fail the healthcheck in case certain dependencies fail (eg: metrics service, s3 etc) but would like to observe the errors in the response for taking further action.

Maintain overall max request timeout (Optional)

Health Checks are called frequently, hence latency should ideally be low. Since most checks are to dependencies over the network, we need to make sure to timeout if the calls are not served within time.

While it's the responsibility of each checker func to maintain it's own TTL, having a max timeout value for a health check request might be a good idea.

This could be an optional feature.