To get the project up and running:
make dev-up
. This will get our kubernetes cluster up and running. Test thismake dev-apply
. Apply changes made to the yaml defining our kubernetes clustermake dev-update-apply
. This will build our docker image, use kind to load the docker image into the k8s cluster, use kustomize to load our base k8s config, and then wait until the cluster has applied the changes.make dev-update
. If all we did is change some code, this is all we needmake dev-down
. This will shut down the project, shutting down the container, deleting the cluster, and the telepresence daemon.
We have our middleware functions contained here functions contained here. These
functions can take in arguments, and they return type web.Middleware
, for
example, our Logging middleware takes the logger as a parameter. This is useful
so we can avoid having to use the singleton pattern for the logger.
To achieve this, we utilize a closure as can be seen below:
func Logger(log *zap.SugaredLogger) web.Middleware {
m := func(handler web.Handler) web.Handler {
h := func(ctx context.Context, w http.ResponseWriter, r *http.Request) error {
// Logging
err := handler(ctx, w, r)
// Some more logging
return err
}
return h
}
return m
}
Simple enough. We log both before and after the call to handler()
, when the
request begins and when the request completes.
Some values are taken from the request
object, while others are stored in the
context at the foundation layer, such as the traceID
.
This middleware accepts a logger as a parameter, because part of handling an error is to log it.
- The first course of action is to log the error.
- We use a switch-case block to determine if the error is trusted or untrusted.
We use this determination to fill up the
ErrorResponse
struct appropriately. - We respond with the appropriate error message and status code.
- Finally, we check if the error was a shutdown error, allowing the app to gracefully shutdown after having logged the error and responded to the request.
This middleware calls recover()
(within a deffered function, otherwise the
call will do nothing). If there is a panic!
then we want to log it, we also
use of the setter functions in package metrics
to safely update the panic
metrics value held in the context of the request.
We want to return an error here, so we need to utilize named return values,
allowing us to update the value of the err
variable from within the defer.
This error is then handled by the errors middleware, which will respond with a
500 Internal Server Error
.
This middleware sets the metrics data on the context of the request. After the handler is called we increment requests, and increment goroutine metrics (which uses a modulo operation to only do the work every 100 requests).
The foundation layer is meant to be usable across many applications, and other developers. Each package in this layer should be as unopinionated as possible. To make sure we maintain this high-degree of reusability, we have strong policies in place for these packages:
- No logging. Logging should be determined by the user of these packages, we don't want these packages to prescribe a specific logging package or implementation pattern.
We are using package httptreemux
as our router, and in package web we create a
small web framework that extends the default funcitonality.
The key entity is the App
struct which is the entry point into our
application. We override the default Handler
function, creating a new
signature which accepts the context
as its first parameter.
type Handler func(ctx context.Context, w http.ResponseWriter, r *http.Request) error
This allows us to write our routes in such a way that we can pass in the context what object object.
We also override the Handle
function of the mux, allowing us to inject
middleware before and after the call to the provided Handler
function. We
differentiate between two layers of middleware:
-
Application-layer middleware that is applied to all routes. This middleware is passed to the
NewApp()
function. -
Local middleware that is applied selectively. We supply this middleware in the
Handle()
function.
While the majority of our middleware is implemented in the business-layer, we are choosing to implement some middleware-like functionality in the foundation layer, as part of our web framework.
The business layer middleware is implemented in a mostly-typical, if not somewhat unique pattern:
-
Write a middleware function that can accept any type of argument, and which returns a type of
web.Middleware
: This is the more familiar pattern: a function that takes in aHandler
and returns aHandler
. -
The
wrapMiddleware()
function in our framework will apply all middlewares, ultimately returning a single wrappedHandler
super-charged with all of the provided functionality.
Meanwhile, the functionality that we are implementing in the foundation layer
uses the context
instead. In our Handle()
function, we create a struct
containing a trace ID, the current time and store it in the context
. We can
then make use of these in other middleware, by retrieving them from the request
context (e.g., as in the logger middleware).
web.Respond
-
Local Layer Middleware. This is for middleware that does not need to be applied to all routes. For example, not all routes will require authentication. This middleware is applied when we register a route via the
Handle
function. -
Application Layer Middleware. This gets applied to all routes attached to an
App
. We use this for logging, error handling, metrics, and panic handling.
If we think of each route servicing incoming requests as an onion with the
Handler
itself at the center, then we should think of the local layer
middleware as closest to the center, as it is applied first, with the
application layer middleware on the outside, as it is applied last.
Our httptreemux
router will create a context for every incoming request, and
every outgoing response. Keeping this information in the foundation layer is a
debatable decision.
For incoming requests, we are keeping track of three things in the context:
-
TraceID
. A unique string that allows us to differentiate requests. We use UUID for this. -
Now
. We track the time and duration. -
Status Code
. By keeping the status code in the context, we can use it in our foundation layerRespond()
function.
This is our base k8s configuration that never changes regardless of the
environment that we are running in. We use kustomize
to patch the base
configuration into different environments (e.g., dev, staging, prod).
We are using package zap for logging. We pass
around a single logger that is intialized in the main()
function of the
server. This single logger is then passed around through the app using various
cfg
structs.
Our metrics.go
file uses a singleton, var m *metrics
is a package level
variable. In this instance, the API of the expvar
package has forced our hand,
because it is using a singleton internally. We can get away with a package level
variable when we tick the following boxes:
-
The order of initialization does not matter. We must not have an order-dependency on our package level variables, our only need is that they are intialized before
main()
. -
They don't rely on anything from the configuration system. If our package level variable requires configuration, then it must be intialized in
main()
(orrun()
, our nominalmain
). -
The only code touching the variable is the code in the same source code file as the variable. Even if it's in the same package, we don't want code in other files touching the package level variable. We won't to keep it centralized because we want to have an easy mental model of how the variable is being mutated.
We store our metrics data in the context of each request, updating individual metrics on every request. Because we are using package expvar, we can safely update these values concurrently.
We make use of the init()
function, which runs before main()
, to initialize
the metrics singleton. We are keeping track of the number of:
- Goroutines
- Requests
- Errors
- Panics
Because we are storing metrics in the context, we want to use getters and setters, because of the lack of type safety involved in the context.
The goroutine metric will slow down any given request because we have to make
use of the runtime package which will internally inspect the runtime state,
adding additional goroutines to do this work. We use a modulo operation on the
request value (if v.requests.Value()%100 == 0
), so that we only have to
actually check how many goroutines are running on every 100th request (this
number is likely way too small for most services, but is just a placeholder).
This is the default. We do not want to expose information about the state of our
application beyond the boundaries of our API. This is a security risk. By
default, unless we can justify a reason otherwise, errors should be returned to
the client as 500 Internal Server Error
.
We trust the messaging of the error to be secure and not leak any sensitive information, such that we can respond to the client with the messaging.
We place this error type in the business layer, and we version it as well. This is because our policies may change in future versions of our API in regards to how we want to handle trusted errors.
If our service is having data integrity issues it should be shut down. When our service is corrupting databases, file systems, etc., then we want to gracefully shutdown. But we do not want any code outside actually shutting down our app, and so instead these errors serve as a suggestion., our application can then determine the appropriate action after inspecting the error.
We place this error type in the foundation layer. A Shutdown Error is a special
case of an Untrusted Error. Like all other errors, it gets handled by the
errors
middleware, because it is untrusted it is logged, an an Internal Server Error
gets returned to the client. The middleware then checks if the
error is indeed a Shutdown Error, and if so, returns the error. This sends the
control flow back to the (*App).Handle
function in the foundation layer.
At this point, we run a function validateShutdown
. The reason we need this is
because it's possible that we may have network problems causing our code to
reach this line, but that are not errors worthy of shutting the app down.
When we do validate the shutdown error, we then use the shutdown
channel in
the App
struct to signal to the application that we need to gracefully
shutdown. Graceful shutdown will begin.
Our guiding principle is that all configuration must happen in main.go
. This
is to enforce simplicity. When all configuration happens in one place, then we
don't have to worry about losing track of it.