mtneug/spate

🌊 Horizontal service autoscaler for Docker Swarm mode

🌊 spate

spate /speɪt/
noun

1. freshet, flood
2. a) a large number or amount
   b) a sudden or strong outburst rush

— Merriam-Webster Dictionary

spate is a horizontal service autoscaler for Docker Swarm mode inspired by Kubernetes' Horizontal Pod Autoscaler.

Currently spate can scale services based on exposed Prometheus metrics. However, the foundations already have been layed for different types. Future versions will, for instance, be able to scale based on the CPU or memory usage.

Installation

For every release static Linux binaries can be downloaded from the release page in GitHub. But the easiest way to run spate is, of course, as a Docker Swarm service with the mtneug/spate image:

$ docker service create \
    --constraint 'node.role == manager' \
    --mount 'type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock' \
    mtneug/spate

spate sets the replica count via the Docker API and needs therefore access to the Unix socket of a manager node. Also, make sure to put spate in the necessary networks so that it can poll data from the running containers is should scale. Prometheus metrics are exposed on port 8080.

Quick Start

After spate is running a reconciliation loop constantly looks for changes in the Docker Swarm cluster. Autoscaling is entirely controlled through service labels. In this way users and scripts only have to directly interact with the Docker CLI client.

$ docker service create \
    --name my-worker \
    --network spate-demo \
    --label "de.mtneug.spate=enable" \
    --label "de.mtneug.spate.metric.load.type=prometheus" \
    --label "de.mtneug.spate.metric.load.kind=replica" \
    --label "de.mtneug.spate.metric.load.prometheus.endpoint=http://localhost:8080/metrics" \
    --label "de.mtneug.spate.metric.load.prometheus.name=jobs_total" \
    --label "de.mtneug.spate.metric.load.target=3" \
    my/worker

In this example a my-worker service is created. Each replica exposes the jobs_total Prometheus metric on port 8080, which should count the number of jobs currently processed. It is the aim, that each replica, in average, processes three jobs. A metric is therefore added with the de.mtneug.spate.metric.load labels. This way, if too many jobs are in the system, spate automatically increase the replica count and vice versa.

spate is highly configurable. Have a look at the documentation for a complete list of options.

Example

An example application is provided with spate-demo. Consult this repository's README for more information.

License

Apache 2.0 (c) Matthias Neugebauer

spate's Issues

The user can declare cool down times after various events

Description

Services might have a cost associated with starting or stopping replicas. Often this simply is the initialization and shutdown time, but one could also imagine other factors. In these cases keeping the number of replicas stable over a period of time is preferable. For this spate allows to specify cool down times that apply after up and down scales. During this time no autoscaling is performed even if the target is not met. This also guards against artificial low or high measures. For instance after a down scale one metric could be artificially low leading to an up scale. If instead some time passes a more realistic measurement is taken. Cool down times can also be set for other events, i.e. after the service is added or updated. They are specified on a per service basis. If no time is provided a default one applies. The default can be set by the administrator globally.

Design

Each cool down time is associated with an event. By default there is 3m cool down time after a scaled_up event while scaled_down wait for 5m (like the pod autoscaler in Kubernetes). service_added and service_updated have no cool down time. Users can change these with the de.mtneug.spate.loop.cooldown.<event> labels. Administrators can set global defaults with the --default-cool-down-time-<event> CLI options. For service_updateds the cool down time of the new service definition is used.

Example

# Setting the global default
$ spate \
   --default-cool-down-time-scaled_up=30s \
   --default-cool-down-time-scaled_down=20m \
   --default-cool-down-time-service_added=2m \
   --default-cool-down-time-service_updated=1m

# Overwriting the default
$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.loop.cooldown.scaled_up=30s \
   --label de.mtneug.spate.loop.cooldown.scaled_down=20m \
   --label de.mtneug.spate.loop.cooldown.service_added=2m \
   --label de.mtneug.spate.loop.cooldown.service_updated=1m \
   ...
   my/service

a

a

A replica metric can be the CPU usage

Description

The CPU usage is a general metric that is provided by Docker for each replica. It might not fully capture the state of a replica, but it can always be used even for applications that expose no other information. In spate the CPU usage is a relative value between 0 and 1. It is also a metric type with a default value of 0.8. Users can only declare the CPU usage metric once per service.

Design

The replica metric CPU usage is also its own metric type called cpu. Using it only requires that the label de.mtneug.spate.metrics.<key>.type=cpu is present. This metric type can only be used once. Using it multiple times results in a deactivation of autoscaling and an error logs message. The default target value of 0.8 can be overwritten.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.cpu.target=0.7 \
   ...
   my/service

Automatic handling of service creations, updates, and deletions

Description

spate observes changes in the Docker Swarm service landscape and reacts to them. Services can be created, updated and removed.

Design

All service specific user input comes via the service labels. The observation of and reaction to service changes thus plays the central role in spate. As this is inherently event based the event loop construct is used. In this programming model the main program consist of an infinit loop (the event loop), events associated handlers. In the loop the program waits until an event occurred. Then the associated handler is executed and the loop starts over. In the case of spate and the events service created, service updated, and service removed it might look like this:

The Docker remote API has an events endpoint, that can be used to watch for events. Unfortunately the used version (TODO: fix version) does not emit any service related events. It is therefore required that spate periodically polls the current state, compares it to the previous, and if necessary emits events. Administrators can control how often it is look for changes by setting a waiting time. This setting is passed as argument to the program and is global. The resulting design is depicted below:

Notice that the event loop is unchanged. The program should be implemented such that, in case service events are added, the events endpoint of the Docker remote API can easily replace the component looking for changes.

As it contains a "diff" step, this approach has the additional benefit that spate can automatically repair any introduced state errors. The simple event loop thus becomes a reconciliation loop. Because of this property future versions of spate might refrain from replacing this subsystem.

To further decouple the two subsystems, emitted events should be placed in an in-memory queue. The queue has a buffer size of 20 (this is arbitrary for lack of any better number).

Administrators can set how often spate should look for changes by passing a time to the --controller-period CLI argument. The default is 5s.

Example

$ spate --controller-period=10s

Automatic scaling up/down if one/all aggregated metric/s is/are above/below target value/s and all conditions are met

Description

spate automatically scales services up and down by comparing each aggregated metric to the corresponding target value. If one aggregated metric is above the target value the service is scaled up. Similarly if all aggregated metrics are below the target value the service is scaled down. Scaling is not applied if the deviation is in the allowed region (see #14) or the autoscaler is currently cooling down (see #13). Also the number of replicas can be altered by the specified minimum/maximum number of replicas (see #12).

Design

The user can declare how measures of a metric should be aggregated

Description

Metrics might fluctuate. To compensate for that multiple measurements are aggregated to the aggregated metric. On a per metric base users are can specify the aggregation method. The default one is averaging, but users can also choose the minimum, maximum, or simple exponential smoothing.

Design

The aggregation method can be set by the de.mtneug.spate.metrics.<metric name>.aggregation.method label. It can either be avg, min, or max. The default is avg.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.cpu.aggregation.method=max \
   ...
   my/service

A replica metric can be the memory usage

Description

The memory usage is a general metric that is provided by Docker for each replica. It might not fully capture the state of a replica, but it can always be used even for applications that expose no other information. In spate the memory usage is a relative value between 0 and 1. It is also a metric type with a default value of 0.8. Users can only declare on memory usage metric per service.

Design

The replica metric memory usage is also its own metric type called memory. Using it only requires that the label de.mtneug.spate.metrics.<key>.type=memory is present. This metric type can only be used once. Using it multiple times results in a deactivation of autoscaling and an error logs message. The default target value of 0.8 can be overwritten.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.memory.type=memory \
   --label de.mtneug.spate.metrics.memory.target=0.7 \
   ...
   my/service

The user can declare one target value per metric

Description

Autoscaling works by keeping specified metrics at a target level by changing the number of replicas of a service. Users therefore must provide one target value for each metric. Some metric types might have a default target value. Target values always refer to a single replica even if the associated metric is a system metric.

Design

The target is set by the de.mtneug.spate.metrics.<metric name>.target label. The value should be within the domain of the metric but spate allows any 64-bit integer.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.cpu.target=0.7 \
   ...
   my/service

The user can declare how often a metric should be measured

Description

The pace of change is different for each metric. Also some services need to be autoscaled quicker than others (see #15). To accurately asses the current situation users must be able to specify how often spate should measure a specific metric. Users can set this rate on a per metric base. If no rate is set, the default one is used. The default can be set globally by an administrator.

Design

The measurement period is set per metric. spate looks for the label de.mtneug.spate.metrics.<key>.observer.period. The value has to be an time entry. The default period can be set via the CLI option --default-observer-period. If no is set 30s is used.

Example

# Setting the global default
$ spate --default-observer-period=10s

# Overwriting the default
$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.cpu.observer.period=10s \
   ...
   my/service

Automatic measuring of each metric

Description

Metrics are measured periodically in a fixed time interval. A fixed number of measurements are kept for aggregation.

Design

The user can declare an allowed upper/lower deviation from the target value

Description

Instead of a single target value users might want to specify a range. This can be done by upper and lower target deviation values. Users can specify these as absolute or percentage values. If no values are provided no deviation is allowed.

Design

Users can specify upper and lower deviations from the target through the de.mtneug.spate.metrics.<key>.target.deviation.upper and de.mtneug.spate.metrics.<key>.target.deviation.lower labels. Both are either unsigned 64-bit integers or percentage values. A percentage is an unsigned integer between 0 and 100 followed by a % sign.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.cpu.target=0.8 \
   --label de.mtneug.spate.metrics.cpu.target.deviation.lower=10% \
   --label de.mtneug.spate.metrics.cpu.target.deviation.upper=0.1 \
   ...
   my/service

The user can declare a maximum/minimum number of replicas

Description

Users might want to use autoscaling but still restrict the number of replicas. A minimum number of replicas could for instance guarantee certain service response times. And a maximum limit could reserve resources for other services. spate lets users set minimum and maximum values on a per service level.

Design

de.mtneug.spate.replicas.max and de.mtneug.spate.replicas.min are the labels to set the maximum and minimum number of replicas. Both have unsigned 64-bit integer values the same range Docker Swarm is using for keeping track of the desired number of replicas.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.replicas.min=2 \
   --label de.mtneug.spate.replicas.max=5 \
   ...
   my/service

The user can activate/deactivate autoscaling per service

Description

Users of spate are able to decide on a service level if they want to activate autoscaling. Autoscaling does not make sense for every service or the number of replicas is to be controlled by the user. Autoscaling is an opt-in, i.e. it is deactivated by default and has to be enabled by the user.

Design

As all user input autoscaling is activated by service labels. Initially it was the plant to have no dedicated label for it and simply enable autoscaling as soon as there is one metric defined (see #3). This would have the benefit that the user is forced to declare at least one metric. However during implementation it became clear that this approach would mean that spate could not use Docker API filters. Because of the lack of a fixed label name (see #3) and no wildcard support for filters spate would have to retrieve all services and filter itself. The design was therefor altered. Users now have to add the de.mtneug.spate label and set its value to enable.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate=enable \
   ...
   my/service

The user can declare metrics as replica or system metrics

Description

Metrics can measure the service at different granularities: system metrics measure the service as a whole while replica metrics only look one particular replica. replica metrics can be turned into system metrics by summing them up. system metrics thus are just pre-aggregated metrics. As such they must measure something that can be divided between replicas. The metric kind can be predetermined by the metric type.

Design

For setting the whether the metric is a system or replica one there exists the de.mtneug.spate.metrics.<key>.kind label. It can either be system or replica. Some metric types might require this setting while others could overwrite it. If this setting is required but missing, autoscaling is deactivated and an error is logged. An overwrite by the metric type is logged as warning.

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.mymetric.type=prometheus \
   --label de.mtneug.spate.metrics.mymetric.kind=replica \
   ...
   my/service

The user can declare the control loop period per service

Description

Some services require to be autoscaled quicker than others. This is controllable by the control loop period. It determines how often the control loop is run. During each iteration spate assesses the service and scales appropriate. The control loop period ought to be at least equal to the slowest rate of metric measurement (see #5). Otherwise the same aggregated metric is used as in the previous iteration. spate should warn but not enforce that. If no value is specified the default one applies. The default can be set globally by the administrator.

Design

How often the control loop is run is determined by the de.mtneug.spate.autoscaler.period label. Its value is a time entry. If the update interval of one metric is slower than the set loop period a warning should be logged. The global default can be set by the CLI option --default-autoscaler-period. The initial default is 30s.

Example

# Setting the global default
$ spate --default-autoscaler-period=10s

# Overwriting the default
$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.autoscaler.period=10s \
   ...
   my/service

A metric can be a Prometheus metric

Description

One monitoring solutions that has gotten a lot of attention in the container ecosystem is Prometheus. It defines formats for pull based metric gathering that spate can also use for autoscaling.

Design

Prometheus metric have the metric type prometheus. To function correctly additional labels have to be specified. In particular spate needs to know the HTTP url exposing the metrics, the name of the metric, and the kind (see #6) as well as the target value (see #10). localhost in urls refers to individual replicas. This is especially necessary for replica metrics, as there each replica needs to be queried individually. If localhost is not used as the host part and the metric is a replica metric, a warning should be logged. If localhost is used and the metric is a system metric any replica is used for querying. If HTTPS is used the full certificate validation must succeed. There is no option for turning this off. The name must be a valid Prometheus metric name. If more than one Prometheus metric with the exact same url and name is declared, autoscaling is deactivated and an error is logged.

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.mymetric.type=prometheus \
   --label de.mtneug.spate.metrics.mymetric.kind=replica \
   --label de.mtneug.spate.metrics.mymetric.prometheus.endpoint=http://localhost:8080/metrics \
   --label de.mtneug.spate.metrics.mymetric.prometheus.name=mymetric \
   --label de.mtneug.spate.metrics.mymetric.target=20 \
   ...
   my/service

The user can declare multiple metrics per service

Description

Autoscaling requires the gathering of application runtime metrics. Per service at least one metric has to be defined. However one metric might not be enough to capture all scaling related aspects of one service. For instance the application could have a low CPU utilization but a too high memory consumption. In this case only observing the CPU leads to the decision to scale down when in reality more replicas are needed. Therefore users are able to specify an arbitrary amount of metrics per service.

Design

For metric definitions users have to specify labels starting with de.mtneug.spate.metrics. and followed by a user defined key. Each unique key declares a new metric. Separate by a dot users can specify options. A dot is therefore a forbidden character in keys.

The only option that is required for each metric is the type. However depending on the type more inputs might be necessary. The type is a well defined set. Currently planed are prometheus (see #9), cpu (see #17), and memory (see #18).

Example

$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.mymetric.type=prometheus \
   ...
   my/service

The user can declare how many measures of a metric should be used for aggregation

Description

Users can control how many measurements are used for aggregation and thus can influence how many historical values influence the aggregation. If no number is set the default one applies. Administrators are able to set the default globally.

Design

For setting the number of aggregated measures there exists the label de.mtneug.spate.metrics.<metric name>.aggregation.amount. Its value is an unsigned 8-bit integer. The global default is set by the --default-aggregation-amount CLI option. The initial default is 5.

Example

# Setting the global default
$ spate --default-aggregation-amount=8

# Overwriting the default
$ docker service create \
   --name service \
   --label de.mtneug.spate.metrics.cpu.type=cpu \
   --label de.mtneug.spate.metrics.cpu.aggregation.amount=8 \
   ...
   my/service

Expose Prometheus metrics

Description

spate exposes its own Prometheus metrics so that administrators can observe the state of spate.

Design

spate listens on port 8080 for HTTP requests. The /metrics endpoint exposes the Prometheus metrics.

TODO

• What metrics to expose?