containersolutions/redis-cluster-operator

A Kubernetes Operator for running Production Redis Clusters

Redis Cluster Operator

The Redis Cluster Operator runs Redis Clusters on Kubernetes.

We've found many operators which either use the redis-cli directly, which makes it hard to customise behaviour, or do not support a full productionised suite of features.

The aim for this operator is to run productionised clusters with most necessary features, as well as providing additions such as RunBooks to help debug issues with Redis Clusters when running them with this Operator, and ready-made load tests to test your Redis Clusters with real traffic.

• Features
• Installing The Operator
  • Bundled Cluster Wide
  • Bundled Namespaced
  • OLM bundle
• Creating your first Redis Cluster
• Documentation
• Contributing

Features this operator supports

• Cluster Creation
• Cluster Management
• Support for replicated clusters (Master-Replica splits)
• 0 Downtime scaling
• 0 Downtime upgrades
• Persistent clusters (Supported through Kubernetes PVC management)
• Backup & Restore capability for persistent clusters
• Documentation on observability for clusters
• Runbooks for common debugging issues and resolutions
• Ready-made k6s load tests to load Redis Clusters

Installing the Operator

bundled cluster-wide

The operator gets bundled for every release together with all of it's crds, rbac, and deployment.

The origin bundle works in cluster mode, and will manage all RedisClusters created in all namespaces.

To install or upgrade the operator

kubectl apply -f https://github.com/ContainerSolutions/redis-cluster-operator/releases/latest/download/bundle.yml

This will install the Operator in a new namespace redis-cluster-operator.

You can also install the operator in a custom namespace.

bundled namespaced

The operator currently works in cluster-wide mode, but the Operator will support namespaced mode in the future.

We know it's quite important for redundancy, reducing single-point of failures, as well as tenanted models, or excluding namespaces from the operator.

The Operator will support Namespaced mode in the future.

OLM bundle

OLM bundling support is a work in progress. There are remnants of OLM due to the initial Operator SDK installation, but we have not specifically tested and looked at it in depth.

Creating your first Redis Cluster

To create your first Redis cluster, you'll need a CRD.

apiVersion: cache.container-solutions.com/v1alpha1
kind: RedisCluster
metadata:
  name: rediscluster-product-api
spec:
  masters: 3
  replicasPerMaster: 1

Once applied, the Operator will create all the necessary nodes, and set up the cluster ready for use.

Remember to check out the documentation page for more information.

redis-cluster-operator's Issues

Add readiness and liveness probes to redis containers

Idea: Add default container label for pods for easy usage

Ability to override all pod settings

This would provide a lot of flexibility for all users.

We can default the normal pod spec, and allow overriding most settings and document how to do that.

We can default the redis container, and then allow overriding the redis pod, as well as adding additional containers for things like Prometheus and InfluxDB metrics and whatever else folks need.

Update namespace and operator names to be more friendly to clusters

By default Operator SDK generated controller-manager as both the namespace and names of most resources.

This is quite unfriendly to folks running many operators, and we should probably update these to be more reflective of the actual operator.

An example of this can be seen in config/manager/manager.yaml

Document how to create persistent clusters

Creating persistent clusters is possible through the new podSpec field.

We should probably document how to create Redis Clusters with persistence for those who want to do that.

Trying to failover master without moving slots

When ensuring the master replica ratio, and there are too many masters, we try to fail over the first master we find.

If that master has slots assigned to it, the failover fails and we end up in a reconciliation loop forever.

We should try to order the masters by slot, so that we can select a master without slot for the failover.

Ability to set pod priorities for Redis clusters

Supply the ability to set pod priority or priority classes in Redis Clusters. This might be useful to some users

Feature: Allow scale down of Redis Cluster

When a Redis Pod restarts, redis-cli check stops working

When a Redis pod restarts, the old Node ID is reset to a new node. The new node successfully meets the cluster, and successfully joins the pool. The old Node though, is not forgotten as it should be, especially when running without any persistence.

If all the nodes are running, and the cluster is successfully serving all slots, we should be confident in removing the old node id and forgetting it.

If a master node fails, it is automatically failed over to it's replica. If a replica restarts, the new node successfully joins the previous master node.

Document advanced usage such as spread constraints and pod priority

Feature: Add Service for Redis cluster

We need a service to connect to the Redis Cluster for initial node bootstrapping in applications

Feature: Rebalance clusters after scale up

idea: Helm Chart option for installation of the Operator

Having another installation option for this Operator would be a better approach.

Document resource request and limit settings

Allow injecting a password for Redis Clusters

Most redis clusters will be using a password, and we need to make sure the operator has the capability to use password.

There are a few spots now where we will need the password.

• Readiness & Liveness Probes
• Redis node communication (we need to inject into the redis.conf)
• Redis clients need to use the password

We also need to provide this capability by providing a secret with interfaced keys.

This allows people using the Operator to inject secrets to Redis cluster however they prefer, whether this is via the External Secrets Operator or normal Kubernetes secret. Their applications can then use that same secret to connect to the Redis Cluster.

Ability to override or set resource requests and limits on redis pods.

We might also be able to default these to redis settings provided, or infer these from Redis Config

Add documentation for how to override Redis settings

It is currently possible to override all of the Redis settings for redis.conf, but this isn't documented anywhere.

It's a good idea to document these for users.

Currently you can override Redis configuration by specifying it in the CRD under config:

apiVersion: cache.container-solutions.com/v1alpha1
kind: RedisCluster
metadata:
  name: rediscluster-sample
spec:
  masters: 3
  replicasPerMaster: 1
  config: |
    maxmemory 50mb
    maxmemory-policy allkeys-lru

Allow overriding Pod parameters

When users use the Operator in certain secure contexts such as OpenShift, they may need to update the security context and parameters of the pods externally.

We need to cater for this in the Redis Operator to allow overriding some of these things

Use topology Spread constraints and nodeSelectors to spread redis nodes across kubernetes nodes

When a Kubernetes node fails, it will take the redis pods with it. The same goes for kubernetes clusters which are spread by availability zones. We could have a mechanism to allow specifying the spread constraints so users can identify these themselves

We need to supply the ability to override spread constraints, affinity, tolerations, node name, and node selectors.

Use Parallel pod management policy for Redis statefulset

When managing the statefulset for the Redis Clusters, we could potentially specify which pod management policy should be used, in case some folks have some specific needs for this.