pravega/zookeeper-operator

Kubernetes Operator for Zookeeper

zookeeper-operator's Issues

Zookeeper pre-stop hook failes when cluster hasn't started

The Zookeeper pre-stop hook de-registers a node from the ensemble before it the node stops. When deleting an instance, the ensemble may not be available to remove the node, creating an warning K8s event.

++ DOMAIN=zinc-chinchilla-zookeeper-headless.default.svc.cluster.local                                                                                                                                     │
++ QUORUM_PORT=2888                                                                                                                                                                                        │
++ LEADER_PORT=3888                                                                                                                                                                                        │
++ CLIENT_HOST=zinc-chinchilla-zookeeper-client                                                                                                                                                            │
++ CLIENT_PORT=9277                                                                                                                                                                                        │
+ source /usr/local/bin/zookeeperFunctions.sh                                                                                                                                                              │
++ set -ex                                                                                                                                                                                                 │
+ DATA_DIR=/data                                                                                                                                                                                           │
+ MYID_FILE=/data/myid                                                                                                                                                                                     │
+ LOG4J_CONF=/conf/log4j-quiet.properties                                                                                                                                                                  │
+ set +e                                                                                                                                                                                                   │
++ zkConnectionString                                                                                                                                                                                      │
++ set +e                                                                                                                                                                                                  │
++ nslookup zinc-chinchilla-zookeeper-client                                                                                                                                                               │
++ [[ 0 -eq 1 ]]                                                                                                                                                                                           │
++ set -e                                                                                                                                                                                                  │
++ echo zinc-chinchilla-zookeeper-client:9277                                                                                                                                                              │
+ ZKURL=zinc-chinchilla-zookeeper-client:9277                                                                                                                                                              │
+ set -e                                                                                                                                                                                                   │
++ cat /data/myid                                                                                                                                                                                          │
+ MYID=1                                                                                                                                                                                                   │
+ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar remove zinc-chinchilla-zookeeper-client:9277 1                                                                            │
Connecting to Zookeeper zinc-chinchilla-zookeeper-client:9277                                                                                                                                              │
, message: "+ source /conf/env.sh\n++ DOMAIN=zinc-chinchilla-zookeeper-headless.default.svc.cluster.local\n++ QUORUM_PORT=2888\n++ LEADER_PORT=3888\n++ CLIENT_HOST=zinc-chinchilla-zookeeper-client\n++ CL│

Wrong Cluster Membership in Dynamic Config File

Problem description
We have suspicions that Pravega components (e.g., BK, SSS) are having trouble to interact with Zookeeper sometimes, which leads to failures. In this sense, an hypothesis we have is that the configuration of Zookeeper may be behind those problems. In a simple experiment, we deployed 3 Zookeeper instances with the Zookeeper Operator, and in the dynamic config file of each instance, we see the following:

(zookeeper-0)

/ # cat /data/zoo.cfg.dynamic
server.1=zookeeper-0.zookeeper-headless.default.svc.cluster.local:2888:3888:participant;2181

(zookeeper-1)

/ # cat /data/zoo.cfg.dynamic
server.1=zookeeper-0.zookeeper-headless.default.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
server.2=zookeeper-1.zookeeper-headless.default.svc.cluster.local:2888:3888:observer;0.0.0.0:2181

(zookeeper-2)

/ # cat /data/zoo.cfg.dynamic
server.1=zookeeper-0.zookeeper-headless.default.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
server.2=zookeeper-1.zookeeper-headless.default.svc.cluster.local:2888:3888:participant;0.0.0.0:2181
server.3=zookeeper-2.zookeeper-headless.default.svc.cluster.local:2888:3888:observer;0.0.0.0:2181

So, apparently every Zookeeper instance has a different view of the cluster, which may lead to problems. According to the documentation of Zookeeper, all the instances should have a consistent configuration.

Problem location
Zookeeper operator dynamic config generation.

Suggestions for an improvement
At first glance, all the instances should have the same membership in zoo.cfg.dynamic with all the members of the Zookeeper cluster. Also, we may need to verify whether the role setting (observer/participant) is correctly set in this context.

Check the entire license header on Go files

Create an automated check that verifies that all Golang files contain the entire license header below.

/**
 * Copyright (c) 2018 Dell Inc., or its subsidiaries. All Rights Reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 */

Handle ZookeeperCluster Custom Resource Deletion

When the ZookeeperCluster resource is deleted the corresponding Kubernetes resources backing the cluster should also be removed.

Allow offline use

Hello,

In an offline env where we have a private registry (usually mirroring dockerhub)

• Add a env var to dockerfile to set the registry to use for emccorp/zookeeper image., which will allow to build a custom image without changing the code

or/and

• make the registry to use in env variables of the operator (idk if possible)

Thank you

Formatting README file

1. TOC will be included
2. Minor formatting changes

Add CI for Docker container builds

We should publish new Docker container images via CI for nightlies and versions.

ZK Cluster fails to come up when re-created with old PVCs

Steps to Reproduce:

1. Create a zookeeper operator and cluster.
2. Once cluster is up and running, delete cluster using kubectl delete zk<zk-cluster-name>
3. Try to re-create cluster using same .yaml file as before.
4. First server (zk-0) comes up correctly, but 2nd server hangs indefinately in crashLoopBackOff state. Other servers are never created.

Expected:
Zk Cluster should get re-created correctly with all previous data from existing PVCs.

Slow Zookeeper Pod Recovery (in PKS)

Problem description
In PR #69, the membership management of a Zookeeper cluster has been fixed. That is, at this point a Zookeeper ensemble has a consistent configuration along time, even in the presence of pod restarts.

However, during the investigation of #69, another problem popped up: the long time it takes for Zookeeper to recover from pod deletions/restarts in PKS. Zookeeper is a critical dependency for Pravega and Bookkeeper, so we have examples of the impact of this problem in numerous reported issues: pravega/pravega#3942, pravega/pravega#3836, pravega/pravega#3783, pravega/pravega#3954. That is, among other symptoms, having an unhealthy Zookeeper service running in the cluster for too long may impact the Controller (e.g., unexpected restarts), Bookkeeper (e.g., inability to access metadata), Segment Store (e.g., problems with container recoveries, container assignments), as well as client applications (e.g., readers may exhaust their retries).

In PKS, this problem can be consistently reproduced. This is an example:

1. We have a complete Pravega deployment in PKS:

λ kubectl.exe get pods
NAME                                                    READY   STATUS    RESTARTS   AGE
alert-seagull-nfs-client-provisioner-5d7745fd84-4dhpn   1/1     Running   0          51m
benchmark-pod-3                                         1/1     Running   0          41m
original-ibex-nfs-client-provisioner-86c46b5d6d-8d5kl   1/1     Running   0          56m
pravega-bookie-0                                        1/1     Running   0          2m45s
pravega-bookie-1                                        1/1     Running   0          2m45s
pravega-bookie-2                                        1/1     Running   0          2m45s
pravega-operator-554d769d4c-gbdj7                       1/1     Running   0          43m
pravega-pravega-controller-d599b4fd6-wj5cd              1/1     Running   0          2m45s
pravega-pravega-segmentstore-0                          1/1     Running   0          2m45s
zookeeper-0                                             1/1     Running   0          5m57s
zookeeper-1                                             1/1     Running   0          5m17s
zookeeper-2                                             1/1     Running   0          4m36s
zookeeper-3                                             1/1     Running   0          4m9s
zookeeper-4                                             1/1     Running   0          3m40s
zookeeper-operator-566694c7ff-stzkn                     1/1     Running   0          59m

2. I have started a workload to verify that everything works fine:

/bin/pravega-benchmark -controller tcp://172.25.194.11:9090 -producers 1 -segments 1 -size 100 -events 100 -time 300 -stream myStream
...
[epollEventLoopGroup-4-1] INFO io.pravega.client.segment.impl.SegmentOutputStreamImpl - Connection setup complete for writer efc88ec9-d8a7-436b-a966-02169cd8140c
     501 records Writing,     100.1 records/sec,   0.01 MB/sec,     8.2 ms avg latency,    87.0 ms max latency
     501 records Writing,     100.0 records/sec,   0.01 MB/sec,     5.0 ms avg latency,    13.0 ms max latency
     501 records Writing,     100.0 records/sec,   0.01 MB/sec,     5.2 ms avg latency,    23.0 ms max latency
     501 records Writing,     100.0 records/sec,   0.01 MB/sec,     4.7 ms avg latency,    11.0 ms max latency
     501 records Writing,     100.1 records/sec,   0.01 MB/sec,     6.4 ms avg latency,    58.0 ms max latency

3. Then, we delete 2 out of 5 Zookeeper pods, expecting the service to automatically recover them:

λ kubectl delete pods zookeeper-2 zookeeper-0 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "zookeeper-2" force deleted
pod "zookeeper-0" force deleted

4. Zookeeper pods and, consequently, Controller pods keep restarting while the service is trying to recover Zookeeper pods:

λ kubectl.exe get pods
NAME                                                    READY   STATUS    RESTARTS   AGE
alert-seagull-nfs-client-provisioner-5d7745fd84-4dhpn   1/1     Running   0          57m
benchmark-pod-3                                         1/1     Running   0          46m
original-ibex-nfs-client-provisioner-86c46b5d6d-8d5kl   1/1     Running   0          61m
pravega-bookie-0                                        1/1     Running   0          8m11s
pravega-bookie-1                                        1/1     Running   0          8m11s
pravega-bookie-2                                        1/1     Running   0          8m11s
pravega-operator-554d769d4c-gbdj7                       1/1     Running   0          49m
pravega-pravega-controller-d599b4fd6-wj5cd              0/1     Running   2          8m11s
pravega-pravega-segmentstore-0                          1/1     Running   0          8m11s
zookeeper-0                                             0/1     Running   4          4m50s
zookeeper-1                                             1/1     Running   0          10m
zookeeper-2                                             0/1     Running   4          4m51s
zookeeper-3                                             1/1     Running   0          9m35s
zookeeper-4                                             1/1     Running   0          9m6s
zookeeper-operator-566694c7ff-stzkn                     1/1     Running   0          64m

5. The restarted Zookeeer pods get stuck in the very same point as in the rest of tests we have done so far (see Connecting to Zookeeper zookeeper-client:2181):

λ kubectl.exe logs zookeeper-0
...
Name:      zookeeper-headless.default.svc.cluster.local
Address 1: 172.25.194.10 172-25-194-10.zookeeper-client.default.svc.cluster.local
Address 2: 172.25.194.9 172-25-194-9.zookeeper-client.default.svc.cluster.local
Address 3: 172.25.194.5 172-25-194-5.zookeeper-client.default.svc.cluster.local
+ [[ 0 -eq 1 ]]
+ set -e
+ set +e
++ zkConnectionString
++ set +e
++ nslookup zookeeper-client
++ [[ 0 -eq 1 ]]
++ set -e
++ echo zookeeper-client:2181
+ ZKURL=zookeeper-client:2181
+ set -e
++ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar get-all zookeeper-client:2181
Connecting to Zookeeper zookeeper-client:2181

After a period of time between 10 minutes to 15 minutes, the system comes back to stability.

---

However, the same experiment in GKE has a different outcome: I measured the time from the deletion of pods to their total recovery in 1min 10secs. In this case, the recovery was fast enough to do not induce any Controller restart:

raul_gracia@cloudshell:~ (pravega-dev)$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
coiling-shark-nfs-server-provisioner-0      1/1     Running   0          5h54m
pravega-bookie-0                            1/1     Running   1          6m22s
pravega-bookie-1                            1/1     Running   0          6m22s
pravega-bookie-2                            1/1     Running   0          6m22s
pravega-operator-687bbf897b-82z69           1/1     Running   0          155m
pravega-pravega-controller-f4dfc887-9gzb4   1/1     Running   0          6m22s
pravega-pravega-segmentstore-0              1/1     Running   0          6m22s
zookeeper-0                                 1/1     Running   0          72s
zookeeper-1                                 1/1     Running   0          8m57s
zookeeper-2                                 1/1     Running   0          23s
zookeeper-3                                 1/1     Running   0          7m49s
zookeeper-4                                 1/1     Running   0          7m5s
zookeeper-operator-5fb6d7cf4d-tlpsr         1/1     Running   0          5h```

---

Observations of this issue so far:

• The issue is reproducible in PKS, not in GKE.
• While in PKS it may take minutes for Zookeeper to stabilize, in general we observed that this is a transient problem and the system recovers.
• In PKS, the slow Zookeeper pod restarts occurs the first time we delete Zookeeper pods. Subsequent Zookeeper pod deletions are recovered fast, without problems. This leads us to think that after the Zookeeper cluster recovers from the first deletion, the resolution for the name zookeeper-client:2181 is cached (this is the point in which Zookeeper pods get stuck when trying to recover). If the name resolution in PKS was slow due to some DNS problem, this may explain why subsequent pod deletions are handled faster.

Problem location
PKS networking configuration? DNS?

Suggestions for an improvement

• Try with a longer readiness probe interval in Zookeeper pods.
• Understand and potentially fix the DNS architecture in our PKS clusters.

Can we use this operator to deploy zookeeper nodes across multiple clusters and multi-data center

Basically is this using reconfig API is being used when zk nodes come up/goes down across cluster/data-center?

Support full zookeeper configuration options

The ZK config map does not currently include the full compliment of configuration options. The remaining config options should be added to the config map spec.

Error trying to run the operator locally

I'm trying to run the operator locally as instructed in the operator-sdk user guide using the operator-sdk up local command. However, the operators panics and outputs the following log:

$ operator-sdk up local
INFO[0000] Go Version: go1.11
INFO[0000] Go OS/Arch: linux/amd64
INFO[0000] operator-sdk Version: 0.0.5+git
INFO[0000] Watching zookeeper.pravega.io/v1beta1, ZookeeperCluster, default, 5
panic: No Auth Provider found for name "gcp"

goroutine 1 [running]:
github.com/pravega/zookeeper-operator/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1alpha1.NewForConfigOrDie(0xc000394000, 0xc0003e62d0)     
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1alpha1/admissionregistration_client.go:58 +0x65
github.com/pravega/zookeeper-operator/vendor/k8s.io/client-go/kubernetes.NewForConfigOrDie(0xc000394000, 0x0)                                                   
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/vendor/k8s.io/client-go/kubernetes/clientset.go:529 +0x49             
github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/k8sclient.mustNewKubeClientAndConfig(0x59, 0xc0001f5c30, 0xe840f0)  
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/k8sclient/client.go:138 +0x68
github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/k8sclient.newSingletonFactory()                                     
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/k8sclient/client.go:52 +0x34
sync.(*Once).Do(0x1ba0390, 0x1146e50)
        /home/adrian/.gvm/gos/go1.11/src/sync/once.go:44 +0xb3
github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/k8sclient.GetResourceClient(0x10e256f, 0x1c, 0x10d7753, 0x10, 0xc00003e470, 0x7, 0xc000430580, 0xc0001f5e20, 0xe8304e, 0xc0004e6410, ...)
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/k8sclient/client.go:70 +0x3d
github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/sdk.Watch(0x10e256f, 0x1c, 0x10d7753, 0x10, 0xc00003e470, 0x7, 0x12a05f200, 0x0, 0x0, 0x0)
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/vendor/github.com/operator-framework/operator-sdk/pkg/sdk/api.go:45 +0x84
main.main()
        /home/adrian/.gvm/pkgsets/go1.11/global/src/github.com/pravega/zookeeper-operator/cmd/zookeeper-operator/main.go:31 +0x215                              
exit status 2
Error: failed to run operator locally: exit status 1

We also experienced this error with the Pravega operator (pravega/pravega-operator#39) and fixed it by importing the gcp package. Will apply the same fix and submit a PR.

Add Travis status badges to README

Add integration smoke tests of zookeeper clients

Add CI for unit tests

We intend to use travis for CI of the project.

Revisit logic to make server participant in zookeeperReady.sh

Based on this thread about the process of making a server a participant in PR #69 :

https://github.com/pravega/zookeeper-operator/pull/69/files#r295982466

It is unclear why we need to remove the server as a observer before we add it as a participant. The first step should not be needed.

Persistence section in manifest seems mandatory

Problem description
The sample manifest in the README does not work with the latest version of the Zookeeper operator. The main reason is that the persistence section seems to be mandatory, probably from PR #62. If it is not set, the Zookeeper operator throws a cryptic error and Zookeeper is not deployed:

{"level":"info","ts":1569916028.5814774,"logger":"controller_zookeepercluster","msg":"Updating zookeeper status","Request.Namespace":"default","Request.Name":"zookeeper","StatefulSet.Namespace":"default","StatefulSet.Name":"zookeeper"}
{"level":"error","ts":1569916028.585655,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"zookeepercluster-controller","request":"default/zookeeper","error":"Operation cannot be fulfilled on zookeeperclusters.zookeeper.pravega.io \"zookeeper\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/pravega/zookeeper-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/pravega/zookeeper-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/pravega/zookeeper-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/pravega/zookeeper-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/pravega/zookeeper-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/pravega/zookeeper-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/pravega/zookeeper-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/pravega/zookeeper-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/pravega/zookeeper-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/pravega/zookeeper-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/pravega/zookeeper-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/pravega/zookeeper-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1569916029.586072,"logger":"controller_zookeepercluster","msg":"Reconciling ZookeeperCluster","Request.Namespace":"default","Request.Name":"zookeeper"}
{"level":"info","ts":1569916029.586174,"logger":"controller_zookeepercluster","msg":"Updating existing config-map","Request.Namespace":"default","Request.Name":"zookeeper","ConfigMap.Namespace":"default","ConfigMap.Name":"zookeeper-configmap"}
{"level":"info","ts":1569916029.5912735,"logger":"controller_zookeepercluster","msg":"Updating StatefulSet","Request.Namespace":"default","Request.Name":"zookeeper","StatefulSet.Namespace":"default","StatefulSet.Name":"zookeeper"}
{"level":"info","ts":1569916029.5961022,"logger":"controller_zookeepercluster","msg":"Updating existing client service","Request.Namespace":"default","Request.Name":"zookeeper","Service.Namespace":"default","Service.Name":"zookeeper-client"}

Adding the persistence section to the manifest fixes the problem and allows me to deploy Zookeeper.

Problem location
Persistence section of the operator.

Suggestions for an improvement
We need to take a decision: should the persistence section be mandatory or not?
In the affirmative case, we need to:

1. Fix the Zookeeper operator README file, concretelly in the sample Zookeeper manifest provided and add the persistence section.
2. Throw a clearer error in the operator logs if a user missed to specify this section.

Otherwise, we need to set proper defaults for the persistence section and allow the deployment of Zookeeper without that section in the manifest.

storageClassName not applied from ZookeeperCluster yaml

I've been trying to get the example zk cluster running while specifying a specific storageClass via the PersistentVolumeClaimSpec that's under "persistence" in the ZookeeperCluster yaml and it doesn't appear to be applying that field. It keeps attempting to apply the cluster's default storage class.

apiVersion: "zookeeper.pravega.io/v1beta1"
kind: "ZookeeperCluster"
metadata:
  name: "example"
  namespace: "default"
spec:
  persistence:
    storageClassName: cluster-storage
    accessModes:
    - ReadWriteMany
    resources:
      requests:
        storage: 5Gi
  size: 3

Handle scale-up/scale-down operations

Depends upon #4. There may be other requirements to handle scale-up and down using kubectl scale. However, the cluster can be scaled by changing the cluster spec and re-applying wtih kubectl apply.

Docker images are released with a wrong tag

Yesterday, @Tristan1900 and I released version 0.2.2. We created an operator release in GitHub, and then a Travis build was triggered and Docker images were created and pushed as part of the deploy stage.

zookeeper-operator/.travis.yml

Line 23 in 43a0bd7

script: make push

However, the image tag was formed incorrectly and the resulting image was pushed as 0.2.2-dirty instead of 0.2.2, making the release inaccessible through the expected tag.

Docker image tag is obtained through git tag information.

zookeeper-operator/Makefile

Line 18 in 43a0bd7

VERSION=$(shell git describe --always --tags --dirty | sed "s/\(.*\)-g`git rev-parse --short HEAD`/\1/")

The above git describe command is using the -dirty flag, which adds the "-dirty" suffix to the git tag if the working tree has local modification.

That means that the Travis build process generates modifications that are not git-ignored. I was able to reproduce this locally and I found out that the make dep can make modifications to the vendor directory and the Gopkg.lock file that are causing make build to create images with the wrong tag.

Two possible solutions to this are:

1. Remove the -dirty flag from the git describe command.
2. Add Gopkg.lock and vendor to the .gitignore file. Maybe we can also remove Gopkg.lock from the repo.

@Tristan1900 @spiegela what do you think?

Upgrade to Zookeeper image 3.5.5

Resource status not updating correctly

Although there is a concept of resource status for the ZookeeperCluster, the status is never updated throughout the lifecycle of the reconciliation.
We should be updating the ZookeeperCluster status with the correct node count of the stateful set here: https://github.com/pravega/zookeeper-operator/blob/master/pkg/zk/sync.go#L38-L44

This is the current status of a fully deployed ZookeeperCluster:

~ kubectl get zookeeper-cluster some-zookeeper-cluster -n some-namespace -o json
. . .
    "status": {
        "members": {
            "ready": null,
            "unready": null
        },
        "size": 0
    }
. . .

Changing Zookeeper resource "storage" value through `kubectl edit` not reflecting

Changing Zookeeper resource "storage" values through kubectl edit not reflecting for new/scaledup zookeeper pods

Increasing the zookeeper storage size does not work for the already deployed zookeeper pods & also for the new/scaledup pods.

# kubectl edit zk some-zookeeper-cluster -n some-namespace
...
    resources:
      requests:
        storage: 20Gi  --> Increased to 50Gi
...

PVC are not deleted when ZK is deleted

When a ZookeeperCluster resource is deleted (i.e. kubectl delete zk example), PVC are not automatically deleted. PVC should have owner reference information so that they are attached to the CR lifecycle.

Print operator version

On startup, the operator should print its version and commit SHA. It will be very useful information for troubleshooting. Moreover, there should be a -version flag to print the version and exit.

Add Travis Build

RBAC Fails to deploy on GKE

When deploying the operator with kubectl apply -f deploy the RBAC resource fails to deploy on GKE:

$> kubectl apply -f deploy
customresourcedefinition.apiextensions.k8s.io "zookeeper-clusters.zookeeper.pravega.io" created
deployment.apps "zookeeper-operator" created
rolebinding.rbac.authorization.k8s.io "default-account-zookeeper-operator" created
Error from server (Forbidden): error when creating "deploy/rbac.yaml": roles.rbac.authorization.k8s.io "zookeeper-operator" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["*"], APIGroups:["zookeeper.pravega.io"], Verbs:["*"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["persistentvolumeclaims"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["events"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["deployments"], APIGroups:["apps"], Verbs:["*"]} PolicyRule{Resources:["daemonsets"], APIGroups:["apps"], Verbs:["*"]} PolicyRule{Resources:["replicasets"], APIGroups:["apps"], Verbs:["*"]} PolicyRule{Resources:["statefulsets"], APIGroups:["apps"], Verbs:["*"]} PolicyRule{Resources:["poddisruptionbudgets"], APIGroups:["policy"], Verbs:["*"]}] user=&{[email protected]  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]

Solution

See prometheus-operator/prometheus-operator#357 for a solution which should be added to the README.

But in short, running the following before deploying the Operator works:

kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin [email protected]

Rbac error in all_ns and default_ns

I am having troubles creating rbac in all_ns and default_ns.
I think problems are located here and here. The first one should have a namespace specified and the second one is using default instead of zookeeper-operator

allow to set a SeviceAccount to cluster Statefulset

We need to be able to define a Kubernetes ServiceAccount in the Statefulset, so we can use images from private repositories (with specific ImagePullSecrets).

Add versioning

Since we are about to open up the repo and make the first release. We need to add versioning to the project.

Add support for data-checking on container restart

Currently the zookeeper-operator assumes that if a restarted instance's zoo.cfg.dynamic is valid, then so is the rest of the information under /data. The zookeeper operator should detect that a pod has been restarted with a valid zoo.cfg.dynamic, and then check the status of the data, restoring or removing the data if necessary.

Should support and default to zookeeper official image

3.5 zookeeper has an official image on Dockerhub here.

the emccorp one its not clear its maintained, the latest is 6 mo old and is pre-release.

should switch to default to the official image.

this also means changing the command

Set appropriate default zookeeper image

The default image used to deploy zookeeper, in the absence of any image being provided in the manifest, is emccorp/zookeeper:3.5.4-beta-operator. This image does not have the fix for issue #66 and hence causes problems when developer is not careful enough to specify the right image in the manifest.
Please change the default zookeeper image to pravega/zookeeper:0.2.2 or later

Resource Status not updated

When deploying a zookeeper cluster with 0.2.0 zookeeper-operator, the ZookeeperCluster resource does not get the Status section updated and always stays at

Status:
  External Client Endpoint:  
  Internal Client Endpoint:  
  Members:
    Ready:         <nil>
    Unready:       <nil>
  Ready Replicas:  0
  Replicas:        0

Example:
We've deployed a zk-cluster in namespace test-project that is supposed to have three replicas.
Inspecting the related stateful set reveals that there are indeed three replicas running in the system:

$kubectl get statefulsets -n test-project
NAME        DESIRED   CURRENT   AGE
zookeeper   3         3         53m

Inspecting the ZookeeperCluster existing in the project, we get the Status section showing zero replicas.

$kubectl describe zk zookeeper -n test-project
Name:         zookeeper
Namespace:    test-project
Labels:       <none>
Annotations:  <none>
API Version:  zookeeper.pravega.io/v1beta1
Kind:         ZookeeperCluster
Metadata:
  Creation Timestamp:  2019-03-01T19:08:21Z
  Generation:          1
  Owner References:
    API Version:           nautilus.dellemc.com/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Project
    Name:                  test-project
    UID:                   60c9fb21-3c55-11e9-a64e-005056bdd0fa
  Resource Version:        620540
  Self Link:               /apis/zookeeper.pravega.io/v1beta1/namespaces/test-project/zookeeper-clusters/zookeeper
  UID:                     6117246e-3c55-11e9-a64e-005056bdd0fa
Spec:
  Config:
    Init Limit:  10
    Sync Limit:  2
    Tick Time:   2000
  Image:
    Pull Policy:  Always
    Repository:   emccorp/zookeeper
    Tag:          3.5.4-beta-operator
  Labels:
    App:      zookeeper
    Release:  zookeeper
  Persistence:
    Access Modes:
      ReadWriteOnce
    Data Source:  <nil>
    Resources:
      Requests:
        Storage:  20Gi
  Pod:
    Affinity:
      Pod Anti Affinity:
        Preferred During Scheduling Ignored During Execution:
          Pod Affinity Term:
            Label Selector:
              Match Expressions:
                Key:       app
                Operator:  In
                Values:
                  zookeeper
            Topology Key:  kubernetes.io/hostname
          Weight:          20
    Labels:
      App:      zookeeper
      Release:  zookeeper
    Resources:
    Termination Grace Period Seconds:  30
  Ports:
    Container Port:  2181
    Name:            client
    Container Port:  2888
    Name:            quorum
    Container Port:  3888
    Name:            leader-election
  Replicas:          3
  Size:              3
Status:
  External Client Endpoint:  
  Internal Client Endpoint:  
  Members:
    Ready:         <nil>
    Unready:       <nil>
  Ready Replicas:  0
  Replicas:        0
Events:            <none>

Support zookeeper rolling upgrades

Currently rolling updates should work, but have not been tested. Rolling updates should be tested an validated.

Add CI for test coverage reporting

Zookeeper health check

This is to add Helm Charts to Zookeeper so as to implement helm test on it so that post an upgrade we can determine whether the upgrade was successful or not by simply running a helm test <release-name>

creating CONTRIBUTING file

1. Inclusion of CONTRIBUTING.md in the root directory
2. Inclusion of Content in wiki

Add zookeeper cluster status support

Currently, the zk Status cluster is blank. This should be populated with data from the associated stateful-set. This is also required for the kubectl scale command to work.

Allow Zookeepers to be run with ephemeral storage.

Currently the operator only allows for persistent volumes, however for certain use cases it can be very useful to have ephemeral storage instead of persistent storage.

This could be implemented like the etcd-operator, where EmptyDir is used when no PVC spec is provided. The issue there is that the change would be backwards incompatible. The API documentation actually states that this is the behavior, but it isn't implemented.

The changes would be relatively minimal and would just require edits to the statefulSet generator and api.

Add recovery handling for restarted pods

Currently, there are not detailed checks of data within a pod's persistent volumes. If a pod starts, and the network identity, zookeeper ordinal and cluster membership all match, then the data is assumed correct. If any of those don't match, the data is removed.

The desired behavior is for the Pod, during startup, to check the consistency of data, and potentially leverage ZK snaps to recover data.

unhealthy zookeepers

Installed Nautilus to a new nightshift cluster, came back ~14 hours later and pravega bookie pods were crashing, zookeeper pods were running but unhealthy. Logs attached.

$ kubectl get pods --namespace nautilus-pravega
NAME                                           READY   STATUS             RESTARTS   AGE
nautilus-bookie-0                              0/1     CrashLoopBackOff   39         18h
nautilus-bookie-1                              0/1     CrashLoopBackOff   45         18h
nautilus-bookie-2                              1/1     Running            0          18h
nautilus-pravega-controller-685d99d988-qg2vq   1/1     Running            2          18h
nautilus-pravega-grafana-pod                   1/1     Running            0          18h
nautilus-pravega-influxdb-pod                  1/1     Running            0          18h
nautilus-pravega-segmentstore-0                1/1     Running            0          18h
nautilus-pravega-segmentstore-1                1/1     Running            0          18h
nautilus-pravega-segmentstore-2                1/1     Running            0          18h
nautilus-pravega-telegraf-pod                  1/1     Running            0          18h
nautilus-pravega-zookeeper-0                   1/1     Running            0          18h
nautilus-pravega-zookeeper-1                   1/1     Running            0          18h
nautilus-pravega-zookeeper-2                   1/1     Running            0          18h
pravega-operator-54cdd8ccf9-9pbx6              1/1     Running            0          18h
pravega-service-broker-c755d8df9-nw9jd         1/1     Running            0          18h

bookies-crashed.log

zk-0.log.zip

Add unit tests

There are no tests for this project right now. We should expect fairly complete coverage.

Currently prioritizing the internal packages:

• v1beta1.ClusterSpec#withDefaults
• v1beta1.ContainerImaage#withDefaults
• v1beta1.PodPolicy#withDefaults
• v1beta1.ZookeeperConfig#WithDefaults
• deploy#makeZkSts
• deploy#makePodSpec
• deploy#makeZkHeadlessSvc
• deploy#makeZkClientSvc
• deploy#makeZkConfigMap
• deploy#makeZkPdb

Fix Windows Cross compilation failure

Since last 2 days, make build-go fails without any changes to zookeeper-operator code, with this error:

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
-ldflags "-X github.com/pravega/zookeeper-operator/pkg/version.Version=0.2.3-10 -X github.com/pravega/zookeeper-operator/pkg/version.GitSHA=68bf700" \
-o bin/zookeeper-operator-linux-amd64 cmd/manager/main.go
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
-ldflags "-X github.com/pravega/zookeeper-operator/pkg/version.Version=0.2.3-10 -X github.com/pravega/zookeeper-operator/pkg/version.GitSHA=68bf700" \
-o bin/zookeeper-exporter-linux-amd64 cmd/exporter/main.go
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build \
-ldflags "-X github.com/pravega/zookeeper-operator/pkg/version.Version=0.2.3-10 -X github.com/pravega/zookeeper-operator/pkg/version.GitSHA=68bf700" \
-o bin/zookeeper-operator-darwin-amd64 cmd/manager/main.go
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build \
-ldflags "-X github.com/pravega/zookeeper-operator/pkg/version.Version=0.2.3-10 -X github.com/pravega/zookeeper-operator/pkg/version.GitSHA=68bf700" \
-o bin/zookeeper-exporter-darwin-amd64 cmd/exporter/main.go
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build \
-ldflags "-X github.com/pravega/zookeeper-operator/pkg/version.Version=0.2.3-10 -X github.com/pravega/zookeeper-operator/pkg/version.GitSHA=68bf700" \
-o bin/zookeeper-operator-windows-amd64.exe cmd/manager/main.go
# github.com/pravega/zookeeper-operator/vendor/golang.org/x/sys/windows
vendor/golang.org/x/sys/windows/dll_windows.go:21:6: missing function body
vendor/golang.org/x/sys/windows/dll_windows.go:24:6: missing function body
Makefile:31: recipe for target 'build-go' failed
make: *** [build-go] Error 2
The command "make build" exited with 2.

This error was seen only with go versions < 1.12.
Hence need to upgrade go version used by zk-operator to 1.12+

Updating README.md

Modification of content in Build operator image Section

Support 'rescale' operation / rename 'size' to 'replicas'

Rescale is a standard operation that some resource types support. Unsure about the specifics. May involve renaming 'size' to 'replicas'.

Add coverage for error conditions in reconcile loop

Add a LICENSE file and headers

Before opening up the repo we'll need to upload a LICENSE file and add the license headers to all Go files. We can apply the same solution as in pravega/pravega-operator#45.

Allow blank watch namespace

A blank watch namespace indicates to watch ALL namespaces for the ZookeeperCluster resource, however the SDK currently blocks a blank namespace.

second zk pod doesn't get ready

I tried starting a simple cluster (using the "latest" image built in late July). The second pod ("zk-cluster-1") never gets ready and the third doesn't start up.

describe pod zk-cluster-2 says:

++ echo zk-cluster-client:2181
+ ZKURL=zk-cluster-client:2181
+ set -e
++ cat /data/myid
+ MYID=2
++ java -Dlog4j.configuration=file:/conf/log4j-quiet.properties -jar /root/zu.jar get-role zk-cluster-client:2181 2
Connecting to Zookeeper zk-cluster-client:2181
Server not found in zookeeper config
+ ROLE=

Logs from the second pod just shows the localhost ruok requests. On the first pod ("zk-cluster-0") it shows the connections from the first pod, but I guess it never successfully joins the cluster.

2019-08-17 03:23:07,288 [myid:1] - INFO  [NIOWorkerThread-1:ZooKeeperServer@1041] - Client attempting to establish new session at /10.42.6.236:43346
2019-08-17 03:23:07,293 [myid:1] - INFO  [CommitProcWorkThread-1:ZooKeeperServer@748] - Established session 0x10004784d030048 with negotiated timeout 4000 for client /10.42.6.236:43346
2019-08-17 03:23:07,690 [myid:1] - WARN  [NIOWorkerThread-1:NIOServerCnxn@366] - Unable to read additional data from client sessionid 0x10004784d030048, likely client has closed socket
2019-08-17 03:23:07,691 [myid:1] - INFO  [NIOWorkerThread-1:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Leader,name3=Connections,name4=10.42.6.236,name5=0x10004784d030048]
2019-08-17 03:23:07,691 [myid:1] - INFO  [NIOWorkerThread-1:NIOServerCnxn@627] - Closed socket connection for client /10.42.6.236:43346 which had sessionid 0x10004784d030048
2019-08-17 03:23:12,201 [myid:1] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:354