Che/CodeReady workspaces operator uses Operator SDK and Go Kube client to deploy, update and manage K8S/OpenShift resources that constitute a multi user Eclipse Che/CodeReady Workspaces cluster.
The operator watches for a Custom Resource of Kind CheCluster
, and operator controller executes its business logic when a new Che object is created, namely:
- creates k8s/OpenShift objects
- verifies successful deployment of Postgres, Keycloak and Che
- runs exec into Postgres and Keycloak pods to provisions databases, users, realm and clients
- updates CR spec and status (passwords, URLs, provisioning statuses etc.)
- continuously watches CR, update Che ConfigMap accordingly and schedule a new Che deployment
- changes state of certain objects depending on CR fields:
- turn on/off TLS mode (reconfigure routes, update ConfigMap)
- turn on/off OpenShift oAuth (login with OpenShift in Che) (create identity provider, oAuth client, update Che ConfigMap)
- updates Che deployment with a new image:tag when a new operator version brings in a new Che tag
The project is in its early development and breaking changes are possible.
IMPORTANT! Cluster Admin privileges are required
./deploy.sh $namespace
The script will create sa, role, role binding, operator deployment, CRD and CR.
Wait until Che deployment is scaled to 1 and Che route is created.
When on pure k8s, make sure you provide a global ingress domain in deploy/crds/org_v1_che_cr.yaml
for example:
k8s:
ingressDomain: '192.168.99.101.nip.io'
The following instructions show how to test Che operator under development using OLM installer. Steps below are applicable to Openshift infrastructure only.
- Build your custom operator image
docker build -t user/che-operator .
and push it to a docker registry.
-
Specify your operator image. Open deploy/operator.yaml, replace default operator image
quay.io/eclipse/che-operator:nightly
with yours (say,docker.io/user/che-operator:latest
). -
Create newer OLM files by executing:
olm/update-nightly-olm-files.sh
-
Build catalog source image. Go to
olm/eclipse-che-preview-openshift
folder and build the image:docker build -t user/custom-catalog-source:latest .
Push it into your docker registry. -
Create custom catalog source yaml:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: eclipse-che-preview-openshift
namespace: che-namespace
spec:
image: docker.io/user/custom-catalog-source:latest
sourceType: grpc
Replace value of image
field with your catalog source image.
- Deploy Che using chectl:
chectl server:start --installer=olm --multiuser --platform=openshift -n che-namespace --catalog-source-yaml /home/user/path/to/custom-catalog-source.yaml --olm-channel=nightly --package-manifest-name=eclipse-che-preview-openshift
Bear in mind that che-operator service account needs to have cluster admin privileges so that the operator can create oauthclient at a cluster scope.
There is oc adm
command in both deploy scripts. Uncomment it if you need this feature.
Make sure your current user has cluster-admin privileges.
TLS is enabled by default. Turning it off is not recommended as it will cause malfunction of some components.
When the cluster is configured to use self-signed certificates for the router, the certificate will be automatically propogated to Che components as trusted.
If cluster router uses certificate signed by self-signed one, then parent/root CA certificate should be added into corresponding config map of additional trusted certificates (see serverTrustStoreConfigMapName
option).
By default self-signed certificates for Che will be generated automatically.
If it is needed to use own certificates, create che-tls
secret (see k8s.tlsSecretName
option) with key.crt
and tls.crt
fields. In case of self-signed certificate self-signed-certificate
secret should be created with public part of CA certificate under ca.crt
key in secret data.
The operator watches all objects it creates and reconciles them with CR state. It means that if you edit a configMap che, the operator will revert changes. Since not all Che configuration properties are custom resource spec fields (there are simply too many of them), the operator creates a second configMap called custom which you can use for any environment variables not supported by CR field. The operator will not reconcile configMap custom.
docker build -t $registry/$repo:$tag .
You can then use the resulting image in operator deployment (deploy/operator.yaml)
There's a little script that will build a local Docker image and deploy an operator to a selected namespace, as well as create service account, role, role binding, CRD and example CR.
oc new-project $namespace
build_deploy_local.sh $namespace
The above method will work only with Docker 17.x (does not works if you want to build in MiniShift/MiniKube). Mostly useful if you run oc cluster up
locally.
You can run/debug this operator on your local machine (not deployed to a k8s cluster), provided that the below pre-reqs are met.
Go client grabs kubeconfig either from InClusterConfig or ~/.kube locally. Make sure you oc login (or your current kubectl context points to a target cluster and namespace), and current user/server account can create objects in a target namespace.
The operator detects namespace to watch by getting value of WATCH_NAMESPACE
environment variable.
You can set it in Run configuration in the IDE, or export this env before executing the binary.
This applies both to Run and Debug.
The Operator takes these files and replaces values to get a string used as the exec
command to configure Keycloak.
Make sure you run the following before running/debugging:
cp templates/keycloak_provision /tmp/keycloak_provision
cp templates/oauth_provision /tmp/oauth_provision
These files are added to a container image, and thus this step is not required when deploying an Operator image.
e2e
directory contains end-to-end tests that create a custom resource, operator deployment, required RBAC.
Pre-reqs to run end-to-end (e2e) tests:
- a running OpenShift instance (3.11+)
- current oc/kubectl context as a cluster admin user
OOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o $GOPATH/src/github.com/eclipse/che-operator/run-tests $GOPATH/src/github.com/eclipse/che-operator/e2e/*.go
Or you can build in a container:
docker run -ti -v /tmp:/tmp -v ${OPERATOR_REPO}:/opt/app-root/src/go/src/github.com/eclipse/che-operator registry.redhat.io/devtools/go-toolset-rhel7:1.11.5-3 sh -c "OOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o /tmp/run-tests /opt/app-root/src/go/src/github.com/eclipse/che-operator/e2e/*.go"
cp /tmp/run-tests ${OPERATOR_REPO}/run-tests
The resulted binary is created in the root of the repo. Make sure it is run from this location since it uses relative paths to yamls that are then deserialized.
There's a script run-okd-local.sh
which is more of a CI thing, however, if you can run oc cluster up
in your environment, you are unlikely to have any issues.
./run-tests
Tests create a number of k8s/OpenShift objects and generally assume that a fresh installation of OpenShift is available. TODO: handle AlreadyExists errors to either remove che namespace or create a new one with a unique name.
A custom resource is created, which signals the operator to deploy Che/CRW with default settings.
Once an successful installation of Che/CRW is verified, tests patch custom resource to:
- enable oAuth
- enable TLS mode
Subsequent checks verify that the installation is reconfigured, for example uses secure routes or ConfigMap has the right Login-with-OpenShift values
TODO: add more scenarios