redhatinsights/clowder

Operator for operating cloud.redhat.com

What is Clowder?

Clowder is a kubernetes operator designed to make it easy to deploy applications running on the cloud.redhat.com platform in production, testing and local development environments.

Learn More

See Clowder in Action

Why use Clowder?

In addition to reducing the effort to maintain a deployment template, Clowder provides a number of other benefits:

• Consistent deployment Whether you’re deploying to production, running smoke tests on a PR, or developing your application locally, Clowder will use the same app definition for all three. No more endlessly tweaking environment variables!

• Focus on development Clowder has the best practices of running an app in a microservices environment as well as specific requirements from the app-sre team, such as pod affinity, rollout parameters etc built-in. Spend less time worrying about configuring deployment templates and more time writing your app.

• Assisting Ops Any dev or SRE that learns how Clowder deploys apps will implicitly understand the deployment of any other app utilizing Clowder.

• Deploy a full environment locally Gone are the days of hacking together scripts that just about get you mocked or partially working dependant services. With Clowder, you can deploy an instance of the cloud.redhat.com platform on your local laptop, or in a dev cluster to use as you wish.

Clowder will provision resources depending on the mode choosen for each provider, and will return a consistently formatted JSON configuration document for each app to consume, leaving teams to focus more on writing code than differences between environments. The Clowder config client can assist with this and currently has support for Python, Go, Javascript and Ruby.

Feature List

Clowder currently features support for:

• Kafka Topics

• Object Storage

• PostgreSQL Database

• In-Memory DB

• Feature Flags

• CronJob support

• Jobs Support

• Istio integration

• TLS Sidecar

• Simple Autoscaling

Roadmap

Our current roadmap looks like this:

• Autoscaling (possibly via Keda)

• Dynamic routing for public web sevices

• Automatic metrics configuration

• Automatic network policy configuration

• Standard, configurable alerting: Error rate, latency, Kafka topic lag, etc

• Canary deployments (possibly via Flagger)

• Operational remediations

• Observe service status from a ClowdApp status, based on sevice dependencies.

• Advanced logging configurations (e.g. logging messages to kafka topics)

Getting Clowder

Clowder is already running in pre-prod/prod environments.

To run Clowder locally in Minikube, obtain and install Minikube.

Clowder is developed on Fedora and the kvm driver has been found to work best initiated with the following options:

minikube start --cpus 4 --disk-size 36GB --memory 16000MB --driver=kvm2 --addons registry --addons ingress  --addons=metrics-server --disable-optimizations

ℹ️	Mac OS is also supported with the virtualbox and hyperkit drivers. A full guide can be found here

To persist these changes for every minikube invocation, run the following:

minikube config set cpus 4
minikube config set memory 16000
minikube config set disk-size 36GB
minikube config set driver kvm2

If you encounter any kvm issues, please take a look at the troubleshooting guide

The kube_setup.sh script then needs to be run by invoking

./build/kube_setup.sh

Clowder can then be installed by running:

# Be sure to get the latest release in the link above!
minikube kubectl -- apply -f $(curl https://api.github.com/repos/RedHatInsights/clowder/releases/latest | jq '.assets[0].browser_download_url' -r) --validate=false

Usage

To use Clowder to deploy an application a ClowdEnvironment resource must be present to define an environment. Once this has been deployed, a ClowdApp resource is authored for the app and deployed alongside the ClowdEnvironment.

Example app developer workflow:

1. Install Clowder on a minikube environment.

2. Use kubectl apply -f clowdenv.yaml to apply a ClowdEnvironment resource to the cluster.

3. Use kubectl apply -f clowdapp.yaml to apply a ClowdApp resource to the cluster.

More details on how to do this are present in the Getting Started section of the documentation.

API Reference

Building Clowder

If you want to run a version of Clowder other than the released version there are a few prerequisites you will need. To learn about developing Clowder please visit the developing clowder page for more detailed instructions.

History

To understand more about the design decisions made while developing Clowder, please visit the design document

Connect

Any questions, please ask one of the Clowder development team

• @psav

• @bsquizz

• @bennyturns

• @adamrdrew

clowder's Issues

Database creds are not randomized

Auto update deployments upon config change

CI pipeline doesn't handle no test results runs

I observed the CI pipeline is failing on #420 , which tackles a small error in the README, which I presume doesn't run any tests as no code changes where made on it. However checking the CI logs it seems the system expects some sort of test result regardless, making the PR validation to fail.

Database creds are regenerated when spec changes

Currently when the spec changes, the creds are reset. This causes the PVC to be useless as the creds have been replaced. The creds should be retrieved if they have already been set and not overwritten.

Broken links in Getting Started doc page

I found what I think are a few broken links in the getting started page.

I tried to investigate where these links should point in order to try and help fix them, as follows:

• In Install clowder in your local box section there is a link for the developer guide that I think should point here: https://github.com/RedHatInsights/clowder/blob/master/docs/antora/modules/ROOT/pages/developer-guide.adoc
• In Create your first ClowdEnvironment there is a link for the ephemeral environments getting started that I think should point here: https://consoledot.pages.redhat.com/docs/dev/getting-started/ephemeral/index.html
• Also in this section, there is a link pointing to the default ephemeral template which may point here instead: https://github.com/RedHatInsights/bonfire/blob/master/bonfire/resources/ephemeral-cluster-clowdenvironment.yaml
• In Create your first ClowdApp section there is a link to ClowdApps API which I think should point here: https://redhatinsights.github.io/clowder/clowder/dev/api_reference.html

Hope this helps

Set rollout parameters based on recommendations from App SRE

Apply default resources to containers when app doesn't set them

docs: broken link to bonfire ephemeral environments

the getting started page (at /docs/antora/modules/usage/pages/getting-started.adoc) links to a broken reference for working with bonfire, in this stanza:

Bonfire is a cli tool used to deploy apps into ephemeral environments. Read about getting started with bonfire on ephemeral environments here

that linked page no longer exists, and there doesn't seem to be any equivalent one on the consoledot.pages docs.

looks like it can be safely replaced with the bonfire repo readme page.

other suggestions (which are probably too general for this purpose):

• IQE testing in ephemeral environments on the redhat engineering docs
• ephemeral environment and service onboarding on the SMQE wiki redhat-source page

Pull Policy defined by InsightsApp

Base liveness and readiness probes point to wrong path and hardcoded port

Add maker for app-interface s3 object store config

nil kafka config blows up operator

Pull kafka resource to get broker info

Implement DB maker for app-interface

Configure manager to watch multiple namespaces

https://sdk.operatorframework.io/docs/building-operators/golang/operator-scope/#watching-resources-in-a-set-of-namespaces

Apply pod anti-affinity configuration to pod templates

Cannot setup in minikube

Description

The kube_setup.sh script fails to deploy the strimzi operator in minikube.

Steps to Reproduce

1. minikube start --cpus 4 --disk-size 36GB --memory 8000MB --addons=registry --addons=ingress --driver=kvm2
2. bash ./build/kube_setup.sh

Expected Results

All pods running.

Actual Results

All pods but strimzi-cluster-operator are running; strimzi-cluster-operator is in a CrashBackoffLoop.

2022-04-19 22:04:18 ERROR Main:153 - Cluster Operator verticle in namespace * failed to start
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/apis/kafka.strimzi.io/v1beta2/kafkas?watch=true. Message: kafkas.kafka.strimzi.io is forbidden: User "system:serviceaccount:strimzi:strimzi-cluster-operator" cannot watch resource "kafkas" in API group "kafka.strimzi.io" at the cluster scope. Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkas, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkas.kafka.strimzi.io is forbidden: User "system:serviceaccount:strimzi:strimzi-cluster-operator" cannot watch resource "kafkas" in API group "kafka.strimzi.io" at the cluster scope, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:693) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:672) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:74) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
        Suppressed: java.lang.Throwable: waiting here
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:164) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:175) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.waitUntilReady(WatcherWebSocketListener.java:120) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:80) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:682) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:657) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:89) ~[io.fabric8.kubernetes-client-5.8.0.jar:?]
                at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:61) ~[io.strimzi.operator-common-0.26.0.jar:0.26.0]
                at io.strimzi.operator.common.AbstractOperator.lambda$createWatch$18(AbstractOperator.java:440) ~[io.strimzi.operator-common-0.26.0.jar:0.26.0]
                at io.strimzi.operator.common.Util.lambda$async$0(Util.java:76) ~[io.strimzi.operator-common-0.26.0.jar:0.26.0]
                at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.1.5.jar:4.1.5]
                at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.1.5.jar:4.1.5]
                at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.1.5.jar:4.1.5]
                at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.1.5.jar:4.1.5]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
                at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.68.Final.jar:4.1.68.Final]
                at java.lang.Thread.run(Thread.java:829) [?:?]

Additional Details

• minikube version 1.25.2, commit 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7
• Fedora 35
• kube_setup.sh version https://github.com/RedHatInsights/clowder/blob/49af63d34dddb74737d622b00b53b3a19fffa315/build/kube_setup.sh

Unable to deploy a job

Context

I'm trying to deploy a job with bonfire and Clowder to an ephemeral cluster. Here is the configuration:

• clowdapp.yaml
• test.yaml:

appsFile:
  host: gitlab
  repo: insights-platform/cicd-common
  path: bonfire_configs/ephemeral_apps.yaml

apps:
- name: ccx-data-pipeline
  components:
    - name: insights-results-aggregator-exporter
      host: github
      repo: RedHatInsights/insights-results-aggregator-exporter
      path: deploy/clowdapp.yaml
      ref: CCXDEV-5895
      parameters:
        CLOWDER_ENABLED: "true"
        IMAGE_TAG: latest
    - name: ccx-insights-results
      host: github
      repo: RedHatInsights/insights-results-aggregator
      path: deploy/clowdapp.yaml
      ref: master
      parameters:
        CLOWDER_ENABLED: "true"
        CLOUDWATCH_ENABLED: "false"
        IMAGE_TAG: qa

• commands:

ns=$(bonfire namespace reserve)
bonfire deploy -c test.yaml -n $ns ccx-data-pipeline 

Results

Running oc get jobs -n $ns returns "No resources found".

Extra information

If I set a schedule it works.

Questions

• Are jobs supported? Or only cronjobs?
• Is this a problem of bonfire?
• Is my clowdapp.yaml badly configured?

Thanks in advance.

The steps to run kube_setup.sh are inaccurate

go over Getting Cloweder (at REDME.adoc) step by step:
the part of installtion Clowder with latest release:
'Clowder can then be installed by visiting the latest release page, copying the link to the manifest, and running something similar to that shown below:'
should locate before running the kube_setup.sh script

the script exist in clowder folder, so you have to install it before

Add maker for minio object store config

Database points to Kafa config

Add deployment for minio provider