Giter Site home page Giter Site logo

budgetkey-k8s's Introduction

The BudgetKey Kubernetes Environment

The Budgetkey Kubernetes environment manages most Budgetkey infrastructure as code.

The root of this repository is a Helm chart with helm charts under charts-external defined as dependencies of this chart. The dependencies are defined in Chart.yaml.

Values are defined in the following Helm value files:

This chart is continuously synced to Hasadna cluster via ArgoCD as defined here.

Local development can be done by installing either this root helm chart or any of the dependant charts to your local Kubernetes cluster using Helm.

Common Tasks

All code assumes you are inside a bash shell with required dependencies and connected ot the relevant environment

Adding an external app

  • Duplicate and modify an existing chart under charts-external directory
  • Setup the external app's continuous deployment
    • Copy the relevant steps from an existing app's .travis.yml
    • Also, suggested to keep deployment notes in the app's README.md
    • Follow the app's README to setup Docker and GitHub credentials on Travis

Modifying secrets

Secrets are stored and managed directly in kubernetes and are not managed via Helm.

To update an existing secret, delete it first kubectl delete secret SECRET_NAME

After updating a secret you should update the affected deployments, you can use ./force_update.sh to do that

All secrets should be optional so you can run the environment without any secretes and will use default values similar to dev environments.

Each environment may include a script to create the environment secrets under environments/ENVIRONMENT_NAME/secrets.sh - this file is not committed to Git.

You can use the following snippet in the secrets.sh script to check if secret exists before creating it:

! kubectl describe secret <SECRET_NAME> &&\
  kubectl create secret generic <SECRET_NAME> <CREATE_SECRET_PARAMS>

budgetkey-k8s's People

Contributors

akariv avatar orihoch avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

budgetkey-k8s's Issues

[incident] pipelines pod stuck in ContainerCreating due to persistent disk unable to mount

  • kubectl get pods
    • pipelines pod stuck in ContainerCreating status
  • kubectl describe pod pipelines-79dd8668bb-dmnc4_budgetkey
    • timeout expired waiting for volumes to attach/mount for pod "budgetkey"/"pipelines-79dd8668bb-dmnc4". list of unattached/unmounted volumes=[data]
  • gcloud compute disks describe budgetkey-pipelines-data-3
    • the persistent disk is still attached to the node
    • users: - https://www.googleapis.com/compute/v1/projects/hasadna-general/zones/europe-west1-b/instances/gke-hasadna-cluster-default-pool-17373db3-dlpj
  • gcloud compute ssh gke-hasadna-cluster-default-pool-17373db3-dlpj
    • sudo journalctl -n 500 -f
    • error should appear periodically - as kubernetes retries everything
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: I0627 10:19:42.835137    1516 reconciler.go:186] operationExecutor.UnmountVolume started for volume "data" (UniqueName: "kubernetes.io/gce-pd/budgetkey-pipelines-data-3") pod "459bd8f9-79e9-11e8
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: E0627 10:19:42.858324    1516 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/gce-pd/budgetkey-pipelines-data-3\" (\"459bd8f9-79e9-11e8-8769-42010a840006\")" failed. No retries pe
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: Unmounting arguments: /var/lib/kubelet/pods/459bd8f9-79e9-11e8-8769-42010a840006/volumes/kubernetes.io~gce-pd/data
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]: Output: umount: /var/lib/kubelet/pods/459bd8f9-79e9-11e8-8769-42010a840006/volumes/kubernetes.io~gce-pd/data: target is busy
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]:         (In some cases useful info about processes that
Jun 27 10:19:42 gke-hasadna-cluster-default-pool-17373db3-dlpj kubelet[1516]:          use the device is found by lsof(8) or fuser(1).)
  • check the pod name - cat /var/lib/kubelet/pods/459bd8f9-79e9-11e8-8769-42010a840006/etc-hosts
  • can inspect other directories in the pod dir as well
  • run lsof to determine which process is keeping it -
  • toolbox - starts a privileged docker container for debugging (due to container optimized os limits)
    • apt install lsof
    • lsof -w | grep 459bd8f9-79e9-11e8-8769-42010a840006
    • kill / investigate the process
    • kubernetes keeps retrying, so wait ~1 minute and it should work

cluster migration plan

  • fonts - copy and expose as themes
  • datapackages - copy from server gz
  • travis - use deploy

[incident] elasticsearch release fails - "has no deployed releases"

  • helm ls --all
    • budgetkey-elasticsearch-budgetkey release is in PENDING_UPGRADE state
  • delete the release and recreate
    • helm delete --purge budgetkey-elasticsearch-budgetkey
    • ./helm_upgrade_external_chart.sh elasticsearch
  • alternatively - you can try to inspect the configmap directly
    • kubectl get cm -n kube-system | grep budgetkey-elasticsearch-budgetkey
    • can try to manually modify the configmap - to force a successful status

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.