illumidesk/helm-chart

The official IllumiDesk helm-chart.

IllumiDesk

This monorepo is used to maintain IllumiDesk's authenticators, spawners, and microservices. This setup assumes that all services are running with Kubernetes. Please refer to our help guides for more information.

Overview

Jupyter Notebooks are a great education tool for a variety of subjects since it offers instructors and learners a unified document standard to combine markdown, code, and rich visualizations. With the proper setup, Jupyter Notebooks allow organizations to enhance their learning experiences.

When combined with the nbgrader package instructors are able to automate much of tasks associated with grading and providing feedback for their users.

Why?

Running a multi-user setup using JupyterHub and nbgrader with containers requires some additional setup. Some of the questions this distribution attempts to answer are:

• How do we manage authentication when the user isn't a system user within the JupyterHub or Jupyter Notebook container?
• How do we manage permissions for student and instructor folders?
• How do we securely syncronize information with the Learning Management System (LMS) using the LTI 1.1 and LTI 1.3 standards?
• How do we improve the developer experience to provide more consistency with versions used in production, such as with Kubernetes?
• How should deployment tools reflect these container-based requirements and also (to the extent possible) offer users an option that is cloud-vendor agnostic?

Our goal is to remove these obstacles so that you can get on with the teaching!

Prerequisites

Kubernetes v1.17+.

Quick Start

This setup only supports Kubernetes-based installations at this time. Refer to the helm-chart repo for installation instructions.

Development Installation

Refer to the contributing guide located in the root of this repo.

Building the JupyterHubs

1. Build the JupyterHub for local testing with docker-compose or docker:

make build-hubs

1. Build the JupyterHub for local testing with Kubernetes:

make build-hubs-k8

General Guidelines

This project enforces the Contributor Covenant. Be kind and build a nice open source community with us. ++

License

Please refer to the included license in this repository's root directory.

helm-chart's Issues

Ignore helm related files

Update .gitignore to ignore files related to helm-chart projects.

Migrate images to the illumidesk-containers repo

Move the images folder and contents to the illumidesk-containers repo.

Add SSL and multi domains to EKS Cluster setup

Update JupyterHub images to reference upstream images

Now that the JupyterHub K8 images are using Python 3.8 we no longer need to maintain a separate Dockerfile to build an image with Py3.8.

Fix the helm package chart page

Fix the chart web page published by GH pages which is currently set to https://illumidesk.github.io/helm-chart but returns a 404 error.

Install and manage keycloak with Operators

Add support for the Keycloak Kubernetes Operator to install and manage Keycloak resources, specifically:

• Keycloak Realm
• Keycloak Client
• Keycloak User

Installing and managing the above resources will help simplify the deployment pipeline by not having to build custom images to import these installation and configuration settings before installing or updating the cluster with the helm chart.

Add Changelog for helm chart

Deprecate upstream bitnami postgres helm-chart in favor of native manifests

Replace the upstream Postgres helmp-chart dependency with native manifests.

Reasoning:

• If we need to deploy a production-grade Postgres instance then we would do so with an externally managed service
• The upstream helm-chart has many dependencies and may cause confusion with our setup, particularly due to the fact that several services connect to Postgres databases.

Add helper template

Add a helper template to the illumidesk/templates directory as a _helpers.tpl file.

Disable the creation of PV / PVCs by default

Overview

When deploying the stack with AWS EKS PVs are created by default as EBS volumes. This adds unnecessary cost and complexity especially when the hub pod deployment does not require a PV/PVC for the database.

Requirement

Update the values.yml file to include the hub.db.url.type key and set it to postgres by default. The JupyterHub sub-chart in this case will not create a hub-db-dir PV since this PV is only required when for deployments that use SQLite.
.

Add fake values to values.yaml example

Provide some fake examples to values.yml located in the examples directory to further assist the end-user.

Add contributing guide

Add contributing guide to describe how contributors (collaborators that contribute both code and documentation) should structure their project and what procedures/validations are in place before merging a pull request.

Add chart testing to CI

Add chart testing to the continuous integration process with GH actions.

Refactor Postgres template settings

Refactor template settings to avoid repetition between connection settings related to an internally deployed Postgres pod and the connection settings related to external Postgres instances.

All Postgres connection URIs should assume that the Postgres instance is running independently, regardless of whether the instance is running as a pod or with an externally managed service, such as AWS RDS.

Update custom configuration to use the extraFiles option

Instead of loading custom configurations and placing those in the extraConfigs section of the configuration's YAML, add a *.py file to the /usr/local/jupyterhub/jupyterhub_config.d/ path so that it's automatically loaded.

This link has an example of how to update the standard configuration to support this option.

Cerate helm repo

Create a helm-chart repository using the GitHub pages option.

Update repo based on the latest best practices

Update repo to include files, docs, and conventions based on the latest Helm v3 patterns. View examples by creating a test chart with the helm create ... command (use helm create -h to view all options) and consult best practices with these docs.

Add Metrics configuration

Add Metrics configuration which should include:

• Metrics settings
• Application-specific service parameters, such as management port and annotations
• Prometheus Operator for ServiceMonitor configuration

feat: Add NFS to illumidesk deployment

link pv and pvc and mount it to respective pods

Add the grader setup service application image

Add the Dockerfile for the grader-setup-app image.

Bump illumidesk package version

Upgrade the illumidesk package to version 1.1.0.

Update helm-chart to remove hard coded grader-setup-service image value

Update the helm-chart so users can override the grader-setup service domain, namespace, and tag in case they would like to pull this image into a registry other than DockerHub.

Add Secrets to store and manage sensitive information

Add Secrets.yaml to the template manifests to manage sensitive information such as passwords and keys.

This feature replaces some settings that are currently in place with string values so this does require some refactoring.

Add datadog helm chart

Add datadog helm chart and lock as a dependency with helm dependency build.

Upgrade the load balancer sub-chart

Upgrade from legacy aws-alb-load-balancer to the newer aws-load-balancer-controller.

Deploy the datadog agents to the kube-system namespace

Overview

Adding the datadog agent to every namespaced deployment does not add value, since the datadog agent can view all resources in the cluster, regardless of namespace, therefore it only needs to be added once.

Request

Update the helm-chart so that the datadog dependency is a kube-system (namespace) resource.

Update the AWS load balancer type

Update the Nginx annotations to use AWS NLB instead of AWS ELB. AWS ELB does not support WebSockets and we need WebSocket support for Jupyter Kernels.

Add post deletion hook to helm chart

1. Create a post deletion hook to delete alb-ingress-controller and external-dns which are two service accounts that are created outside of the namespace
   Possible Issue: Since alb-ingress- controller and external-dns are part of the helm chart, this may require refactorization when adding 2 or 3 additional domains.

Update JupyterHub and Notebook base images

Update jupyterhub/k8s-hub image version to version 1.3.0 of the JupyterHub package.

Refactor chart to update options when using Keycloak

Overview

Update the chart to use Keycloak manifests or sub-chart to enable the correct communication when terminating TLS sessions with Nginx when using AWS NLB + Nginx.

Requirement

This is the standard ingress-controller manifest for NLB. We used the same settings, except for one minor change:

For the Service update the targetPort for the http to use to tohttps instead of http.

Before:

...
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
...

After:

...
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: tohttps
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
...

The ingress resource is specified below. Notice the annotations and the secretName to specify the TLS secret. This TLS secret should be located in the same namespace as the ingress resource.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: keycloak
  namespace: keycloak
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/server-snippet: |
      if ( $server_port = 80 ) {
         return 308 https://$host$request_uri;
      }
spec:
  tls:
    - hosts:
      - test-nlb.illumidesk.com
      secretName: tls-secret
  rules:
  - host: test-nlb.illumidesk.com
    http:
      paths:
      - backend:
          serviceName: keycloak
          servicePort: 8080

This is the Keycloak Deployment/Service:

apiVersion: v1
kind: Service
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app: keycloak
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
      - name: keycloak
        image: quay.io/keycloak/keycloak:12.0.3
        env:
        - name: KEYCLOAK_USER
          value: "admin"
        - name: KEYCLOAK_PASSWORD
          value: "admin"
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        readinessProbe:
          httpGet:
            path: /auth/realms/master
            port: 8080

Steps to Replicate the Setup (ingress-nginx with TLS + AWS NLB):

• Create a new namespace, called ingress-nginx: kubectl create namespace ingress-nginx
• Deploy ingress-nginx with the AWS NLB option as documented here. To confirm the command to deploy is kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/aws/deploy.yaml
• (Optional) Add custom Apple annotations to the ingress-nginx manifests in the file above.
• Create TLS private key, root, and certificate for the nginx resource. This section of the docs has an example of how you could create a cert with the openssl command.
• Create the secret in the namespace where the application is deployed: kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE} -n {namespace-where-app-is-deployed}
• Deploy the ingress resource and the app itself.

Update values.yaml to use the GenericOAuthenticator by default

Document the use of the standard JupyterHub GenericOAuthenticator so that it's used by default instead of LTI 1.1 or LTI 1.3.

Update the values.yaml to default to the GenericOAuthenticator

Update the values.yaml file to use the GenericOAuthenticator by default.

The values to integrate with a third-party service such as Auth0 or Keycloak are managed separately with IlluimiDesk's deployment scripts and services.

Update requirements to include saml authenticator

Update the requirements.in file to include the jupyterhub-samlauthenticator package.

Add ImagePullSecrets to helm chart.

Private Registry Reference: http://docs.heptio.com/content/private-registries/pr-gcr.html
AWS Secrets Controller: https://aws.amazon.com/blogs/containers/aws-secrets-controller-poc/
Tasks:

1. Add Secret manifest for docker registry
2. Reference Secret for both grader and jupyterhub
3. Integrate AWS secrets controller to secure secrets

Update external-dns manifest to accept dynamic namespaces

Update the external-dns.yaml manifest to replace the hardcoded default namespace with {{ .Release.Namespace }}.

Addinto gitops and flux into EKSCTL cluster

gitops

Remove place holder configs from ALB ingress

Remove the following section from the ALB ingress resource template:

      - path: /*
        backend:
          serviceName: ssl-redirect
          servicePort: use-annotation

Add comments to values.yaml to clarify options

In addition to providing documentation in the README provide comments in the values.yaml file to briefly describe what each value is used for. The CONTRIBUTING guide should describe how these comments should be structured.

#60

Add datadog agent

Add datadog agent sub-chart to add optional monitoring support.

Remove infrastructure items from helm-chart repo

Remove the items and documentation that are related to infrastructure. Specifically, we need to remove:

• IAM folder and policy files
• cluster folder and configuration file

Add Postgres operators

Add Postgres operators to automate management tasks for Postgres databases required for the system:

• Ensure databases are created and ready for new deployments
• Update databases if there is a name change for a related service, such as the JupyterHub or Keycloak

AWS upgrades

Update the helm-chart repo so that it updates AWS resources:

• Deprecate the use of Key IDs and secrets for AWS user accounts and use IAM roles
• Update the ALB ingress controller version

Configure a separate external database key for each service that requires Postgres

Services have three options for databases:

• Default internal database for dev purposes. For example, the JupyterHub uses the SQLite database if none are defined and Keycloak uses the H2 database if non are defined
• Internal Postgres pods
• External databases such as with AWS RDS

Updating the configuration when the database is internal or external can get tricky, particularly when dealing with the POSTGRES_HOST value. For this reason, it's probably best to define a new externalDatabase key to define the user, password, etc.

Convert ALB ingress controller to cluster resource

Add the ingress-nginx as an ingress controller. This is particularly useful for development environments and/or environments that do not rely on AWS and therefore removes the explicit dependency of having to use the aws-alb-ingress-controller.

Proof of concept with FusionAuth

Test a deployment using FusionAuth instead of Keycloak using the GenericOAuthenticator.

Move all environment variables to a ConfigMap

Create a ConfigMap to manage all environment variables. This will help centralize environment variables that are currently located in the custom JupyterHub configuration as well as in various other locations.

For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: illumidesk-keycloak
  namespace: {{ .Release.Namespace }}
  labels: {{- include "common.labels.standard" . | nindent 4 }}
    app.kubernetes.io/component: keycloak
    {{- if .Values.commonLabels }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
    {{- end }}
  {{- if .Values.commonAnnotations }}
  annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
  {{- end }}
data:
  KEYCLOAK_CREATE_ADMIN_USER: {{ ternary "true" "false" .Values.auth.createAdminUser | quote }}
  KEYCLOAK_ADMIN_USER: {{ .Values.auth.adminUser | quote }}
  POSTGRES_USER:  {{ .Values.auth.postgresUser | quote }}
  ...

Add Prometheus and Grafana

Replace AWS users with legacy keys/secrets with IAM roles

Replace the use of AWS key ids and secrets to authenticate users with IAM roles. Utilize tools such as kube2iam, etc., or an AWS native solution if possible.

Create Helm Hook to generate and upload LTI1.1 config xml

Example XML FIle

match PV storage capacity to PVC storage requests

• Have a default option of 20gi
• template out storage capacity and storage requests to the values.YAML