aisingapore/kapitan-hull-admin

Administrator's section of the Kapitan Hull stack

Kapitan Hull Administrator

Table of Contents

• Kapitan Hull Administrator
  • Table of Contents
  • Preface
  • Usage

Preface

This repository contains Helm charts and Docker builds for global components of AI Singapore's end-to-end ML projects.

This repository is to setup the following by the MLOps team within the K8S clusters created by the DataOps and InfraOps teams:

• MLFlow
• VSCode

This has to be installed after the cluster is set up with its persistence storage and an orchestrator of choice (RunAI, Polyaxon, etc.).

Usage

To be added...

kapitan-hull-admin's Issues

[Feature]: Coder enhancements

• change to 8gb home disk
• set cache location of pip, conda and huggingface to the main PVC

[Feature]: Adding a node selector for Coder and MLFlow deployments

Add a node selector option for the Terraform deployments for Coder and MLFlow so that it doesn't get deployed in GPU nodes.

[Documentation]: Write a guide for Code Server documentation

To write a guide with regards to customising code server and deploy it in Coder:

• installing and persisting packages
• updating to newer versions
• etc.

[Chore]: Test Coder, MLFlow and Code Server components/Docker images on GCP

Moved from #7 seperating the test for GCP.

• MLFlow
  • Test new version on GCP
    • Runs successfully
    • Artifacts stored successfully
• Coder
  • Test new version on GCP
    • Runs successfully
• Code server
  • Test new version on GCP
  • Runs successfully

CI/CD enhancements and documentation update

Moved from the Gitlab to here.

• Addition of CI/CD to push master copies of Docker images and Helm charts to Harbor/MLOps GCP project
  • Helm charts:
    • onprem: oci://registry.aisingapore.net/mlops-pub/kapitan-hull
    • GCP: asia-southeast1-docker.pkg.dev/machine-learning-ops/pub-charts
  • Docker images:
    • onprem: registry.aisingapore.net/mlops-pub
    • GCP: asia-southeast1-docker.pkg.dev/machine-learning-ops/pub-images
  • We'll start the semantic versioning to 0.2.0 and start using tags to freeze code for every milestone
    • To list the version of packages within the Tags section to crosscheck
    • Any change of major/minor versioning (x.y.0) of packages would warrant a new minor version bump
    • Any change of patch versioning (0.0.z) of packages would warrant a new patch version bump
• To use Gitlab CI to automatically spin up MLOps components by just passing through files/variables
• Updating documentation of using this repo and publish it in GitLab/GitHub pages

Helm chart enhancements

https://github.com/norwoodj/helm-docs

Update Terraform + Coder templates

This is to update the Terraform + Coder templates in light of the complications we've experienced last month.

This includes:

• fsgroup issues
• increasing Coder PostgreSQL CPU + memory

feat: Add RunAI kubeconfig creation script

Based on this section of the documentation:

https://docs.run.ai/latest/admin/runai-setup/authentication/researcher-authentication/

[Chore]: Upgrade Coder, MLFlow and Code Server components/Docker images

To upgrade internal services as well.

• MLFlow
  • Build new version
  • Test new version on-prem
    • Runs successfully
    • Artifacts stored successfully
• Coder
  • Test new version on-prem
    • Runs successfully
• Code server
  • Test new version on-prem
  • Runs successfully
• Terraform
  • Update the changes as default after testing (latest -> stable)