dell/omnia

An open-source toolkit for deploying and managing high performance clusters for HPC, AI, and data analytics workloads.

Ansible playbook-based deployment of Slurm and Kubernetes on servers running an RPM-based Linux OS

Omnia (Latin: all or everything) is a deployment tool to turn servers with RPM-based Linux images into functioning Slurm/Kubernetes clusters.

Omnia Documentation

Omnia Documentation is hosted on Read The Docs.

Current Status:

Licensing

Omnia is made available under the Apache 2.0 license

Contributing To Omnia

We encourage everyone to help us improve Omnia by contributing to the project. Contributions can be as small as documentation updates or adding example use cases, to adding commenting and properly styling code segments all the way up to full feature contributions. We ask that contributors follow our established guidelines for contributing to the project.

Contributions to Omnia are made through Pull Requests (PRs) to "devel" branch. "devel" is the bleeding edge branch of Omnia packed with experimental and untested features".

Omnia Community Members:

Contributors

Our thanks go to everyone who makes Omnia possible (emoji key):

John Lockman ⚠️ 💻 📝 🤔 🚧 🧑‍🏫 🎨 👀 📢 🐛	Lucas A. Wilson 💻 🎨 🚧 🤔 📝 📖 🧑‍🏫 📆 👀 📢 🐛	Sujit Jadhav 🤔 📖 💻 👀 🚧 📆 🧑‍🏫 📢 💬 ⚠️ 🐛	Deepika K 💻 ⚠️ 🐛 🛡️ 📢 👀 🧑‍🏫	Abhishek SA 💻 🐛 📖 ⚠️ 🚧 📢 🧑‍🏫 👀	Sakshi Arora 💻 🐛 📢	Shubhangi Srivastava 💻 🚧 🐛 📢
Cassey Goveas 📖 🐛 🚧 📢	Khushboo Dholi 💻	Prasoon Kumar Sinha 🤔 📢	SajithDas 📆 📢	i3igpete 💼 📢	renzo-granados 🐛	Aditya-DP 💻
Katakam Rakesh Naga Sai 💻	araji 💻	Mike Renfro 📖	Lee Reynolds 💻 📖 ✅	blesson-james 💻 ⚠️ 🐛	avinashvishwanath 📖	abhishek-s-a 💻 📖 ⚠️
Franklin-Johnson 💻 📝	teiland7 💻 📝	VishnupriyaKrish 💻 ⚠️	Ishita Datta 📖	William Dizon ✅	bssitton-BU 🐛	John Hearns 🐛
kris buggenhout 🐛	jiad-vmware 🐛	Justin Lecher 🤔	Kavyabr23 💻 ⚠️	vedaprakashanp ⚠️ 💻	Bhagyashree-shetty ⚠️ 💻	Nihal Ranjan ⚠️ 💻 📢
ptrinesh 💻	Ikko Ashimine 💻	Lakshmi-Patneedi 💻	Jie Li 💻	Yong Chen 🎨	nvtngan 💻 🔌	tamilarasansubrama1 ⚠️ 💻
shemasr 🐛 💻 ⚠️	Naresh Sharma 🐛	Jon Hass 📖 🎨	KalyanKonatham 🐛	Rahul Akolkar 🐛	srinandini-karumuri 💻	Rishabhm47 ⚠️ 💻
vaishakh-pm ⚠️ 💻	shridhar-sharma ⚠️ 💻	Jaya.Dayyala ⚠️ 💻	fasongan 💻	rahuldell21 💻 ⚠️	diptiman12 💻	Supriya Parthasarathy 📆
Subhankar-Adak 💻	priti-parate 💻	Lavanya Adhikari 💻	preeti-thankachan ⚠️	Boris Glimcher 💻 🚧📖	Moshi Binyamini 💻🚧

omnia's Issues

Commands for quickly switching node personality

Is your feature request related to a problem? Please describe.
We need a way to switch nodes from Kubernetes to Slurm and back

Describe the solution you'd like
Solution is a pair of scripts which move nodes from K8s->Slurm and Slurm->K8s

Describe alternatives you've considered
I considered using a playbook, but since this will be run on the master node a Bash script seemed most appropriate

Additional context
N/A

provide NVIDIA NGC Tensor Flow example

Is your feature request related to a problem? Please describe.
Omnia should provide and example for running Tensorflow in NVIDIA NGC containers

Describe the solution you'd like
provide yaml or notebook for how to use NGC containers on omnia

Describe alternatives you've considered

Additional context
https://ngc.nvidia.com/

use versionlock for k8s and kernel

Is your feature request related to a problem? Please describe.
we need to lock the versions of
kernel
kubeadm
kubectl
kubelet

Describe the solution you'd like
yum versionlock kubelet-1.16.7 kubectl-1.16.7 kubeadm-1.16.7

Describe alternatives you've considered

Additional context
we need to settle on a kernel to lock as well

Add device plugin for Xilinx Alveo FPGAs

Is your feature request related to a problem? Please describe.
Support desired for Alveo FPGAs

Describe the solution you'd like
Device plugin here: https://github.com/Xilinx/FPGA_as_a_Service

Describe alternatives you've considered
N/A

Additional context
N/A

Updating docs on contributing to the project

We need to very simply lay out how contributions to the project can be made. We have fairly detailed instructions in CONTRIBUTING.md, but we should be more specific about how to handle the fork/branch/PR process.

We should also include a figure which describes the process.

Separate applications into individual playbooks?

Should we have separate single-task playbooks for each application, or should we have a single playbook with multiple tags?

Install mpi-operator

Is your feature request related to a problem? Please describe.
We need to be able to run MPI jobs on the Kubernetes cluster

Describe the solution you'd like
mpi-operator from KubeFlow should do the trick: https://github.com/kubeflow/mpi-operator

Describe alternatives you've considered
We could build our own operator, but that seems unnecessary

Additional context
I think we should install the MPI operator separate from KubeFlow, as some users may want to run MPI jobs independently of using KubeFlow's other features.

cmd to join new computes

Is your feature request related to a problem? Please describe.
we need a command to join new computes after the cluster has been created.

Describe the solution you'd like
something like grow_cluster -f ./new_inventory_file

Describe alternatives you've considered
maybe just a tag in the ansible playbook?

Additional context
this could apply to different scenarios:

1. adding new compute hardware to existing cluster
2. adding cloud infrastructure to existing cluster

Create CPU example

Is your feature request related to a problem? Please describe.
we should have an example yaml demonstrating CPU workload. Bonus if it uses MPI-Operator

Describe the solution you'd like
something similar to the tensorflow example that uses GPUs

Describe alternatives you've considered

Additional context

INSTALL.md needs updating

Is your feature request related to a problem? Please describe.
https://github.com/dellhpc/omnia/blob/devel/docs/INSTALL.md has some outdated information and needs to be updated.

Describe the solution you'd like
change overlay network explanation
add metalLB
check for other correctness

Describe alternatives you've considered

Additional context

Support for Fedora CoreOS

Is your feature request related to a problem? Please describe.
Would be ideal to have a lightweight base OS for Kubernetes clusters.

Describe the solution you'd like
Originally wanted to use CoreOS, but Core is EoL. Project has moved to Fedora CoreOS

Describe alternatives you've considered
We could also build our own minimal spin, but that seems like unnecessary work

Additional context
Fedora CoreOS automatically updates. Will this be a concern?

Update docs to support github pages

Is your feature request related to a problem? Please describe.
We should support a GitHub page for Omnia

Describe the solution you'd like
Built-in feature. We just have to update the format of the docs to ensure it works.

kfserving-ingressgateway does not request enough resources

Describe the bug
when launching kubeflow the kfserving-ingressgateway is OOMKilled. It needs more resources than are originally requested.

To Reproduce
launch kubeflow
observe kf-serving-ingressgateway OOMKilled

Expected behavior
this should not happen

Screenshots
istio-system kfserving-ingressgateway-6b469d64d-fgjqk 0/1 OOMKilled 1 19m 10.244.159.199 compute005.localdomain <none> <none>
Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context

jupyter_config.yaml containers are stale

Describe the bug
the containers described in jupyter_config.yaml are old and need to be updated

To Reproduce
Steps to reproduce the behavior:
try to start a jupyterhub container

Expected behavior
the container should start

Screenshots

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context

OpenHPC repos

Is your feature request related to a problem? Please describe.
Use OpenHPC RPM repositories to provide HPC libraries and tools

Describe the solution you'd like
Enable the OpenHPC RPM repos in /etc/yum.repos.d/

Describe alternatives you've considered
We could rebuild the packages, but seems like duplicating work

Additional context
N/A

Use Omnia to deploy multiple clusters

Is your feature request related to a problem? Please describe.
Instead of using Omnia to install a single cluster, Omnia should be able to install multiple clusters from a single inventory of nodes.

Describe the solution you'd like
Rather than running Omnia on a cluster's master node, Omnia should be run on a separate management server. In order to accomplish this, we should:

• Eliminate assumption of execution on master node (see #58, first bullet)
• Provide means of handling multiple inventory files
  • Possible solution is AWX
    • AWX also provides potential for multi-inventory management (i.e., hybrid cloud)

Describe alternatives you've considered
Currently Omnia would have to be run independently from each cluster. This could be troublesome in a multi-cluster environment

Additional context
N/A

Switch to GPU Operator

Is your feature request related to a problem? Please describe.
we have a branch using nvidia-device-plugin but not GPU Operator

Describe the solution you'd like
a new branch that uses GPU Operator not nvidia-device-plugin

Describe alternatives you've considered

Additional context

allow variable for master node IP

Is your feature request related to a problem? Please describe.
omnia currently has hardcoded 10.0.0.1 and p3p1 for the master node. This should be a variable in the host_inventory_file

Describe the solution you'd like
replace hardcoded values with variables

Describe alternatives you've considered
none

Additional context
none

Install Intel FPGA device plugin

Is your feature request related to a problem? Please describe.
Need to add support for Intel FPGAs

Describe the solution you'd like
https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/cmd/fpga_plugin/README.md

Describe alternatives you've considered
No alternatives

Additional context
We will need to move to CRI-O (see #76 or #57) in order for the plugin to work.

Setup TLS certificate management in the cluster

Is your feature request related to a problem? Please describe.
The k8s cluster needs to manage TLS certs for other issues such as #19 - install local registries

Describe the solution you'd like
start with: https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#a-note-to-cluster-administrators

Describe alternatives you've considered

Additional context

Mark nfs-provisioner as default StorageClass

Is your feature request related to a problem? Please describe.
there is currently no default storageClass defined but we are using nfs-provisioner. We should set nfs-provisioner as default StorageClass so that applications such as KubeFlow can take advantage of the service.

Describe the solution you'd like
kubectl patch storageclasses.storage.k8s.io nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Describe alternatives you've considered
we could force every application to create their own volumes, this is problematic.

Additional context
https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

Document network config as an example

Let's be sure we document that the metallb config is an example and must be adjusted to fit the user's network scheme.

This resolves concerns in PR #60

Batch scheduling for Kubernetes

Is your feature request related to a problem? Please describe.
Add batch scheduling for Kubernetes jobs via Volcano

Auto-detect accelerators

Is your feature request related to a problem? Please describe.
How to we determine which packages (e.g., operators, device-plugins, drivers. etc.) to install on each compute node?

Describe the solution you'd like
Gather facts about each node and dynamically determine which packages to install based on accelerators in the box.

Describe alternatives you've considered
Alternative would be to have node groups for each, but this solution wouldn't allow for corner cases, such as DSS8440 with a mix of accelerators.

Additional context
This is not an immediate concern, but future nodes may be more heterogeneous in their accelerator makeup. Let's explore a more dynamic way of inspecting each node. Also simplifies inventory management.

autoscale coreDNS

Is your feature request related to a problem? Please describe.
coreDNS needs to run on all the compute nodes, we need some mechanisim to autoscale coredns to the total number of nodes in the cluster or find another alternative.

Describe the solution you'd like
when initializing cluster coredns should be replicated to every node.

Describe alternatives you've considered

Additional context

Update jupyterhub to latestes

Is your feature request related to a problem? Please describe.
jupyterhub group released an updated helm chart v0.9.0 that gets jhub v1.1.0

Describe the solution you'd like
update the version

Describe alternatives you've considered

Additional context

Add omnia logo

Omnia needs a logo, and we've already made one! Just need to put it in the repo.

Install Local Registry

Is your feature request related to a problem? Please describe.
Omnia needs an option to install a local registry

Describe the solution you'd like
a playbook for installing a local registry such as Harbor - https://github.com/goharbor/harbor

Describe alternatives you've considered
local registry - with simple docker container

Additional context

Examples for storage connectors

Is your feature request related to a problem? Please describe.
Would like examples for other storage connectors such as HDFS, NoSQL and S3

Describe the solution you'd like
HDFS, NoSQL and S3 for cloud

Describe alternatives you've considered

Additional context

Validate Polyaxon

https://github.com/polyaxon/polyaxon

Current version is 0.6.1

Refactor playbooks

Is your feature request related to a problem? Please describe.
In order to support more community contributions, we should make playbooks easier to read/extend.

Describe the solution you'd like
Let's follow the best practices guidelines for Ansible: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html

Describe alternatives you've considered
N/A

Additional context

• One directory structure for each major component, i.e., kubernetes, slurm, tools, etc.
• Eliminate shell and command plays whenever possible
  ** Extra benefit - yum plays could have version tagged and placed within vars.yml, providing a solution to #28 without forcing versionlock via yum.
• Isolate different tasks of each component in order to avoid accidental modification

User Management

How should we handle user management?

One option is to provide an openLDAP service for other applications to leverage: https://github.com/helm/charts/tree/master/stable/openldap

Be able to install different SDNs

Is your feature request related to a problem? Please describe.
Be able to have the choice to pick from different SDNs during install.

Describe the solution you'd like
Maybe pass a keyword argument for the ansible script which identifies the kind of SDN network to be installed. To get started, Omnia could support Calico, Cilium, Contiv-VPP, Kube-router, Weave Net. I think these are commonly used ones, supported by Kubeadm installation path.

Provide branch with v1.16 and GPU Operator

It looks like KF1.0 will work on K8s 1.16. Do we want to just upgrade going forward (post v0.1)?

Provide Object Storage

Is your feature request related to a problem? Please describe.
Provide object storage deployment

Describe the solution you'd like
a helm chart that deploys object storage with the k8s cluster

Describe alternatives you've considered
MinIO ?

Additional context

JupyterHub Install as separate playbook

Is your feature request related to a problem? Please describe.
jupyterhub is currently installed with the init tag. Only base k8s and supporting services should be started in init

Describe the solution you'd like
pull the commands out of startservices and make them a new playbook

Describe alternatives you've considered
it could stay in the current playbook

Additional context

Fix Licensing

Need to update LICENSE file to reference Dell Technologies copyright.

Also need to add appropriate license clauses to the top of all source files.

Auto Cull Users from JupyterHub

Is your feature request related to a problem? Please describe.
we should automatically remove idle users.

Describe the solution you'd like
add cull instructions in the jupyter_config.yaml to automatically remove idle users.

Describe alternatives you've considered

Additional context

kfserving-gateway does not change limits

Describe the bug
limits are not set in kfserving-gateway

To Reproduce

Expected behavior

Screenshots

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context

Create playbook for installing tools

Is your feature request related to a problem? Please describe.
Need a way to install tools like #22

Describe the solution you'd like
Playbooks for copying tools to local filesystem on admin/service nodes

Describe alternatives you've considered
Shell script installer was considered, but doesn't scale to multiple admin nodes

Additional context
N/A

nfs-provisioner ip variable in host_inventory_file

Is your feature request related to a problem? Please describe.
the IP address for nfs-provisioner-client is hard coded in the startservices playbook and should be a variable in the inventory file or vars

Describe the solution you'd like
add variable to startservices group to allow users to point to different NFS servers

Describe alternatives you've considered
none

Additional context
none

scuttle should be a playbook

Is your feature request related to a problem? Please describe.
scuttle is currently a bash script that uses clustershell. This needs to be updated to use a playbook, we should not depend on clustershell.

Describe the solution you'd like
rewrite scuttle in a yaml playbook

Describe alternatives you've considered

Additional context
https://github.com/dellhpc/omnia/blob/devel/kubernetes/scuttle

Include dockerfiles for custom containers

Is your feature request related to a problem? Please describe.
Add dockerfiles for customized JupyterLab containers

Describe the solution you'd like
Include dockerfiles for customized JupyterLab containers. This is related to issue #42

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Create additional documentation on system setup and pre-installation prep

Is your feature request related to a problem? Please describe.
Not related to a particular feature.

Describe the solution you'd like
Documentation needs to be updated to include instructions on server/network preparation prior to running Ansible playbooks.

Describe alternatives you've considered
No alternatives. Documentation needs to be provided.

Additional context
No additional context.

make network interfaces parameter or get from inventory

Is your feature request related to a problem? Please describe.
the network interface for k8s is currently hard coded in startmaster. It should either be a user defined variable or do some magic with ansible's inventory. We have several adapters on most nodes so I would prefer to start with a user defined variable.

Describe the solution you'd like
create variable in inventory file
use variable in startmaster playbook

Describe alternatives you've considered

Additional context
Add any other context or screenshots about the feature request here.

Installing Omnia on a Single Node

Is your feature request related to a problem? Please describe.
If there was a way to install this on a single node that would be great!

Describe the solution you'd like
Perhaps an installation strategy with README to do it on a single node.

Describe alternatives you've considered
N/A

Additional context
N/A

Validate KubeFlow 1.0

KubeFlow 1.0 is now officially released.

https://github.com/kubeflow/kubeflow/releases/tag/v1.0

Container runtime alternatives to Docker

Is your feature request related to a problem? Please describe.
Switch the container runtime from Docker/containerd to CRI-O

Describe the solution you'd like
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o

Describe alternatives you've considered
We could stick with Docker, but would probably need to move off of the CentOS repos and onto CE.

Additional context
N/A

Install Prometheus

We should look into installing Prometheus by default when deploying Omnia

add --wait to jupyterhub install

Is your feature request related to a problem? Please describe.
jupyterhub currently fails if you do not have all containers downloaded. There is a timeout and it just quits. Add the --wait flag to helm install to force it to wait until all containers have been downloaded

Describe the solution you'd like
add --wait to helm install

Describe alternatives you've considered
we could prepull images elsewhere in the playbooks but jupyterhub has a hook that will pull them all down for us

Additional context