converged-computing/kubescaler

Supporting classes in Python for easily scaling Kubernetes clusters (under development)

KubeScaler

Make your Kubernetes cluster YUGE! Or Smol. :)

This is a set of helper Python classes that make it easy to add elasticity, or scaling up and down, of your Kubernetes clusters in Python. We currently have support for the clouds we use, namely:

• Google (GKE)
• Amazon (EKS)

🚧️ under development 🚧️

This tool is under development and is not ready for production use. Documentation and examples coming soon!

😁️ Contributors 😁️

We use the all-contributors tool to generate a contributors graphic below.

Vanessasaurus 💻

Md Rajib Hossen 💻

TODO

• fix up GKE scale function to only be one function, we don't need to reset max and min again
• run experiments for scaling on EKS

License

HPCIC DevTools is distributed under the terms of the MIT license. All new contributions must be made under this license.

See LICENSE, COPYRIGHT, and NOTICE for details.

SPDX-License-Identifier: (MIT)

LLNL-CODE- 842614

kubescaler's Issues

Experiment with GKE scaler to add spot instance groups

I recently adopted the linear programming algorithm to calculate optimized instance groups for Google Cloud, and (superficially) it looks like we might have some promising combinations, meaning groups of spot instances that beat on demand prices. https://github.com/converged-computing/metrics-operator-experiments/tree/main/google/spot-instances/run0. To support this, we will need to do similar to what I added to AWS to create the cluster and then add on / delete managed node groups. GKE, unlike AWS, does not allow for an empty control plane, so I might need to create a cluster with one tiny node first, not sure because I haven't looked at this library in a while (but will soon).

TODO:

• try creating with sizes 0 (did not work)
• if get error make size of 1 (this did work, used c2d-standard-2 instance type with 1 node)
• then try to delete (not worth it)
• if does not work, try keeping size 1 and use https://github.com/googleapis/google-cloud-python/blob/461c76bbc6bd7cda3ef6da0a0ec7e2418c1532aa/packages/google-cloud-container/google/cloud/container_v1/services/cluster_manager/client.py#L3131
• then create/delete node pools as needed.

Create better install logic for aws vs google

Right now they are both attempted to be imported, so having them distinct isn't meaningful. We either need to:

• have all cloud dependencies come with the standard install
• separate the import logic to allow importing one but not the other (ideal)

For the time being I'm just using pip install -e .[all] after a clone, but I'd like to fix this.

Node fails to join

Describe the bug
Resource handler returned message: "[Issue(Code=NodeCreationFailure, Message=Instances failed to join the kubernetes cluster, ResourceIds=[i-04b80822945ab1727, i-06145e8f375e3b501, i-08f235094b0381923, i-0b9f48b3839d9ad34])] (Service: null, Status Code: 0, Request ID: null)" (RequestToken: 6d549180-af57-b3a7-098b-f1b340e02f12, HandlerErrorCode: GeneralServiceException)

To Reproduce
Steps to reproduce the behavior:
eksctl create cluster -f eks-efa-cluster-config-hpc7g.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: hossen1-cluster-4
  region: us-east-1
  version: "1.27"

iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
      name: cluster-autoscaler
      namespace: kube-system
    attachPolicy:
      Version: "2012-10-17"
      Statement:
      - Effect: Allow
        Action:
        - "autoscaling:DescribeAutoScalingGroups"
        - "autoscaling:DescribeAutoScalingInstances"
        - "autoscaling:DescribeLaunchConfigurations"
        - "autoscaling:DescribeTags"
        - "autoscaling:SetDesiredCapacity"
        - "autoscaling:TerminateInstanceInAutoScalingGroup"
        - "ec2:DescribeLaunchTemplateVersions"
        Resource: '*'

availabilityZones: ["us-east-1a", "us-east-1b"]
managedNodeGroups:
  - name: hossen1-cluster-4
    availabilityZones: ["us-east-1a"]
    instanceType: hpc7g.16xlarge
    minSize: 4
    maxSize: 8
    desiredCapacity: 4
    efaEnabled: true
    placement:
      groupName: eks-efa-testing
    labels: { "flux-operator": "true" }
    tags:
      kubernetes.io/cluster/hossen1-cluster-4: "owned"
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/scaling-study-efa: "owned"
      Name: 'hossen1-cluster-4'
    propagateASGTags: true
    ssh:
      allow: true
      publicKeyPath: ~/.ssh/id_rsa.pub

Expected behavior
A clear and concise description of what you expected to happen.

Version of eksctl
0.149.0-dev+2adbe411.2023-09-11T16:14:07Z

Anything else?