hnaderi/scala-k8s

Kubernetes client, data models and typesafe manifest generation for scala, scalajs, and scala native

Scala K8S Kubernetes client, data models and typesafe manifest generation for scala, scalajs, and scala native

What

This library provides a full blown extensible client that you can use to interact directly with kubernetes API server, to create operators or accomplish other automation tasks, also you can use it to create or manipulate manifests in scala.

Why

Kubernetes spec is large enough to fit in one's brain, and YAML and helm are a joke! instead of using different tooling and languages with no to little IDE and compiler support, you can use this library to have the latest kubernetes API spec under your tool belt, in scala! so the most complex templates are just simple functions, and you can use whatever abstraction you like to create objects; and create manifests easily.
for easy to use recipes and integration with sbt, visit this project

Goals

• to become the defacto k8s integration library in all scala ecosystems

Design principles

• As extensible as possible
• As dependency free as possible
• As Un-opinionated as possible
• Provide seamless integrations
• All specs are generated from the spec directly and will be in sync with kubernetes all the time

Getting started

This library is currently available for Scala binary versions 2.12, 2.13 and 3.2 on JVM/JS/Native.
This library is architecured in a microkernel fashion and all the main kubernetes stuff are implemented/generated in pure scala, and integration modules are provided separately.
main modules are:

• objects raw kubernetes objects, which has no dependency
• client raw kubernetes client and requests, requests can also be extended in user land easily!

libraryDependencies ++= Seq(
  "dev.hnaderi" %% "scala-k8s-objects" % "@VERSION@", // JVM, JS, Native ; raw k8s objects
  "dev.hnaderi" %% "scala-k8s-client" % "@VERSION@", // JVM, JS, Native ; k8s client kernel and requests
  )

The following integrations are currently available:

libraryDependencies ++= Seq(
  "dev.hnaderi" %% "scala-k8s-http4s-ember" % "@VERSION@", // JVM, JS, Native ; http4s ember client integration
  "dev.hnaderi" %% "scala-k8s-http4s-netty" % "@VERSION@", // JVM ; http4s netty client integration
  "dev.hnaderi" %% "scala-k8s-http4s-blaze" % "@VERSION@", // JVM; http4s blaze client integration
  "dev.hnaderi" %% "scala-k8s-http4s-jdk" % "@VERSION@", // JVM; http4s jdk-client integration
  "dev.hnaderi" %% "scala-k8s-http4s" % "@VERSION@", // JVM, JS, Native ; http4s core and fs2 integration
  "dev.hnaderi" %% "scala-k8s-zio" % "@VERSION@", // JVM ; ZIO native integration using zio-http and zio-json 
  "dev.hnaderi" %% "scala-k8s-sttp" % "@VERSION@", // JVM, JS, Native ; sttp integration using jawn parser
  "dev.hnaderi" %% "scala-k8s-circe" % "@VERSION@", // JVM, JS ; circe integration
  "dev.hnaderi" %% "scala-k8s-json4s" % "@VERSION@", // JVM, JS, Native; json4s integration
  "dev.hnaderi" %% "scala-k8s-spray-json" % "@VERSION@", // JVM ; spray-json integration
  "dev.hnaderi" %% "scala-k8s-play-json" % "@VERSION@", // JVM ; play-json integration
  "dev.hnaderi" %% "scala-k8s-zio-json" % "@VERSION@", // JVM, JS ; zio-json integration
  "dev.hnaderi" %% "scala-k8s-jawn" % "@VERSION@", // JVM, JS, Native ; jawn integration
  "dev.hnaderi" %% "scala-k8s-manifests" % "@VERSION@", // JVM, JS, Native ; yaml manifest reading and generation
  "dev.hnaderi" %% "scala-k8s-scalacheck" % "@VERSION@" // JVM, JS, Native; scalacheck instances
)

visit project site to see more tutorials and docs, also please drop a ⭐ if this project interests you. I need encouragement.

sbt integration

see this project

Future plans

• more integrations (akka-http, ...)!

scala-k8s's Issues

list deployments matching labelSelector doesn't appear to work

I'm trying to do the equivalent of kubectl get deployment -A -l app=foolish based on the sttp client example. It appears to be returning all deployments, not just deployments with an app=foolish label.

import io.circe.Json
import dev.hnaderi.k8s.circe._
import dev.hnaderi.k8s.client.APIs
import dev.hnaderi.k8s.client.SttpJdkURLClientBuilder
import sttp.client3.circe._

val client = SttpJdkURLClientBuilder.load((os.home / ".kube" / "scratch-config.yaml").toNIO)

val p1 = APIs.deployments.list(labelSelector = List("app=foolish")).send(client)
p1.body.items.flatMap(_.metadata).flatMap(_.labels).flatMap(_.get("app")).map(println)
// res0: Seq[Unit] = List((), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), ())
// authelia
// tivan
// cert-manager
// cainjector
// webhook
// azure-policy
// azure-policy-webhook
// csi-blob-controller
// kube-prometheus-stack-operator
// foolish
// foolish
// nox-dox
// nox-dox
// nox-dox
// redirector-test
// selenium-chrome-node
// selenium-edge-node
// selenium-firefox-node
// selenium-hub

JS Kubeconfig

watchNodes is throwing ConnectException

I'm experiencing the following ConnectException when trying use watchNodes code provided in the documentation. However getNodes code works perfectly fine.

Exception:

K8s watch nodes
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
dev.hnaderi.k8s.client.HttpClient$$anon$2@1c17b411
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishConnect(UnixAsynchronousSocketChannelImpl.java:256)
        at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:202)
        at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:217)
        at java.base/sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:312)
        at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:113)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
        at async_ @ fs2.io.net.SocketGroupCompanionPlatform$AsyncSocketGroup.connect$1$$anonfun$1(SocketGroupPlatform.scala:70)
        at println @ uk.sky.poc.k8s.ControllerTest$.run(ControllerTest.scala:30)
        at map @ com.comcast.ip4s.SocketAddressCompanionPlatform$ResolveOps.resolve(SocketAddressPlatform.scala:39)
        at flatMap @ fs2.io.net.SocketGroupCompanionPlatform$AsyncSocketGroup.connect$1(SocketGroupPlatform.scala:71)
        at delay @ fs2.io.net.SocketGroupCompanionPlatform$AsyncSocketGroup.setup$1$$anonfun$3(SocketGroupPlatform.scala:56)
        at onError$extension @ org.typelevel.keypool.KeyPool$Builder.keepRunning$1(KeyPool.scala:370)
        at delay @ fs2.io.net.SocketGroupCompanionPlatform$AsyncSocketGroup.setup$1(SocketGroupPlatform.scala:53)
        at main$ @ uk.sky.poc.k8s.ControllerTest$.main(ControllerTest.scala:24)
        at main$ @ uk.sky.poc.k8s.ControllerTest$.main(ControllerTest.scala:24)

Code:

object ScalaK8sTest extends IOApp:

  val buildClient = EmberKubernetesClient.defaultConfig[IO, Json]

  def run(args: List[String]): IO[ExitCode] =
    for {
      _ <- IO.println("K8s watch nodes")

      watchNodes <- fs2.Stream
        .resource(buildClient)
        .evalTap(IO.println)
        .flatMap(APIs.nodes.list().listen)
        .evalTap(IO.println)
        .compile
        .toList
        
      _ <- IO.println(s"nodes list: $watchNodes")
    } yield ExitCode.Success

@hnaderi could you help please? Thanks

APIs for CustomResourceDefinition

related #96

Add Remained API verbs

Currently not all API verbs are implemented

Support for Patch verb

Support the following operations:

• Json patch
• Json merge patch
• Strategic json merge patch
• Server side apply

For more detail on how kubectl uses this verb, read this

zio remote cluster authentication and pod listing

Hi @hnaderi, documentation for listing pods (simplest example) in zio when we have a remote cluster is missing. the following code was the best I came up with.

object Main extends ZIOAppDefault {
  def run = {
    val token = "sha256~xxxx"

    val namespace = "my-namespace"
    val host = "https://cluster_url:443"
    val auth = AuthenticationParams.bearer(token)

    val client = ZClient.default
    val backend = ZIOBackend.make
    val k8sLayer = ZLayer.fromFunction(HttpClient[Task](host, _, auth))
    (for {
      k8s <- ZIO.service[HttpClient[Task]]
      x <- k8s.send(PodAPI(namespace).list)
      _ <- ZIO.logInfo(x.toString)
    } yield ())
      .provide(
        client,
        backend,
        k8sLayer
      )
}

and I encounter the following error:

timestamp=2023-12-05 level=ERROR thread=#zio-fiber-1 message="" cause="Exception in thread "zio-fiber-4" dev.hnaderi.k8s.client.ErrorResponse: Request failed!
status: Other(403)
details: Status(Status,v1,Some(Forbidden),Some(403),Some(Failure),Some(StatusDetails(None,None,None,Some(pods),None,None)),Some(ListMeta(None,None,None,None)),Some(pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "my-namespace"))

        at dev.hnaderi.k8s.client.ZIOBackend.$anonfun$expect$5(ZIOBackend.scala:137)
        at zio.ZIO.$anonfun$map$2(ZIO.scala:960)
        at dev.hnaderi.k8s.client.ZIOBackend.expect(ZIOBackend.scala:137)
        at testK8s.Main.run.client(main.scala:50)
        at testK8s.Main.run(main.scala:65)"

any help will be highly appreciated.

Listing pods not working

Tried to list the pods like below but it doesn't seem to work.

buildClient.use(APIs.pods.list().send).map(println)

I could list the pods using the curl command!

curl -s http://localhost:8080/api/v1/namespaces/default/pods/ | jq -rM '.items[].metadata.name'

Http4sKubernetesClient.defaultConfig should allow passing context

First of all, thanks for this nice library, really easy to get started with!

I have one suggestion on the Http4sKubernetesClient, especially its defaultConfig: It should optionally allow passing a context name like load (which it uses internally) so that you can specify which context to use. As both homeConfig and podConfig are private, it's not trivial to recreate defaultConfig with a specific context. Or is that intended for some good reason I'm not aware of?

Integration with akka http

ZIO kubeconfig

ZIO json tests fail randomly

[Question] Is there a way to use your object generator to generate objects for an arbitrary OpenAPI Spec?

I'm writing an integration with Argo Workflows and I was thinking it'd be handy to reuse your object generator to process their OpenAPI spec but I wasn't able to find a way to integrate with it (I'm still fairly new to the scala ecosystem so I may just have missed it)

CustomResource with WatchEvent support for CustomResourceAPI

At the moment CustomResourceAPI only support CRDs not CRs. Is it possible to add CRs to the API similar to CRDs so that we can watch on the CR events using listen function?

@hnaderi Do you think it is feasible? Thanks

Container images sizeBytes to large to fit into Int, causing decode error

Hi,

I've just tried to list nodes from a cluster via the API like this:

val client = SttpJdkURLClientBuilder.from("http://localhost:8001")
val nodes = APIs.nodes.list().send(client)

Triggering the following error:

...
dev.hnaderi.k8s.client.SttpKBackend$DecodeError: Expected Integer, but got: 3241619679

Looking at the original api server output, I think in this specific case it is the sizeBytes field in generated ContainerImage, but there might be other fields as well, that should use Long instead of Int.

final case class ContainerImage(
  names : Option[Seq[String]] = None,
  sizeBytes : Option[Int] = None
) {
...

Add authorization and TLS support

Native kubeconfig

Improve CI/CD build time

This project has a huge code base thanks to vast amount of object definitions in kubernetes spec. Count that for each platform and language version to get a grasp of build time.
Build and test in CI/CD takes almost all of the available resources of the worker machine and makes pipeline runs very unreliable. It's very rare to have a pipeline complete without several retries! CI step simply hang up and are killed by the 6 hours timeout of the github actions.
There are some possible solutions and workarounds:

• Set an explicit lower timeout, for example 30 minutes. This is possible using https://github.com/sbt/sbt-github-actions/releases/tag/v0.15.0 But sbt-typelevel doesn't include this version yet.
• reduce build time, which needs investigation.

Websocket support

Websocket support allows us to implement pod exec API.
Here is one example

fieldSelector not working

I use the code below to watch pod changes, seems like fieldSelector donesn't work, can you help ?

fs2.Stream
      .resource(buildClient)
      .flatMap(
        CoreV1.events
          .list(fieldSelector = List(s"metadata.namespace=${ns}"))
          .listen
      )

Integration with http4s-jdk-client

Code generation misses kind field in the `CustomResourceDefinitionNames`

Hi @hnaderi - I'm back from a long weekend, hope you're well. 😄

So I've done some more digging and the culprit is here!

final case class CustomResourceDefinitionNames(
  plural : String,
  singular : Option[String] = None,
  listKind : Option[String] = None,
  categories : Option[Seq[String]] = None,
  shortNames : Option[Seq[String]] = None
)

Note that this is missing the kind field. What's interesting is that I followed your plugin code and that led me to the following understanding:

1. CustomResourceDefinitionNames is a generate class
2. The KubernetesSpecPlugin generates it by reading the k8s spec based on the version supplied in the build.sbt file.
3. Having downloaded the spec as json, it parses and writes the details into src_managed.

So far so good. (I like how you defined the plugin to do that, btw - nice!)

The trouble is that it isn't working correctly. If you go to the spec, you can clearly see the kind field defined, and also declared as required in the JSON schema definition - but for some reason, this is being ignored.

Here's the spec, search for CustomResourceDefinitionNames.
https://raw.githubusercontent.com/kubernetes/kubernetes/v1.27.1/api/openapi-spec/swagger.json

...and you get this:

    "io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.CustomResourceDefinitionNames": {
      "description": "CustomResourceDefinitionNames indicates the names to serve this CustomResourceDefinition",
      "properties": {
        "categories": {
          "description": "categories is a list of grouped resources this custom resource belongs to (e.g. 'all'). This is published in API discovery documents, and used by clients to support invocations like `kubectl get all`.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "kind": {
          "description": "kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the `kind` attribute in API calls.",
          "type": "string"
        },
        "listKind": {
          "description": "listKind is the serialized kind of the list for this resource. Defaults to \"`kind`List\".",
          "type": "string"
        },
        "plural": {
          "description": "plural is the plural name of the resource to serve. The custom resources are served under `/apis/<group>/<version>/.../<plural>`. Must match the name of the CustomResourceDefinition (in the form `<names.plural>.<group>`). Must be all lowercase.",
          "type": "string"
        },
        "shortNames": {
          "description": "shortNames are short names for the resource, exposed in API discovery documents, and used by clients to support invocations like `kubectl get <shortname>`. It must be all lowercase.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "singular": {
          "description": "singular is the singular name of the resource. It must be all lowercase. Defaults to lowercased `kind`.",
          "type": "string"
        }
      },
      "required": [
        "plural",
        "kind"
      ],
      "type": "object"
    }

Note:

        "kind": {
          "description": "kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the `kind` attribute in API calls.",
          "type": "string"
        },

and

      "required": [
        "plural",
        "kind"
      ],

What do you think is going on?

Originally posted by @davesmith00sky in #96 (comment)

Add ZIO native streaming

ZIO based client doesn't support streaming responses currently, it can be implemented after zio/zio-http#1500

Listen function is timing out after 45 sec.

I'm trying to listen to K8s nodes / pods continuously but the listen function times out after 45 seconds. Is this the expected behaviour? Is there a way to listen indefinitely and print the any new nodes are added to the cluster?

Timeout Exception:

java.util.concurrent.TimeoutException: 45 seconds
        at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(Util.scala:104)
        at main$ @ poc.k8s.ControllerTest$.main(ControllerTest.scala:24)
        at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(Util.scala:104)
        at timeout @ org.http4s.ember.core.Util$.timeoutMaybe(Util.scala:104)

Code:

object ControllerTest extends IOApp:

  val clientResource = EmberKubernetesClient.defaultConfig[IO, Json]

  def run(args: List[String]): IO[ExitCode] =
    for {
      _ <- IO.println("Watch K8s nodes")

      watchNodes <- fs2.Stream.resource(clientResource)
        .flatMap(APIs.nodes.list().listen)
        // Debugging
        .flatMap(ev =>
          exec(
            IO.println("event => " + ev.event) >> 
            IO.println("name => " + ev.payload.metadata.flatMap(_.name))
          )
        )
        .compile
        .drain

    } yield ExitCode.Succes

@hnaderi Could you help with this please? Thanks

Example of how to create a CRD

Hi @hnaderi,

Between being fairly new to k8s and scala-k8s, I'm struggling to figure out how to create a CRD from a manifest. I've tried a few things, but essentially what I'm trying to do is simply the equivalent of:

kubectl apply -f my-crd-manifest.yaml

Where my-crd-manifest.yaml contains the definition of a CRD.

Could you point me in the right direction please?

Once I've got it set up, I've got some custom look ups defined that are based on the example in the readme. Can I just check if that is the right way to go about that?

Thanks in advance.

Add sttp support