chronosphereio/calyptia-core-docker-desktop-extension

Docker Desktop extension for Calyptia Core automation.

calyptia-core-docker-desktop-extension's Issues

Vivo image is 1.33 GB

Current Vivo image is too big:

calyptia/vivo        1.1.0              d62920d6069d   2 weeks ago     1.33GB

Docker Desktop / Extend CLI usability with container and registration command

Calyptia CLI command is useful when already installed and the user already knows its own API keys. But we aim to simplify the process for users to try out Calyptia Core in Docker Desktop.

I am working on building a Docker Desktop extension for Calyptia Core (feel free to take over this) and I found some missing pieces to automate the process, and make it easier. What is missing to simplify the onboarding process ?

• Calyptia CLI command container image. Our public docker hub organization does not have the CLI tool exposed
• CLI should support a core-register option that redirects the user in the browser to the core.calyptia.com domain. This command must check if Core is already installed or not.
• Once the user authenticates, the CLI waits for the credentials set in the cloud. So it can auto-install Core in the local Kubernetes cluster defined by Docker Desktop

With those components in place the workflow in Docker Desktop should be:

• Docker Desktop: add Calyptia Core extension

• the extension runs our container and automates registration, e.g:

  docker run calyptia/calyptia-cli core-register

• the user register or re-login, the CLI receives the credentials to install Core

• upon successful installation:

  • Docker Desktop opens a browser Tab with Core UI

Move kubectl commands to Calyptia CLI

Move kubectl commands to the Calyptia CLI and remove kubectl dependency.

Auto create release on tag

Add CI to automatically create a Github release for tags

Document UI development outside of DD

Please document how a UI developer can modify the UI, CSS and general frontend development and test it properly without deploying the extension.

You can add the information in a DEVELOPERS.md file

Intermittent Vivo container crashes

When I use Core Docker Desktop Extension, Vivo container is intermittent crashing.
This is caused by EPIPE on node application (that is Vivo?):

$ kubectl logs calyptia-internal-vivo-7b79dd94ff-4qjs4
<snip>
2/14 05:31:22] [ info] [output:stdout:stdout.0] thread worker #0 stopped\n"}
Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

And frequently crashing vivo container causes lots of waiting loop of CrashLoopBackOff:

$ kubectl get po
NAME                                                              READY   STATUS             RESTARTS       AGE
calyptia-amazing-secret-cc95-default-deployment-75b56677b778cjc   1/1     Running            17 (39m ago)   22d
calyptia-bright-haven-44a6-default-deployment-74ffc78774-lvlp6    1/1     Running            19 (39m ago)   27d
calyptia-cluster-logging-5095-6bdmt                               1/1     Running            68 (39m ago)   27d
calyptia-cluster-logging-f7c1-mmz6w                               1/1     Running            34 (39m ago)   22d
calyptia-complete-multiple-man-1898-default-deployment-5c6gxb45   1/1     Running            19 (39m ago)   22d
calyptia-easy-cypher-8c2d-default-deployment-57997d68d5-m8nkm     1/1     Running            20 (39m ago)   22d
calyptia-giving-tombstone-0ff0-default-deployment-7c58ddcfrh854   1/1     Running            19 (39m ago)   22d
calyptia-health-check-0cdb-7f9dd4664c-zj99s                       1/1     Running            97 (38m ago)   22d
calyptia-health-check-2360-58b68c669c-sdnqj                       1/1     Running            65 (38m ago)   22d
calyptia-health-check-4c53-7f84d596c4-rgc6z                       1/1     Running            66 (38m ago)   22d
calyptia-health-check-4c6a-796f95c9f7-lbplx                       1/1     Running            88 (38m ago)   22d
calyptia-health-check-4ca8-86cb8cb5b4-k2hpx                       1/1     Running            68 (38m ago)   22d
calyptia-health-check-6414-76b5d7998b-dhxl4                       1/1     Running            77 (38m ago)   22d
calyptia-health-check-7640-64766d5564-bn9tw                       1/1     Running            78 (38m ago)   22d
calyptia-health-check-7bd2-56bf586fc7-hrmkt                       1/1     Running            69 (38m ago)   22d
calyptia-health-check-805d-b97d99f9b-6lhzg                        1/1     Running            72 (38m ago)   22d
calyptia-health-check-8f01-6694768df4-84klw                       1/1     Running            71 (38m ago)   22d
calyptia-health-check-9cf7-5bb47ccb5d-nzhx6                       1/1     Running            75 (38m ago)   22d
calyptia-health-check-a3fa-65dc8869d6-c9cwn                       1/1     Running            73 (38m ago)   27d
calyptia-health-check-acd0-5459b999c9-74g4k                       1/1     Running            91 (38m ago)   22d
calyptia-health-check-ea7f-d6996cc89-t76kb                        1/1     Running            66 (38m ago)   22d
calyptia-internal-vivo-7b79dd94ff-4qjs4                           0/1     CrashLoopBackOff   15 (81s ago)   3h38m
calyptia-known-catwoman-51c3-default-deployment-687798ffd8jvwkm   1/1     Running            20 (39m ago)   22d
calyptia-mint-apocalypse-70d0-default-deployment-5b8f8c66f2wgpb   1/1     Running            19 (39m ago)   22d
calyptia-modern-calamity-c5d8-default-deployment-8585466692nc22   1/1     Running            18 (39m ago)   22d
calyptia-optimal-andromeda-b6ac-default-deployment-5f54946fqksr   1/1     Running            19 (39m ago)   22d
calyptia-present-frenzy-a444-default-deployment-5bc9476667sfrg2   1/1     Running            20 (39m ago)   22d
calyptia-primary-micromax-3812-default-deployment-58f5f9f5qh76m   1/1     Running            18 (39m ago)   22d
calyptia-supreme-fenris-a033-default-deployment-56f9bdfcddldk7m   1/1     Running            18 (39m ago)   22d
calyptia-well-copperhead-0c72-default-deployment-68447689dvt9fc   1/1     Running            17 (39m ago)   22d

RFC: product: let's add more value to the extension for users

The Docker Desktop extension idea was born with the desire to amplify the message about Calyptia Core. From the beginning, it was easy to realize that this new "local-market-place" for users could be a good starting point for many initiatives.

The first value of this extension as of today for __us__is that any customer or prospects will be able to try out our product without the need to have access to a Kubernetes Cluster to try Core; note that getting a cluster is not that easy and sometimes takes time, the user has to take many steps to start testing our product, which is not ideal.

Docker Desktop provides a local Kubernetes cluster ready to go, so for us is easy to provide a 2-3 click UI experience to let the user try out the product. This is where we are today:

I think this is a perfect starting point, and there is plenty of room for improvement: we need to add more value to users to keep using the extension and our technology. I see Docker Desktop as a Sandbox environment; ideally, we want to bundle more features that help users.

Today the extension is helpful for us to pitch prospects, but the next exciting step is: what value/feature can we add so users will try out the extension themselves without our intervention? How do we make it a great add-on?

Some raw ideas need further technical elaboration

1. Bundle Core UI inside the extension

Before considering the technical challenges, let's focus on the values this could bring.

I think this would add a smooth experience for the user, where they can play with pipeline automation or custom pipelines right there, with no need to go to the browser; this is about simplicity and a better right-out-of-the-box experience.

2. Add Vivo project as a destination

Maybe this is not a specific Docker Extension challenge but something generic for Core as a product. To try Core , you must have a destination to receive data; I think we need to reduce friction **here.

This is the current state when building a pipeline:

What if we add a Local category on top and add Vivo as an option?

Vivo becomes a local service that receives data flow from the sources or automated agents, and the user can start experimenting and see value right away

NOTE: once Vivo is launched or available, we should have an easy way to access it through Core UI.

Please comment on this, and make the extension get to the next level have a high priority; it's not just a side project.

vivo: add a "clear" button

add a button that allows to "clear" the records buffer.

Icon and screenshot URLs required for publishing

We need to provide an icon image and set of screenshot URLs as part of the container metadata: https://docs.docker.com/desktop/extensions-sdk/extensions/labels/

Need to upload these somewhere public for anyone to see (it's part of the marketplace listing) and then update the Dockerfile labels to reference them.
Screenshots likely need to wait for a stable UI.

Deleting core instance leaves error page

Deleting a core instance then leaves an error page until we refresh the page (e.g. switch away and back):

Sometimes CPU usage of Virtual Machine Service spikes up

In M1/M2 Mac, I often experienced that CPU usage of Virtual Machine Service spikes up:

This causes surprisingly consuming battery on macOS....

Note: This phenomenon tends to occur after Vivo container crash. The beginning of using this Docker Desktop extension does not occurred this CPU usage spike.

Core Desktop Extension Version: 0.4.2
macOS M1/M2:

% sw_vers
ProductName:		macOS
ProductVersion:		13.1
BuildVersion:		22C65

UI improvements.

• After successful authentication, the extension should install (deploy) Core automatically, so we avoid one click (Deploy Core).
• On removing the extension, remove Core from the Kubernetes cluster
• In the view of "manage core", create a table components with the metadata/environment json info that includes; name, tags, k8s metadata, status of the instance.

design: revisit design guidelines

Docker Desktop team provides the following design guidelines, let's make sure we are aligned:

https://docs.docker.com/desktop/extensions-sdk/design/design-guidelines/

Timeout for "Waiting Authorization"

If the user is installing the extension just closes the authorization window without any Cancel/Accept action, the extension in "Waiting Authorization" hangs forever:

Here is a quick repro:

• https://www.loom.com/share/881523bc4a5545f78fa25136bf9eb794

We need better logic to handle this kind of case, maybe a timeout before the button "Log in with browser" get's active again?

The current workaround is to go out of the extension to another menu and then back to the extension, the login option is available.

Remove Core components when uninstalling the extension

After uninstalling the extension the Core pods/containers are still present:

expectation is that after removing the extension, no Core components should exist in the user environment.

Kubernetes filter not retrieving cluster metadata

Cluster logging makes use of the kubernetes filter, for some reason
when using the DD provided kubernetes cluster the metadata information
cannot be retrieved from the api-server.

Investigate why and fix it.

blockers to be published

The following feedback has been received from Docker team that blocks the publish of this extension:

• Remove the double quotes from the title and fix typo: “Calytia Core”
• Remove the double quotes from the description: “Use Calyptia Core within Docker Desktop to manage observability.”
• The detailed description is not accurate: “It will allow you to use, evaluate and test Calyptia Core without needing to deploy or use a Kubernetes cluster”. Apparently the extension requires the user to have the built-in Kubernetes cluster in Docker Desktop enabled.
• If you remove the cloud instance from the web and re-install the extension, the extension will not create a new cloud instance. It seems there are workloads of Calyptia running in K8s after the extension is removed (hardy-piledriver-a5a3_calyptia and vivo_calyptia-internal-vivo). Given the Extensions SDK doesn’t provide a hook to clean-up custom resources as part of the uninstall process, I’d advise on improving the extension to check whether Calpytia is already running on the K8s cluster.
• Add a dark mode version of the extension. cc: @ariCrespo79

Other Suggestions / Non-blocking issues:

• After opening “Vivo” in the extension and sending multiple events, the “Clear” button doesn’t clear the logs.
• To use the UI component styles from the Docker extension styleguide wherever possible such the primary blue buttons etc https://www.figma.com/file/U7pLWfEf6IQKUHLhdateBI/Docker-Extensions-UI-Styleguide?node-id=1%3A28771
• Change CTA from "Sure" to "Delete" as that is a more standard UX practice.

Remove device login Auth

Calyptia authentication uses auth0 which works fine. But when is about to authenticate external applications like the Docker Desktop extension we depend on a Devide-based authentication mechanism that adds an extra step for the user.

Talking with the authors of Meshery, they mention they are using this Golang component to implement a straight OAuth2 layer:

• https://github.com/markbates/goth

can we host that service ? the good thing is that goth seems to allow to use Auth0 as the backend

Add acceptance or automated tests for releases

Ensure we verify basic behaviour, potentially with podman desktop.

On deploy, set a default tag "docker-desktop"

when deploying Core, there is no tag associated to the instance, please set a default: docker-desktop.

unreachable / auto-healing ?

I found my local cluster in an unreachable state:

Inspecting the logs:

~/coding/core-docker-desktop-extension (main) » kubectl get pods                                                                              edsiper@monox-2
NAME                                                             READY   STATUS             RESTARTS        AGE
calyptia-giving-elite-4711-default-deployment-748d49885b-mzcw6   1/1     Running            3 (165m ago)    8h
calyptia-health-check-d80d-7fdc948c9-44bvh                       0/1     CrashLoopBackOff   92 (115s ago)   8h
calyptia-www-f455b6569-r9cvf                                     1/1     Running            2 (164m ago)    177m

~/coding/core-docker-desktop-extension (main) » kubectl logs calyptia-health-check-d80d-7fdc948c9-44bvh                                       edsiper@monox-2
Calyptia Core v22.9, patch_level=1
* https://calyptia.com

[2022/10/15 02:57:11] [error] [config] error in /config/fluent-bit.conf:3: undefined value - check config is in valid classic format
[2022/10/15 02:57:11] [error] configuration file contains errors, aborting.

what can be done to avoid this problem ?

fix icon in extension listing

and.. fix the description

Provide Vivo deployment

Use the K8S YAML to deploy Vivo within the local cluster in the same namespace as a simple option to deploy: https://github.com/calyptia/vivo/blob/master/vivo-k8s.yaml
The suggestion is to use the YAML directly in the code or provide it in a public location to consume.

My suggestion is to always deploy it with Core in the same namespace using a fixed vivo service name to simplify workflows.

This will need integration then to display the Vivo UI as a tab - currently Vivo will not accept data without a valid websocket connection but hopefully this will be resolved: https://github.com/calyptia/vivo/issues/12

Vivo CrashLoopBackOff

» kubectl get pods                                                                                                                            edsiper@monox
NAME                                                              READY   STATUS             RESTARTS      AGE
calyptia-health-check-4560-67b9d99d5f-gng8t                       1/1     Running            0             2m52s
calyptia-pipeline-655fbb9ff-jcn9b                                 1/1     Running            0             2m32s
calyptia-relevant-hardball-12d8-default-deployment-7fb7b4666vwq   1/1     Running            0             3m4s
calyptia-vivo-78bbc95cdd-z6x4m                                    0/1     CrashLoopBackOff   4 (41s ago)   3m4s

kubectl logs calyptia-vivo-78bbc95cdd-z6x4m                                                                                                 edsiper@monox
Using VIVO_TOKEN variable for token.

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  3m46s                  default-scheduler  Successfully assigned default/calyptia-vivo-78bbc95cdd-z6x4m to docker-desktop
  Normal   Pulled     3m35s                  kubelet            Successfully pulled image "ghcr.io/calyptia/vivo" in 9.933675714s
  Warning  Unhealthy  3m25s                  kubelet            Liveness probe failed: Get "http://10.1.0.254:5489/healthz": dial tcp 10.1.0.254:5489: connect: connection refused
  Normal   Pulled     3m21s                  kubelet            Successfully pulled image "ghcr.io/calyptia/vivo" in 1.04460325s
  Normal   Pulled     2m57s                  kubelet            Successfully pulled image "ghcr.io/calyptia/vivo" in 1.156501459s
  Normal   Pulling    2m21s (x4 over 3m45s)  kubelet            Pulling image "ghcr.io/calyptia/vivo"
  Normal   Created    2m20s (x4 over 3m35s)  kubelet            Created container vivo
  Normal   Pulled     2m20s                  kubelet            Successfully pulled image "ghcr.io/calyptia/vivo" in 1.230841876s
  Normal   Started    2m19s (x4 over 3m35s)  kubelet            Started container vivo
  Warning  BackOff    116s (x8 over 3m11s)   kubelet            Back-off restarting failed container

Expose Kubernetes Cluster name

Cluster names are a bit tricky in Kubernetes and seem there is no clear API to retrieve it.

Since the extension have access to kubectl, we can grab that value by using the following command:

kubectl config view --minify -o jsonpath='{.clusters[].name}'

Data ingestion not working

I've created a simple pipeline HTTP <> HTTP and the configured ports are not routing the data:

This is how the source was configured:

When ingesting data with curl I get an empty response from the server:

~ » curl -i -XPOST -d '{"key": "val"}' -H "content-type: application/json" http://127.0.0.1:9880                                              edsiper@monox-2
curl: (52) Empty reply from server

NOTE: the same command works fine when sending data to a normal Fluent Bit instance

The service seems to be up and running, but not sure why there is no response:

~ » kubectl get services                                                                                                                 52 ↵ edsiper@monox-2
NAME                                            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
app                                             ClusterIP      10.101.55.14    <none>        5489/TCP,24224/TCP   15h
kubernetes                                      ClusterIP      10.96.0.1       <none>        443/TCP              38h
tcp-2020-2b3a4032-0f5f-4135-ab83-0b8d9c4a891d   LoadBalancer   10.102.201.39   localhost     2020:31964/TCP       28h
tcp-9880-c6852113-3163-43fd-a03c-7ce3e70fe251   LoadBalancer   10.101.76.20    localhost     9880:30564/TCP       37m

Note that in my local cluster I have some weird issues from a previous deployment run:

~ » kubectl get pods                                                                                                                    130 ↵ edsiper@monox-2
NAME                                                              READY   STATUS             RESTARTS        AGE
app-c454c47f5-7dsdm                                               0/1     ImagePullBackOff   0               15h
calyptia-deep-star-lord-4d6d-default-deployment-6fdb4f5ffb5xxcq   1/1     Running            1               28h
calyptia-docker-desktop-http-694d5d6bc7-9xrmx                     1/1     Running            0               56s
calyptia-health-check-fca3-fd6c957d6-wnfx4                        0/1     CrashLoopBackOff   324 (62s ago)   28h

Image size is 479MB

The main calyptia/core-docker-desktop image size is 479MB:

calyptia/core-docker-desktop        1.0.1               62c33894e1a3   6 hours ago     479MB

The purpose of the docker extension is to simplify the onboarding or demoing the product to prospects, if the user needs to download 0.5G is not the way to go.

Vivo image is also around 571MB.

Let's fix this.

Fix fonts

Fonts type and size are not following the design proposed, this is how it looks today:

Please fix the look & feel according to the Figma designs:

• https://www.figma.com/file/p4AwEYeF8qvSViMcANamX7/Cloud-%5Bjun2022%5D?node-id=8872%3A61499