chronosphereio / calyptia-core-docker-desktop-extension Goto Github PK
View Code? Open in Web Editor NEWDocker Desktop extension for Calyptia Core automation.
License: Other
Docker Desktop extension for Calyptia Core automation.
License: Other
Current Vivo image is too big:
calyptia/vivo 1.1.0 d62920d6069d 2 weeks ago 1.33GB
Calyptia CLI command is useful when already installed and the user already knows its own API keys. But we aim to simplify the process for users to try out Calyptia Core in Docker Desktop.
I am working on building a Docker Desktop extension for Calyptia Core (feel free to take over this) and I found some missing pieces to automate the process, and make it easier. What is missing to simplify the onboarding process ?
core-register
option that redirects the user in the browser to the core.calyptia.com domain. This command must check if Core is already installed or not.With those components in place the workflow in Docker Desktop should be:
Docker Desktop: add Calyptia Core extension
the extension runs our container and automates registration, e.g:
docker run calyptia/calyptia-cli core-register
the user register or re-login, the CLI receives the credentials to install Core
upon successful installation:
Move kubectl commands to the Calyptia CLI and remove kubectl dependency.
Add CI to automatically create a Github release for tags
Please document how a UI developer can modify the UI, CSS and general frontend development and test it properly without deploying the extension.
You can add the information in a DEVELOPERS.md file
When I use Core Docker Desktop Extension, Vivo container is intermittent crashing.
This is caused by EPIPE on node application (that is Vivo?):
$ kubectl logs calyptia-internal-vivo-7b79dd94ff-4qjs4
<snip>
2/14 05:31:22] [ info] [output:stdout:stdout.0] thread worker #0 stopped\n"}
Error: write EPIPE
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}
And frequently crashing vivo container causes lots of waiting loop of CrashLoopBackOff:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
calyptia-amazing-secret-cc95-default-deployment-75b56677b778cjc 1/1 Running 17 (39m ago) 22d
calyptia-bright-haven-44a6-default-deployment-74ffc78774-lvlp6 1/1 Running 19 (39m ago) 27d
calyptia-cluster-logging-5095-6bdmt 1/1 Running 68 (39m ago) 27d
calyptia-cluster-logging-f7c1-mmz6w 1/1 Running 34 (39m ago) 22d
calyptia-complete-multiple-man-1898-default-deployment-5c6gxb45 1/1 Running 19 (39m ago) 22d
calyptia-easy-cypher-8c2d-default-deployment-57997d68d5-m8nkm 1/1 Running 20 (39m ago) 22d
calyptia-giving-tombstone-0ff0-default-deployment-7c58ddcfrh854 1/1 Running 19 (39m ago) 22d
calyptia-health-check-0cdb-7f9dd4664c-zj99s 1/1 Running 97 (38m ago) 22d
calyptia-health-check-2360-58b68c669c-sdnqj 1/1 Running 65 (38m ago) 22d
calyptia-health-check-4c53-7f84d596c4-rgc6z 1/1 Running 66 (38m ago) 22d
calyptia-health-check-4c6a-796f95c9f7-lbplx 1/1 Running 88 (38m ago) 22d
calyptia-health-check-4ca8-86cb8cb5b4-k2hpx 1/1 Running 68 (38m ago) 22d
calyptia-health-check-6414-76b5d7998b-dhxl4 1/1 Running 77 (38m ago) 22d
calyptia-health-check-7640-64766d5564-bn9tw 1/1 Running 78 (38m ago) 22d
calyptia-health-check-7bd2-56bf586fc7-hrmkt 1/1 Running 69 (38m ago) 22d
calyptia-health-check-805d-b97d99f9b-6lhzg 1/1 Running 72 (38m ago) 22d
calyptia-health-check-8f01-6694768df4-84klw 1/1 Running 71 (38m ago) 22d
calyptia-health-check-9cf7-5bb47ccb5d-nzhx6 1/1 Running 75 (38m ago) 22d
calyptia-health-check-a3fa-65dc8869d6-c9cwn 1/1 Running 73 (38m ago) 27d
calyptia-health-check-acd0-5459b999c9-74g4k 1/1 Running 91 (38m ago) 22d
calyptia-health-check-ea7f-d6996cc89-t76kb 1/1 Running 66 (38m ago) 22d
calyptia-internal-vivo-7b79dd94ff-4qjs4 0/1 CrashLoopBackOff 15 (81s ago) 3h38m
calyptia-known-catwoman-51c3-default-deployment-687798ffd8jvwkm 1/1 Running 20 (39m ago) 22d
calyptia-mint-apocalypse-70d0-default-deployment-5b8f8c66f2wgpb 1/1 Running 19 (39m ago) 22d
calyptia-modern-calamity-c5d8-default-deployment-8585466692nc22 1/1 Running 18 (39m ago) 22d
calyptia-optimal-andromeda-b6ac-default-deployment-5f54946fqksr 1/1 Running 19 (39m ago) 22d
calyptia-present-frenzy-a444-default-deployment-5bc9476667sfrg2 1/1 Running 20 (39m ago) 22d
calyptia-primary-micromax-3812-default-deployment-58f5f9f5qh76m 1/1 Running 18 (39m ago) 22d
calyptia-supreme-fenris-a033-default-deployment-56f9bdfcddldk7m 1/1 Running 18 (39m ago) 22d
calyptia-well-copperhead-0c72-default-deployment-68447689dvt9fc 1/1 Running 17 (39m ago) 22d
The Docker Desktop extension idea was born with the desire to amplify the message about Calyptia Core. From the beginning, it was easy to realize that this new "local-market-place" for users could be a good starting point for many initiatives.
The first value of this extension as of today for __us__is that any customer or prospects will be able to try out our product without the need to have access to a Kubernetes Cluster to try Core; note that getting a cluster is not that easy and sometimes takes time, the user has to take many steps to start testing our product, which is not ideal.
Docker Desktop provides a local Kubernetes cluster ready to go, so for us is easy to provide a 2-3 click UI experience to let the user try out the product. This is where we are today:
I think this is a perfect starting point, and there is plenty of room for improvement: we need to add more value to users to keep using the extension and our technology. I see Docker Desktop as a Sandbox environment; ideally, we want to bundle more features that help users.
Today the extension is helpful for us to pitch prospects, but the next exciting step is: what value/feature can we add so users will try out the extension themselves without our intervention? How do we make it a great add-on?
Some raw ideas need further technical elaboration
1. Bundle Core UI inside the extension
Before considering the technical challenges, let's focus on the values this could bring.
I think this would add a smooth experience for the user, where they can play with pipeline automation or custom pipelines right there, with no need to go to the browser; this is about simplicity and a better right-out-of-the-box experience.
2. Add Vivo project as a destination
Maybe this is not a specific Docker Extension challenge but something generic for Core as a product. To try Core , you must have a destination to receive data; I think we need to reduce friction **here.
This is the current state when building a pipeline:
What if we add a Local category on top and add Vivo as an option?
Vivo becomes a local service that receives data flow from the sources or automated agents, and the user can start experimenting and see value right away
NOTE: once Vivo is launched or available, we should have an easy way to access it through Core UI.
Please comment on this, and make the extension get to the next level have a high priority; it's not just a side project.
add a button that allows to "clear" the records buffer.
We need to provide an icon image and set of screenshot URLs as part of the container metadata: https://docs.docker.com/desktop/extensions-sdk/extensions/labels/
Need to upload these somewhere public for anyone to see (it's part of the marketplace listing) and then update the Dockerfile labels to reference them.
Screenshots likely need to wait for a stable UI.
In M1/M2 Mac, I often experienced that CPU usage of Virtual Machine Service spikes up:
This causes surprisingly consuming battery on macOS....
Note: This phenomenon tends to occur after Vivo container crash. The beginning of using this Docker Desktop extension does not occurred this CPU usage spike.
Core Desktop Extension Version: 0.4.2
macOS M1/M2:
% sw_vers
ProductName: macOS
ProductVersion: 13.1
BuildVersion: 22C65
Docker Desktop team provides the following design guidelines, let's make sure we are aligned:
https://docs.docker.com/desktop/extensions-sdk/design/design-guidelines/
If the user is installing the extension just closes the authorization window without any Cancel/Accept action, the extension in "Waiting Authorization" hangs forever:
Here is a quick repro:
We need better logic to handle this kind of case, maybe a timeout before the button "Log in with browser" get's active again?
The current workaround is to go out of the extension to another menu and then back to the extension, the login option is available.
Cluster logging makes use of the kubernetes filter, for some reason
when using the DD provided kubernetes cluster the metadata information
cannot be retrieved from the api-server.
Investigate why and fix it.
The following feedback has been received from Docker team that blocks the publish of this extension:
Other Suggestions / Non-blocking issues:
Calyptia authentication uses auth0 which works fine. But when is about to authenticate external applications like the Docker Desktop extension we depend on a Devide-based authentication mechanism that adds an extra step for the user.
Talking with the authors of Meshery, they mention they are using this Golang component to implement a straight OAuth2 layer:
can we host that service ? the good thing is that goth seems to allow to use Auth0 as the backend
Ensure we verify basic behaviour, potentially with podman desktop.
when deploying Core, there is no tag associated to the instance, please set a default: docker-desktop
.
I found my local cluster in an unreachable state:
Inspecting the logs:
~/coding/core-docker-desktop-extension (main) » kubectl get pods edsiper@monox-2
NAME READY STATUS RESTARTS AGE
calyptia-giving-elite-4711-default-deployment-748d49885b-mzcw6 1/1 Running 3 (165m ago) 8h
calyptia-health-check-d80d-7fdc948c9-44bvh 0/1 CrashLoopBackOff 92 (115s ago) 8h
calyptia-www-f455b6569-r9cvf 1/1 Running 2 (164m ago) 177m
~/coding/core-docker-desktop-extension (main) » kubectl logs calyptia-health-check-d80d-7fdc948c9-44bvh edsiper@monox-2
Calyptia Core v22.9, patch_level=1
* https://calyptia.com
[2022/10/15 02:57:11] [error] [config] error in /config/fluent-bit.conf:3: undefined value - check config is in valid classic format
[2022/10/15 02:57:11] [error] configuration file contains errors, aborting.
what can be done to avoid this problem ?
Use the K8S YAML to deploy Vivo within the local cluster in the same namespace as a simple option to deploy: https://github.com/calyptia/vivo/blob/master/vivo-k8s.yaml
The suggestion is to use the YAML directly in the code or provide it in a public location to consume.
My suggestion is to always deploy it with Core in the same namespace using a fixed vivo
service name to simplify workflows.
This will need integration then to display the Vivo UI as a tab - currently Vivo will not accept data without a valid websocket connection but hopefully this will be resolved: https://github.com/calyptia/vivo/issues/12
» kubectl get pods edsiper@monox
NAME READY STATUS RESTARTS AGE
calyptia-health-check-4560-67b9d99d5f-gng8t 1/1 Running 0 2m52s
calyptia-pipeline-655fbb9ff-jcn9b 1/1 Running 0 2m32s
calyptia-relevant-hardball-12d8-default-deployment-7fb7b4666vwq 1/1 Running 0 3m4s
calyptia-vivo-78bbc95cdd-z6x4m 0/1 CrashLoopBackOff 4 (41s ago) 3m4s
kubectl logs calyptia-vivo-78bbc95cdd-z6x4m edsiper@monox
Using VIVO_TOKEN variable for token.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m46s default-scheduler Successfully assigned default/calyptia-vivo-78bbc95cdd-z6x4m to docker-desktop
Normal Pulled 3m35s kubelet Successfully pulled image "ghcr.io/calyptia/vivo" in 9.933675714s
Warning Unhealthy 3m25s kubelet Liveness probe failed: Get "http://10.1.0.254:5489/healthz": dial tcp 10.1.0.254:5489: connect: connection refused
Normal Pulled 3m21s kubelet Successfully pulled image "ghcr.io/calyptia/vivo" in 1.04460325s
Normal Pulled 2m57s kubelet Successfully pulled image "ghcr.io/calyptia/vivo" in 1.156501459s
Normal Pulling 2m21s (x4 over 3m45s) kubelet Pulling image "ghcr.io/calyptia/vivo"
Normal Created 2m20s (x4 over 3m35s) kubelet Created container vivo
Normal Pulled 2m20s kubelet Successfully pulled image "ghcr.io/calyptia/vivo" in 1.230841876s
Normal Started 2m19s (x4 over 3m35s) kubelet Started container vivo
Warning BackOff 116s (x8 over 3m11s) kubelet Back-off restarting failed container
I've created a simple pipeline HTTP <> HTTP
and the configured ports are not routing the data:
This is how the source was configured:
When ingesting data with curl
I get an empty response from the server:
~ » curl -i -XPOST -d '{"key": "val"}' -H "content-type: application/json" http://127.0.0.1:9880 edsiper@monox-2
curl: (52) Empty reply from server
NOTE: the same command works fine when sending data to a normal Fluent Bit instance
The service seems to be up and running, but not sure why there is no response:
~ » kubectl get services 52 ↵ edsiper@monox-2
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app ClusterIP 10.101.55.14 <none> 5489/TCP,24224/TCP 15h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 38h
tcp-2020-2b3a4032-0f5f-4135-ab83-0b8d9c4a891d LoadBalancer 10.102.201.39 localhost 2020:31964/TCP 28h
tcp-9880-c6852113-3163-43fd-a03c-7ce3e70fe251 LoadBalancer 10.101.76.20 localhost 9880:30564/TCP 37m
Note that in my local cluster I have some weird issues from a previous deployment run:
~ » kubectl get pods 130 ↵ edsiper@monox-2
NAME READY STATUS RESTARTS AGE
app-c454c47f5-7dsdm 0/1 ImagePullBackOff 0 15h
calyptia-deep-star-lord-4d6d-default-deployment-6fdb4f5ffb5xxcq 1/1 Running 1 28h
calyptia-docker-desktop-http-694d5d6bc7-9xrmx 1/1 Running 0 56s
calyptia-health-check-fca3-fd6c957d6-wnfx4 0/1 CrashLoopBackOff 324 (62s ago) 28h
The main calyptia/core-docker-desktop image size is 479MB
:
calyptia/core-docker-desktop 1.0.1 62c33894e1a3 6 hours ago 479MB
The purpose of the docker extension is to simplify the onboarding or demoing the product to prospects, if the user needs to download 0.5G is not the way to go.
Vivo image is also around 571MB.
Let's fix this.
Fonts type and size are not following the design proposed, this is how it looks today:
Please fix the look & feel according to the Figma designs:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.