spegel-org / spegel Goto Github PK
View Code? Open in Web Editor NEWStateless cluster local OCI registry mirror.
License: MIT License
Stateless cluster local OCI registry mirror.
License: MIT License
Spegel does currently not automatically deal with registry outages, if the registry which has an outage is the same one that the Spegel image is pulled from. The mirror configuration is written by Spegel, and for that to happen the image has to be present on the node.
This becomes a catch 22 scenario where scaling a cluster will be impossible because new nodes will not be able to pull images from the registry with the outage.
We need a solution to protect from these types of situations.
current configmap name for leader election is kube-system-leader-election
for kube-system
namespace
providing helm-chart name as prefix would be more meaningful
Knowing if Spegel is working or not may be a bit difficult as images fall back to the original registry. For debugging purposes it may be nice to after an image is pulled inspect if it was fetched through Spegel or not. One option could be to annotate or label the image to indicate when it is fetched with Spegel.
Good day!
Is it possible somehow configure spegel on AWS Bottlerocket nodes? If not - are you planning add this support?
Thanks!
Currently the Pod IP is used in the peer key for routing. This means that every time that the Spegel instance is restarted the IP changes and the peer identifier changes. This seems wasteful as the Pod is going to run on the same node with the same images advertised. We should explore what the impact is of the current solutions and if there are any benefits of changing to using the Node IP as the peer key.
Currently Spegel only supports Containerd. CRI-O is another popular container runtime. This should also be supported as an alternative backened. It has the benefit of also forcing the code architecture to not be centered around Containerd, enabling other runtimes to also be implemented.
LMK if there is a better way to ask this, but my team was wondering how images are stored on nodes in the cluster. If you pull an image to a node, and its not in the spegel cache in the cluster, does that image + all associated layers stay on that one node, or does spegel try to distribute the layers throughout all nodes in the cluster?
Wondering because our application runs as 1 pod / node a lot of times, and was wondering if all the layers of that pod's containers are kept on that node. We have ML compute pods + nodes coming up and down a lot so if the images all stay on the same node, then we would have a lot of cold start situations.
Thanks!
topologyKeys on service chart template, should be optional, since not supported by all potential configurations
Hi team,
We are using Spegel on top of Scaleway Kapsule cluster. Behavior was great until we tried to upgrade from 0.0.9 to 0.0.11 or 0.0.12.
The upgrade failed, getting this error in registry container:
Search...
{"level":"error","ts":1694005879.5690522,"caller":"build/main.go:72","msg":"","error":"Containerd registry config path needs to be set for mirror configuration to take effect"
Since containerd settings can be managed by end users, any idea how to handle this ? Any possibilité to go back the the 0.0.9 behavior which was not facing such issue ?
Thansk for your support
Hi,
Spegel looks great! But I'm on a test cluster and have noticed that it's not starting up quickly enough on a fresh node to get any benefit from it - by the time Spegel has started and registered the mirrors the pods that I want to pull from Spegel have already started pulling from the default registry
The startup process seems to be blocking on my CNI driver (Calico) starting up:
Warning NetworkNotReady 16s (x12 over 38s) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
This more or less happens in parallel to my pods starting up, so by the time Spegel is running my pods have pulled from the default registry and bypassed Spegel.
My workaround so far is to enable hostNetwork: true
in the Spegel daemonset spec - this stops it having a dependency on Calico, and it starts more or less immediately (before my pods at least)
I assume not everyone uses Calico so changing Spegel to use hostNetwork by default may not be desired? Could a flag be added to the helm chart to enable this instead? Bear in mind that service.registry.port
needs setting to 30020
for this to work, but otherwise it does seem to work in some limited testing..
imagepull second time from another node in same cluster should be fast but it took same time as of downloading from remote repo, looks like caching didnt work
here are the logs from spegel pod for xxx.yyy.io/app-image-k8:dev_123
image (masked repo and image details)
any pointers to look into what can be the issue?
{"level":"error","ts":1696247240.4278097,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2/app-image-k8/manifests/dev_123","status":404,"method":"HEAD","latency":5.000973923,"ip":"10.14.130.153","handler":"mirror","error":"could not resolve mirror for key: xxx.yyy.io/app-image-k8:dev_123","stacktrace":"github.com/xenitab/pkg/gin.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2936\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1995"}
{"level":"error","ts":1696247731.0288842,"caller":"registry/registry.go:211","msg":"mirror failed attempting next","error":"expected mirror to respond with 200 OK but received: 500 Internal Server Error","stacktrace":"github.com/xenitab/spegel/internal/registry.(*Registry).handleMirror.func2\n\t/build/internal/registry/registry.go:211\nnet/http/httputil.(*ReverseProxy).modifyResponse\n\t/usr/local/go/src/net/http/httputil/reverseproxy.go:324\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/local/go/src/net/http/httputil/reverseproxy.go:490\ngithub.com/xenitab/spegel/internal/registry.(*Registry).handleMirror\n\t/build/internal/registry/registry.go:217\ngithub.com/xenitab/spegel/internal/registry.(*Registry).registryHandler\n\t/build/internal/registry/registry.go:137\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/xenitab/spegel/internal/registry.(*Registry).metricsHandler\n\t/build/internal/registry/registry.go:271\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/slok/go-http-metrics/middleware/gin.Handler.func1.1\n\t/go/pkg/mod/github.com/slok/[email protected]/middleware/gin/gin.go:17\ngithub.com/slok/go-http-metrics/middleware.Middleware.Measure\n\t/go/pkg/mod/github.com/slok/[email protected]/middleware/middleware.go:117\ngithub.com/slok/go-http-metrics/middleware/gin.Handler.func1\n\t/go/pkg/mod/github.com/slok/[email protected]/middleware/gin/gin.go:16\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/xenitab/pkg/gin.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:28\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2936\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1995"}
A Spegel instance with an empty routing table has little to no use, as it will never be able to discover any other peers. For this reason it would be good for the readiness probe to only give a 200 response only when the routing table is not empty.
This looks like a great solution to a common problem! Do you think it would be worthwhile to aim for an upstream contribution to containerd?
Currently Spegel requires access to the Containerd sock to function properly. This mean that it is basically root on the node. In reality Spegel only needs read only access to all of the layers and tags. The issue is that the containerd client is required to do tag resolution.
One option would be to switch over to using the CRI API. One drawback currently is that it does not have an event subscription service which we currently use with the containerd client.
If tags could be listed without the use of the containerd client the blobs directory could be mounted read only to the container and new layers could be detected with the help of a file watcher.
With the new feature enabling Spegel to fallback on mirroring from another node, we end up with an interesting situation. Currently every instance will filter out itself when mirroring requests. This is not useful if the request is already coming from another node. In these situation we should actually allow proxying to the same instance.
One solution is to allow mirroring of all requests. The other is to add a client ip check or a header to the request to allow proxying to the same instance.
Hi there,
I have a k3s cluster, where I want to use spegel. I install spegel as described in README using helm upgrade --create-namespace --namespace spegel --install --version v0.0.11 spegel oci://ghcr.io/xenitab/helm-charts/spegel
However, the pod gets into CrashLoopBackOff
with the following error, when I check the logs:
Defaulted container "registry" out of: registry, configuration (init)
{"level":"error","ts":1695750332.7763488,"caller":"build/main.go:72","msg":"","error":"rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService","stacktrace":"main.main\n\t/build/main.go:72\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
How can I solve this problem? I would appreciate any help.
Thanks a lot!
Hello,
Thanks for this awesome project :)
We started to PoC spegel in our environments, but discovered that some of our devs were relying on the infamous combo
imagePullPolicy: Always
image: myimage:latest
it seems that in that case, the pod still gets the old image.
I assume that is because the manifest is also cached and not pulled from the registry every time ?
Can/could we optionally change the behavior and always request the manifests to the registry ?
not sure if that is the right way, but, allowing to switch from :
capabilities = ["pull", "resolve"]
to
capabilities = ["pull"]
?
https://github.com/XenitAB/spegel/blob/v0.0.9/internal/oci/containerd.go#L278
Thanks !
Hi there,
Spegel sounds like a great project and we are interested in using it to avoid Docker Hub's rate limiting on Azure AKS which comes by the end of September.
During regular operations, we should not run into limits. However, we fear that this may happen in case of a disaster recovery. E.g. the whole cluster burns down and everything needs to be set up from scratch asap.
This would be another great use case for Spegel. However, if I got it right, Spegel does not provide a persistence layer. Is it possible to add persistence to Spegel such that a recovery via k8s level backup tools like Velero would be possible?
I hope that this might be a quick win simply by providing parts of the file system via a PV? Or is it all in-memory and there is no easy way to persist the local images?
All the best! Continue the great work.
Niklas
Both the image and Helm chart should be signed during release.
Hi! I just installed spegel to my cluster, and some of the pods in the daemonset get this cryptic error and crash loop
Defaulted container "registry" out of: registry, configuration (init)
{"level":"info","ts":1688689161.8643885,"logger":"p2p","caller":"routing/p2p.go:63","msg":"starting p2p router","id":"/ip4/:)/tcp/5001/p2p/12D3KooWD6pXCqqMcVeBXtoNQQJRX2fTKfj5Dc98WCs7yoTQLGcy"}
{"level":"info","ts":1688689161.8646646,"caller":"build/main.go:160","msg":"running registry","addr":":5000"}
I0707 00:19:21.865702 1 leaderelection.go:245] attempting to acquire leader lease spegel/spegel-leader-election...
{"level":"info","ts":1688689161.8717268,"caller":"state/state.go:44","msg":"running scheduled image state update"}
{"level":"error","ts":1688689162.1034093,"caller":"build/main.go:68","msg":"","error":"failed to update all images: failed to walk image manifests: content digest sha256:190d5c74da93b56c3fcd4e08603834c9b20a5149214a369a5705b37ed62c2c72: not found","stacktrace":"main.main\n\t/build/main.go:68\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
Any idea what this is about? Thanks!
Currently the mirror handler will forward the HTTP request to the first node it discovers. If the request to the node fails Containerd will fallback to the original registry. A better solution would be to retry the request to a different node which is discovered later.
There is a new project https://github.com/plprobelab/go-kademlia that aims to build a standardized generic Kademlia implementation. It looks like libp2p2 kad will refactor to use the new library according to libp2p/go-libp2p-kad-dht#856. I am currently not sure but I think we could just switch to using go-kademlia if it offers all of the features that we require. This may simplify some of the routing implementations as the new API is simpler.
It is currently possible to run mixed Kubernetes clusters with nodes running WASM containers using krustlet. This is for example supported in AKS. The runwasi project has however made the process a lot simpler by enabling WASM workloads to run in containerd.
As there is already support for packaging WASM in an OCI format support for WASM would be trivial. In the case of runwasi I think it would just work out of the box as it would be Containerd who would to the actual image pulling.
This feature is currently in the early days of development so it may take a while until Spegel supports WASM containers.
Currently it does not seem like there is an easy way to support GKE. When creating a cluster the Containerd config looks like the following.
version = 2
required_plugins = ["io.containerd.grpc.v1.cri"]
# Kubernetes doesn't use containerd restart manager.
disabled_plugins = ["io.containerd.internal.v1.restart"]
oom_score = -999
[debug]
level = "info"
[grpc]
gid = 412
[plugins."io.containerd.grpc.v1.cri"]
stream_server_address = "127.0.0.1"
max_container_log_line_size = 262144
sandbox_image = "gke.gcr.io/pause:3.6@sha256:10008c36b4611b44db1229451675d5d7d86c7ddf4ef00f883d806a01547203f6"
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/home/kubernetes/bin"
conf_dir = "/etc/cni/net.d"
conf_template = "/home/containerd/cni.template"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://mirror.gcr.io","https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = false
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
The issue here is that it does not configure the prerequisite config_path
. Without it Containerd will not look for any mirror configuration.
One interesting point to notice is that GKE is using the old mirror syntax to configure a specific mirror for Docker Hub. This is most likely a part of a feature they offer as they will cache Docker Hub images to avoid rate limiting. It would be better if they used the newer mirror configuration method. As even if Spegel would override this, it could be configured to when run in a GKE cluster keep the last mirror fallback for Docker Hub.
Hello,
I try to install spegel using helm upgrade --create-namespace --namespace spegel --install --version v0.0.12 spegel oci://ghcr.io/xenitab/helm-charts/spegel
however I get the following error. I tried on 2 different machines. Both giving me the same error. Am I missing something?
Error: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Auser%2Fimage%3Apull&scope=repository%3Axenitab%2Fhelm-charts%2Fspegel%3Apull&service=ghcr.io: 403 Forbidden
Spegel makes persistent changes to the hosts filesystem. This is required to setup the mirror configuration. The mirror configuration should be present on the node for as long as Spegel is being used. A good practice however would be to revert the mirror configuration back to its pre Spegel state if Spegel ever was removed. This is especially useful for those who are just evaluating Spegel but chose not to use it. Keeping the mirror configuration will not break the cluster as it would always fallback to the original registry, and the configuration would be removed when old nodes are removed from the cluster.
The old solution was to add an option to remove the configuration files during shutdown. While a good option it wasn't optimal as it is beneficial to keep the configuration during versions updates of Spegel, as it would fallback to other Spegel instances in the cluster.
An alternative solution is to create a Helm uninstall hook which would run and remove the configuration from all nodes when the Helm chart is removed. This has the benefit of cleaning up stateful changes and returning the node configuration to the state it existed in before Spegel.
I'm getting a mixture of 404 and 500 when trying to pull image. It only happens for my own image and I think I can see some successful image pulls.
Logs from one spegel pod:
{"level":"info","ts":1699687666.3606877,"logger":"p2p","caller":"routing/p2p.go:59","msg":"starting p2p router","id":"/ip4/10.22.130.68/tcp/5001/p2p/12D3KooWKxpkvZfxSMMYutnYGpfsdRJP4cZJ46NGZStL8N8D6hie"}
{"level":"info","ts":1699687666.3609054,"caller":"build/main.go:169","msg":"running registry","addr":":5000"}
I1111 07:27:46.361268 1 leaderelection.go:245] attempting to acquire leader lease spegel/spegel-leader-election...
{"level":"info","ts":1699687666.3617876,"caller":"state/state.go:42","msg":"running scheduled image state update"}
{"level":"error","ts":1699687671.4085758,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/manifests/sha256:d138dcd6666c3050e31fa2e30963aabd33aa23dba34bce0904cb90429622b5c5","status":500,"method":"GET","latency":3.177211296,"ip":"10.22.136.66","handler":"mirror","error":"mirror resolve retries exhausted for key: sha256:d138dcd6666c3050e31fa2e30963aabd33aa23dba34bce0904cb90429622b5c5","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"info","ts":1699687671.4949071,"caller":"state/state.go:49","msg":"received image event","image":"gcr.io/datadoghq/agent:7.46.0@sha256:4e5f6127e43348b78fea1da5393dd2ff038dff72b78b42de5b11c7600412830a"}
{"level":"info","ts":1699687671.5828915,"caller":"state/state.go:49","msg":"received image event","image":"gcr.io/datadoghq/agent:7.46.0@sha256:4e5f6127e43348b78fea1da5393dd2ff038dff72b78b42de5b11c7600412830a"}
{"level":"info","ts":1699687671.6040404,"caller":"state/state.go:49","msg":"received image event","image":"gcr.io/datadoghq/agent"}
{"level":"error","ts":1699687683.8613865,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:6967698a904318aed4c1547be609141f22d40a9588e0787dcb801550a1bdca23","status":404,"method":"GET","latency":5.001035143,"ip":"10.22.136.66","handler":"mirror","error":"request closed for key: sha256:6967698a904318aed4c1547be609141f22d40a9588e0787dcb801550a1bdca23","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1699687690.478658,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:dee98d4b974d4ac8e95c28a17e5dddc4e818169b6a8a9542c25eafd4fc2b1a58","status":404,"method":"GET","latency":5.00098084,"ip":"10.22.136.66","handler":"mirror","error":"request closed for key: sha256:dee98d4b974d4ac8e95c28a17e5dddc4e818169b6a8a9542c25eafd4fc2b1a58","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1699687690.4786735,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:d05bbff9997e08bbd4b8ddf12f496fd312eb5f691d236dd8d3e29b1ac8b91b9d","status":500,"method":"GET","latency":4.992058285,"ip":"10.22.136.66","handler":"mirror","error":"mirror resolve retries exhausted for key: sha256:d05bbff9997e08bbd4b8ddf12f496fd312eb5f691d236dd8d3e29b1ac8b91b9d","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1699687690.4786735,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:e7808094cb5d351220c95581e5edd2fdd6f0cbc53154400bf0842a171127e238","status":500,"method":"GET","latency":4.996645105,"ip":"10.22.136.66","handler":"mirror","error":"mirror resolve retries exhausted for key: sha256:e7808094cb5d351220c95581e5edd2fdd6f0cbc53154400bf0842a171127e238","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1699687697.153541,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:1d98726e0d51e9317902923837e48db6ba3080104d83cab6e95beb91c744bb2a","status":500,"method":"GET","latency":1.386007186,"ip":"10.22.136.66","handler":"mirror","error":"mirror resolve retries exhausted for key: sha256:1d98726e0d51e9317902923837e48db6ba3080104d83cab6e95beb91c744bb2a","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1699687697.153541,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:077b5992cbfc562fac5ca434250dfc92acb83b59d7de408c2495a1a873186e55","status":404,"method":"GET","latency":5.000501557,"ip":"10.22.136.66","handler":"mirror","error":"request closed for key: sha256:077b5992cbfc562fac5ca434250dfc92acb83b59d7de408c2495a1a873186e55","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
{"level":"error","ts":1699687701.3775723,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2//REDACTED/server/blobs/sha256:64d6ec39685a5d38dc24f9f5987cd01d6bc84d7a0dc21398f977400549cb22eb","status":500,"method":"GET","latency":3.272260102,"ip":"10.22.136.66","handler":"mirror","error":"mirror resolve retries exhausted for key: sha256:64d6ec39685a5d38dc24f9f5987cd01d6bc84d7a0dc21398f977400549cb22eb","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
Events from my pod:
Normal Pulling 2m6s kubelet Pulling image "us-docker.pkg.dev/agones-images/release/agones-sdk:1.34.0"
Normal Pulled 118s kubelet Successfully pulled image "us-docker.pkg.dev/agones-images/release/agones-sdk:1.34.0" in 7.60820591s (7.608215832s including waiting)
Normal Created 118s kubelet Created container agones-gameserver-sidecar
Normal Started 118s kubelet Started container agones-gameserver-sidecar
Normal Pulling 118s kubelet Pulling image "us-docker.pkg.dev/REDACTED/server:b25e9d50444856962c9861b6e0842f289490f014"
Normal Pulled 92s kubelet Successfully pulled image "us-docker.pkg.dev/REDACTED/server:b25e9d50444856962c9861b6e0842f289490f014" in 26.130419751s (26.130431392s including waiting)
Normal Created 92s kubelet Created container days
Normal Started 91s kubelet Started container days
Environment is:
AWS EKS
Karpenter
I installed spegel 0.0.9
from Helm. It started on most of the nodes, but on some it failed:
{"level":"error","ts":1691505526.569151,"caller":"build/main.go:68","msg":"","error":"failed to update all images: failed to walk image manifests: unexpected media type for digest: sha256:17e484fedc7ecaa8c83404aa85f0b615743ce723c05d3735436426f12d9e9c77","stacktrace":"main.main\n\t/build/main.go:68\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
I don't know how to troubleshoot this further.
Using cosign to sign OCI artifacts is very useful, especially when using registry mirrors, as a way to verify that the artifact received comes from a trusted source. Currently Spegel has not been verified with cosign. After having a quick look it seems like it should be an issue, but it should be verified to work and tested.
Currently the image tag is used in the Helm chart. This is not good practice and it is better to use the image digest instead. The release GitHub action should be updated to include the digest.
Hello, trying out spegel and facing these errors across the map:
{"level":"info","ts":1699886843.334517,"caller":"registry/registry.go:174","msg":"handling mirror request from external node","path":"/v2/p2m-qa/silo-p2m-qa-subtemplate-silo/manifests/sha256:2520965b7b02cd4c5090f635622c92c4dfd2446a2d9332106ae140a1e1eb7bc4","ip":"100.96.35.1"}
{"level":"error","ts":1699886843.3367004,"caller":"[email protected]/logger.go:62","msg":"","path":"/v2/p2m-qa/silo-p2m-qa-subtemplate-silo/manifests/sha256:2520965b7b02cd4c5090f635622c92c4dfd2446a2d9332106ae140a1e1eb7bc4","status":500,"method":"GET","latency":0.002227242,"ip":"100.96.35.1","handler":"mirror","error":"mirror resolve retries exhausted for key: sha256:2520965b7b02cd4c5090f635622c92c4dfd2446a2d9332106ae140a1e1eb7bc4","stacktrace":"github.com/xenitab/pkg/gin.NewEngine.Logger.func1\n\t/go/pkg/mod/github.com/xenitab/pkg/[email protected]/logger.go:62\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2009"}
Containers:
p2m-qa-subtemplate-silo:
Container ID: containerd://09110087199b09a31ce246c2be47d4046116f9bc64d927b4262036f56028e20b
Image: privateharborrepo.com/p2m-qa/silo-p2m-qa-subtemplate-silo:70
Image ID: privateharborrepo.com/p2m-qa/silo-p2m-qa-subtemplate-silo@sha256:2520965b7b02cd4c5090f635622c92c4dfd2446a2d9332106ae140a1e1eb7bc4
Containerd config.toml:
## template: jinja
# Use config version 2 to enable new configuration fields.
# Config file is parsed as version 1 by default.
version = 2
imports = ["/etc/containerd/conf.d/*.toml"]
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "projects.registry.vmware.com/tkg/pause:3.6"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
[plugins."io.containerd.grpc.v1.cri".containerd]
discard_unpacked_layers = false
spegel-values.yaml:
...
spegel:
# -- Registries for which mirror configuration will be created.
registries:
- https://docker.io
- https://ghcr.io
- https://quay.io
- https://mcr.microsoft.com
- https://public.ecr.aws
- https://gcr.io
- https://registry.k8s.io
- https://k8s.gcr.io
- https://lscr.io
- https://privateharborrepo.com
...
Is there something obvious that we're missing here? It seems to work just fine at other times, although I am still trying to figure out what defines "fine" :)
self hosted Harbor Version v2.7.1-6015b3ef - changed the url for obvious reasons
nodes: v1.23.10+vmware.1 Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.6.6
Changes were made to defaulting to https in Containerd 1.7.7 which we need to verify is not an issue.
Hi,
Is it possible to have private images (the ones that can only be pulled with a password) to not be cached? Seems like spegel is making it possible to pull the private images without any password from cache.
Somehow all spegel's pods in my test cluster have linearly increasing memory usage that already killed one of pods twice in 1 day due to OOM (limit is 320Mi).
How can I help to debug such issue?
SOCI Snapshotter is an interesting project which could reduce Pod startup time. At a glance it seems like Spegel should be able to support SOCI ontop of running a normal registry. The main question is if there are users who want this.
Currently a reverse proxy request is only done once. This means that a failed request will fallback to going out to the source registry to complete the request. It would be good to retry x number of times to different nodes which have the requested resource.
With the old service topology feature in Kubernetes it was possible to create a Service which would "prefer local". What that means is that a NodePort service could be created which when called upon would always route to the node local Pod if possible. If that were to fail it would route to a Pod in the cluster. This was a great feature that was sadly left out of the new solution as stated here.
https://discuss.kubernetes.io/t/deprecated-topologykeys-alternative-for-node-locality/18184
This feature may be re introduced in the future, and when that happens we should use it instead of having to mirror registry configurations.
The pods crashloop with configuration exec ./spegel: exec format error
, which is usually a sign that they have been built with the wrong arch in mind.
docker inspect
shows that it's built with arm in mind though:
"Architecture": "arm64",
"Os": "linux",
Reverting to 0.0.14 immediately fixes the issue.
Broken digest: sha256:f7e32e17ee8b9a64efdc7958f6fa1099cfeaaea3854841b3d47bcb7dba31fcbc
Working digest: sha256:18d64359225e16155ed2a138fb88bba163353c8fb79913197148c4ee9c73f600
Currently it is difficult to measure performance differences between using Spegel and not using Spegel.
With the merging of containerd/containerd/pull/7313 it will be easier to measure pull speed per registry. This feature will be released in containerd 1.7. When the version is released the performance measuring should be documented for end users to compare the end result.
spegel pods are in crashlookbackoff with below error
Defaulted container "registry" out of: registry, configuration (init)
{"level":"error","ts":1696142954.9523215,"caller":"build/main.go:72","msg":"","error":"Containerd registry config path is /etc/containerd/certs.d:/etc/docker/certs.d but needs to be /etc/containerd/certs.d for mirror configuration to take effect","stacktrace":"main.main\n\t/build/main.go:72\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
helm chart version used to install
helm upgrade --install --namespace kube-system --version v0.0.11 spegel oci://ghcr.io/xenitab/helm-charts/spegel
image: ghcr.io/xenitab/spegel@sha256:8c73772fbf07b8b0458a26dc991b48f06b407082f3f554a0ee3bffd1ff89bf91
containerd config file:
# cat /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
[grpc]
address = "/run/containerd/containerd.sock"
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
Should the appVersion get updated to the latest version for every release that is cut? Right now it is still at 0.0.1 and the image is not working with the later helm chart releases. I am manually changing the tag on my end, but would be nice if it just worked without having to do that.
Currently Spegel does not work with Bottlerocket when run with EKS.
https://github.com/bottlerocket-os/bottlerocket/blob/develop/README.md
This needs to be explored further to see if it is possible to support.
Is it possible to disable libp2p_* prefix in /metrics response? It have too much not-so-helpfull data for us, so it would be handy to enable it only for debugging purposes.
And thank you for great project!
After carefully reviewing the following containerd-related errors, I deleted the configurations related to "mirror" in
/etc/rancher/k3s/registries.yaml
and/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
. Now,spegel
is working properly.time="2023-11-20T23:28:55.907708354+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `mirrors` cannot be set when `config_path` is provided"
That's a bit unfortunate since the docs state:
Spegel does not aim to replace projects like Harbor or Zot but instead complements them.
I was hoping to use spegel in my cluster and zot as a pull thru cache deployed elsewhere. For what it's worth this is my current containerd mirrors:
mirrors:
docker.io:
endpoint:
- https://zot.domain.tld/v2/docker.io
ghcr.io:
endpoint:
- https://zot.domain.tld/v2/ghcr.io
quay.io:
endpoint:
- https://zot.domain.tld/v2/quay.io
gcr.io:
endpoint:
- https://zot.domain.tld/v2/gcr.io
registry.k8s.io:
endpoint:
- https://zot.domain.tld/v2/registry.k8s.io
public.ecr.aws:
endpoint:
- https://zot.domain.tld/v2/public.ecr.aws
I don't see a way to have spegel take over this responsibility it seems like you either have spegel or a pull thru cache, maybe this can be a feature request?
Originally posted by @onedr0p in #212 (comment)
Hi
I tried Spegel recently and found that when a node pulls an image (etcd in this case), the Spegel instance on the node detects it as an external request:
{"level":"info","ts":1697181744.1945643,"caller":"registry/registry.go:174","msg":"handling mirror request from external node","path":"/v2/etcd-development/etcd/blobs/sha256:47ba7aff063ffd3883d25edc17bc0a92a4b76d48bcc89e8a21a149db234576bd","ip":"10.65.10.213"}
The IP (10.65.10.213
) is the one assigned to the cilium_host
interface.
I'm not sure how to fix this. Is someone here using Spegel with Cilium?
serviceMonitor object needs to have support to add extra labels for prometheus to discover target, current helm chart doesn't support that
I observe constant timeouts while containerd trying to access 127.0.0.1:30021 local mirror:
root@myhostname:~# journalctl -r -u containerd |grep 30021 |head
Nov 01 00:29:36 myhostname containerd[2413588]: time="2023-11-01T00:29:36.775166395Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/rtl/marketing-promocode/be-1301/manifests/49ad4e18?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:33 myhostname containerd[2413588]: time="2023-11-01T00:29:33.774039526Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/mkk/loan/feature-mkk-3870/manifests/bf6460d0?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:26 myhostname containerd[2413588]: time="2023-11-01T00:29:26.775053471Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/opendatacollecting/eparser/odc-2259/manifests/ee2d26a6?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:24 myhostname containerd[2413588]: time="2023-11-01T00:29:24.843621330Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/inventory/api/supply-return-readmodel/inv-2632/manifests/ddd7f161?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:29:05 myhostname containerd[2413588]: time="2023-11-01T00:29:05.787353142Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/wms/svc/logistics-megasort-facade/wms-41662/manifests/ca271126?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:28:59 myhostname containerd[2413588]: time="2023-11-01T00:28:59.053739919Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/eea/platform/exteca.platform.orchestrator.sublots/master/manifests/9c2bfe17?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:28:26 myhostname containerd[2413588]: time="2023-11-01T00:28:26.772423981Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/eea/platform/exteca.platform.orchestrator.sublots/master/manifests/9c2bfe17?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:28:01 myhostname containerd[2413588]: time="2023-11-01T00:28:01.977061019Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/lsplt/trip-container-service/trip-container-business-layer/master/manifests/f07adc9d?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:27:49 myhostname containerd[2413588]: time="2023-11-01T00:27:49.779983205Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/pricing/services/pricing-comments/f-ppric-3865/manifests/ed7e4e82?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Nov 01 00:27:07 myhostname containerd[2413588]: time="2023-11-01T00:27:07.147305757Z" level=info msg="trying next host" error="failed to do request: Head \"http://127.0.0.1:30021/v2/kdp/api/onec-gateway-api/master/manifests/c495844d?ns=gitlab-registry.mycomp.ru\": dial tcp 127.0.0.1:30021: i/o timeout" host="127.0.0.1:30021"
Seems like access from hostnetwork to 127.0.0.1:nodeport working only with proxy-mode=iptables (see: kubernetes/kubernetes#111840). As result, deploying spegel in cluster with ipvs make image pull times much worse (1s->60s).
How can we handle this issue without falling back to iptables?
It is pretty difficult to determine if Spegel is working or not. There are many things that can go wrong and image pulling may still work as it falls back to the original registry. A good solution would be to extend the OCI interface to add a validate function. This function would somehow validate that the mirror configuration added works properly. I am not sure what the best way to do this is. We would want to validate that the configuration is properly picked up and the request is reaching Spegel. I have listed some ideas for how this could be done.
Hey just thought I'd file a bug report:
Cluster: v1.27.4-eks-2d98532
Nodes: v1.27.3-eks-a5565ad
Log
registry {"level":"error","ts":1693316449.014054,"caller":"build/main.go:71","msg":"","error":"Containerd registry config path is /etc/containerd/certs.d:/etc/docker/certs.d but needs to be /etc/containerd/certs.d for mirror configuration to take effect","stacktrace":"main.main\n\t/build/main.go:71\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
configuration {"level":"info","ts":1693316396.3836858,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://docker.io","path":"/etc/containerd/certs.d/docker.io/hosts.toml"}
configuration {"level":"info","ts":1693316396.3838172,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://ghcr.io","path":"/etc/containerd/certs.d/ghcr.io/hosts.toml"}
configuration {"level":"info","ts":1693316396.3839626,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://quay.io","path":"/etc/containerd/certs.d/quay.io/hosts.toml"}
configuration {"level":"info","ts":1693316396.3840818,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://mcr.microsoft.com","path":"/etc/containerd/certs.d/mcr.microsoft.com/hosts.toml"}
configuration {"level":"info","ts":1693316396.3842103,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://public.ecr.aws","path":"/etc/containerd/certs.d/public.ecr.aws/hosts.toml"}
configuration {"level":"info","ts":1693316396.3843098,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://gcr.io","path":"/etc/containerd/certs.d/gcr.io/hosts.toml"}
configuration {"level":"info","ts":1693316396.3844151,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://registry.k8s.io","path":"/etc/containerd/certs.d/registry.k8s.io/hosts.toml"}
configuration {"level":"info","ts":1693316396.3845265,"caller":"oci/containerd.go:286","msg":"added containerd mirror configuration","registry":"https://k8s.gcr.io","path":"/etc/containerd/certs.d/k8s.gcr.io/hosts.toml"}
configuration {"level":"info","ts":1693316396.3846445,"caller":"build/main.go:74","msg":"gracefully shutdown"}
Flux Config to Deploy
---
apiVersion: v1
kind: Namespace
metadata:
name: spegel
labels:
toolkit.fluxcd.io/tenant: sre-team
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: spegel
namespace: spegel
spec:
type: "oci"
interval: 5m0s
url: oci://ghcr.io/xenitab/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: spegel
namespace: spegel
spec:
interval: 1m
chart:
spec:
chart: spegel
version: "v0.0.11"
interval: 5m
sourceRef:
kind: HelmRepository
name: spegel
Deploy is failing with the same error log as above on both managed node group nodes and karpenter provisioned spot instances.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.