uhthomas / automata Goto Github PK

View Code? Open in Web Editor NEW

28.0 3.0 2.0 6.31 MB

Monorepo for Starjunk and subsidiaries

Starlark 3.41% Go 0.01% CUE 96.34% HCL 0.25% Shell 0.01%

gitops kubernetes cue cuelang k8s bazel jellyfin

automata's People

Contributors

Stargazers

Watchers

Forkers

s4rd1nh4 telecomfreak

automata's Issues

chore: replace vault with bitwarden

They're both secret stores, and one is much easier to manage.

promtail?

Desire is currently using fluent-bit, which has been working fine admittedly. The reason for considering change is that it's a Helm chart, and needs to be rewritten in CUE. I want to understand what the best logging agent is in order to use my time effectively.

I had considered vector as it's extremely fast, but it doesn't seem to play nice with loki. Loki ships promtail which supposedly comes with lots of things for free, but I would like to understand the performance implications.

feat: monitor etcd

It looks like etcd does not run in-cluster with Talos, and therefore is not easy to monitor.

Casper: Improve metrics collection

Prometheus isn't fully configured yet, and annotations need to be added to various services for scraping.

chore(k8s/unwind/grafana): use declarative config

Grafana is currently configured through the UI, with lots of dashboards manually added. It would be nice to move this configuration into the Kubernetes manifests and load dashboard directly from grafana.net.

feat: victoria logs

https://docs.victoriametrics.com/VictoriaLogs/QuickStart.html

feat(kube-system): monitor kube-controller-manager and kube-scheduler

They both listen on localhost only by default, which means they cannot be connected to from within the cluster. The service scrape configs were removed in 8e1e45e. The nodes will need to be reconfigured to listen on 0.0.0.0 and then these service scrapes can be added back.

dev/kipp: Large/long uploads result in 500 or 502

When an upload is large (100MB+), or takes a while, one of two things will happen:

NGINX returns a 502.
Kipp returns a 500 because it got a 502 from S3 (Linode).

I've ruled out that Linkerd is not the cause by removing the sidecar injection. Not sure if it's a timeout in ingress-nginx, or something greater.

Here's some example responses:

upload: MultipartUpload: upload multipart failed
    upload id: 2~uS5fEhLYQ2me9YEDAKu8KVHG7yP3wia
caused by: SignatureDoesNotMatch: 
    status code: 403, request id: tx0000000000000019011e5-005ed2e4fc-5041f0-default, host id:

upload: MultipartUpload: upload multipart failed
    upload id: 2~C-GeAYog6R-kJNNF66uJXJV7i796gt2
caused by: SerializationError: failed to unmarshal error message
    status code: 502, request id: , host id: 
caused by: UnmarshalError: failed to unmarshal error message
    00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 32  20 42 61 64 20 47 61 74  |itle>502 Bad Gat|
00000020  65 77 61 79 3c 2f 74 69  74 6c 65 3e 3c 2f 68 65  |eway</title></he|
00000030  61 64 3e 0d 0a 3c 62 6f  64 79 3e 0d 0a 3c 63 65  |ad>..<body>..<ce|
00000040  6e 74 65 72 3e 3c 68 31  3e 35 30 32 20 42 61 64  |nter><h1>502 Bad|
00000050  20 47 61 74 65 77 61 79  3c 2f 68 31 3e 3c 2f 63  | Gateway</h1></c|
00000060  65 6e 74 65 72 3e 0d 0a  3c 68 72 3e 3c 63 65 6e  |enter>..<hr><cen|
00000070  74 65 72 3e 6f 70 65 6e  72 65 73 74 79 3c 2f 63  |ter>openresty</c|
00000080  65 6e 74 65 72 3e 0d 0a  3c 2f 62 6f 64 79 3e 0d  |enter>..</body>.|
00000090  0a 3c 2f 68 74 6d 6c 3e  0d 0a                    |.</html>..|

caused by: expected element type <Error> but have <html>

rules_k8s?

Rather than using flux to automate deployments, it might be far better to use rules_k8s to hermetically build and deploy manifests.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Regex Manager contains disallowed fields: extractVersion

vault?

feat: smart monitoring

Something like https://github.com/AnalogJ/scrutiny could be useful. It needs a daemonset for collection, a deployment for the web ui and influx db to work correctly.

casper: Add BuildBarn

Currently the old 6f.io server is running a Bazel remote cache server, which should be replaced by remote cache and execution (see BuildBarn).

dev/kipp: Ingress annotations aren't being applied

The timeouts aren't set, and neither is the max body size.

➜  ~ kubectl -n ingress-nginx exec ingress-nginx-controller-77b69ddf57-g9dcc -- cat nginx.conf | grep client_max_body_size
Defaulting container name to controller.
Use 'kubectl describe pod/ingress-nginx-controller-77b69ddf57-g9dcc -n ingress-nginx' to see all of the containers in this pod.
			client_max_body_size                    1m;
			client_max_body_size        1m;
			client_max_body_size                    1m;
			client_max_body_size                    1m;
			client_max_body_size                    1m;
			client_max_body_size                    21m;

casper: Add buildkite runners

Cleanup Terraform plans

The current Terraform plans were quite ad-hoc and messy.

Let's organize main.tf into a tree of modules and clean everything up. It won't scale otherwise.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

WARN: Error updating branch: update failure
WARN: Package lookup failures

Errored

These updates encountered an error and will be retried. Click on a checkbox below to force a retry now.

chore(deps): pin dependencies (actions/cache, actions/checkout, actions/download-artifact, actions/github-script, actions/setup-node, actions/upload-artifact, bazel_gazelle, bazelbuild/setup-bazelisk, com_github_bazelbuild_buildtools, docker/login-action, io_bazel_rules_docker, io_bazel_rules_go, io_bazel_rules_k8s, rules_python, tailscale/github-action)

Warning

Renovate failed to look up the following dependencies: Could not determine new digest for update (go package github.com/crunchydata/postgres-operator), Could not determine new digest for update (docker package ghcr.io/prymitive/karma).

Files affected: go.mod, k8s/amour/karma/list.cue

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

chore(deps): update module github.com/libopenstorage/external-storage to v5

Detected dependencies

bazel

container_deps.bzl

io_docker_index_library_debian_bookworm_slim bookworm-slim@sha256:d6a343a9b7faf367bd975cadb5c9af51874a8ecf1a2b2baa96877d578ac96722

io_gcr_distroless_base_debian11 latest@sha256:d08c10f03c27271160993f294e0eb120af71217d0cf4587c484cc5b7cb3fe5ee

deps.bzl

bazel_gazelle v0.35.0

com_github_bazelbuild_buildtools v6.4.0

com_github_tnarg_rules_cue a687771e1b85f7552f9f128f5231fe0e27ec97df

rules_proto f9b0b880d1e10e18daeeb168cef9d0f8316fdcb5

rules_python 0.5.0

io_bazel_rules_docker v0.25.0

io_bazel_rules_go v0.45.1

io_bazel_rules_k8s v0.7

bazelisk

.bazelversion

bazel 6.5.0

github-actions

.github/workflows/k8s-diff.yaml

actions/checkout v3

bazelbuild/setup-bazelisk v2

actions/cache v3

actions/upload-artifact v3

actions/checkout v3

bazelbuild/setup-bazelisk v2

actions/cache v3

actions/upload-artifact v3

actions/download-artifact v3

actions/setup-node v3

actions/github-script v6

.github/workflows/k8s.yaml

actions/checkout v3

tailscale/github-action v2

bazelbuild/setup-bazelisk v2

actions/cache v3

docker/login-action v2

.github/workflows/test.yaml

actions/checkout v3

bazelbuild/setup-bazelisk v2

gomod

go.mod

go 1.21

cuelang.org/go v0.7.0

github.com/1Password/onepassword-operator v1.8.0

github.com/NVIDIA/gpu-operator v1.11.1

github.com/VictoriaMetrics/operator/api v0.0.0-20231128174956-7965dba77210@7965dba77210

github.com/backube/volsync v0.8.0

github.com/cert-manager/cert-manager v1.13.3

github.com/cilium/cilium v1.14.6

github.com/crunchydata/postgres-operator v0.0.0-00010101000000-000000000000@000000000000

github.com/external-secrets/external-secrets v0.9.11

github.com/grafana/grafana-operator/v5 v5.8.0

github.com/prometheus/prometheus v0.49.1

github.com/rook/rook/pkg/apis v0.0.0-20240118185538-f46c0845eea7@f46c0845eea7

k8s.io/api v0.29.3

k8s.io/apiextensions-apiserver v0.29.3

k8s.io/client-go v12.0.0+incompatible

k8s.io/kube-aggregator v0.29.1

k8s.io/kubernetes v1.29.1

k8s.io/api v0.29.1

k8s.io/apiextensions-apiserver v0.29.1

k8s.io/apimachinery v0.29.1

k8s.io/apiserver v0.29.1

k8s.io/cli-runtime v0.29.1

k8s.io/client-go v0.29.1

k8s.io/cloud-provider v0.29.1

k8s.io/cluster-bootstrap v0.29.1

k8s.io/code-generator v0.29.1

k8s.io/component-base v0.29.1

k8s.io/component-helpers v0.29.1

k8s.io/controller-manager v0.29.1

k8s.io/cri-api v0.29.1

k8s.io/csi-translation-lib v0.29.1

k8s.io/dynamic-resource-allocation v0.29.1

k8s.io/kms v0.29.1

k8s.io/kube-aggregator v0.29.1

k8s.io/kube-controller-manager v0.29.1

k8s.io/kube-proxy v0.29.1

k8s.io/kube-scheduler v0.29.1

k8s.io/kubectl v0.29.1

k8s.io/kubelet v0.29.1

k8s.io/legacy-cloud-providers v0.29.1

k8s.io/metrics v0.29.1

k8s.io/mount-utils v0.29.1

k8s.io/pod-security-admission v0.29.1

k8s.io/sample-apiserver v0.29.1

k8s.io/sample-cli-plugin v0.29.1

k8s.io/sample-controller v0.29.1

github.com/libopenstorage/external-storage v0.20.4-rc1

github.com/portworx/sched-ops v0.20.4-openstorage-rc3

github.com/crunchydata/postgres-operator v1.3.3-0.20230629151007-94ebcf2df74d@94ebcf2df74d

github.com/openshift/api v0.0.0-20240401200911-ab1b479a063f@ab1b479a063f

regex

k8s/amour/backup/breakfast/statefulset_list.cue

syncthing/syncthing edge@sha256:a200af1e5b2aee7c184c848c3af179d6fedca55e899c15a9c2851c35501f1943

k8s/amour/backup/immich_unwind/statefulset_list.cue

syncthing/syncthing edge@sha256:a200af1e5b2aee7c184c848c3af179d6fedca55e899c15a9c2851c35501f1943

k8s/amour/backup/legacy/statefulset_list.cue

syncthing/syncthing edge@sha256:a200af1e5b2aee7c184c848c3af179d6fedca55e899c15a9c2851c35501f1943

k8s/amour/backup/lola/statefulset_list.cue

syncthing/syncthing edge@sha256:a200af1e5b2aee7c184c848c3af179d6fedca55e899c15a9c2851c35501f1943

k8s/amour/backup/melonade/statefulset_list.cue

syncthing/syncthing edge@sha256:a200af1e5b2aee7c184c848c3af179d6fedca55e899c15a9c2851c35501f1943

k8s/amour/backup/synology/statefulset_list.cue

syncthing/syncthing edge@sha256:a200af1e5b2aee7c184c848c3af179d6fedca55e899c15a9c2851c35501f1943

k8s/amour/cert_manager_csi_driver/daemon_set_list.cue

registry.k8s.io/sig-storage/csi-node-driver-registrar v2.5.0

k8s.gcr.io/sig-storage/livenessprobe v2.6.0

quay.io/jetstack/cert-manager-csi-driver v0.5.0

k8s/amour/cilium/hubble_ui/deployment_list.cue

quay.io/cilium/hubble-ui-backend v0.12.0@sha256:8a79a1aad4fc9c2aa2b3e4379af0af872a89fcec9d99e117188190671c66fc2e

k8s/amour/dcgm_exporter/daemon_set_list.cue

nvcr.io/nvidia/k8s/dcgm-exporter 3.3.0-3.2.0-ubuntu22.04

k8s/amour/external_secrets/deployment_list.cue

ghcr.io/external-secrets/external-secrets v0.9.5

k8s/amour/external_secrets/external-secrets.cue

ghcr.io/external-secrets/external-secrets v0.9.5

ghcr.io/external-secrets/external-secrets v0.9.5

k8s/amour/external_secrets/webhook/deployment_list.cue

ghcr.io/external-secrets/external-secrets v0.9.5

k8s/amour/karma/list.cue

ghcr.io/prymitive/karma 0.116

ghcr.io/prymitive/karma 0.116@sha256:ddfb0a874d24ca314457a74db351d59db1b9609206f4c01fc272b59a6867d374

k8s/amour/media/recyclarr/cron_job_list.cue

alpine 3.17.2@sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501

mikefarah/yq 4.33.1@sha256:ddf60fa876a4f73414477fab551bcfb864a179cad6ce998b13ba4180e0f5702d

k8s/amour/metrics_server/deployment_list.cue

registry.k8s.io/metrics-server/metrics-server v0.6.3

k8s/amour/minecraft/cf_atm9/stateful_set_list.cue

curlimages/curl 8.1.2

k8s/amour/onepassword_connect/deployment_list.cue

1password/connect-api 1.7.2@sha256:6aa94cf713f99c0fa58c12ffdd1b160404b4c13a7f501a73a791aa84b608c5a1

1password/connect-sync 1.7.2@sha256:fe527ed9d81f193d8dfbba4140d61f9e8c8dceb0966b3009259087504e5ff79c

k8s/amour/rook_ceph/ceph_cluster_list.cue

docker.io/rkachach/ceph v18.2.1_patched_v1

k8s/amour/rook_ceph/deployment_list.cue

quay.io/ceph/ceph v18.2.0

k8s/amour/snapshot_controller/deployment_list.cue

registry.k8s.io/sig-storage/snapshot-controller v6.2.1

k8s/amour/volsync_system/deployment_list.cue

quay.io/brancz/kube-rbac-proxy v0.14.0

k8s/unwind/cert_manager_csi_driver/daemon_set_list.cue

registry.k8s.io/sig-storage/csi-node-driver-registrar v2.5.0

k8s.gcr.io/sig-storage/livenessprobe v2.6.0

quay.io/jetstack/cert-manager-csi-driver v0.5.0

k8s/unwind/csi_snapshotter/stateful_set_list.cue

registry.k8s.io/sig-storage/csi-provisioner v3.4.0

registry.k8s.io/sig-storage/csi-snapshotter v6.2.1

registry.k8s.io/sig-storage/hostpathplugin v1.11.0

k8s/unwind/dragonfly_operator_system/deployment_list.cue

gcr.io/kubebuilder/kube-rbac-proxy v0.13.1

docker.dragonflydb.io/dragonflydb/operator v0.0.6

k8s/unwind/grafana_agent/grafana_agent_list.cue

grafana/agent v0.32.1

k8s/unwind/immich/postgres_cluster_list.cue

k8s/unwind/immich/redis_failover_list.cue

redis 7.0.11-alpine@sha256:e20345b7ec692815860c07f0209eb0465687b0c28cd85df412811ae1ac7b653e

redis 7.0.11-alpine@sha256:e20345b7ec692815860c07f0209eb0465687b0c28cd85df412811ae1ac7b653e

k8s/unwind/kube_system/metrics_server/deployment_list.cue

registry.k8s.io/metrics-server/metrics-server v0.6.3

k8s/unwind/kubernetes_dashboard/deployment_list.cue

kubernetesui/metrics-scraper v1.0.9@sha256:9b599f50dc7bfdfe71f021a4859fe19f74baf2135a8538ba1c1013832b7a66b4

k8s/unwind/loki/backend/stateful_set_list.cue

grafana/loki 2.8.2@sha256:dc4328febf349d9198ef0f1c893160483fc7b2180d7e31485325f6e702ee73c4

k8s/unwind/loki/gateway/deployment_list.cue

nginxinc/nginx-unprivileged 1.23.4-alpine3.17-slim@sha256:7c85fc22f25023a120c45fac6616f2b6fd8e37429259a9d2c333681994d1e9e1

k8s/unwind/loki/read/deployment_list.cue

grafana/loki 2.8.2@sha256:dc4328febf349d9198ef0f1c893160483fc7b2180d7e31485325f6e702ee73c4

k8s/unwind/loki/write/stateful_set_list.cue

grafana/loki 2.8.2@sha256:dc4328febf349d9198ef0f1c893160483fc7b2180d7e31485325f6e702ee73c4

k8s/unwind/minecraft/cf_atm8/stateful_set_list.cue

curlimages/curl 8.1.2

k8s/unwind/postgres_operator/deployment_list.cue

k8s/unwind/rook_ceph/ceph_cluster_list.cue

quay.io/ceph/ceph v17.2.6

k8s/unwind/rook_ceph/deployment_list.cue

quay.io/ceph/ceph v17.2.6

k8s/unwind/secrets_store_csi_driver/daemon_set_list.cue

registry.k8s.io/sig-storage/csi-node-driver-registrar v2.7.0

registry.k8s.io/sig-storage/livenessprobe v2.9.0

k8s/unwind/snapshot_controller/deployment_list.cue

registry.k8s.io/sig-storage/snapshot-controller v6.2.1

k8s/unwind/thomas/breakfast_backup/statefulset_list.cue

syncthing/syncthing 1.23.6@sha256:88d6c8516d27876f6dacf7b9b544075d70e0d42480a2e85ec4dbb313764cc1e6

k8s/unwind/thomas/melonade_backup/statefulset_list.cue

syncthing/syncthing 1.23.6@sha256:88d6c8516d27876f6dacf7b9b544075d70e0d42480a2e85ec4dbb313764cc1e6

k8s/unwind/thomas/synologybackup/job_list.cue

rclone/rclone 1.62.2@sha256:f6322df9af20b551049c2746f15facc9be1154aed3ab79e0d2529edbc8433935

k8s/amour/cilium/hubble_relay/list.cue

cilium/cilium 1.15.0-rc.0

k8s/amour/cilium/hubble_ui/list.cue

cilium/hubble 0.12.3

k8s/amour/cilium/list.cue

cilium/cilium 1.15.0-rc.0

k8s/amour/emqx/list.cue

emqx/emqx 5.6.0

k8s/amour/emqx_exporter/list.cue

emqx/emqx-exporter 0.2.7

k8s/amour/frigate/list.cue

blakeblackshear/frigate 0.13.2

k8s/amour/grafana/list.cue

grafana/grafana 10.0.2

k8s/amour/home_assistant/list.cue

home-assistant/core 2023.12.4

k8s/amour/karma/list.cue

prymitive/karma 0.116

k8s/amour/kube_state_metrics/list.cue

kubernetes/kube-state-metrics 2.8.2

k8s/amour/nvidia_device_plugin/list.cue

nvcr.io/nvidia/k8s-device-plugin 0.14.3

k8s/amour/ping_exporter/list.cue

czerwonk/ping_exporter 1.1.0

k8s/amour/smartctl_exporter/list.cue

prometheus-community/smartctl_exporter 0.11.0

k8s/amour/speedtest_exporter/list.cue

MiguelNdeCarvalho/speedtest-exporter 3.5.4

k8s/amour/tailscale/deployment_list.cue

tailscale/tailscale v1.56.0

k8s/amour/vm_operator/list.cue

VictoriaMetrics/operator 0.42.3

k8s/unwind/grafana/list.cue

grafana/grafana 10.0.2

k8s/unwind/home_assistant/list.cue

home-assistant/core 2023.12.4

k8s/unwind/immich/immich_machine_learning/list.cue

immich-app/immich 1.66.1

terraform

tf/backend.tf

hashicorp/terraform 1.7.1

tf/main.tf

cloudflare 4.23.0

Check this box to trigger a request for Renovate to run again on this repository

linkerd: Automatically renew trust anchor?

I want this cluster to be as low maintenance as possible, and not fall over just because I forgot to renew a certificate or something. Is it possible to automatically renew the Linkerd trust anchor? My current thinking is a cron job which will create a new trust anchor periodically and commit it to git, letting Flux take care of actually applying it. Will chat to the authors and see what they think of the idea.

ingress-nginx/ingress-nginx: Remote IP addresses are always 127.0.0.1

2020/05/30 00:53:26 [error] 38#38: *3263 client intended to send too large body: 48687665 bytes, client: 127.0.0.1, server: kipp.dev.6f.io, request: "POST / HTTP/2.0", host: "kipp.dev.6f.io"

Either some missing config or Linkerd2.

chore(k8s/unwind): use nvme storage class where needed

The storage class rook-ceph-hdd-ec-delete-block is used a lot as it was the first available storage class. With the recent introduction of an nvme storage class, it should be used instead for the majority case. It's much faster and can make a significant positive impact on performance as seen with VictoriaMetrics.

chore(k8s/unwind): use stateful sets

A lot of applications like Grafana and Jellyfin use a deployment with a single replica and a RWO PVC. These should be stateful sets.

replace Grafana Agent

Given the recent work to implement VictoriaMetrics and VMAgent, Grafana Agent doesn't do much besides collect logs. In preparation for Victoria Logs, an alternate log collector like fluent-bit or vector should be deployed instead. This would mean both Grafana Agent and the Grafana Agent Operator could be deleted which would be nice.

cmd/resolver: Undefined behavior if app.kubernetes.io/managed-by label already exists

Authenticate with SSO?

https://github.com/dexidp/dex

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.