manno/fleet

Introduction

Fleet is GitOps at scale. Fleet is designed to manage multiple clusters. It's also lightweight enough that it works great for a single cluster too, but it really shines when you get to a large scale. By large scale we mean either a lot of clusters, a lot of deployments, or a lot of teams in a single organization.

Fleet can manage deployments from git of raw Kubernetes YAML, Helm charts, or Kustomize or any combination of the three. Regardless of the source all resources are dynamically turned into Helm charts and Helm is used as the engine to deploy everything in the cluster. This gives a high degree of control, consistency, and auditability. Fleet focuses not only on the ability to scale, but to give one a high degree of control and visibility to exactly what is installed on the cluster.

Quick Start

For more information, have a look at Fleet's documentation.

Install

Get helm if you don't have it. Helm 3 is just a CLI and won't do bad insecure things to your cluster.

For instance, using Homebrew:

brew install helm

Install the Fleet Helm charts (there's two because we separate out CRDs for ultimate flexibility.)

helm -n cattle-fleet-system install --create-namespace --wait \
    fleet-crd https://github.com/rancher/fleet/releases/download/v0.9.0/fleet-crd-0.9.0.tgz
helm -n cattle-fleet-system install --create-namespace --wait \
    fleet https://github.com/rancher/fleet/releases/download/v0.9.0/fleet-0.9.0.tgz

Add a Git Repo to watch

Change spec.repo to your git repo of choice. Kubernetes manifest files that should be deployed should be in /manifests in your repo.

cat > example.yaml << "EOF"
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: sample
  # This namespace is special and auto-wired to deploy to the local cluster
  namespace: fleet-local
spec:
  # Everything from this repo will be run in this cluster. You trust me right?
  repo: "https://github.com/rancher/fleet-examples"
  paths:
  - simple
EOF

kubectl apply -f example.yaml

Get Status

Get status of what Fleet is doing:

kubectl -n fleet-local get fleet

You should see something like this get created in your cluster.

kubectl get deploy frontend

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
frontend   3/3     3            3           116m

Enjoy and read the docs.

License

For developer and maintainer documentation, see DEVELOPING.md.

fleet's Issues

Agent panicks when CI updates

Happens sometimes during e2e setup:
gha-e2e-logs-3891e0158855b19f08c639042d7024c814939634-2314915879.zip

runner@fv-az42-358:~/work/fleet/fleet$ kubectl get pods -A
NAMESPACE             NAME                                      READY   STATUS             RESTARTS      AGE
kube-system           local-path-provisioner-84bb864455-cz2wd   1/1     Running            0             10m
kube-system           coredns-96cc4f57d-pp4gv                   1/1     Running            0             10m
kube-system           helm-install-traefik-crd--1-svdfc         0/1     Completed          0             10m
kube-system           metrics-server-ff9dbcb6c-n2tht            1/1     Running            0             10m
kube-system           helm-install-traefik--1-drvgh             0/1     Completed          2             10m
kube-system           svclb-traefik-pk85q                       2/2     Running            0             10m
kube-system           svclb-traefik-gdwtx                       2/2     Running            0             10m
kube-system           traefik-56c4b88c4b-v5494                  1/1     Running            0             10m
cattle-system         cattle-cluster-agent-75b655997d-xqn5v     1/1     Running            0             4m47s
cattle-system         cattle-cluster-agent-75b655997d-zmnwg     1/1     Running            0             4m36s
cattle-fleet-system   fleet-agent-67c5c794-225jq                0/1     CrashLoopBackOff   3 (48s ago)   104s
runner@fv-az42-358:~/work/fleet/fleet$ kubectl logs -n cattle-fleet-system   fleet-agent-67c5c794-225jq
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x807e6a]

goroutine 1 [running]:
github.com/modern-go/reflect2.(*frozenConfig).Type2(0x212b400, {0x28c85a8, 0x2046220})
        /home/runner/go/pkg/mod/github.com/modern-go/[email protected]/reflect2.go:157 +0x4a
github.com/modern-go/reflect2.(*UnsafeMapType).Key(0xc000e23400)
        /home/runner/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:53 +0x49
github.com/json-iterator/go.decoderOfMap(0xc000ea3200, {0x28c4118?, 0xc000e23400})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_map.go:14 +0xe5
github.com/json-iterator/go._createDecoderOfType(0xc000ea3200, {0x28c4118?, 0xc000e23400})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:182 +0x2c8
github.com/json-iterator/go.createDecoderOfType(0xc000ea3200, {0x28c4118, 0xc000e23400})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:142 +0xdf
github.com/json-iterator/go.decoderOfType(0xc000ea3200, {0x28c4118, 0xc000e23400})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:124 +0x45
github.com/json-iterator/go.describeStruct(0xc000ea2ed0, {0x28c42f8?, 0xc000e6b890})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_extension.go:380 +0x6d0
github.com/json-iterator/go.decoderOfStruct(0xc000ea2ed0, {0x28c42f8, 0xc000e6b890})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_decoder.go:14 +0xb4
github.com/json-iterator/go._createDecoderOfType(0xc000ea2ed0, {0x28c42f8?, 0xc000e6b890})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:176 +0x410
github.com/json-iterator/go.createDecoderOfType(0xc000ea2ed0, {0x28c42f8, 0xc000e6b890})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:142 +0xdf
github.com/json-iterator/go.decoderOfType(0xc000ea2ed0, {0x28c42f8, 0xc000e6b890})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:124 +0x45
github.com/json-iterator/go.describeStruct(0xc000dea610, {0x28c42f8?, 0xc000e6b740})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_extension.go:380 +0x6d0
github.com/json-iterator/go.decoderOfStruct(0xc000dea610, {0x28c42f8, 0xc000e6b740})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_decoder.go:14 +0xb4
github.com/json-iterator/go._createDecoderOfType(0xc000dea610, {0x28c42f8?, 0xc000e6b740})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:176 +0x410
github.com/json-iterator/go.createDecoderOfType(0xc000dea610, {0x28c42f8, 0xc000e6b740})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:142 +0xdf
github.com/json-iterator/go.decoderOfType(0xc000dea610, {0x28c42f8, 0xc000e6b740})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:124 +0x45
github.com/json-iterator/go.decoderOfSlice(0xc000ea2e40, {0x28c4258?, 0xc000e23280})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_slice.go:12 +0xcd
github.com/json-iterator/go._createDecoderOfType(0xc000ea2e40, {0x28c4258?, 0xc000e23280})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:180 +0x431
github.com/json-iterator/go.createDecoderOfType(0xc000ea2e40, {0x28c4258, 0xc000e23280})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:142 +0xdf
github.com/json-iterator/go.decoderOfType(0xc000ea2e40, {0x28c4258, 0xc000e23280})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:124 +0x45
github.com/json-iterator/go.describeStruct(0xc000deacc8, {0x28c42f8?, 0xc000e2dc20})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_extension.go:380 +0x6d0
github.com/json-iterator/go.decoderOfStruct(0xc000deacc8, {0x28c42f8, 0xc000e2dc20})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_decoder.go:14 +0xb4
github.com/json-iterator/go._createDecoderOfType(0xc000deacc8, {0x28c42f8?, 0xc000e2dc20})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:176 +0x410
github.com/json-iterator/go.createDecoderOfType(0xc000deacc8, {0x28c42f8, 0xc000e2dc20})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:142 +0xdf
github.com/json-iterator/go.decoderOfType(0xc000deacc8, {0x28c42f8, 0xc000e2dc20})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:124 +0x45
github.com/json-iterator/go.(*frozenConfig).DecoderOf(0xc000238140, {0x28c41b8, 0xc000e2db60})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:114 +0x108
github.com/json-iterator/go.(*Iterator).ReadVal(0xc0008de1b0, {0x225d6e0, 0xc000381970})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:72 +0xe5
github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc000238140, {0xc000e9e000?, 0xc000e6ca98?, 0x2?}, {0x225d6e0, 0xc000381970})
        /home/runner/go/pkg/mod/github.com/json-iterator/[email protected]/config.go:348 +0xa5
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc000372000, {0xc000e87500, 0x95d, 0x95d}, 0xc000e81380, {0x28a20f0?, 0xc000e3cf60?})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:264 +0x84c
k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*codec).Decode(0xc000396280, {0xc000e87500?, 0x95d?, 0x95d?}, 0x0?, {0x28a20f0, 0xc000e3cf60})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/versioning/versioning.go:136 +0x1f9
k8s.io/client-go/tools/clientcmd.Load({0xc000e87500, 0x95d, 0x95d})
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/tools/clientcmd/loader.go:409 +0x18f
k8s.io/client-go/tools/clientcmd.NewClientConfigFromBytes({0xc000e87500?, 0x0?, 0x212a560?})
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/tools/clientcmd/client_config.go:111 +0x25
github.com/rancher/fleet/modules/agent/pkg/register.testClientConfig({0xc000e87500?, 0xc000d960c0?, 0xc000e5c500?})
        /home/runner/work/fleet/fleet/modules/agent/pkg/register/register.go:281 +0x33
github.com/rancher/fleet/modules/agent/pkg/register.createClusterSecret({0x28b6190, 0xc000a53680}, {0x0, 0x0}, {0x28bf850, 0xc0006d14c0}, 0xc000b15040)
        /home/runner/work/fleet/fleet/modules/agent/pkg/register/register.go:196 +0x8eb
github.com/rancher/fleet/modules/agent/pkg/register.runRegistration({0x28b6190, 0xc000a53680}, {0x28bf850?, 0xc0006d14c0?}, {0xc00005a00a, 0x13}, {0x0, 0x0})
        /home/runner/work/fleet/fleet/modules/agent/pkg/register/register.go:67 +0x1af
github.com/rancher/fleet/modules/agent/pkg/register.tryRegister({0x28b6190, 0xc000a53680}, {0xc00005a00a, 0x13}, {0x0, 0x0}, 0x3d3b300?)
        /home/runner/work/fleet/fleet/modules/agent/pkg/register/register.go:80 +0x18b
github.com/rancher/fleet/modules/agent/pkg/register.Register({0x28b6190, 0xc000a53680}, {0xc00005a00a, 0x13}, {0x0, 0x0}, 0xc000b1c750?)
        /home/runner/work/fleet/fleet/modules/agent/pkg/register/register.go:49 +0x97
github.com/rancher/fleet/modules/agent/pkg/agent.Start({0x28b6190, 0xc000a53680}, {0x0?, 0x1ebd265?}, {0xc00005a00a, 0x13}, {0x0, 0x0}, 0xc0009b79f8?)
        /home/runner/work/fleet/fleet/modules/agent/pkg/agent/app.go:56 +0x16a
main.(*FleetAgent).Run(0xc000a0ba40, 0xc000b16f00, {0x0?, 0x0?, 0x0?})
        /home/runner/work/fleet/fleet/cmd/fleetagent/main.go:52 +0x10d
github.com/rancher/wrangler-cli.bind.func1(0xc000b16f00?, {0x3d77f00, 0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/builder.go:269 +0x127
github.com/spf13/cobra.(*Command).execute(0xc000b16f00, {0xc00004c230, 0x0, 0x0})
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000b16f00)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:960 +0x39c
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:897
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:890
github.com/rancher/wrangler-cli.Main(0xc000b16f00)
        /home/runner/go/pkg/mod/github.com/rancher/[email protected]/builder.go:72 +0x58
main.main()
        /home/runner/work/fleet/fleet/cmd/fleetagent/main.go:64 +0x116