fluxcd/source-controller

The GitOps Toolkit source management component

Source controller

The source-controller is a Kubernetes operator, specialised in artifacts acquisition from external sources such as Git, OCI, Helm repositories and S3-compatible buckets. The source-controller implements the source.toolkit.fluxcd.io API and is a core component of the GitOps toolkit.

APIs

Kind	API Version
GitRepository	source.toolkit.fluxcd.io/v1
OCIRepository	source.toolkit.fluxcd.io/v1beta2
HelmRepository	source.toolkit.fluxcd.io/v1
HelmChart	source.toolkit.fluxcd.io/v1
Bucket	source.toolkit.fluxcd.io/v1

Features

• authenticates to sources (SSH, user/password, API token, Workload Identity)
• validates source authenticity (PGP, Cosign, Notation)
• detects source changes based on update policies (semver)
• fetches resources on-demand and on-a-schedule
• packages the fetched resources into a well-known format (tar.gz, yaml)
• makes the artifacts addressable by their source identifier (sha, version, ts)
• makes the artifacts available in-cluster to interested 3rd parties
• notifies interested 3rd parties of source changes and availability (status conditions, events, hooks)
• reacts to Git, Helm and OCI artifacts push events (via notification-controller)

Guides

• Get started with Flux
• Setup Webhook Receivers
• Setup Notifications
• How to build, publish and consume OCI Artifacts with Flux

Roadmap

The roadmap for the Flux family of projects can be found at https://fluxcd.io/roadmap/.

Contributing

This project is Apache 2.0 licensed and accepts contributions via GitHub pull requests. To start contributing please see the development guide.

source-controller's Issues

Allow S3 scheme for HelmRepository source

It is possible to create a helm repo inside an S3 bucket. It would be nice to be able to use this instead of the Bucket source.

[Feature] Include VCS files and scrub them

From #74:

• It would be nice to have the ability to include the bare git repository as a part of the working copy
  • It would also be nice to be able to prune remote content and configuration, either optionally or by default

Support Helm charts from OCI registries

The language in the Helm docs about registries suggests that the plan is support using OCI (https://helm.sh/docs/topics/registries/). This feature still experimental in Helm, but it looks like the plan is to graduate it in a release soon: helm/helm#7613, helm/community#136.

Source controller fails to create some helm package from git repository

Following #194 : after upgrade to 0.7.5 (that includes source-controller 0.7.2 that include #269), it seems not yet working properly:

With a chart layout in git like that

Chart.yaml # <= reference to `dependency` with a conditional
Chart.lock
charts/
   dependency/
      templates/
templates/

Source controller generates a package like that:

 Chart.yaml 
 Chart.lock
 charts/
    <parent-chart-name>/
        charts/
           dependency/
               templates/
    dependency/
       templates/
templates/

Parent chart looks included as dependency, which cause an already exists in helm reconciliation

Note: a helm dep build <chart-path> in the git repository is not doing such a thing.

source-controller pod restarting (OOMKilled)

I have noticed that the source-controller pod of my gotk deployment restarting a huge number of times over the weekend (148 times -- version 0.1.1). I've re-deployed a newer version (0.2.1) but the restarts keep happening (about 2 every half hour).

$> k describe po -n gotk-system source-controller-5cc54c757c-ccwz8
Name:         source-controller-5cc54c757c-ccwz8
Namespace:    gotk-system
Priority:     0
Node:         my-node/10.0.10.11
Start Time:   Mon, 02 Nov 2020 13:57:18 +0000
Labels:       app=source-controller
              pod-template-hash=5cc54c757c
Annotations:  prometheus.io/port: 8080
              prometheus.io/scrape: true
Status:       Running
IP:           10.0.10.12
IPs:
  IP:           10.0.10.12
Controlled By:  ReplicaSet/source-controller-5cc54c757c
Containers:
  manager:
    Container ID:  docker://6b4a1a89311360cb832fe1d540b4f4cb96c9b8a6591fb01349390ffcdfc99b90
    Image:         my-registry.com/fluxcd/source-controller:v0.2.1
    Image ID:      docker-pullable://my-registry.com/fluxcd/source-controller@sha256:e8b708159f6d651a9577695af14bf3291ef844ca5cd7e85f182416b76561d27c
    Ports:         9090/TCP, 8080/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --events-addr=
      --watch-all-namespaces=true
      --log-level=info
      --log-json
      --enable-leader-election
      --storage-path=/data
    State:          Running
      Started:      Mon, 02 Nov 2020 14:34:16 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 02 Nov 2020 14:13:59 +0000
      Finished:     Mon, 02 Nov 2020 14:34:15 +0000
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      50m
      memory:   64Mi
    Liveness:   http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      RUNTIME_NAMESPACE:  gotk-system (v1:metadata.namespace)
      HTTPS_PROXY:        http://http.my-proxy.com:8000
      NO_PROXY:           10.0.0.0/8,172.0.0.0/8
    Mounts:
      /data from data (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bfr47 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-bfr47:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bfr47
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/arch=amd64
                 kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age                 From                                       Message
  ----    ------     ----                ----                                       -------
  Normal  Scheduled  <unknown>           default-scheduler                          Successfully assigned gotk-system/source-controller-5cc54c757c-ccwz8 to my-node
  Normal  Pulled     4m6s (x3 over 41m)  kubelet, my-node  Container image "my-registry.com/fluxcd/source-controller:v0.2.1" already present on machine
  Normal  Created    4m6s (x3 over 41m)  kubelet, my-node  Created container manager
  Normal  Started    4m6s (x3 over 41m)  kubelet, my-node  Started container manager

This causes the helm-controller to not be able to reconcile HelmReleases:

$> k get hr --all-namespaces
NAMESPACE                NAME                                       READY   STATUS                                                                                                                                                                                                                                      AGE
namespace1           chart1        False   Get "http://source-controller.gotk-system/helmchart/namespace1/chart1/chart1-v0.15.5.tgz": dial tcp 172.20.225.87:80: connect: connection refused              2d18h
( . . .)
( . . .)
( . . .)
namespace2          chart11       False   Get "http://source-controller.gotk-system/helmchart/namespace2/chart11/chart11-v0.1.3.tgz": dial tcp 172.20.225.87:80: connect: connection refused              2d18h

The source controller manages one GitRepository and two HelmRepositories.
The helm controller takes care of 11 HelmReleases, each with similar configuration:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: my-release
  namespace: namespace1
spec:
  install:
    remediation:
      retries: -1
  upgrade:
    remediation:
      retries: -1
  interval: 1m0s
  releaseName: my-release
  chart:
    spec:
      version: 1.0.2
      chart: my-chart
      sourceRef:
        kind: HelmRepository
        name: my-repository
        namespace: namespace1
  valuesFrom:
  - kind: ConfigMap
    name: my-values
    valuesKey: environment
    targetPath: myEnv
  values:
    my-value: 30

While writing up this issue the source-controller restarted 3 more times
Logs from the source controller don't indicate any errors:

{"level":"info","ts":"2020-11-02T15:00:17.646Z","logger":"controllers.HelmChart","msg":"Reconciliation finished in 364.572799ms, next run in 1m0s","controller":"helmchart","request":"namespace1/chart1"}
( . . . )
( . . . )
( . . . )
{"level":"info","ts":"2020-11-02T15:00:17.646Z","logger":"controllers.HelmChart","msg":"Reconciliation finished in 364.572799ms, next run in 1m0s","controller":"helmchart","request":"namespace1/chart11"}
{"level":"info","ts":"2020-11-02T15:01:12.527Z","logger":"controllers.GitRepository","msg":"Reconciliation finished in 1.631398488s, next run in 3m0s","controller":"gitrepository","request":"namespace1/my-git-repo"}
{"level":"info","ts":"2020-11-02T15:01:12.870Z","logger":"controllers.HelmRepository","msg":"Reconciliation finished in 1.165803995s, next run in 3m0s","controller":"helmrepository","request":"namespace1/my-repository"}

helm: support Helm chart from GitRepository

The current HelmChart spec and its implementation do only support Helm charts from HelmRepository resources. Extending this to also support a chart in the path of a GitRepository would make the resource more suitable for usage by the Helm Operator or its successor.

The HelmChart should have a new field, or a field supporting a type definition with a reference to the GitRepository. In addition, it should be possible to configure the path to the chart relative to the root of the GitRepository artifact.

The chart path should be packaged as a Helm chart using the package action in the action Helm package, or a combination of the chartutil methods, depending on the overhead and availability of package configuration options.

source controller needs proxy to access github.com when running inside corporate network

In order to get the source controller to access a github.com repository from within my client's corporate network I needed to add the following environmental variables to the source-controller deployment.

    - name: HTTPS_PROXY
      value: http://http.proxy.abc.com:8000
    - name: NO_PROXY
      value: 10.0.0.0/8

Maybe gotk install could be enhanced to generate this?

GCP Bucket support

Hi there,

I am trying to use a GCP bucket as Bucket Repository for my helm releases. I do it like this:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: Bucket
metadata:
  name: bucket-name
  namespace: sparkle
spec:
  interval: 1m
  bucketName: bucket-name
  endpoint: storage.googleapis.com
  provider: generic
  region: europe-west4
  secretRef:
    name: my-secret

Unfortunately it failes with this log:

{
  "namespace": "sparkle",
  "name": "bucket-name",
  "level": "error",
  "reconcilerKind": "Bucket",
  "error": "listing objects from bucket 'bucket-name' failed: A header or query you provided requested a function that is not implemented.",
  "msg": "Reconciler error",
  "controller": "bucket",
  "ts": "2020-11-18T10:05:40.021Z",
  "reconcilerGroup": "source.toolkit.fluxcd.io",
  "logger": "controller"
}

After some googling I stumbled across this issue:
minio/mc#3073

There it states:

This looks like GCS S3 layer doesn't implement AWS S3 compatible ListObjectsV2 implementation

See https://cloud.google.com/storage/docs/interoperability (Note: While most actions are interoperable with the Amazon S3 V2 SDK, listing objects can only be performed using the Amazon S3 V1 list objects method.)

Which seem to be the issue here as well.

Am I doing something wrong within the configuration or is this an issue on your side?

Thanks for your answers in advance :)

CC: @stefanprodan

Unable to deploy from nested Artifactory Helm repository

Issue

Using JFrog private Helm repo is not working with flux but works with helm install

Reproduction
Using following configuration
Helm repo

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: jfrog-helm
  namespace: flux-system
spec:
  interval: 1m
  secretRef:
    name: jfrog-helm-creds
  url: https://xxxx.jfrog.io/artifactory/helm

Using following release

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: elasticsearch
  namespace: test-9
spec:
  interval: 1m
  releaseName: elasticsearch
  chart:
    spec:
      chart: elastic/elasticsearch
      version: 7.9.2
      sourceRef:
        kind: HelmRepository
        name: jfrog-helm
        namespace: flux-system
      interval: 1m
  values:
    clusterName: "elasticsearch"
    nodeGroup: "master"
    replicas: 1
    minimumMasterNodes: 1
    image: "docker.elastic.co/elasticsearch/elasticsearch"
    imageTag: "6.8.8"
    imagePullPolicy: "IfNotPresent"
    overideName: "elasticsearch"
    esJavaOpts: "-Xmx1g -Xms1g"
    volumeClaimTemplate:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 500Gi
    resources:
      requests:
        cpu: "1000m"
        memory: "1Gi"
      limits:
        cpu: "1000m"
        memory: "3Gi"
    lifecycle:
      postStart:
        exec:
          command:
             - bash
             - -c
             - |
               #!/bin/bash
               ES_URL=http://localhost:9200
               while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done
               curl -XPOST "$ES_URL/_scripts/connections_threshold" -H 'Content-Type: application/json' -d'{"script": {"lang": "painless","source": "return params.total >= params.threshold;"}}'
               curl -XPOST "$ES_URL/_scripts/misconfigurations_threshold" -H 'Content-Type: application/json' -d'{"script": {"lang": "painless","source": "return params.misconfigurations_count >= params.threshold;"}}'

The helm release fails deployment because the chart is not fetched:

➜  fluxv2_test git:(main) kubectl get helmcharts.source.toolkit.fluxcd.io -A
NAMESPACE     NAME                   CHART                   VERSION   SOURCE KIND      SOURCE NAME   READY     STATUS                       AGE
flux-system   test-9-elasticsearch   elastic/elasticsearch   7.9.2     HelmRepository   jfrog-helm    Unknown   reconciliation in progress   14m
➜  fluxv2_test git:(main) kubectl get helmcharts.source.toolkit.fluxcd.io -A
NAMESPACE     NAME                   CHART                   VERSION   SOURCE KIND      SOURCE NAME   READY   STATUS                                                                                                                                                                                                                             AGE
flux-system   test-9-elasticsearch   elastic/elasticsearch   7.9.2     HelmRepository   jfrog-helm    False   storage error: symlink /data/helmchart/flux-system/test-9-elasticsearch/elastic/elasticsearch-7.9.2.tgz /data/helmchart/flux-system/test-9-elasticsearch/elastic/elastic/elasticsearch-latest.tgz.tmp: no such file or directory   14m

Notes

• Secret jfrog-helm-creds is set correctly
• Using helm install with the same configuration (i.e. password, repo and chart) works prefectly
• Using a public repo (i.e. bitnami) with the above resources works

Any idea how to resolve this ?
My hair is turning even grayer....and it's pretty gray already... 😢

Question: Why symlinks are excluded ?

Hello,

I would like to understand why symlinks are excluded from the artifact package actually ?

We would like to push a standard configuration to all our clusters (like external dns, autoscaler, etc, etc, etc) and to achieve this we have a bootstrap folder symlinked to each cluster configuration directory like this:

\ bootstrap
   \ external-dns
   \ autoscaler
\ clusters
  \ cluster-01
     \ bootstrap (symlinked to ../../bootstrap
     \ ...
  \ cluster-02
     \ bootstrap (same)

Due to the exclude of symlink in archive, this doesn't work.
Thanks :)

`HelmChart` from `GitRepository` with local dependencies does not package

Apologies if this belongs in an issue on a different repository, but I'm running into an issue when deploying a HelmRelease sourced from a GitRepository.

The HelmRelease is an umbrella chart containing other HelmReleases, minimal configuration shown below:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: umbrella
  namespace: flux-system
spec:
  ref:
    branch: main
  url: https://<gitrepository>.git
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: umbrella
  namespace: flux-system
spec:
  chart:
    spec:
      chart: chart/
      sourceRef:
        kind: GitRepository
        name: umbrella
        namespace: flux-system

The basic premise of the umbrella chart is:

Chart.yaml
Chart.lock
charts/
|-- app a
     |-- Chart.yaml
     |-- templates/
|-- app b
     |-- Chart.yaml
     |-- templates/
...

All dependencies are merely locally sourced subcharts in charts/.

I have successfully gotten it to work in several similar scenarios:

• install chart directly (helm install umbrella -n flux-system chart/)
• package (helm package chart/) and commit the .tgz, point HelmRelease directly to the .tgz
• copy the *.tgz from the source-controller pod (/data/gitrepository/flux-system/umbrella/<cmmit sha>.tar.gz) to my local machine and helm install pointing to the tgz

The only errors are coming from source-controller:

source-controller-6cdb6c8889-7nx76 manager {"level":"error","ts":"2020-11-03T04:38:04.135Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"source.toolkit.fluxcd.io","reconcilerKind":"HelmChart","controller":"helmchart","name":"flux-system-bigbang","namespace":"flux-system","error":"scheme \"\" not supported","errorVerbose":"scheme \"\" not supported\nhelm.sh/helm/v3/pkg/getter.Providers.ByScheme\n\t/go/pkg/mod/helm.sh/helm/[email protected]/pkg/getter/getter.go:134\ngithub.com/fluxcd/source-controller/internal/helm.NewChartRepository\n\t/workspace/internal/helm/repository.go:54\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).reconcileFromTarballArtifact\n\t/workspace/controllers/helmchart_controller.go:501\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).Reconcile\n\t/workspace/controllers/helmchart_controller.go:150\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"}

Encryption

When delegating decryption to source-controller, it will be easier to add more decryption providers and all the consumers will be getting already decrypted artefacts.

semver pattern 1.0.x admits the version 1.1.0-prerelease

The library used by source-controller for semver ranges includes prereleases in ranges, unconditionally. That is, the range 1.0.x includes 1.1.0-prerelease; and worse, there's no syntax for excluding prereleases.

This is not the behaviour expected or wanted -- you would expect 1.0.x to mean "all patch releases of 1.0". Failing that particular pattern working, you would at least want some way of expressing the latter, but there is none.

I suggest either 1. using Masterminds/semver (with strict parsing); or, if there are just too many problems with that, forking blang and merging the PR that fixes ranges; or failing that, filtering prereleases before comparing them to the range (with, I suppose, the option to include them). The latter really should be implicit per the pattern, but it can at least be a sensibly defaulted option.

valuesFile repackages an empty values.yaml in the published helm chart.

When the source controller detects that someone has declared spec.chart.spec.valuesFile in their helm release, it attempts to read the file and overwrite the file. Instead, currently it incorrectly writes a zero byte length file to disk.

-rw-r--r--    1 controll nogroup          0 Jan 20 23:09 values.yaml
-rw-r--r--    1 controll nogroup        370 Jan 20 23:09 values-test.yaml

I dug into the code a bit and think i figured out where...
Read is supplied with a empty slice of zero bytes length.
It should have a length derived from src.

...
		var valuesData []byte
		if _, err := src.Read(valuesData); err != nil {
...

This means io Read returns an empty slice

// Read reads up to len(b) bytes from the File.
// It returns the number of bytes read and any error encountered.
// At end of file, Read returns 0, io.EOF.
func (f *File) Read(b []byte) (n int, err error) {
	if err := f.checkValid("read"); err != nil {
		return 0, err
	}
	n, e := f.read(b)
	return n, f.wrapErr("read", e)
}

And finally this is written to disk as a zero byte file.

isValuesFileOverriden, err = helm.OverwriteChartDefaultValues(helmChart, valuesData)

I was thinking maybe replace this

source-controller/controllers/helmchart_controller.go

Lines 507 to 515 in 778b07c

	src, err := os.Open(srcPath)
	if err != nil {
	err = fmt.Errorf("failed to open values file '%s': %w", chart.Spec.ValuesFile, err)
	return sourcev1.HelmChartNotReady(chart, sourcev1.StorageOperationFailedReason, err.Error()), err
	}
	defer src.Close()
	var valuesData []byte
	if _, err := src.Read(valuesData); err != nil {

with ioutil readfile?

Support packaged charts (`.tgz`) for `HelmChart` artifacts from `GitRepository` and `Bucket` sources

We currently assume that the spec.chart in a HelmChart from a GitRepository or Bucket source is the path to a Helm chart directory.

The feature would be more useful if it also accepted .tgz files, especially because this is how they are often stored in S3 (compatible storage) buckets.

Add guidance for fetching and untar artifacts

Add instructions to dev guide for developing a controller that reacts to source events and fetches & untars the artifacts. All this code is in kustomize controller, we just need to extract some code snippets and explain how it works.

HelmChart ValuesFile no longer accepts relative path in GitRepository

Regression bug introduced in #178 and reported on Slack.

During the refactor review the detail of the ValuesFile was missed, and part of the functionality got lost:

	// Alternative values file to use as the default chart values, expected to be a
	// relative path in the SourceRef. Ignored when omitted.
	// +optional
	ValuesFile string `json:"valuesFile,omitempty"`

Resulting in the following for valuesFile: ./infrastructure/vault/values-vault-operator.yaml:

NAMESPACE       NAME                                    READY   MESSAGE       REVISION  SUSPENDED
infra-system    infra-system-vault-operator             False   failed to locate override values file: values-vault-operator.yaml False

Azure DevOps compatibility issues

In source-controller we use go-git to perform Git operations.

According to these issues go-git/go-git#64, src-d/go-git#335, src-d/go-git#1058 Azure DevOps doesn't implement Git v1 protocol and go-git doesn't implement v2 that supposed to work with Azure. As shelling out to the git binary is not an option for us, we have to figure out a way to Azure DevOps.

Support S3 buckets

Proposal: fluxcd/flux2#242

Hash HelmRepository index contents instead of relying on 'generated' timestamp

Ran in to an interesting issue earlier where a Helm repo was updating the contents of their index.yaml but not updating the generated timestamp. My reading of the source is that this causes the source controller to ignore the changes.

Clearly, forgetting to update this value is a bug on the part of the repository maintainer. However it strikes me that we could avoid this relatively easy to make mistake by hashing the contents of the index.yaml and updating if the hash changes.

Thoughts? Happy to submit a PR.

RUNTIME_NAMESPACE overrides storage-addr flag.

Currently, if RUNTIME_NAMESPACE evn var is set, source-controller ignores the storage-addr flag.
Both host and port are disregarded.
Also, if RUNTIME_NAMESPACE evn var is set there is no way to specify a port.

source-controller/main.go

Line 236 in d326ae4

if os.Getenv("RUNTIME_NAMESPACE") != "" {

Support for bearer token authentication

CodeFresh.io seems to only support token authentication for their hosted ChartMuseum installation. Would you accept a PR that adds a bearer token authentication mechanism to the source-controller?

[Bug] source-controller is dependent on master branch existing

This is the output I get on repositories without a master branch:

(the default branch is named main in my case, and master does not exist)

Will you take a patch to rectify the behavior?

       /home/erikh/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2020-07-13T09:28:31.258Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"GitRepository","namespace":"runs","name":"default-538-git","uid":"3ae19115-8fc5-4b22-ab0d-441f0f70b96a","apiVersion":"source.fluxcd.io/v1alpha1","resourceVersion":"866174"}, "reason": "error", "message": "git clone error: couldn't find remote ref \"refs/heads/master\""}
2020-07-13T09:28:31.262Z        ERROR   controller-runtime.controller   Reconciler error        {"controller": "gitrepository", "request": "runs/default-538-git", "error": "git clone error: couldn't find remote ref \"refs/heads/master\""}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/erikh/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/erikh/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/erikh/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /home/erikh/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /home/erikh/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /home/erikh/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /home/erikh/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
        /home/erikh/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

Dependencies of local dependency are not packaged.

Hello,

I'm having an issue with a HelmChart which is targeting a folder in a GitRepository.
The chart root located in this folder has a dependency on another local chart, component, specified as such :

dependencies:
- name: component
  version: 0.1.0
  repository: file://../component
  alias: mycomponent

This works fine and if I download the package tgz from the source-controller, it contains the component chart in the charts subfolder.

But, this component chart also has a dependency on postgresql, specified as such :

dependencies:
- name: postgresql
  condition: postgresql.deploy
  version: "9.8.3"
  repository: https://charts.bitnami.com/bitnami

And this is not included in the tgz, which cause my deployment to fail.

For what it's worth, calling helm package on root does package recursively : root/charts/component/charts/postgresql/Chart.yaml, which means a traditional helm install works.

I looked at the code, it seems it is because the dependency resolution is not recursive and

source-controller/internal/helm/dependency_manager.go

Line 80 in aed11a1

func chartForLocalDependency(dep *helmchart.Dependency, cp string) (*helmchart.Chart, error) {

does not resolve dependency before loading the subchart.

Is there something I am missing ?

Thank you.

Feature requests around CRD API WRT file inclusion/exclusion

No rush on this, but the ability to include the git repo and all files (no automatic ignore lists) would be first stage goals I would really love to do for ya'll and could probably deliver in a week or two.

I do have use cases for all these but if they are poorly explained, let's discuss please. I would really like these features in the controller functionality, and am happy to make effort towards building them.

• It would be nice to have the ability to include the bare git repository as a part of the working copy
  • It would also be nice to be able to prune remote content and configuration, either optionally or by default
• It would be nice to set the mask for file exclusion
• It would also be nice to set a mask for file inclusion, the inverse where everything is excluded when this option is set.
  • Merging the two results would be applying a exclusion first to all files, then inclusion of the list, then exclusion of the specified files, so that both options hold POLS properties.

I could pass this upwards to my CI users who could then ensure no content gets into CI runs that is not intended to be shared with, say, anonymous parties.

Azure Container Registry returns HTTP 500

My Helm release is not installed because the generated Helm chart source is not downloaded when the referenced Helm repository is an Azure Container Registry.
I use version 0.2.1 of source controller.

The source controller fails to download the chart which is in this state :

NAME          CHART   VERSION          SOURCE KIND      SOURCE NAME   READY   STATUS                                                                                                       AGE
default-cdn   cdn     >=1.0.0 <2.0.0   HelmRepository   contoso       False   failed to fetch https://contoso.azurecr.io/helm/v1/repo/_blobs/cdn-1.0.0.tgz : 500 Internal Server Error     15m

And here is the logs of the source controller :

{"level":"error","ts":"2020-11-03T13:10:01.741Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"source.toolkit.fluxcd.io","reconcilerKind":"HelmChart","controller":"helmchart","name":"default-cdn","namespace":"cd","error":"failed to fetch https://contoso.azurecr.io/helm/v1/repo/_blobs/cdn-1.0.0.tgz : 500 Internal Server Error","errorVerbose":"failed to fetch https://contoso.azurecr.io/helm/v1/repo/_blobs/cdn-1.0.0.tgz : 500 Internal Server Error\nhelm.sh/helm/v3/pkg/getter.(*HTTPGetter).get\n\t/go/pkg/mod/helm.sh/helm/[email protected]/pkg/getter/httpgetter.go:73\nhelm.sh/helm/v3/pkg/getter.(*HTTPGetter).Get\n\t/go/pkg/mod/helm.sh/helm/[email protected]/pkg/getter/httpgetter.go:41\ngithub.com/fluxcd/source-controller/internal/helm.(*ChartRepository).DownloadChart\n\t/workspace/internal/helm/repository.go:175\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).reconcileFromHelmRepository\n\t/workspace/controllers/helmchart_controller.go:312\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).Reconcile\n\t/workspace/controllers/helmchart_controller.go:148\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"}

The state of the Helm repository is :

NAME        URL                                         READY   STATUS                                   AGE
contoso     https://contoso.azurecr.io/helm/v1/repo     True    Fetched revision: 2020-11-03T11:48:23Z   3h50m

If I place my chart and the corresponding index.yaml on a static server (GitHub raw files), and change the URL of my Helm repository, all works fine : the chart is downloaded and and the release installed (and auto upgraded according to SemVer expression).

Propose renaming `GitRepository` Kind to `SourceRepository`

The fact that the source control system can be Git is an implementation detail that I don't believe needs to be in the name of the Kind.

I think more appropriate names for the Kind would be SourceRepository or maybe ManifestRepository which indicate what is contained in the repository as opposed to the source control system used to manage it (which could be a SourceRepository.Spec.Driver field)

Readiness probe should fail if the pod is not the current leader

When running source-controller in HA mode (two or more pods), the readiness probe should fail until the current pod becomes leader. Stand-by replicas must not be added to the LB to avoid "faild to download artifact" errors (ref: fluxcd/flux2#836).

Generate API reference documentation

Besides having a written spec, it would help users if we also generate a reference from our API code. This can be done using "refdocs", as done by the Helm Operator.

• Create make api-docs target to generate reference document to docs/api/source.md
• Run make api-docs in CI so that we ensure they are always up-to-date with code changes

SSH URL parse issue when using Azure DevOps Git source

Attempting to add a git source to an Azure DevOps repository using ssh results invalid port ":v3" after host.

$> flux create source git flux-system \
  --url=ssh://[email protected]:v3/example/personal/fluxv2 \
  --ssh-key-algorithm=rsa \
  --branch=master \
  --interval=1m \
  --git-implementation=libgit2
✗ git URL parse failed: parse "ssh://[email protected]:v3/example/personal/fluxv2": invalid port ":v3" after host

Emit events

Source controller should emit events and integrate with notification controller.

Create events for:

• artifact acquisition failures
• source revision changes

Source controller resource for AWS ECR helm registry

We are currently managing our charts in AWS ECR buckets. We used the helm-ecr to support fetching the chart with the old helm-operator. Is HelmRepository already supporting ECR as a chart source? Is there an example how to use AWS ECR with flux v2?

SemVer of chart for prerelease

I want to install a Helm chart with auto update when new chart versions are available. I would want to install all pre-release versions of 1.x.
So I defined a HelmRelease resource like that :

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: cdn
spec:
  chart:
    spec:
      chart: cdn
      version: '>=1.0.0-0 <2.0.0'
      sourceRef:
        kind: HelmRepository
        name: contoso
        namespace: flux-system

As I want pre-releases, I added the suffix -0 to the version.

On my Helm repository, I have two versions of my chart : 1.0.0-staging1 and 1.1.0-staging1. The list of charts is well pulled by the source controller. Here is a extract of the file /data/helmrepository/flux-system/contoso/index.yaml in the source controller pod :

  cdn:
  - apiVersion: v2
    appVersion: 1.0.0
    created: "2020-11-13T14:32:33.27052869Z"
    digest: cbc12b443972e1cc8694228a78ca3f8fc33e917ce409f27aa36b84e2d44c3ac8
    name: cdn
    urls:
    - https://charts.contoso.local/cdn-1.1.0-staging1.tgz
    version: 1.1.0-staging1
  - apiVersion: v2
    appVersion: 1.0.0
    created: "2020-11-16T22:11:21.539448196Z"
    digest: aec4d24dcce55a3cd4b7d8feddac026689021240b5b884496bc431eb5f44c8b5
    name: cdn
    urls:
    - https://charts.contoso.local/cdn-1.0.0-staging1.tgz
    version: 1.0.0-staging1

But the reconcialiation of the Helm chart never succeeds, as mentionned in the logs of source controller :

{"level":"error","ts":"2020-11-16T22:23:19.460Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"source.toolkit.fluxcd.io","reconcilerKind":"HelmChart","controller":"helmchart","name":"default-cdn","namespace":"flux-system","error":"no chart version found for cdn->=1.0.0-0 <2.0.0"}

Why no chart version is accepted ? Do I have an error in my version range expression ?

Expose funcs/methods for cloning git repository given *GitRepository

In https://github.com/squaremo/image-automation-controller I refer to the GitRepository type for access to a git repository. But it's only reusable as a bag of fields -- the code for cloning a repository is all internal, so I have to duplicate it.

Every exported name is additional API surface to maintain, of course; but this seems pretty uncontroversial code and unlikely to change drastically, since it's tightly coupled to the fields in GitRepository.

git: support .tarignore

When creating artifacts from Git repositories, we could detect if the .tarignore file is present in the root dir and pass to the tar command.

Ref: --exclude-ignore=.tarignore

Watch GitRepository and HelmRepository resources for changes

At present, if someone for example forgets to commit their HelmRepository while creating a HelmChart with a reference to it, and later notices their mistake and commits the forgotten HelmChart. This would take until the next .spec.interval to be noticed by the HelmChart.

To make the source-controller act speedier on such mistakes, the HelmChartReconciler should watch GitRepository and HelmRepository resources for revision changes, and instantly queue a reconciliation for the HelmChart resources that depend on it.

[Bug] missing support for relative URL paths in helm repo chart definitions

jetstack helm repo at https://charts.jetstack.io has this (example) chart entry:

  - appVersion: v0.5.2
    created: "2018-12-11T17:31:41.836Z"
    description: A Helm chart for cert-manager
    digest: 5a2081b9532be8ba5b1cf2f37d812fe679eddb2640eecc297bcd8ebe79cb7719
    home: https://github.com/jetstack/cert-manager
    keywords:
    - cert-manager
    - kube-lego
    - letsencrypt
    - tls
    maintainers:
    - email: [email protected]
      name: munnerz
    name: cert-manager
    sources:
    - https://github.com/jetstack/cert-manager
    urls:
    - charts/cert-manager-v0.5.2.tgz
    version: v0.5.2

Note the relative URL. This makes source-controller choke with this error message:

{"level":"debug","ts":"2020-08-24T22:07:50.565Z","logger":"controller-runtime.manager.events","msg":"Normal","object":{"kind":"HelmChart","namespace":"gitops-system","name":"cert-manager-cert-manager","uid":"fe4ab123-5381-4e18-8c78-5067648e7b91","apiVersion":"source.toolkit.fluxcd.io/v1alpha1","resourceVersion":"11929773"},"reason":"error","message":"scheme \"\" not supported"}
{"level":"error","ts":"2020-08-24T22:07:50.568Z","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"helmchart","name":"cert-manager-cert-manager","namespace":"gitops-system","error":"scheme \"\" not supported","errorVerbose":"scheme \"\" not supported\nhelm.sh/helm/v3/pkg/getter.Providers.ByScheme\n\t/go/pkg/mod/helm.sh/helm/[email protected]/pkg/getter/getter.go:134\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).reconcile\n\t/workspace/controllers/helmchart_controller.go:213\ngithub.com/fluxcd/source-controller/controllers.(*HelmChartReconciler).Reconcile\n\t/workspace/controllers/helmchart_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

This repo and its charts work fine with the helm CLI, flux's helm-operator and rancher's helm-operator. Other repos I've found that trigger this also include https://grafana.github.io/loki/charts

Support for git repository update notifications based on selective file patterns

The custom resource status (transition timestamp and url) should only change when a change occurred to a file that is included by the ignorePatterns specification, see issue #74

Handle Helm chart dependencies

At present, we require chart dependencies to be available in the source for HelmChart resources that make use of a GitRepository or Bucket. This limitation was introduced for the sake of development speed, but we can do better by making use of the HelmRepository resources we have at our disposal in the same namespace as the HelmChart.

The solution would look something like the following (simplified, pseudo and non optimized steps):

1. Index the HelmRepository resources by their (normalized) URL
2. Before building the chart
   1. Look up the HelmRepository resources by their (normalized) URL based on the Chart.yaml data, and load the credentials and the artifact from the resources into a helm.ChartRepository
   2. Assume an "anonymous" HelmRepository for any item we did not find a resource for, create a helm.ChartRepository for those, and download the index from remote into memory using ChartRepository.DownloadIndex
   3. Download the dependency charts using the client and index from each ChartRepository, while taking into account the contents of the charts/ directory
3. Continue with the chart build

Note that this all will likely result in our own version of downloader.Manager, specifically designed for our way of operating and working with Helm.

[Feature Request] No polling of already-fetched references

Hi, I'm working on integrating the source-controller with a CI project of mine. My goal is to fetch specific SHAs once, get the tarball, and then delete the object in the CRD. This would be my ideal workflow; but it may not be realistic with the way the source-controller is currently made, and I'm hoping I can answer that with this ticket.

What I'm worried about is the notion of syncing. I don't want it to call out every time for something that already exists, and ideally, not do any work after the first work is finished and its state is reported.

I took a look at the code; it looks like there's a high likelihood that it won't sync twice for a specific SHA, but might symlink it as "latest.tar.gz" for the repo URL when I request it (which is fine, if it's only managing the one SHA). Additionally, it doesn't seem like I can name multiple repositories with the same URL and not run into conflicts, which is a big problem as new SHAs come in that need to be tested on the same repository.

I may also be confused. :) I talked to @squaremo a bit tonight about it, but would be happy to jump on a call with my use case, too, if anyone is interested in discussing further.

Happy to provide patches to make my use case viable if the idea has merit. This seems like about 95% of what I need to make git+kubernetes doable on my CI system.

Panic with v0.4.1 and k8s 1.19.0

E1130 17:57:48.354795       6 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 172 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x18f1120, 0x2830180)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x18f1120, 0x2830180)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).reconcile(0xc00044a000, 0x1d541a0, 0xc0000b8010, 0x17c37ec, 0xe, 0xc000795bc0, 0x20, 0xc0002c0920, 0x14, 0x0, ...)
    /workspace/controllers/helmrepository_controller.go:199 +0x110
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).Reconcile(0xc00044a000, 0xc00067e79c, 0x4, 0xc0002c0920, 0x14, 0xc000762300, 0x0, 0x0, 0x0)
    /workspace/controllers/helmrepository_controller.go:131 +0x99d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004d2a20, 0x195a2e0, 0xc0006738a0, 0xc00066d200)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244 +0x2a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004d2a20, 0x203000)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0004d2a20)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0007623a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0007623a0, 0x1d15700, 0xc0003f7830, 0x1d15e01, 0xc000094480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007623a0, 0x3b9aca00, 0x0, 0xc0003a2701, 0xc000094480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0007623a0, 0x3b9aca00, 0xc000094480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:179 +0x416
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1771f50]
goroutine 172 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x18f1120, 0x2830180)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).reconcile(0xc00044a000, 0x1d541a0, 0xc0000b8010, 0x17c37ec, 0xe, 0xc000795bc0, 0x20, 0xc0002c0920, 0x14, 0x0, ...)
    /workspace/controllers/helmrepository_controller.go:199 +0x110
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).Reconcile(0xc00044a000, 0xc00067e79c, 0x4, 0xc0002c0920, 0x14, 0xc000762300, 0x0, 0x0, 0x0)
    /workspace/controllers/helmrepository_controller.go:131 +0x99d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004d2a20, 0x195a2e0, 0xc0006738a0, 0xc00066d200)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244 +0x2a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004d2a20, 0x203000)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0004d2a20)
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0007623a0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0007623a0, 0x1d15700, 0xc0003f7830, 0x1d15e01, 0xc000094480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007623a0, 0x3b9aca00, 0x0, 0xc0003a2701, 0xc000094480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0007623a0, 0x3b9aca00, 0xc000094480)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:179 +0x416

I believe it has been introduced by 01d0053, which is weird because this commit is supposed to be in v0.2.2 but I don't have the problem on 2 other clusters running source-controller v0.2.2.

Support Git submodules

In our practice we compose fleet repository from cluster wide manifests and application/team manifests tracked in external repositories. In such scenario fleet repository contains main application kustomization in which external manifests is used as base or referenced somehow else.
As so we need flat file structure of fleet repository with all submodules inited and updated.
It is possible or our use case is invalid?

Provide a prefix filter for the Bucket spec

I would like to use the same S3 bucket accross multiple projects and only give access to a specific prefix for each one of the projects.
Currently, the bucket controller downloads the whole contents of the bucket and the filtering is done on the HelmChart spec or on the Ignore option, but this is only applied after all the contents of the bucket have been downloaded.
Adding a prefix filter would also make the process more efficient.
Thank you

Source Controller artifact status remains unchanged when spec changes

Testing my code that used a GitRepository custom resource and retrieves artifacts from the source controller I changed the GitRepository spec to a nonexistent branch. However the status.artifact did not change...

    "spec": {
        "interval": "1m0s",
        "ref": {
            "branch": "not-present"
        },
        "secretRef": {
            "name": "kraan-http"
        },
        "url": "https://github.com/fidelity/kraan.git"
    },
    "status": {
        "artifact": {
            "checksum": "5285096c6386caeaaedb8cffc6f85d8f2f44e2ae",
            "lastUpdateTime": "2020-11-10T12:27:03Z",
            "path": "gitrepository/gotk-system/addons-config/88c34877fa35ccd9534dfb9b901711ecece213c5.tar.gz",
            "revision": "gh-pages/88c34877fa35ccd9534dfb9b901711ecece213c5",
            "url": "http://source-controller.gotk-system/gitrepository/gotk-system/addons-config/88c34877fa35ccd9534dfb9b901711ecece213c5.tar.gz"
        },
        "conditions": [
            {
                "lastTransitionTime": "2020-11-10T12:32:03Z",
                "message": "unable to clone 'https://github.com/fidelity/kraan.git', error: couldn't find remote ref \"refs/heads/not-present\"",
                "reason": "GitOperationFailed",
                "status": "False",
                "type": "Ready"
            }
        ],
        "observedGeneration": 4
    }

Following the guidelines at https://toolkit.fluxcd.io/dev-guides/source-watcher/ I am watching for changes to GitRepositories and my watch function is being called but per the example it is only looking at status.artifact.revision and not detecting that the git fetch has failed.

Should the source-controller be clearing the artifact sub element when the spec changes or should my code be checking the condition? If latter the dev guide should be updated?

Distribute entire git repository artifacts

Hi,

Unless I'm mistaken the source controller exclude the .git dir from the artifacts it generates.

This way of doing this will limit the ability to make kustomize use local paths using git references when go-getter will have merged its support.

So I'm proposing an optional property for GitRepository which will allows to create artifacts containing all the git data.

Regards.

GitLab self-hosted: git clone unknown error

Hi,

I tried to bootstrap the toolkit components in a GitLab (self-hosted) repository and got the following error in the last step:

$ tk bootstrap gitlab --verbose --namespace=gitops-toolkit --owner=devops --repository=gitops-toolkit --hostname=gitlab.mydomain --ssh-hostname=ssh.gitlab.mydomain
► connecting to gitlab.mydomain
✔ repository cloned
✚ generating manifests
✔ components manifests pushed
► installing components in gitops-toolkit namespace
namespace/gitops-toolkit created
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io created
role.rbac.authorization.k8s.io/crd-controller-gitops-toolkit created
rolebinding.rbac.authorization.k8s.io/crd-controller-gitops-toolkit created
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-gitops-toolkit created
service/notification-controller created
service/source-controller created
service/webhook-receiver created
deployment.apps/helm-controller created
deployment.apps/kustomize-controller created
deployment.apps/notification-controller created
deployment.apps/source-controller created
networkpolicy.networking.k8s.io/deny-ingress created
Waiting for deployment "source-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "source-controller" successfully rolled out
deployment "kustomize-controller" successfully rolled out
deployment "helm-controller" successfully rolled out
deployment "notification-controller" successfully rolled out
✔ install completed
► configuring deploy key
✔ deploy key configured
► generating sync manifests
✔ sync manifests pushed
► applying sync manifests
◎ waiting for cluster sync
✗ git clone error: unknown error: remote:

The status of the GitRepository CR:

$ kubectl get gitrepository gitops-toolkit -o yaml
apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  creationTimestamp: "2020-08-24T09:32:20Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  name: gitops-toolkit
  namespace: gitops-toolkit
  resourceVersion: "61696"
  selfLink: /apis/source.toolkit.fluxcd.io/v1alpha1/namespaces/gitops-toolkit/gitrepositories/gitops-toolkit
  uid: f5029ed7-a07b-46d7-9b64-8a681df30521
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: gitops-toolkit
  url: ssh://[email protected]/devops/gitops-toolkit
status:
  conditions:
  - lastTransitionTime: "2020-08-24T10:44:33Z"
    message: 'git clone error: unknown error: remote: '
    reason: GitOperationFailed
    status: "False"
    type: Ready

As of the source-controller logs:

{"level":"error","ts":"2020-08-24T10:27:52.897Z","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"gitrepository","name":"gitops-toolkit","namespace":"gitops-toolkit","error":"git clone error: unknown error: remote: ","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}

$ tk --version
tk version 0.0.18

Bug: Helm Dependency update occurs even when dependencies are present

The source controller tries to update dependencies for helm charts even when dependencies are present.

Using istio, I turned off all egress traffic not related to the git respository, and saw this error occur:

source-controller-5844b4d7bd-lmw2t manager {"level":"error","ts":"2021-02-02T14:02:10.774Z","logger":"controller.helmchart","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","name":"monitoring","namespace":"monitoring","error":"Get \"https://charts.helm.sh/stable/index.yaml\": read tcp 10.42.3.22:55546->185.199.111.153:443: read: connection reset by peer","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:248\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99"}

when deploying the chart here: https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring/-/tree/main/chart

which has all dependencies locally stored in the charts folder.

Azure Devops SSH URL Invalid Value

This happens when I run the following:

flux create source git flux-system  \
--git-implementation=libgit2   \
--url=ssh.dev.azure.com/v3/andoing/fluxv2-poc/fluxv2-manifests   \
--branch=master   \
--interval=1m

✚ generating GitRepository source
► applying GitRepository source
✗ GitRepository.source.toolkit.fluxcd.io "flux-system" is invalid: spec.url: Invalid value: "": spec.url in body should match '^(http|https|ssh)://'

I got my url syntax straight from the docs.
The default ssh URL provided by Azure DevOps doesn't work either, that's in the form [email protected]. I messed around with the syntax and got ssh://[email protected]/v3 to get past this error, BUT this link doesn't work for actually connecting to the Azure DevOps repo.

Any help is much appreciated.

EDIT:
What I think is required to resolve this issue is for the source controller to support the ssh url syntax provided by Azure Devops, [email protected]/v3/org/project/repo.

Optional TLS cert for Git over HTTPS

When using Git over HTTPS, the controller should look for the cert files inside the credential secret, the same way we do for Helm repositories.

apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  url: https://github.com/stefanprodan/podinfo
  secretRef:
    name: https-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: https-credentials
  namespace: default
type: Opaque
data:
  username: <BASE64> 
  password: <BASE64> 
  certFile: <BASE64>
  keyFile:  <BASE64>
  caFile:   <BASE64>

The certFile, keyFile and caFile should be optional, when present, the controller will use them to connect to the Git HTTPS server.

Azure Git Repo are not working giving a git-upload-pack error

Basically, I tried to add a git repo from azure in different manner, but by the official way, I am stuck with:

{"level":"error","ts":"2020-11-07T04:58:09.548Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"source.toolkit.fluxcd.io","reconcilerKind":"GitRepository","controller":"gitrepository","name":"XXXXX-k8s","namespace":"flux-system","error":"unable to clone 'ssh://ssh.dev.azure.com/XXXXXX/YYYY/MyK8sClusterConf', error: unknown error: remote: Command git-upload-pack '/XXXXXX/YYYY/MyK8sClusterConf' is not in expected format."}

Usually the ssh url is: [email protected]:v3/XXXXXX/YYYY/MyK8sClusterConf
But in Flux2, I had to change the url to: ssh://ssh.dev.azure.com/XXXXXX/YYYY/MyK8sClusterConf

I tried to hack a bit and removed the validation on the URL in order to use the url provided by azure, but I got an another issue.

Does anyone know how to use flux2 with azure?