stolostron/multicloud-operators-subscription

Fork of https://github.com/open-cluster-management-io/multicloud-operators-subscription

multicloud-operators-subscription

• Overview
• Architecture
• Stand-alone deployment
• Multi-cluster deployment
  • Prerequisite
  • Operator Deployment
  • Add-on Deployment
  • What is next
• GitOps subscription
• Object storage subscription
• Community, discussion, contribution, and support

Overview

Subscriptions (subscription.apps.open-cluster-management.io) allow clusters to subscribe to a source repository channel that can be the following types: Git repository, Helm release registry, or Object storage repository.

Subscriptions can point to a channel for identifying new or updated resource templates. The subscription operator can then download directly from the storage location and deploy to targeted managed clusters without checking the hub cluster first. With a subscription, the subscription operator can monitor the channel for new or updated resources instead of the hub cluster.

Architecture

Stand-alone deployment

Deploy the subscription operator.

$ git clone https://github.com/open-cluster-management-io/multicloud-operators-subscription.git
$ cd multicloud-operators-subscription
$ make deploy-standalone
$ kubectl -n open-cluster-management get deploy  multicluster-operators-subscription
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
multicluster-operators-subscription   1/1     1            1           21m

Create a Helm channel and subscribe to it.

kubectl apply -f ./examples/helmrepo-channel

Find the nginx pods that are deployed to the current namespace. You should have 3 backend pods with the controller.

$ kubectl get pods -l app.kubernetes.io/name=ingress-nginx
NAME                                                   READY   STATUS    RESTARTS   AGE
ingress-nginx-simple-controller-84bbdd59f7-ccnwt       1/1     Running   0          4m36s
ingress-nginx-simple-defaultbackend-78669bfbbb-b6rwh   1/1     Running   0          4m36s
ingress-nginx-simple-defaultbackend-78669bfbbb-hb9g6   1/1     Running   0          4m36s
ingress-nginx-simple-defaultbackend-78669bfbbb-nmkng   1/1     Running   0          4m36s

Multi-cluster deployment

Prerequisite

Install the clusteradm CLI tool. For more information see here.

Using clusteradm, deploy a cluster manager on your hub cluster and deploy a klusterlet agent on your managed cluster. For more information see here.

Operator Deployment

Deploy the subscription operator on the hub cluster.

$ kubectl config use-context <hub cluster context> # kubectl config use-context kind-hub
$ clusteradm install hub-addon --names application-manager
$ kubectl -n open-cluster-management get deploy  multicluster-operators-subscription
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
multicluster-operators-subscription   1/1     1            1           25s
$ kubectl -n open-cluster-management wait deploy multicluster-operators-subscription --for condition=available

Add-on Deployment

Enable the subscription add-on for managed cluster. For the value of <managed cluster name>, choose the managed cluster you want to install the add-on to by running the command kubectl get managedclusters on the hub cluster.

$ kubectl config use-context <hub cluster context> # kubectl config use-context kind-hub
$ kubectl get managedclusters
NAME                        HUB ACCEPTED   MANAGED CLUSTER URLS      JOINED   AVAILABLE   AGE
<managed cluster name>      true           https://127.0.0.1:38745   True     True        21s
$ clusteradm addon enable --names application-manager --clusters <managed cluster name> # clusteradm addon enable --names application-manager --clusters cluster1
$ kubectl -n <managed cluster name> get managedclusteraddon # kubectl -n cluster1 get managedclusteraddon
NAME                  AVAILABLE   DEGRADED   PROGRESSING
application-manager   True

Check the the subscription add-on deployment on the managed cluster.

$ kubectl config use-context <managed cluster context> # kubectl config use-context kind-cluster1
$ kubectl -n open-cluster-management-agent-addon get deploy application-manager
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
application-manager   1/1     1            1           103s

What is next

After a successful deployment, test the subscription operator with a helm subscription. Run the following command:

git clone https://github.com/open-cluster-management-io/multicloud-operators-subscription.git
cd multicloud-operators-subscription
kubectl config use-context <hub cluster context> # kubectl config use-context kind-hub
kubectl apply -f examples/helmrepo-hub-channel

After a while, you should see the subscription propagated to the managed cluster and the Helm app installed. By default, when a subscription deploys subscribed applications to target clusters, the applications are deployed to that subscription namespace. To confirm, run the following command:

$ kubectl config use-context <managed cluster context> # kubectl config use-context kind-cluster1
$ kubectl get subscriptions.apps
NAME        STATUS       AGE    LOCAL PLACEMENT   TIME WINDOW
nginx-sub   Subscribed   2m50s   true
$ kubectl get pod
NAME                                                   READY   STATUS    RESTARTS   AGE
ingress-nginx-simple-controller-84bbdd59f7-l7t9f       1/1     Running   0          3m34s
ingress-nginx-simple-defaultbackend-78669bfbbb-d498w   1/1     Running   0          3m34s
ingress-nginx-simple-defaultbackend-78669bfbbb-n47r2   1/1     Running   0          3m34s
ingress-nginx-simple-defaultbackend-78669bfbbb-r2xjn   1/1     Running   0          3m34s

GitOps subscription

You can subscribe to public or enterprise Git repositories that contain Kubernetes resource YAML files or Helm charts, or both. See Git repository channel subscription for more details.

Object storage subscription

You can subscribe to cloud object storage that contain Kubernetes resource YAML files. See Object storage channel subscription for more details.

Community, discussion, contribution, and support

Check the CONTRIBUTING Doc for how to contribute to the repo.

Communication channels

Slack channel: #open-cluster-mgmt

License

This code is released under the Apache 2.0 license. See the file LICENSE for more information.

multicloud-operators-subscription's Issues

Unexpected log for a GitHub channel

We've created the following appsub:

apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  name: cnv-appsub
  namespace: kubevirt
  annotations:
    apps.open-cluster-management.io/github-path: kubevirt
    apps.open-cluster-management.io/github-branch: master
spec:
  channel: ec/ec-channel
  placement:
    placementRef:
      kind: PlacementRule
      name: cnf-placement

That refers to the following channel of type GitHub:

apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
  name: ec-channel
  namespace: ec
spec:
  type: GitHub
  pathname: https://github.com/danielerez/ocm-gitops-policy.git

And we see the following log:
W0421 17:06:26.798052 1 helmrepo.go:373] subsciption.spec.name is missing for subscription: ec/cnv-appsub

This looks weird as cnv-appsub has nothing to do with helm...

😿 Failed to sync the upstream PRs: #228, #231, #232

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#228
• open-cluster-management-io/multicloud-operators-subscription#231
• open-cluster-management-io/multicloud-operators-subscription#232

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 54fbd317f22ab072fe68488ff2375720465a844a~13..54fbd317f22ab072fe68488ff2375720465a844a --allow-empty --keep-redundant-commits
git cherry-pick -x d0e4d0d719f114a6069b4e21cbd4457af5036544~2..d0e4d0d719f114a6069b4e21cbd4457af5036544 --allow-empty --keep-redundant-commits
git cherry-pick -x 267a21a7eecd07e023f2a25bd9f4cff93e3fb6f8~1..267a21a7eecd07e023f2a25bd9f4cff93e3fb6f8 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #263

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#263

The pull-request (#848) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

[RFE] Creating applications with custom namespace in Git

As today, If one wants to define a custom namespace for the objects deployed using a subscription pointing to a GitHub Channel, the subscription controller overrides the destination namespace to honour the one where the subscription was created (https://github.com/open-cluster-management/multicloud-operators-subscription/blob/master/pkg/subscriber/github/subscriber_item.go#L445-L447).

I would like the subscription controller to not override the namespace if the objects have a namespace already defined in their yaml definition. Or at least make it configurable thru the subscription, so the user can decide if namespaces are overridden by the controller.

Thanks in advance,

😿 Failed to sync the upstream PRs: #301

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#301

The pull-request (#906) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #294

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#294

The pull-request (#892) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #259

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#259

The pull-request (#842) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #316

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#316

The pull-request (#930) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Appsubs fail to apply CRDs: schemas are required

We've defined an appsub that applies CRD with:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    operator.kubevirt.io: ""
  name: kubevirts.kubevirt.io
spec:
  additionalPrinterColumns:
  - JSONPath: .metadata.creationTimestamp
    name: Age
    type: date
  - JSONPath: .status.phase
    name: Phase
    type: string
  group: kubevirt.io
  names:
    categories:
    - all
    kind: KubeVirt
    plural: kubevirts
    shortNames:
    - kv
    - kvs
    singular: kubevirt
  scope: Namespaced
  version: v1alpha3
  versions:
  - name: v1alpha3
    served: true
    storage: true

It failed due to:
E0421 17:05:01.346887 1 synchronizer.go:461] Failed to apply kind template&{map[apiVersion:apiextensions.k8s.io/v1 kind:CustomReso urceDefinition metadata:map[annotations:map[apps.open-cluster-management.io/hosting-deployable:ec/ec-channel-CustomResourceDefinition-ku bevirts.kubevirt.io apps.open-cluster-management.io/hosting-subscription:ec/cnv-appsub apps.open-cluster-management.io/sync-source:subgb k8s-ec/cnv-appsub] labels:map[operator.kubevirt.io:] name:kubevirts.kubevirt.io] spec:map[additionalPrinterColumns:[map[JSONPath:.metada ta.creationTimestamp name:Age type:date] map[JSONPath:.status.phase name:Phase type:string]] group:kubevirt.io names:map[categories:[all ] kind:KubeVirt plural:kubevirts shortNames:[kv kvs] singular:kubevirt] scope:Namespaced version:v1alpha3 versions:[map[name:v1alpha3 se rved:true storage:true]]]]}with error:CustomResourceDefinition.apiextensions.k8s.io "kubevirts.kubevirt.io" is invalid: spec.versions[0] .schema.openAPIV3Schema: Required value: schemas are required

by changing the versions to:

versions:
  - name: v1alpha3
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              cronSpec:
                type: string
              image:
                type: string
              replicas:
                type: integer

It works but it doesn't sound right to require this as k8s doesn't require it.

Support for multi-namespace application

Issues we expect the ongoing effort to enable defining an appsub that is able to apply resources on various namespaces to solve:

1. Having one appsub that retrieves other appsubs from a git repository - this way OCM can define one appsub for essential configurations and the other appsubs would be configured within the git repo (e.g., for CNV, CNF).

2. Support appsubs that apply resources on various namespaces - OCM currently seems to support appsubs that define resources in a namespace that is associated with it, the default namespace and non-namespaced resources. But we need to support CNF's performance-addon-operator that applies non-namespaced resources as well as resources in the operator's namespace (openshift-performance-addon) and in the openshift-marketplace namespace.

@jnpacker fyi

😿 Failed to sync the upstream PRs: #312

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#312

The pull-request (#924) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #317

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the pull-request (#932) couldn't be merged: HttpError: Pull Request is not mergeable:

• open-cluster-management-io/multicloud-operators-subscription#317

The pull-request (#932) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #308

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#308

The pull-request (#916) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Creating a subscription shows failure in operator log

I have deployed a hub cluster (named hub) and a managed cluster (named cluster0) that has joined to it.

[rwsu@mini-f31 rwsu]$ kubectl get clusterjoinrequests.mcm.ibm.com
NAME                                                      CLUSTER NAME   CLUSTER NAMESPACE   STATUS     AGE
clusterjoin-KWv601Zh1icnNOcAQ9fW2QLj8_IxhIM54woqj2vgOFk   cluster0       cluster0            Approved   1m

[rwsu@mini-f31 rwsu]$ kubectl get cluster --all-namespaces
NAMESPACE   NAME       ENDPOINTS            STATUS   AGE
cluster0    cluster0   192.168.39.85:8443   Ready    22s

The multicloud channel, deployable, subscription, and placementrule operators have been installed on the hub cluster.

And the multicloud subscription operator has been installed on the managed cluster.

All the operators except the subscription operator were deployed by checking out their git repo and applyng the following:
kubectl apply -f deploy/crds
kubectl apply -f deploy

The subscription operator was deployed using this command with --standalone commented out.
kubectl apply -f deploy/standalone

All operators are using the latest images from quay.io/multicloudlab/.

My goal is to create a deployable on the hub cluster and have the underlying resource, a secret, be deployed on the managed cluster. From my understanding, this can be done by (1) creating a namespace type Channel, (2) creating a Subscription mapping the Channel to the managed cluster, and (3) creating a Deployable in the namespace defined in the Channel.

After creating the namespace, Channel, Subscription, and Deployable, I don't see the Secret that is wrapped by the Deployable on the managed cluster. I see the following:

[rwsu@mini-f31 rwsu]$ cat <<EOF | kubectl create -f -
> apiVersion: v1
> kind: Namespace
> metadata:
>   name: namespace-for-cluster0
> EOF
namespace/namespace-for-cluster0 created

[rwsu@mini-f31 rwsu]$ cat <<EOF | kubectl create -f -
> apiVersion: app.ibm.com/v1alpha1
> kind: Channel
> metadata:
>   name: channel-for-cluster0
>   namespace: default
> spec:
>   sourceNamespaces:
>   - namespace-for-cluster0
>   type: Namespace
>   pathname: default
> EOF
channel.app.ibm.com/channel-for-cluster0 created
[rwsu@mini-f31 rwsu]$ kubectl get channels
NAME                   TYPE        PATHNAME   AGE
channel-for-cluster0   Namespace   default    5s

[rwsu@mini-f31 rwsu]$ cat <<EOF | kubectl create -f -
> apiVersion: app.ibm.com/v1alpha1
> kind: Subscription
> metadata:
>   name: subscription-for-cluster0
>   namespace: default
> spec:
>   channel: default/channel-for-cluster0
>   placement:
>     clusters:
>     - name: cluster0
> EOF
subscription.app.ibm.com/subscription-for-cluster0 created
[rwsu@mini-f31 rwsu]$ kubectl get appsub
NAME                        STATUS       AGE
subscription-for-cluster0   Propagated   12s

[rwsu@mini-f31 rwsu]$ cat <<EOF | kubectl create -f -
> apiVersion: app.ibm.com/v1alpha1
> kind: Deployable
> metadata:
>   annotations:
>     app.ibm.com/is-local-deployable: "false"
>   name: deployable-containing-secret
>   namespace: namespace-for-cluster0
> spec:
>   template:
>     apiVersion: v1
>     kind: Secret
>     metadata:
>       name: test-secret
>       namespace: default
>     data:
>       username: YWRtaW4=
>       password: MWYyZDFlMmU2N2Rm
> EOF
deployable.app.ibm.com/deployable-containing-secret created
[rwsu@mini-f31 rwsu]$ kubectl get deployables -A
NAMESPACE                NAME                                         TEMPLATE-KIND   TEMPLATE-APIVERSION    AGE   STATUS
cluster0                 subscription-for-cluster0-deployable-fkgnx   Subscription    app.ibm.com/v1alpha1   98s   
default                  subscription-for-cluster0-deployable         Subscription    app.ibm.com/v1alpha1   98s   Propagated
namespace-for-cluster0   deployable-containing-secret                 Secret          v1                     31s   

The subscription operator on the hub cluster shows a failure:

I0117 14:58:18.478217 1 manager.go:54] Go Version: go1.13
I0117 14:58:18.479616 1 manager.go:55] Go OS/Arch: linux/amd64
I0117 14:58:18.480256 1 manager.go:56] Version of operator-sdk: v0.12.0
I0117 14:58:18.847043 1 manager.go:105] Starting ... Registering Components for cluster: /
I0117 14:58:22.263957 1 manager.go:131] Could not generate and serve custom resource metricserrordiscovering resource information failed for HelmChartSubscription in app.ibm.com/v1alpha1: apiVersion app.ibm.com/v1alpha1 and kind HelmChartSubscription not found available in Kubernetes cluster
time="2020-01-17T14:58:23Z" level=info msg="Could not create ServiceMonitor objecterrorno ServiceMonitor registered with the API" source="manager.go:161"
I0117 14:58:23.248187 1 manager.go:165] Install prometheus-operator in your cluster to create ServiceMonitor objectserrorno ServiceMonitor registered with the API
I0117 14:58:23.248209 1 manager.go:169] Starting the Cmd.
I0117 15:06:20.192521 1 mcmhub_controller.go:183] MCM Hub Reconciling subscription: default/subscription-for-cluster0
I0117 15:06:20.717145 1 mcmhub_controller.go:239] MCM Hub updating subscriptoin status to {Propagated 0001-01-01 00:00:00 +0000 UTC map[]}
I0117 15:06:20.721281 1 event.go:258] Event(v1.ObjectReference{Kind:"Subscription", Namespace:"default", Name:"subscription-for-cluster0", UID:"c16ec797-2e65-4fea-8aa2-3e12c73cbd4d", APIVersion:"app.ibm.com/v1alpha1", ResourceVersion:"1835", FieldPath:""}): type: 'Normal' reason: 'Deploy' Depolyable default/subscription-for-cluster0-deployable created in the subscription namespace for deploying the subscription to managed clusters
I0117 15:06:20.727955 1 mcmhub_controller.go:183] MCM Hub Reconciling subscription: default/subscription-for-cluster0
I0117 15:06:20.728259 1 hub.go:459] subscription updated for default/subscription-for-cluster0 new deployables:default/subscription-for-cluster0-deployable
I0117 15:06:20.746397 1 hub.go:132] Updating Subscription failed. subscription: &v1alpha1.Subscription{TypeMeta:v1.TypeMeta{Kind:"Subscription", APIVersion:"app.ibm.com/v1alpha1"}, ObjectMeta:v1.ObjectMeta{Name:"subscription-for-cluster0", GenerateName:"", Namespace:"default", SelfLink:"/apis/app.ibm.com/v1alpha1/namespaces/default/subscriptions/subscription-for-cluster0", UID:"c16ec797-2e65-4fea-8aa2-3e12c73cbd4d", ResourceVersion:"1835", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63714870380, loc:(*time.Location)(0x35e1820)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"app.ibm.com/deployables":"default/subscription-for-cluster0-deployable"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1alpha1.SubscriptionSpec{Channel:"default/channel-for-cluster0", Package:"", PackageFilter:(*v1alpha1.PackageFilter)(nil), PackageOverrides:[]*v1alpha1.Overrides(nil), Placement:(*v1alpha1.Placement)(0xc000980fc0), Overrides:[]v1alpha1.Overrides(nil), TimeWindow:(*v1alpha1.TimeWindow)(nil)}, Status:v1alpha1.SubscriptionStatus{Phase:"", Message:"", Reason:"", LastUpdateTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Statuses:v1alpha1.SubscriptionClusterStatusMap(nil)}}, error: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"Operation cannot be fulfilled on subscriptions.app.ibm.com "subscription-for-cluster0": the object has been modified; please apply your changes to the latest version and try again", Reason:"Conflict", Details:(*v1.StatusDetails)(0xc000a86420), Code:409}}
I0117 15:06:20.746626 1 mcmhub_controller.go:239] MCM Hub updating subscriptoin status to {Failed Operation cannot be fulfilled on subscriptions.app.ibm.com "subscription-for-cluster0": the object has been modified; please apply your changes to the latest version and try again 0001-01-01 00:00:00 +0000 UTC map[]}
E0117 15:06:20.751457 1 mcmhub_controller.go:246] Failed to update status for mcm hub subscription default/subscription-for-cluster0 with error: Operation cannot be fulfilled on subscriptions.app.ibm.com "subscription-for-cluster0": the object has been modified; please apply your changes to the latest version and try again retry after 1 seconds
I0117 15:06:20.751582 1 mcmhub_controller.go:183] MCM Hub Reconciling subscription: default/subscription-for-cluster0
I0117 15:06:20.751922 1 hub.go:459] subscription updated for default/subscription-for-cluster0 new deployables:default/subscription-for-cluster0-deployable
I0117 15:06:20.876197 1 mcmhub_controller.go:239] MCM Hub updating subscriptoin status to {Propagated 2020-01-17 15:06:20.876183004 +0000 UTC m=+482.616752210 map[]}
I0117 15:06:20.884864 1 mcmhub_controller.go:183] MCM Hub Reconciling subscription: default/subscription-for-cluster0
I0117 15:06:20.885849 1 mcmhub_controller.go:239] MCM Hub updating subscriptoin status to {Propagated 2020-01-17 15:06:20.885832455 +0000 UTC m=+482.626401704 map[]}
I0117 15:06:20.936378 1 mcmhub_controller.go:183] MCM Hub Reconciling subscription: default/subscription-for-cluster0
I0117 15:06:20.937433 1 mcmhub_controller.go:239] MCM Hub updating subscriptoin status to {Propagated 2020-01-17 15:06:20.937412642 +0000 UTC m=+482.677981858 map[cluster0:0xc0000a8650]}
I0117 15:06:21.751775 1 mcmhub_controller.go:183] MCM Hub Reconciling subscription: default/subscription-for-cluster0

I am having trouble understanding the failure.

Is my subscription spec incorrect?

Both the hub and managed cluster shows this line

I0117 14:58:18.847043 1 manager.go:105] Starting ... Registering Components for cluster: /

Looking at the code (https://github.com/IBM/multicloud-operators-subscription/blob/master/cmd/manager/exec/manager.go#L105), the cluster namespace and cluster name should appear on this line.

Have I configured the subscription operator incorrectly?

"make build" fails with +incompatible: invalid pseudo-version: preceding tag (v3.9.0) not found

I'm on Fedora 31 and using go version go1.13.5 linux/amd64.

[rwsu@mini-f31 multicloud-operators-subscription]$ make build
+ OUT=build/_output/bin/multicloud-operators-subscription
+ shift
+ set -e
+ BUILD_GOOS=linux
+ BUILD_GOARCH=amd64
+ GOBINARY=go
+ BUILDINFO=
+ STATIC=1
+ LDFLAGS='-extldflags -static'
+ GOBUILDFLAGS=
+ IFS=' '
+ read -r -a GOBUILDFLAGS_ARRAY
+ GCFLAGS=
+ export CGO_ENABLED=0
+ CGO_ENABLED=0
+ [[ 1 != \1 ]]
+ GOOS=linux
+ GOARCH=amd64
+ go build -x -o build/_output/bin/multicloud-operators-subscription -ldflags '-extldflags -static' ./cmd/manager
WORK=/tmp/go-build398595899
mkdir -p /home/rwsu/go/pkg/mod/cache/vcs # git3 https://github.com/openshift/api
# lock /home/rwsu/go/pkg/mod/cache/vcs/aec9f97d8120c1be2931401817e2528afb2e0d4b45f89114a59bf32585b17b5a.lock# /home/rwsu/go/pkg/mod/cache/vcs/aec9f97d8120c1be2931401817e2528afb2e0d4b45f89114a59bf32585b17b5a for git3 https://github.com/openshift/api
cd /home/rwsu/go/pkg/mod/cache/vcs/aec9f97d8120c1be2931401817e2528afb2e0d4b45f89114a59bf32585b17b5a; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 77b8897ec79a --
0.031s # cd /home/rwsu/go/pkg/mod/cache/vcs/aec9f97d8120c1be2931401817e2528afb2e0d4b45f89114a59bf32585b17b5a; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 77b8897ec79a --
cd /home/rwsu/go/pkg/mod/cache/vcs/aec9f97d8120c1be2931401817e2528afb2e0d4b45f89114a59bf32585b17b5a; git ls-remote -q https://github.com/openshift/api
1.133s # cd /home/rwsu/go/pkg/mod/cache/vcs/aec9f97d8120c1be2931401817e2528afb2e0d4b45f89114a59bf32585b17b5a; git ls-remote -q https://github.com/openshift/api
go: github.com/operator-framework/[email protected] requires
	github.com/operator-framework/[email protected] requires
	github.com/openshift/[email protected]+incompatible: invalid pseudo-version: preceding tag (v3.9.0) not found

real	0m1.203s
user	0m0.186s
sys	0m0.049s
make: *** [Makefile:118: build] Error 1

😿 Failed to sync the upstream PRs: #236

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the pull-request (#808) couldn't be merged: HttpError: Pull Request is not mergeable:

• open-cluster-management-io/multicloud-operators-subscription#236

The pull-request (#808) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #235

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#235

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 065c92bec507fad83c19e814390e879ffb0de406~28..065c92bec507fad83c19e814390e879ffb0de406 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #269

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#269

The pull-request (#862) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Provide more detail for how to get linting tools

In https://github.com/IBM/multicloud-operators-subscription/blob/master/docs/development.md#development-guide , the linting tools only include tools but does not include link where to get those tools, we should provide links for how to get those tools.

/assign @kuanf

FYI @itdove

😿 Failed to sync the upstream PRs: #299

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#299

The pull-request (#902) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Entities in (policy.)mcm.ibm.com group are rejected

The following scenario that used to work with previous versions of OCM:

1. Have a local appsub with a github channel
2. Store policies and placementbindings in the repo
3. The policies and placementbindings would be added to the hub cluster.

Doesn't work with the latest version, it fails with this error

@mdelder @jnpacker fyi

😿 Failed to sync the upstream PRs: #267

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#267

The pull-request (#858) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Add a label that will leave subscription created objects when it is removed

Going to add a label to the subscription that will not remove resources when the subscription itself is deleted.

subscription-delete-resources: true/false

I'm open to a better name. The default is "true"

Subscriptions are propagated with finalizers in v2.3

When using v2.3 In noticed that when propagating subscriptions they're propagated with their finalizers. This results with the subscribed objects staying on the mc after the subscription was deleted on the hub itself. From checking v2.4 code I think that the issue is still there as well.

Added a pr here:
#623

😿 Failed to sync the upstream PRs: #286

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#286

The pull-request (#884) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #249

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#249

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x ba1906183da1a10ddd2ef09d57077658ba014e90~1..ba1906183da1a10ddd2ef09d57077658ba014e90 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #277

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#277

The pull-request (#868) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #266

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#266

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 7e86faa64705cbae989f40a65f573a41e0d506c5~1..7e86faa64705cbae989f40a65f573a41e0d506c5 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #251

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#251

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 086534d92801bf147acfe082a343c8fd98b7dd5e~4..086534d92801bf147acfe082a343c8fd98b7dd5e --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #303

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#303

The pull-request (#910) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #262

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#262

The pull-request (#851) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

HelmRepo channel and subscriptions do not support relative urls in helm chart index.yaml

we have been testing against multiple helm chart repositories and it appears that charts repos that specify relative url's in their index.yaml cannot be deployed by open cluster manager

nexus for example creates relative urls

-- nexus helm-chart index.yaml

apiVersion: v1
entries:
  pet-battle-api:
  - appVersion: 1.0-SNAPSHOT
    created: 2020-05-18T04:45:49.800Z
    description: A Helm chart for Kubernetes
    digest: 414ad02c1e957ad078ec069c4d64e4b32a45bc3526d38f1a40e942c7807c1537
    maintainers:
    - name: eformat
    name: pet-battle-api
    urls:
    - pet-battle-api-1.0.0.tgz
    version: 1.0.0
generated: 2020-05-22T01:23:13.395Z

-- helm generated index.yaml

apiVersion: v1
entries:
  pet-battle-api:
  - apiVersion: v2
    appVersion: 1.0.0
    created: "2020-05-22T12:36:52.297103117+10:00"
    description: A Helm chart for Kubernetes
    digest: 73bb90bc3e1f517eb4d97be5184d0abaf245a19c1984586792f76bb38c592421
    home: https://eformat.github.io/helm-charts
    maintainers:
    - name: eformat
    name: pet-battle-api
    type: application
    urls:
    - https://eformat.github.io/helm-charts/pet-battle-api-1.0.0.tgz
    version: 1.0.0
generated: "2020-05-22T12:36:52.283165021+10:00"    

it appears that helm3 supports either type using the --url flag when creating the index.yaml

when using the nexus repo - the error in ocm tracks back to here:

{Irreconcilable True ReconcileError failed to download chart from helm repo. - url: pet-battle-api-1.0.0.tgz error: unsupported scheme 2020-05-21 23:50:58 +0000 UTC}] }}

https://github.com/open-cluster-management/multicloud-operators-subscription-release/blob/master/pkg/utils/helmrepoutils.go#L240

given the Channel definition already specifies the helm repo pathname, is it possible to support relative urls in ocm ?

Support insecure-skip-verify while access a git server in subscription controller

This is a customer requirement against MCM 1.3 and the channel controller has been added this feature.
In my fork of this repo, I fixed the issue through this commit frankwg@3a88e87.

Is it possible to create a branch name 'ibm-3.3.0' from the commit 2602450, so, I can raise a PR to merge my change to it?

😿 Failed to sync the upstream PRs: #240

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#240

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x fd29e047a161c0ff5f39fa9b14c2f9db70b4c0f0~24..fd29e047a161c0ff5f39fa9b14c2f9db70b4c0f0 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #298

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#298

The pull-request (#899) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #310

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#310

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x c3d980d75d3340f1440b49b692134bbd9d5daaaa~1..c3d980d75d3340f1440b49b692134bbd9d5daaaa --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #260

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#260

The pull-request (#844) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Not all resources are removed upon appsub removal

When adding an appsub, all its resources are given an annotation that specifies their relation with that appsub, so they will also be removed when removing the appsub.

This is fine in the general case but might fail when the application itself spawns other resources like pods without propagating that annotation to the spawned resources.

One may argue that when removing the appsub the appsub should only be responsible to remove its direct resources and the application is responsible to remove the spawned resources; or that the application is responsible to propagate the aforementioned annotation into its spawned resources. But this is hard to enforce.

A possible solution for this may be to remove all entities within the appsub's namespace, no matter if they have that annotation or not. Need to be careful not to remove things that will be part of the admin-namespace (for multi-namespace appsubs) though.

multicluster-operators-subscription image does not have git installed

When using Kustomize it's possible to point to git repos in order to get the base files.

eg: https://github.com/mvazquezc/acm-testing/blob/stage/apps/reversewords/kustomization.yaml#L3

When the App Manager tries to run the kustomization it fails with the following error:

E0330 16:23:55.208403       1 subscriber_item.go:252] Failed to applying kustomization, error: couldn't make loader for github.com/mvazquezc/acm-testing/apps/reversewords?ref=config: no 'git' program on path: exec: "git": executable file not found in $PATH

I patched the image adding git to it, and after that, it worked as expected:

I0330 16:54:46.398444       1 event.go:255] Event(v1.ObjectReference{Kind:"Subscription", Namespace:"gitops-apps", Name:"reversewords-dev-app-subscription", UID:"b0b2b9fd-344a-4f20-b76f-7de68f8b8f81", APIVersion:"apps.open-cluster-management.io/v1", ResourceVersion:"85925", FieldPath:""}): type: 'Normal' reason: 'CreateResource' Synchronizer created resource reversewords-dev-app-subscription of gvk:apps.open-cluster-management.io/v1, Kind=Subscription
I0330 16:54:46.409408       1 subscription_controller.go:134] Standalone/Endpoint Reconciling subscription: gitops-apps/reversewords-dev-app-subscription
I0330 16:54:46.414057       1 subscriber_item.go:100] Polling on SubscriberItem reversewords-dev-app-subscription
I0330 16:54:46.414315       1 subscriber_item.go:826] Subscribing to branch: refs/heads/stage
I0330 16:54:46.427998       1 subscription_controller.go:134] Standalone/Endpoint Reconciling subscription: gitops-apps/reversewords-dev-app-subscription
I0330 16:54:47.534023       1 subscriber_item.go:223] Applying kustomization /tmp/gitops-apps/mvazquez-gitops-github/apps/reversewords/

😿 Failed to sync the upstream PRs: #296

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#296

The pull-request (#896) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #314

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#314

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 92ebae0b2354c3c78f4318e8359cc8e4caf005c5~1..92ebae0b2354c3c78f4318e8359cc8e4caf005c5 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #276

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#276

The pull-request (#870) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

😿 Failed to sync the upstream PRs: #250

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#250

The pull-request (#832) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Need to have performance test in place

With the refactor of the synchronizer, we feel that we need a way to tell the performance differences between the new and old synchronizers.

Speaking of the performance, we are talking about the deployment delay between a batch of subscriptions. For instance,

If we apply 100 subscriptions at the same time, then,
a. would the code be able to handle these many subscriptions at once?(would all the subscriptions be processed eventually?)
b. if the first subscription deployed at t=0s, would the last subscription deploy at t=1hr?

In addition, we are assuming

1. the test is running on 1 hub and 1 managed clusters(meaning the kubeconfig of both clusters will be provided) and we are measuring the result on the managed cluster side.
2. each subscription is deployed via namespace channel
3. each subscription is targeted to deploy a configmap at the spoke cluster

@xiangjingli
@rokej
@mikeshng
@evelinec

If you guys have any other ideas please also let me know.

Appsub requires an ordered list of resources

The current mechanism assumes entities to be defined are given in a logical order, i.e., a resource is identified after the resources it depends on were identified. This doesn't hold when the application is composed of several (yaml) files.

For instance, let's say that a CR is defined in one file and its CRD is defined in another file. It may be the case that OCM will try to apply the CR from the first file before applying the CRD from the other file.

This can be solved by adding a random mechanism by which OCM tries to apply resources that spread across different files.

😿 Failed to sync the upstream PRs: #264, #265

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#264
• open-cluster-management-io/multicloud-operators-subscription#265

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 958635a3c4ad475d44a4f5745a14fe9ed91bf77d~1..958635a3c4ad475d44a4f5745a14fe9ed91bf77d --allow-empty --keep-redundant-commits
git cherry-pick -x 6acfbbf34d6993ee83e35a1f5b049168547bbb18~1..6acfbbf34d6993ee83e35a1f5b049168547bbb18 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #320

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#320

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x 97e7432837758a3540b85f4c9f720c6dbea7b468~1..97e7432837758a3540b85f4c9f720c6dbea7b468 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #229

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#229

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x e139ae083aec43e31cba45c8990d640dff8eda13~15..e139ae083aec43e31cba45c8990d640dff8eda13 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #282, #284, #283

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because one or more patches couldn't cleanly apply:

• open-cluster-management-io/multicloud-operators-subscription#282
• open-cluster-management-io/multicloud-operators-subscription#284
• open-cluster-management-io/multicloud-operators-subscription#283

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.

Commands to recreate the issue:

git cherry-pick -x f596c46d2b79871d0acae07a33e353b8a9ec0fb7~1..f596c46d2b79871d0acae07a33e353b8a9ec0fb7 --allow-empty --keep-redundant-commits
git cherry-pick -x 9fa64624224d5a9e784bfb9d9aeb1f5b2a4599ce~1..9fa64624224d5a9e784bfb9d9aeb1f5b2a4599ce --allow-empty --keep-redundant-commits
git cherry-pick -x 8cf3a227f2e6314c4861ff1206e235193c8ce634~1..8cf3a227f2e6314c4861ff1206e235193c8ce634 --allow-empty --keep-redundant-commits

😿 Failed to sync the upstream PRs: #280

🪞 Magic Mirror 🪞 failed to sync the following upstream pull-requests because the PR CI failed:

• open-cluster-management-io/multicloud-operators-subscription#280

The pull-request (#877) can be reviewed for more information.

Syncing is paused for the branch main on stolostron/multicloud-operators-subscription until the issue is manually resolved and this issue is closed.