loft-sh / vcluster Goto Github PK

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.

Home Page: https://www.vcluster.com

License: Apache License 2.0

Dockerfile 0.17% Go 98.17% Shell 0.31% Just 0.19% Smarty 1.17%

kubernetes vcluster virtual-clusters multi-tenancy cloud-native platform-engineering helm kubectl k8s k3s

vcluster's Introduction

Website • Quickstart • Documentation • Blog • Twitter • Slack

Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.

Why Virtual Kubernetes Clusters?

Cluster Scoped Resources: much more powerful than simple namespaces (virtual clusters allow users to use CRDs, namespaces, cluster roles etc.)
Ease of Use: usable in any Kubernetes cluster and created in seconds either via a single command or cluster-api
Cost Efficient: much cheaper and efficient than "real" clusters (single pod and shared resources just like for namespaces)
Lightweight: built upon the ultra-fast k3s distribution with minimal overhead per virtual cluster (other distributions work as well)
Strict isolation: complete separate Kubernetes control plane and access point for each vcluster while still being able to share certain services of the underlying host cluster
Cluster Wide Permissions: allow users to install apps which require cluster-wide permissions while being limited to actually just one namespace within the host cluster
Great for Testing: allow you to test different Kubernetes versions inside a single host cluster which may have a different version than the virtual clusters

Learn more on www.vcluster.com.

Learn more in the documentation.

⭐️ Do you like vcluster? Support the project with a star ⭐️

Features

Certified Kubernetes Distribution - vcluster itself is a certified Kubernetes distribution and is 100% Kubernetes API conform. Everything that works in a regular Kubernetes cluster works in vcluster
Lightweight & Low-Overhead - Based on k3s, bundled in a single pod and with super-low resource consumption. Other distributions such as k0s or vanilla k8s are also supported
No Performance Degradation - Pods are scheduled in the underlying host cluster, so they get no performance hit at all while running
Reduced Overhead On Host Cluster - Split up large multi-tenant clusters into smaller vclusters to reduce complexity and increase scalability. Since most vcluster api requests and objects will not reach the host cluster at all, vcluster can greatly decrease pressure on the underlying Kubernetes cluster
Easy Provisioning - Create via vcluster CLI, helm, kubectl, cluster api, Argo CD or any of your favorite tools (it is basically just a StatefulSet)
No Admin Privileges Required - If you can deploy a web app to a Kubernetes namespace, you will be able to deploy a vcluster as well
Single Namespace Encapsulation - Every vcluster and all of its workloads are inside a single namespace of the underlying host cluster
Easy Cleanup - Delete the host namespace and the vcluster plus all of its workloads will be gone immediately
Flexible & Versatile - vcluster supports different storage backends (such as sqlite, mysql, postgresql & etcd), plugins, customizable sync behaviour, vcluster within vcluster setups and has many more additional configuration options to fit a multitude of different use cases

Quick Start (~ 1 minute)

To learn more about vcluster, open the full getting started guide.

1. Download vcluster CLI

VCluster has a hard dependency on helm (v3.10.0+), if it is not installed on your machine, you can install it beforehand or let vCluster install it for you.

Use one of the following commands to download the vcluster CLI binary from GitHub:

Mac (Intel/AMD)

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin

Mac (Silicon/ARM)

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin

Linux (AMD)

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin

Linux (ARM)

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin

Windows (Powershell)

md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';
Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe;
$env:Path += ";" + $Env:APPDATA + "\vcluster";
[Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);

If you get the error that Windows cannot find vcluster after installing it, you will need to restart your computer, so that the changes to the PATH variable will be applied.

Alternatively, you can download the binary for your platform from the GitHub Releases page and add this binary to your PATH.

2. Create a vcluster

vcluster create my-vcluster

# OR: Use --expose to create a vcluster with an externally accessible LoadBalancer
vcluster create my-vcluster --expose

# OR: Use --isolate to create an isolated environment for the vcluster workloads
vcluster create my-vcluster --isolate

Take a look at the vcluster docs to see how to deploy a vcluster using Helm or Kubectl instead.

3. Use the vcluster

Run in a terminal:

# Run any kubectl, helm, etc. command in your vcluster
kubectl get namespace
kubectl get pods -n kube-system
kubectl create namespace demo-nginx
kubectl create deployment nginx-deployment -n demo-nginx --image=nginx
kubectl get pods -n demo-nginx

4. Cleanup

vcluster delete my-vcluster

Alternatively, you could also delete the host-namespace using kubectl.

Architecture

Contributing

Thank you for your interest in contributing! Please refer to CONTRIBUTING.md for guidance.

This project is open-source and licensed under Apache 2.0, so you can use it in any private or commercial projects.

vcluster's People

Contributors

Stargazers

Watchers

Forkers

lukasgentele richburroughs cloudxo eddie-he fpy-loft-sh samkitk developgo fabiankramm l0ckpix aslafy-z escap-data-hub swipswaps danpon rogervaas syllogy hien brunobva tpiperatgod devopstoday11 ab-hishek misteruly googlebabaio morningsong matskiv danielchudc mcbenjemaa mevech reloadbrain henriaw zhihanz fossabot magnologan cjkennedy1972 johsonluo thiruselvaa zhm022726 elifish4 ploch cndoit18 casualuser huangyingting thesanedenis developer-guy yydzhou neotheelder xiv rmb938 garysu nasermirzaei89 doytsujin moustafab hasheddan crd477 woshizilong jeloba jxt518 rkrmr33 mixitgit olljanat tannevaled eightzero moteesh-reddy janwillies pratikjagrut osmanatam 1337andre kuzaxak verchol alpha412826 pengtaoman lngramos di-mati nehashirsat rafpe tobru chrisjohnson sitedata alexrogalskiy wcarlsen rothgar watermeion jdsmithit 3b295 kuuji log1cb0mb mtougeron julienlefur mistermx htreu regalius danielhelfand iomarian bschwenn awx-dewey-du crt-fork johnkk84 devopsapp84 petrovmikhail nikhilm-in desponda

vcluster's Issues

feat: vcluster --version

Would be good to be able to check the CLI version and potentially also useful to include a command or --server flag for the server side version of a vcluster

Add high availability support for k3s

k3s supports HA and currently we only support that for the k8s distribution. Would be great if we can support it for k3s as well.

Invalid DownwardAPI metadata.labels in StatefulSet Pods as of v0.3.0-beta.3

As of the latest release of v0.3.0-beta.3, DownwardAPI metadata.labels for StatefulSet Pods would reference wrong label names. Here's a minimum reproducer:

apiVersion: v1
kind: Namespace
metadata:
  name: vcluster-bug
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: vcluster-bug
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: vcluster-bug
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        env:
          - name: K8S_STATEFUL_SET_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']

A Pod launched inside the host cluster would then look like this:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    vcluster.loft.sh/hosts-rewritten: "true"
    vcluster.loft.sh/labels: |-
      app="nginx"
      controller-revision-hash="web-5d76df5b6d"
      statefulset.kubernetes.io/pod-name="web-0"
    vcluster.loft.sh/name: web-0
    vcluster.loft.sh/namespace: vcluster-bug
    vcluster.loft.sh/owner-set-kind: StatefulSet
    vcluster.loft.sh/service-account-name: default
    vcluster.loft.sh/uid: c683994e-91ab-4343-b0cf-d22bd6916a51
  creationTimestamp: "2021-06-18T08:28:36Z"
  labels:
    vcluster.loft.sh/label-my-vcluster-x-67b9ac9205: web-5d76df5b6d
    vcluster.loft.sh/label-my-vcluster-x-a172cedcae: nginx
    vcluster.loft.sh/label-my-vcluster-x-ec0342c544: web-0
    vcluster.loft.sh/managed-by: my-vcluster
    vcluster.loft.sh/namespace: vcluster-bug-x-my-vcluster
  name: web-0-x-vcluster-bug-x-my-vcluster
  namespace: my-vcluster
  ownerReferences:
  - apiVersion: apps/v1
    kind: StatefulSet
    name: my-vcluster
    uid: 0a0910eb-e109-4637-9557-93cfec84cd1b
  resourceVersion: "48331478"
  uid: e94a26be-c551-4a05-9cb3-56b0615bf9db
spec:
  automountServiceAccountToken: false
  containers:
  - env:
    - name: K8S_STATEFUL_SET_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels['vcluster.loft.sh/label-my-vcluster-x-4992f265ef']
... [REDACTED] ...

Notice that the rewritten fieldPath is incorrect. It's referencing a non-existent label of vcluster.loft.sh/label-my-vcluster-x-4992f265ef, instead of vcluster.loft.sh/label-my-vcluster-x-ec0342c544, which corresponds to the original statefulset.kubernetes.io/pod-name label.

Provide reasonable resource requests and limits

Often there are resource quotas assigned to the namespace into which you install vcluster. These are sometimes set to be quite strict, and there is also a mandatory LimitRange.

vcluster's Helm chart lacks a resources section. It is not clear how much performance the two containers (syncer and vcluster) need, for a typical scenario. For instance, I found that a memory limit of 256 results in an OOM kill, with Run: k3s kubectl being the last entry in the container log.

Helm Chart: statefulset template, broken logic

https://github.com/loft-sh/vcluster/blob/main/chart/templates/statefulset.yaml#L32

spec.volumeClaimTemplates.0.spec.storageClassName by default renders as null instead of a value or empty string. This breaks kustomize validation.

  {{- if .Values.storage.persistence }}  <== default renders as to IF TRUE
  {{- if not .Values.storage.volumeClaimTemplates }}  <== defaults renders to IF NOT FALSE
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: {{ .Values.storage.className }} <== undefined in values file
        resources:
          requests:
            storage: {{ .Values.storage.size }}
  {{- else }}
  volumeClaimTemplates:
{{ toYaml .Values.volumeClaimTemplates | indent 4 }}
  {{- end }}
  {{- end }}

I'm actually surprised Helm doesn't complain about a null value to the key, seems like a break of yaml spec.

K8s doesn't care if the field is defined or not, and the docs simply state to use a string.
storageClassName: "" # Empty string must be explicitly set otherwise default StorageClass will be set
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1
https://v1-21.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#persistentvolumeclaimspec-v1-core

Proposed:

Define the key 'storageClassName' only if a name is defined in values OR
Define a default in values "Filesystem" is the inferred value when undefined per: https://github.com/kubernetes/api/blob/release-1.21/core/v1/types.go#L489

Invalid ConfigMap / Secret ref names in Pods when the 52th character is a period (.)

As of the latest release of v0.3.0-beta.3, when a ConfigMap / Secret contains a period character (.) at index 51, vcluster will rewrite the Pod definition with a bad ref name. Here's a minimum reproducer:

apiVersion: v1
kind: Namespace
metadata:
  name: vcluster-bug
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres.admin-pg.credentials.postgresql.acid.zalan.do
  namespace: vcluster-bug
data:
  username: foobar
---
apiVersion: v1
kind: Pod
metadata:
  name: vcluster-bug
  namespace: vcluster-bug
spec:
  containers:
  - name: nginx
    image: k8s.gcr.io/nginx-slim:0.8
    env:
      - name: USERNAME_FROM_ENV
        valueFrom:
          configMapKeyRef:
            key: username
            name: postgres.admin-pg.credentials.postgresql.acid.zalan.do

This Pod will then be stuck at "Pending" state with this error message:

Error syncing to physical cluster: Pod "vcluster-bug-x-vcluster-bug-x-my-vcluster" is invalid:
spec.containers[0[].env[0[].valueFrom.configMapKeyRef.name: 
Invalid value: "postgres.admin-pg.credentials.postgresql.acid.zalan.-ac00eb36f8": 
a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' 
or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex 
used for validation is '[a-z0-9[]([-a-z0-9[]*[a-z0-9[])?(\.[a-z0-9[]([-a-z0-9[]*[a-z0-9[])?)*')

Notice that the newly referenced ConfigMap name is postgres.admin-pg.credentials.postgresql.acid.zalan.-ac00eb36f8 which is indeed an invalid RFC 1123 subdomain.

A long ConfigMap / Secret name is truncated to a length of 52, then appended with a generated suffix of length 11 in the form of -XXXXXXXXXX (for a total of 63 chars). If the truncated result ends with a period, we'll end up with this issue.

We actually ran into this while installing postgres-operator. This operator generates all Secrets with a (non-configurable) suffix of .credentials.postgresql.acid.zalan.do.

feat: context flag on vcluster

It would be nice to have the ability to specify the kube-context to use when calling vcluster. Currently the default one is used so you have to ensure you are setup correctly before calling vcluster.

Example

vcluster --context my-cluster connect test --namespace test

would use the my-cluster context from kubectl config get-contexts

IPv6-only clusters not supported

Hi,
I wanted to give vcluster a try, however it seems that IPv6-only kubernetes clusters are not yet fully supported;

kl -n host-namespace-1 logs -f vcluster-1-0 vcluster
time="2021-05-18T09:56:40.133915808Z" level=info msg="Starting k3s v1.18.16+k3s1 (8c7dd139)"
time="2021-05-18T09:56:40.161845141Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2021-05-18T09:56:40.162067426Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2021-05-18T09:56:40.193732078Z" level=info msg="Database tables and indexes are up to date"
time="2021-05-18T09:56:40.200156204Z" level=info msg="Kine listening on unix://kine.sock"
time="2021-05-18T09:56:40.620849868Z" level=info msg="Active TLS secret  (ver=) (count 7): map[listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-2001:1600:caca:1002:1d:12e:ffff:4b6d:2001:1600:caca:1002:1d:12e:ffff:4b6d listener.cattle.io/cn-2001:1600:caca:50da:1d:fffe:0:1:2001:1600:caca:50da:1d:fffe:0:1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:58e35fc7fbc214c1d0e3c5f72f84a49a8ee8f2ec2eb4cfe34a546604d09d32f4]"
time="2021-05-18T09:56:40.644266915Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --basic-auth-file=/data/server/cred/passwd --bind-address=127.0.0.1 --cert-dir=/data/server/tls/temporary-certs --client-ca-file=/data/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/data/server/tls/server-ca.crt --kubelet-client-certificate=/data/server/tls/client-kube-apiserver.crt --kubelet-client-key=/data/server/tls/client-kube-apiserver.key --proxy-client-cert-file=/data/server/tls/client-auth-proxy.crt --proxy-client-key-file=/data/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/data/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/data/server/tls/service.key --service-account-signing-key-file=/data/server/tls/service.key --service-cluster-ip-range=2001:1600:caca:50da:1d:fffe::/110 --storage-backend=etcd3 --tls-cert-file=/data/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/data/server/tls/serving-kube-apiserver.key"
Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
I0518 09:56:40.650575       1 server.go:645] external host was not specified, using 2001:1600:caca:1002:1d:12e:ffff:4b6d
I0518 09:56:40.651761       1 server.go:162] Version: v1.18.16+k3s1
I0518 09:56:41.065909       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0518 09:56:41.066022       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0518 09:56:41.081038       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0518 09:56:41.081190       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0518 09:56:41.138573       1 master.go:270] Using reconciler: lease
I0518 09:56:41.323665       1 rest.go:113] the default service ipfamily for this cluster is: IPv6
W0518 09:56:42.195315       1 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources.
W0518 09:56:42.216467       1 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0518 09:56:42.245175       1 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0518 09:56:42.288855       1 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0518 09:56:42.297130       1 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0518 09:56:42.333489       1 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0518 09:56:42.380667       1 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
W0518 09:56:42.380700       1 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
I0518 09:56:42.403280       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0518 09:56:42.403322       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0518 09:56:46.561540       1 secure_serving.go:178] Serving securely on 127.0.0.1:6444
I0518 09:56:46.561672       1 crd_finalizer.go:266] Starting CRDFinalizer
I0518 09:56:46.561764       1 dynamic_cafile_content.go:167] Starting request-header::/data/server/tls/request-header-ca.crt
I0518 09:56:46.561808       1 dynamic_serving_content.go:130] Starting serving-cert::/data/server/tls/serving-kube-apiserver.crt::/data/server/tls/serving-kube-apiserver.key
I0518 09:56:46.562047       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0518 09:56:46.563094       1 controller.go:86] Starting OpenAPI controller
I0518 09:56:46.563180       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0518 09:56:46.563205       1 naming_controller.go:291] Starting NamingConditionController
I0518 09:56:46.563224       1 establishing_controller.go:76] Starting EstablishingController
I0518 09:56:46.563242       1 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I0518 09:56:46.563259       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0518 09:56:46.563286       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/data/server/tls/client-ca.crt
I0518 09:56:46.578483       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0518 09:56:46.578511       1 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller
I0518 09:56:46.578563       1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0518 09:56:46.578570       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0518 09:56:46.578597       1 available_controller.go:404] Starting AvailableConditionController
I0518 09:56:46.578602       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0518 09:56:46.578657       1 autoregister_controller.go:141] Starting autoregister controller
I0518 09:56:46.578662       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0518 09:56:46.578694       1 controller.go:81] Starting OpenAPI AggregationController
I0518 09:56:46.581360       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/data/server/tls/client-ca.crt
I0518 09:56:46.581417       1 dynamic_cafile_content.go:167] Starting request-header::/data/server/tls/request-header-ca.crt
I0518 09:56:46.581779       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0518 09:56:46.584247       1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister
E0518 09:56:46.658946       1 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIP: Invalid value: "2001:1600:caca:50da:1d:fffe:0:1": cannot allocate resources of type serviceipallocations at this time
E0518 09:56:46.667561       1 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/2001:1600:caca:1002:1d:12e:ffff:4b6d, ResourceVersion: 0, AdditionalErrorMsg: 
I0518 09:56:46.679560       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0518 09:56:46.679613       1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller 
I0518 09:56:46.681339       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0518 09:56:46.681383       1 cache.go:39] Caches are synced for autoregister controller
I0518 09:56:46.688337       1 shared_informer.go:230] Caches are synced for crd-autoregister 
I0518 09:56:47.558741       1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0518 09:56:47.558788       1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0518 09:56:47.589383       1 storage_scheduling.go:134] created PriorityClass system-node-critical with value 2000001000
I0518 09:56:47.606356       1 storage_scheduling.go:134] created PriorityClass system-cluster-critical with value 2000000000
I0518 09:56:47.606385       1 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
I0518 09:56:49.765228       1 controller.go:609] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0518 09:56:49.889568       1 controller.go:609] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0518 09:56:50.156090       1 lease.go:224] Resetting endpoints for master service "kubernetes" to [2001:1600:caca:1002:1d:12e:ffff:4b6d]
I0518 09:56:50.157759       1 controller.go:609] quota admission added evaluator for: endpoints
I0518 09:56:50.174678       1 controller.go:609] quota admission added evaluator for: endpointslices.discovery.k8s.io
time="2021-05-18T09:56:50.618681934Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/data/server/tls/server-ca.crt --cluster-signing-key-file=/data/server/tls/server-ca.key --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle --kubeconfig=/data/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --root-ca-file=/data/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/data/server/tls/service.key --use-service-account-credentials=true"
I0518 09:56:50.626095       1 controllermanager.go:161] Version: v1.18.16+k3s1
I0518 09:56:50.628217       1 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
time="2021-05-18T09:56:50.655583671Z" level=info msg="Creating CRD addons.k3s.cattle.io"
time="2021-05-18T09:56:50.681262455Z" level=info msg="Creating CRD helmcharts.helm.cattle.io"
time="2021-05-18T09:56:50.711929954Z" level=info msg="Creating CRD helmchartconfigs.helm.cattle.io"
time="2021-05-18T09:56:50.743876926Z" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
I0518 09:56:50.899874       1 plugins.go:100] No cloud provider specified.
I0518 09:56:50.903791       1 shared_informer.go:223] Waiting for caches to sync for tokens
I0518 09:56:50.935018       1 controller.go:609] quota admission added evaluator for: serviceaccounts
I0518 09:56:50.942311       1 controllermanager.go:533] Started "replicationcontroller"
W0518 09:56:50.942566       1 controllermanager.go:525] Skipping "root-ca-cert-publisher"
W0518 09:56:50.942626       1 controllermanager.go:512] "nodelifecycle" is disabled
I0518 09:56:50.942381       1 replica_set.go:182] Starting replicationcontroller controller
I0518 09:56:50.943366       1 shared_informer.go:223] Waiting for caches to sync for ReplicationController
I0518 09:56:51.008386       1 shared_informer.go:230] Caches are synced for tokens 
I0518 09:56:51.032146       1 controllermanager.go:533] Started "disruption"
W0518 09:56:51.032173       1 controllermanager.go:512] "bootstrapsigner" is disabled
I0518 09:56:51.032480       1 disruption.go:331] Starting disruption controller
I0518 09:56:51.032498       1 shared_informer.go:223] Waiting for caches to sync for disruption
I0518 09:56:51.103384       1 controllermanager.go:533] Started "job"
I0518 09:56:51.103424       1 job_controller.go:145] Starting job controller
I0518 09:56:51.103449       1 shared_informer.go:223] Waiting for caches to sync for job
I0518 09:56:51.165042       1 controllermanager.go:533] Started "replicaset"
I0518 09:56:51.165297       1 replica_set.go:182] Starting replicaset controller
I0518 09:56:51.165340       1 shared_informer.go:223] Waiting for caches to sync for ReplicaSet
time="2021-05-18T09:56:51.267636533Z" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
time="2021-05-18T09:56:51.267687711Z" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
I0518 09:56:51.286705       1 controllermanager.go:533] Started "horizontalpodautoscaling"
W0518 09:56:51.286739       1 controllermanager.go:512] "tokencleaner" is disabled
W0518 09:56:51.286747       1 controllermanager.go:512] "persistentvolume-binder" is disabled
W0518 09:56:51.286753       1 controllermanager.go:512] "attachdetach" is disabled
I0518 09:56:51.287081       1 horizontal.go:169] Starting HPA controller
I0518 09:56:51.287111       1 shared_informer.go:223] Waiting for caches to sync for HPA
I0518 09:56:51.750642       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
I0518 09:56:51.750755       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0518 09:56:51.750798       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
I0518 09:56:51.750881       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io
I0518 09:56:51.750930       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for addons.k3s.cattle.io
I0518 09:56:51.750994       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
I0518 09:56:51.751097       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
I0518 09:56:51.751197       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
I0518 09:56:51.751231       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0518 09:56:51.751259       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
I0518 09:56:51.751306       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
I0518 09:56:51.751342       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
I0518 09:56:51.751404       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
I0518 09:56:51.751463       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0518 09:56:51.751494       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
I0518 09:56:51.751521       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
I0518 09:56:51.751554       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
I0518 09:56:51.751584       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
I0518 09:56:51.751613       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0518 09:56:51.751651       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io
I0518 09:56:51.751725       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
I0518 09:56:51.751758       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
I0518 09:56:51.751785       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
I0518 09:56:51.751839       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0518 09:56:51.751862       1 controllermanager.go:533] Started "resourcequota"
I0518 09:56:51.751903       1 resource_quota_controller.go:272] Starting resource quota controller
I0518 09:56:51.751933       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I0518 09:56:51.751970       1 resource_quota_monitor.go:303] QuotaMonitor running
time="2021-05-18T09:56:51.779410685Z" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2021-05-18T09:56:51.779456015Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
time="2021-05-18T09:56:52.289946016Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
time="2021-05-18T09:56:52.324247973Z" level=info msg="Writing static file: /data/server/static/charts/traefik-1.81.0.tgz"
time="2021-05-18T09:56:52.345976569Z" level=info msg="Writing manifest: /data/server/manifests/rolebindings.yaml"
time="2021-05-18T09:56:52.350008423Z" level=info msg="Writing manifest: /data/server/manifests/ccm.yaml"
time="2021-05-18T09:56:52.359111781Z" level=info msg="Writing manifest: /data/server/manifests/coredns.yaml"
time="2021-05-18T09:56:52.461556884Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2021-05-18T09:56:52.463023570Z" level=info msg="Node token is available at /data/server/token"
time="2021-05-18T09:56:52.463609283Z" level=info msg="To join node to cluster: k3s agent -s https://2001:1600:caca:1002:1d:12e:ffff:4b6d:6443 -t ${NODE_TOKEN}"
I0518 09:56:52.529683       1 controllermanager.go:533] Started "garbagecollector"
I0518 09:56:52.530331       1 garbagecollector.go:133] Starting garbage collector controller
I0518 09:56:52.530353       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I0518 09:56:52.530391       1 graph_builder.go:282] GraphBuilder running
2021-05-18 09:56:52.550448 I | http: TLS handshake error from 127.0.0.1:37086: remote error: tls: bad certificate
time="2021-05-18T09:56:52.563547895Z" level=info msg="Wrote kubeconfig /k3s-config/kube-config.yaml"
time="2021-05-18T09:56:52.563609983Z" level=info msg="Run: k3s kubectl"
time="2021-05-18T09:56:52.563627200Z" level=info msg="k3s is up and running"
I0518 09:56:52.597942       1 controllermanager.go:533] Started "statefulset"
I0518 09:56:52.598518       1 stateful_set.go:146] Starting stateful set controller
I0518 09:56:52.598548       1 shared_informer.go:223] Waiting for caches to sync for stateful set
I0518 09:56:52.779406       1 controller.go:609] quota admission added evaluator for: addons.k3s.cattle.io
I0518 09:56:52.781435       1 controllermanager.go:533] Started "csrapproving"
I0518 09:56:52.781684       1 certificate_controller.go:119] Starting certificate controller "csrapproving"
I0518 09:56:52.781696       1 shared_informer.go:223] Waiting for caches to sync for certificate-csrapproving
time="2021-05-18T09:56:52.833523204Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2021-05-18T09:56:52.833897214Z" level=info msg="Starting /v1, Kind=Node controller"
time="2021-05-18T09:56:52.834071443Z" level=info msg="Starting /v1, Kind=Service controller"
time="2021-05-18T09:56:52.834142579Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2021-05-18T09:56:52.834232913Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2021-05-18T09:56:52.863677201Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
time="2021-05-18T09:56:52.863887128Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2021-05-18T09:56:52.863967949Z" level=info msg="Starting batch/v1, Kind=Job controller"
I0518 09:56:52.882701       1 controllermanager.go:533] Started "ttl"
W0518 09:56:52.882733       1 controllermanager.go:512] "persistentvolume-expander" is disabled
W0518 09:56:52.882749       1 controllermanager.go:525] Skipping "ttl-after-finished"
I0518 09:56:52.882747       1 ttl_controller.go:118] Starting TTL controller
I0518 09:56:52.882877       1 shared_informer.go:223] Waiting for caches to sync for TTL
I0518 09:56:52.898746       1 request.go:621] Throttling request took 1.040939299s, request: GET:https://127.0.0.1:6444/apis/admissionregistration.k8s.io/v1?timeout=32s
I0518 09:56:52.940150       1 controllermanager.go:533] Started "endpointslice"
I0518 09:56:52.940367       1 endpointslice_controller.go:213] Starting endpoint slice controller
I0518 09:56:52.940456       1 shared_informer.go:223] Waiting for caches to sync for endpoint_slice
I0518 09:56:52.976228       1 controllermanager.go:533] Started "daemonset"
I0518 09:56:52.976453       1 daemon_controller.go:286] Starting daemon sets controller
I0518 09:56:52.976469       1 shared_informer.go:223] Waiting for caches to sync for daemon sets
I0518 09:56:53.099704       1 controllermanager.go:533] Started "pv-protection"
W0518 09:56:53.099900       1 controllermanager.go:512] "nodeipam" is disabled
I0518 09:56:53.099876       1 pv_protection_controller.go:83] Starting PV protection controller
I0518 09:56:53.100147       1 shared_informer.go:223] Waiting for caches to sync for PV protection
I0518 09:56:53.131029       1 controllermanager.go:533] Started "clusterrole-aggregation"
I0518 09:56:53.131293       1 clusterroleaggregation_controller.go:149] Starting ClusterRoleAggregator
I0518 09:56:53.131305       1 shared_informer.go:223] Waiting for caches to sync for ClusterRoleAggregator
I0518 09:56:53.168358       1 controllermanager.go:533] Started "csrcleaner"
I0518 09:56:53.168634       1 cleaner.go:82] Starting CSR cleaner controller
I0518 09:56:53.180249       1 controller.go:609] quota admission added evaluator for: deployments.apps
I0518 09:56:53.233142       1 controllermanager.go:533] Started "pvc-protection"
I0518 09:56:53.233470       1 pvc_protection_controller.go:101] Starting PVC protection controller
I0518 09:56:53.233490       1 shared_informer.go:223] Waiting for caches to sync for PVC protection
I0518 09:56:53.313134       1 gc_controller.go:89] Starting GC controller
I0518 09:56:53.313319       1 shared_informer.go:223] Waiting for caches to sync for GC
I0518 09:56:53.313843       1 controllermanager.go:533] Started "podgc"
time="2021-05-18T09:56:53.330485996Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
I0518 09:56:53.433840       1 controllermanager.go:533] Started "namespace"
I0518 09:56:53.433898       1 namespace_controller.go:200] Starting namespace controller
I0518 09:56:53.433915       1 shared_informer.go:223] Waiting for caches to sync for namespace
I0518 09:56:53.466497       1 controllermanager.go:533] Started "endpoint"
I0518 09:56:53.466677       1 endpoints_controller.go:181] Starting endpoint controller
I0518 09:56:53.466695       1 shared_informer.go:223] Waiting for caches to sync for endpoint
I0518 09:56:53.484052       1 controllermanager.go:533] Started "cronjob"
I0518 09:56:53.484301       1 cronjob_controller.go:97] Starting CronJob Manager
I0518 09:56:53.504434       1 controllermanager.go:533] Started "csrsigning"
I0518 09:56:53.504650       1 certificate_controller.go:119] Starting certificate controller "csrsigning"
I0518 09:56:53.504664       1 shared_informer.go:223] Waiting for caches to sync for certificate-csrsigning
I0518 09:56:53.504706       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/server-ca.crt::/data/server/tls/server-ca.key
E0518 09:56:53.553865       1 core.go:89] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
W0518 09:56:53.553891       1 controllermanager.go:525] Skipping "service"
W0518 09:56:53.553907       1 core.go:243] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes.
W0518 09:56:53.553915       1 controllermanager.go:525] Skipping "route"
W0518 09:56:53.553921       1 controllermanager.go:512] "cloud-node-lifecycle" is disabled
I0518 09:56:53.588354       1 controllermanager.go:533] Started "serviceaccount"
I0518 09:56:53.588647       1 serviceaccounts_controller.go:117] Starting service account controller
I0518 09:56:53.588660       1 shared_informer.go:223] Waiting for caches to sync for service account
I0518 09:56:53.639884       1 controllermanager.go:533] Started "deployment"
I0518 09:56:53.640326       1 shared_informer.go:223] Waiting for caches to sync for resource quota
I0518 09:56:53.640420       1 deployment_controller.go:153] Starting deployment controller
I0518 09:56:53.640435       1 shared_informer.go:223] Waiting for caches to sync for deployment
I0518 09:56:53.687679       1 shared_informer.go:230] Caches are synced for HPA 
I0518 09:56:53.690451       1 shared_informer.go:230] Caches are synced for certificate-csrapproving 
I0518 09:56:53.694704       1 shared_informer.go:230] Caches are synced for service account 
I0518 09:56:53.695090       1 shared_informer.go:230] Caches are synced for TTL 
I0518 09:56:53.700682       1 shared_informer.go:230] Caches are synced for PV protection 
I0518 09:56:53.710284       1 shared_informer.go:230] Caches are synced for certificate-csrsigning 
I0518 09:56:53.713943       1 shared_informer.go:230] Caches are synced for GC 
I0518 09:56:53.732358       1 shared_informer.go:230] Caches are synced for ClusterRoleAggregator 
I0518 09:56:53.733737       1 shared_informer.go:230] Caches are synced for PVC protection 
I0518 09:56:53.733857       1 shared_informer.go:230] Caches are synced for disruption 
I0518 09:56:53.733876       1 disruption.go:339] Sending events to api server.
I0518 09:56:53.734456       1 shared_informer.go:230] Caches are synced for namespace 
I0518 09:56:53.741283       1 shared_informer.go:230] Caches are synced for deployment 
I0518 09:56:53.744736       1 shared_informer.go:230] Caches are synced for endpoint_slice 
I0518 09:56:53.746240       1 shared_informer.go:230] Caches are synced for ReplicationController 
I0518 09:56:53.771405       1 shared_informer.go:230] Caches are synced for ReplicaSet 
I0518 09:56:53.905065       1 controller.go:609] quota admission added evaluator for: replicasets.apps
E0518 09:56:53.928396       1 clusterroleaggregation_controller.go:181] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
I0518 09:56:53.937637       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"1fb191d7-431c-487d-b2d3-199da1c21b9c", APIVersion:"apps/v1", ResourceVersion:"222", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-7944c66d8d to 1
I0518 09:56:53.966966       1 shared_informer.go:230] Caches are synced for endpoint 
E0518 09:56:53.976003       1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
I0518 09:56:53.976667       1 shared_informer.go:230] Caches are synced for daemon sets 
I0518 09:56:53.996759       1 shared_informer.go:223] Waiting for caches to sync for garbage collector
I0518 09:56:53.998815       1 shared_informer.go:230] Caches are synced for stateful set 
I0518 09:56:54.028963       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-7944c66d8d", UID:"82703c86-f233-40d3-acfb-1a681563aed9", APIVersion:"apps/v1", ResourceVersion:"261", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-7944c66d8d-g5f4n
I0518 09:56:54.152214       1 shared_informer.go:230] Caches are synced for resource quota 
I0518 09:56:54.203926       1 shared_informer.go:230] Caches are synced for job 
I0518 09:56:54.240639       1 shared_informer.go:230] Caches are synced for resource quota 
I0518 09:56:54.297126       1 shared_informer.go:230] Caches are synced for garbage collector 
I0518 09:56:54.330647       1 shared_informer.go:230] Caches are synced for garbage collector 
I0518 09:56:54.330683       1 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
W0518 09:57:00.128646       1 lease.go:224] Resetting endpoints for master service "kubernetes" to [2001:1600:caca:1002:1d:12e:ffff:4b6d]
time="2021-05-18T09:57:08.598544058Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:57:23.712596609Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:57:38.831005487Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:57:53.934357168Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:58:09.147813795Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:58:24.307792410Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:58:39.481314918Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:58:54.668060028Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:59:09.810203979Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:59:24.986142765Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"
time="2021-05-18T09:59:40.138485865Z" level=error msg="failed to process config: failed to process /data/server/manifests/coredns.yaml: failed to create kube-system/kube-dns /v1, Kind=Service for  kube-system/coredns: Service \"kube-dns\" is invalid: spec.clusterIP: Invalid value: \"0.0.0.10\": provided IP is not in the valid range. The range of valid IPs is 2001:1600:caca:50da:1d:fffe::/110"

Don't hesitate if you need more information !

Cheers

Ingress pods and deployment pods are coming up with the same IPs in a vCluster

I am trying to create an ingress controller, ingress and related objects in a vCluster. Ingress controller and the related SA, namespace, etc.. are all deployed successfully. It resides in its own namespace "ingress-nginx"

**dev-user@ubuntu:~/vcluster$ kubectl get all -n ingress-nginx --kubeconfig kubeconfig.yaml
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-w65dn 0/1 Completed 0 7m34s
pod/ingress-nginx-admission-patch-x7rqg 0/1 Completed 0 7m34s
pod/ingress-nginx-controller-7fcdd96bcf-m2sgn 1/1 Running 0 7m44s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller-admission ClusterIP 10.109.123.12 443/TCP 7m44s
service/ingress-nginx-controller NodePort 10.109.100.180 80:31651/TCP,443:30824/TCP 7m44s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 7m44s

NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-7fcdd96bcf 1 1 1 7m44s

NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 2s 7m44s
job.batch/ingress-nginx-admission-patch 1/1 3s 7m44s**

I deployed workload in default namespace and found the pods of this workload and the job pods created as part of ingress deployment have the same IPs.

**dev-user@ubuntu:~/vcluster$ kubectl get pods -o wide --kubeconfig kubeconfig.yaml
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-world-service-single-6c58c555f8-dhv4l 1/1 Running 0 117s 172.17.0.12 minikube
hello-world-service-single-6c58c555f8-2wwkd 1/1 Running 0 117s 172.17.0.13 minikube

dev-user@ubuntu:~/vcluster$ kubectl get pods -o wide -n ingress-nginx --kubeconfig kubeconfig.yaml
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-w65dn 0/1 Completed 0 3m22s 172.17.0.11 minikube
ingress-nginx-admission-patch-x7rqg 0/1 Completed 0 3m22s 172.17.0.12 minikube
ingress-nginx-controller-7fcdd96bcf-m2sgn 1/1 Running 0 3m32s 172.17.0.11 minikube **

The service which i create to expose my deployment is giving no response on doing a curl.

**dev-user@ubuntu:~/vcluster$ curl -vvvv http://10.97.207.45

Rebuilt URL to: http://10.97.207.45/
Trying 10.97.207.45...
TCP_NODELAY set**

Any idea why these IPs are overlapping ( between ingress job pods and workload pods? )

CoreDNS does not start due to podPriority mismatch

First of all: Very nice project, great initial docs! Looking forward to dive deep. (trying it out here)

On initial deploy the CoreDNS pod was stuck in pending, on further inspection I found this error:

E0519 23:16:45.423428 1 controller.go:301] controller-runtime: manager: reconciler group reconciler kind Pod: controller: pod-forward: name coredns-5b66977746-6wd9x namespace kube-system: Reconciler error pods "coredns-5b66977746-6wd9x-x-kube-system-x-vcluster-0" is forbidden: the integer value of priority (0) must not be provided in pod spec; priority admission controller computed 2000 from the given PriorityClass name

On the host cluster we have specified custom PriorityClasses and set the default class to value 2000, as highlighted.

Workaround: Deploy the custom PriorityClasses to the virtual cluster.

It would be nice if we could set this right from the beginning, or rather not at all and take the default from the host cluster.

Select nodes for the initial k3s StatefulSet

The documentation describes how to select nodes that will receive deployments and other resources. But is it possible to similarly force the initial k3s StatefulSet to be restricted to node using a nodeSelector?

Manually modifying the StatefulSet once it has been created as of vcluster create works, but the UX would benefit from being able to pick nodes at creation time already.

The content of my values.yml file was as follows (I have a windows/linux hybrid cluster at hand).

rbac:
  clusterRole:
    create: true

syncer:
  extraArgs:
    - --fake-nodes=false
    - --node-selector=beta.kubernetes.io/os=linux
    - --enforce-node-selector

Support service account token audience

Currently we replace service account token volume projections with the default service account token of the namespace. However, while this works most of the time it is not 100% correct and in the v1.21 conformance test suite is an e2e test that explicitly tests this, which is why vcluster currently does not pass all of the v1.21 conformance tests. In order to solve this we need to create custom tokens that are idependent of the default service account token and manage them ourselves which gives vcluster more control.

Add --connect flag to vcluster create

Could be very convenient to create and connect to a vcluster in the same command, especially with the new --expose flag this can become quite handy. Because vcluster create also just upgrades the release if it already exists, you could also run the command multiple times without any problems.

# Create a new vcluster and immediately start port forwarding
vcluster create my-vcluster -n my-vcluster --connect

# Create a new vcluster and immediately create a new kube config for the exposed endpoint
vcluster create my-vcluster -n my-vcluster --expose --connect

Sync namespace labels to pods

Currently namespaceSelectors are not supported in vcluster, which means that networkpolicies and spec.affinity.podAffinitity are not fully supported.

One way to translate a namespaceSelector into a pod labelSelector is by syncing all namespace labels to all pods within that namespace in the host cluster. This makes it then possible to rewrite a namespaceSelector.

In order to implement this, we would need a new controller that makes sure that as soon as labels are changed on a virtual namespace, all labels are changed on the pods within that namespace in the host cluster as well.

Support node ports on fake kubelet endpoints

Currently vcluster will exchange the node ip to redirect to traffic to itself (to rewrite cluster metrics). One problem with this is that those fake ips do not forward traffic in created node ports.

In order to implement this, we would need a controller that reconciles all virtual cluster node services and opens a new port on the vcluster that redirects traffic to the actual node ip.

Cannot deploy vcluster when running as root is not allowed

I'm trying to deploy vcluster in a cluster that does not allow running containers as root, so the pods are not deploying with the current version of vcluster.

I have been checking the helm chart for an option to change this through values, but it looks like there isn't.

Is running as root an unavoidable requirement for vcluster or could this be fixed? (Running k3s as non-root is possible, although still experimental.)

Fully support Pod affinity label translation

Currently, label names within affinity of Pod specs are not translated at all, which is a deal-breaker for a lot of real-world use cases. Given that Pod Topology Constraints support was recently landed, is this part of the near-term plan for the next release?

Support syncing PersistentVolumes to the host cluster

Currently it does not work to create a PersistentVolume within a vcluster.
Allow creating PersistentVolumes within the vcluster and sync them back to the host cluster just like the sync-node-changes parameter.

Change HostRewrite Image

With docker hubs pull rate limits having a hard coded image tag becomes an issue on larger deployments where we constantly can not pull from docker hub such as https://github.com/loft-sh/vcluster/blob/main/pkg/controllers/resources/pods/translate.go#L31 having this be a configuruable field so that we can properly proxy the image I think makes sense.

vcluster not working - chmod kine.sock: no such file or directory

I try to create vcluster with vcluster cmd.
It is not working with below message.
I can't solve this problem.

time="2021-05-15T13:48:57.625861730Z" level=fatal msg="starting kubernetes: preparing server: creating storage endpoint: chmod kine.sock: no such file or directory"

Set server URL in kubeconfig secret to hostname that would be used to access vcluster.

At present the --tls-san option can be used to supply an alternate hostname to the generated certificate to match hostname that would be used when accessing the vcluster.

If this option is used then perhaps the kubeconfig secret if generated could also use that hostname rather than default of localhost. This would make the kubeconfig secret when mounted into a different deployment usable immediately without actually needing to copy it and run kubectl config set-cluster --server to fix up the server name.

Because though the default actually includes a port of 8443, which would not be what vcluster is accessed by when using the Kubernetes service inside the cluster, then piggy backing on --tls-san may not be appropriate . In this case then maybe a separate option should instead be added specifically to set what the server should be that is set in the kubeconfig secret when generated although this is duplicating the hostname in two options.

Either way, would be nice that an option existed to be able to set what the server URL is in the kubeconfig secret when generated so that it works as is, rather than needing to be copied and modified before it can be used.

feat: vcluster check (pre-flight checklist for installation)

I am trying to use vcluster and according to the home page "As long as you can create a deployment inside a single namespace, you will be able to create a virtual cluster and become admin of this virtual cluster."

I have a limited account in Okteto.com where I have access only to a namespace but not to the whole cluster. While a simple deployment works, vcluster doesn't install.

~ [1] $ kubectl create deployment nginx --image=nginx -n virtual-kostis-codefresh
deployment.apps/nginx created

~ $ vcluster create sample1 -n virtual-kostis-codefresh
[info]   execute command: helm upgrade sample1 vcluster --repo https://charts.loft.sh --version 0.3.0-beta.0 --kubeconfig /tmp/416075950 --namespace virtual-kostis-codefresh --install --repository-config='' --values /tmp/271665205
[fatal]  error executing helm upgrade sample1 vcluster --repo https://charts.loft.sh --version 0.3.0-beta.0 --kubeconfig /tmp/416075950 --namespace virtual-kostis-codefresh --install --repository-config='' --values /tmp/271665205: Release "sample1" does not exist. Installing it now.
Error: admission webhook "role.webhook.okteto.com" denied the request: rule 0 for role 'sample1' is not allowed: {Verbs:[*] APIGroups:[] Resources:[configmaps secrets services services/proxy pods pods/proxy pods/attach pods/portforward pods/exec pods/log events endpoints persistentvolumeclaims] ResourceNames:[] NonResourceURLs:[]}

I think it would be great if vcluster had a "check" option that mentions if it can be installed in the cluster and reports on what is needed and what is missing. Other tools have a similar feature as well. See for example https://linkerd.io/2.10/reference/cli/check/

Namespace deletion stuck due to pod out of sync ?

Hi,

we are currently facing the following issue when running an integration test against a vcluster (version 0.3.1). The test deploys some workload to the cluster, in this case istio. After executing the actual test, it deletes all resources one after another.

During this deletion phase, it gets stuck when deleting a namespace (istio-system). When looking at the namespace, it cannot be deleted because there exists still a pod (Some resources are remaining: pods. has 1 resource instances).

The pod is according to vcluster in the phase: "Running", however when looking at the host cluster, the pod is already gone.

The logs of the vcluster controller show the following lines over and over again:

08:32:06.985 E0819 08:32:06.985112       1 controller.go:302] controller-runtime: manager: reconciler group  reconciler kind Pod: controller: pod-forward: name ingressgateway-6d8b9cf54c-j6lfl namespace istio-system: Reconciler error get service account ingressgateway-service-account: ServiceAccount "ingressgateway-service-account" not found
08:32:08.377 E0819 08:32:08.377566       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:11.602 E0819 08:32:11.602302       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:12.717 E0819 08:32:12.717352       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:16.002 E0819 08:32:16.002687       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:21.238 E0819 08:32:21.238534       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:23.981 E0819 08:32:23.980936       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:24.142 E0819 08:32:24.142580       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:25.381 E0819 08:32:25.381046       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:27.038 E0819 08:32:27.038698       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:33.481 E0819 08:32:33.481459       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:35.854 E0819 08:32:35.853780       1 reflector.go:138] k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
08:32:39.094 E0819 08:32:39.094549       1 namespace_controller.go:162] deletion of namespace istio-system failed: unexpected items still remain in namespace: istio-system for gvr: /v1, Resource=pods

Could it be that the missing (most likely in the meantime deleted service account) leads to the pods not being synced anymore? Let me know when you need more information!

Thanks, Fabian

users can break out of virtual cluster

I've outlined in this gist here how to gain root access from a virtual cluster to the node that hosts the cluster that actually runs the pods: https://gist.github.com/protosam/a97d9c3db588d475fe686eb32280318a

This seems problematic for multi-tenancy.

Respect underlying node taints

Hi,
Not sure if I'm missing a setting (newish to k3s) however nodes that I specify with --node-taint CriticalAddonsOnly=true:NoExecute in the underlying K3S still receive pods from vCluster (and thus these pods do not start).

vcluster connect lost connection to pod

vcluster connect after some time, the connection is automatically lost

[root@kube-master-01 vcluster]# vcluster connect vcluster-1 -n host-namespace-1
[done] √ Virtual cluster kube config written to: ./kubeconfig.yaml. You can access the cluster via `kubectl --kubeconfig ./kubeconfig.yaml get namespaces`
[info]   Starting port forwarding: kubectl port-forward --namespace host-namespace-1 vcluster-1-0 8443:8443
Forwarding from 127.0.0.1:8443 -> 8443
Forwarding from [::1]:8443 -> 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
E0602 15:24:40.949886   31681 portforward.go:233] lost connection to pod

Support Pod Disruption Budgets (PDBs)

Pod disruption budgets[1] are very important for high availability kubernetes services.
They can be created inside a vcluster, but if I understand vcluster architecture correctly,
they would need to be synced into host cluster to be effective.

[1] https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets

add completion for cli

basic completion for bash, zsh would be nice to have.

Allow vcluster to expose the vcluster endpoint as a k8s service of the host cluster

My use case is the following: I want to create a vcluster in my real cluster in which I want to give full access to some newcomers play with.

By using vcluster connect, it uses my own machine to act as a proxy, and thus they can't connect to this vcluster as it's only accessible through my machine (if we could tweak the kubectl port-forward command to add an --address statement, I could set 0.0.0.0 then they would be able to connect on the vcluster by passing through my host machine. But still, it depends on my machine.

It would be super awesome if vcluster create could have an option as --expose-service which would make the vcluster connect create a kubeconfig file that does not depend on my machine.

Does Software license keep Apache-2.0 License ?

Thanks for making the loft core project open source
I would like to ask what is this project will keep license Apache-2.0 License ?
thx.

Create an example how to use vcluster with regular k8s components instead of k3s

Delete kubeconfig secret when vcluster is deleted.

When the vcluster control plain is deleted the secret used for the kubeconfig file is not deleted. This means that if you start a new vcluster immediately after using the same secret name it is updating the existing secret from the prior instance and not creating a new secret.

This is a problem because if the secret is being mounted into an application (deployed after the vcluster) then Kubernetes may not update quickly enough the values of the secret as seen by that deployment and so the old values of the secret are used and the application could fail.

For the case of where the --target-namespace is not set and the secret is in the same namespace, should perhaps look at setting the owner references of the secret to be the stateful set if --owning-statefulset is provided (much like other resources created from the vcluster) so the secret is also cleaned up.

For the case of where --target-namespace is set, would need to be documented that is up to the user to delete the secret, although in that case it is likely the whole separate target namespace would be getting deleted anyway.

Add support for Horizontal Pod Autoscaler

vcluster crashloop

Hi,

I installed loft today on our onprem upstream kubernetes 1.19.10 cluster without any problem. Afterwards, I tried to create a vcluster which crashes constantly

I see lots of timeouts in the logs yet apiservices seem to be up and running

v1.admissionregistration.k8s.io failed with : Timeout: request did not complete within requested timeout 34s v1beta1.apiextensions.k8s.io failed with : Timeout: request did not complete within requested timeout 34s v1.apiextensions.k8s.io failed with : context deadline exceeded v1beta1.admissionregistration.k8s.io failed with : context deadline exceeded

loft.log

help would be highly appreciated

vcluster does not start in limited RKE cluster

I got a restricted namespace in our internal RKE cluster managed by Rancher. However, vcluster won't start up. I have no idea what the concrete reason is, given that the log contains a massive output.

Things seem to start going wrong with this log entry:
cluster_authentication_trust_controller.go:493] kube-system/extension-apiserver-authentication failed with : Internal error occurred: resource quota evaluation timed out

But probably the attached log file will indicate the underlying reason better.
vcluster1.log

The syncer log is very short:

I0629 13:25:32.393511       1 main.go:223] Using physical cluster at https://10.43.0.1:443
I0629 13:25:32.575521       1 main.go:254] Can connect to virtual cluster with version v1.20.4+k3s1
F0629 13:25:32.587987       1 main.go:138] register controllers: register secrets indices: no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"```

Any ideas?

Unable to expose vcluster using steps in documentation

Hello,
I followed the steps mentioned in External Access - Ingress documentation to access vcluster directly without port forwarding.

Latest release did not have some flags specified in the documentation, hence I built the vcluster cli from master and tried to use the same.

$ cat vcluster1/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: vcluster1-ingress
spec:
  rules:
  - host: vcluster1.<my-clyster-domain-redacted>
    http:
      paths:
      - backend:
          serviceName: vcluster1
          servicePort: 8443
        path: /

$ cat vcluster1/values.yaml
syncer:
  extraArgs:
  - --tls-san=vcluster1.<mycluster-domain-redacted>


$ kubectl apply -f vcluster1/ingress.yaml -n vcluster1
ingress.networking.k8s.io/vcluster1-ingress configured

$ vcluster create vcluster1 -n vcluster1 -f vcluster1/values.yaml
[info]   execute command: helm upgrade vcluster1 vcluster --repo https://charts.loft.sh --version 0.3.0-beta.4 --kubeconfig /tmp/346816742 --namespace vcluster1 --install --repository-config='' --values /tmp/431627021 --values vcluster1/values.yaml
[done] √ Successfully created virtual cluster vcluster1 in namespace vcluster1. Use 'vcluster connect vcluster1 --namespace vcluster1' to access the virtual cluster

Though the vcluster helm installation is done successfully by setting extraArgs in syncer, and the kubeconfig.yaml is also generated with the correct ingress hostname, kubectl is not able to connect to the vcluster due to TLS handshake timeout error as shown below

$ kubectl get ns -v9
I0620 09:38:44.333233 2175166 loader.go:379] Config loaded from file:  /home/myuser/kubeconfig.yaml
I0620 09:38:44.341869 2175166 round_trippers.go:425] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.20.5 (linux/amd64) kubernetes/6b1d87a" 'https://vcluster1.<redacted-my-cluster-domain>/api?timeout=32s'
I0620 09:38:54.349872 2175166 round_trippers.go:445] GET https://<redacted-my-cluster-domain>/api?timeout=32s  in 10007 milliseconds
I0620 09:38:54.349895 2175166 round_trippers.go:451] Response Headers:
I0620 09:38:54.349957 2175166 cached_discovery.go:121] skipped caching discovery info due to Get "https://vcluster1.<redacted-my-cluster-domain>/api?timeout=32s": net/http: TLS handshake timeout

I can see that the vcluster server accepts --tls-san here and also sets it while creating the proxy server here
The secret generated by vcluster is always having https://localhost:$port ( reference:source code)

Right now everything looks good with port-forwarding option but we are looking into vcluster for running some automation scenarios, and direct access without port-forwarding would simplify things for us to a large extent.

Request some guidance to understand what I am doing wrong, so that I can connect to the vcluster directly via kubectl.

Waiting for vCluster to come up...

I can't connect to the vcluster:

vcluster connect vcluster-2 --update-current --debug
[info]   Waiting for vCluster to come up...
[fatal]  timed out waiting for the condition
wait for vCluster
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.(*ConnectCmd).Run
	/Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/connect.go:106
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.NewConnectCmd.func1
	/Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/connect.go:61
github.com/spf13/cobra.(*Command).execute
	/Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/cobra/command.go:850
github.com/spf13/cobra.(*Command).ExecuteC
	/Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/cobra/command.go:958
github.com/spf13/cobra.(*Command).Execute
	/Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/cobra/command.go:895
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.Execute
	/Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/root.go:36
main.main
	/Users/runner/work/vcluster/vcluster/cmd/vclusterctl/main.go:16
runtime.main
	/Users/runner/hostedtoolcache/go/1.16.3/x64/src/runtime/proc.go:225
runtime.goexit
	/Users/runner/hostedtoolcache/go/1.16.3/x64/src/runtime/asm_amd64.s:1371

All pod seem to be started fine:

k get po -o wide
NAME                                                  READY   STATUS    RESTARTS   AGE   IP           NODE                    NOMINATED NODE   READINESS GATES
coredns-854c77959c-9vh84-x-kube-system-x-vcluster-2   1/1     Running   0          16m   10.42.0.10   t320.xxx   <none>           <none>
vcluster-2-0                                          2/2     Running   1          17m   10.42.0.8    t320.xxx   <none>           <none>

To logs from the controle plane:

k logs -f vcluster-01-0 -c vcluster
time="2021-06-04T21:55:41.921673812Z" level=info msg="Starting k3s v1.20.4+k3s1 (838a906a)"
time="2021-06-04T21:55:41.930902651Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2021-06-04T21:55:41.930934416Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2021-06-04T21:55:41.934174422Z" level=info msg="Database tables and indexes are up to date"
time="2021-06-04T21:55:41.935341771Z" level=info msg="Kine listening on unix://kine.sock"
time="2021-06-04T21:55:41.960523842Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.961718174Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.962882249Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.964065893Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.965215732Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.966185326Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.967318198Z" level=info msg="certificate CN=cloud-controller-manager signed by CN=k3s-client-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.969107829Z" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.970873906Z" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.972648892Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.973729851Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:41.975470544Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:41 +0000 UTC"
time="2021-06-04T21:55:42.105260943Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1622843741: notBefore=2021-06-04 21:55:41 +0000 UTC notAfter=2022-06-04 21:55:42 +0000 UTC"
time="2021-06-04T21:55:42.105588248Z" level=info msg="Active TLS secret  (ver=) (count 7): map[listener.cattle.io/cn-10.42.0.8:10.42.0.8 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=BE9B09BF2487E6F5BD28D40EF63D5F11C05D60B6]"
time="2021-06-04T21:55:42.111807715Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/data/server/tls/temporary-certs --client-ca-file=/data/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --feature-gates=ServiceAccountIssuerDiscovery=false --insecure-port=0 --kubelet-certificate-authority=/data/server/tls/server-ca.crt --kubelet-client-certificate=/data/server/tls/client-kube-apiserver.crt --kubelet-client-key=/data/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/data/server/tls/client-auth-proxy.crt --proxy-client-key-file=/data/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/data/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/data/server/tls/service.key --service-account-signing-key-file=/data/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/data/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/data/server/tls/serving-kube-apiserver.key"
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
I0604 21:55:42.113032       1 server.go:659] external host was not specified, using 10.42.0.8
I0604 21:55:42.113285       1 server.go:196] Version: v1.20.4+k3s1
I0604 21:55:42.842273       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0604 21:55:42.843434       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0604 21:55:42.843449       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0604 21:55:42.844685       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0604 21:55:42.844699       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0604 21:55:42.876519       1 instance.go:289] Using reconciler: lease
I0604 21:55:42.961512       1 rest.go:131] the default service ipfamily for this cluster is: IPv4
W0604 21:55:43.317390       1 genericapiserver.go:419] Skipping API batch/v2alpha1 because it has no resources.
W0604 21:55:43.333438       1 genericapiserver.go:419] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.348550       1 genericapiserver.go:419] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.359028       1 genericapiserver.go:419] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.363156       1 genericapiserver.go:419] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.369765       1 genericapiserver.go:419] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.372706       1 genericapiserver.go:419] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0604 21:55:43.378781       1 genericapiserver.go:419] Skipping API apps/v1beta2 because it has no resources.
W0604 21:55:43.378797       1 genericapiserver.go:419] Skipping API apps/v1beta1 because it has no resources.
I0604 21:55:43.389090       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0604 21:55:43.389119       1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time="2021-06-04T21:55:43.400014907Z" level=info msg="Waiting for API server to become available"
time="2021-06-04T21:55:43.400037410Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/data/server/tls/client-ca.crt --cluster-signing-key-file=/data/server/tls/client-ca.key --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle --kubeconfig=/data/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/data/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/data/server/tls/service.key --use-service-account-credentials=true"
time="2021-06-04T21:55:43.401426513Z" level=info msg="Node token is available at /data/server/token"
time="2021-06-04T21:55:43.401510420Z" level=info msg="To join node to cluster: k3s agent -s https://10.42.0.8:6443 -t ${NODE_TOKEN}"
time="2021-06-04T21:55:43.403352418Z" level=info msg="Wrote kubeconfig /k3s-config/kube-config.yaml"
time="2021-06-04T21:55:43.403659056Z" level=info msg="Run: k3s kubectl"
I0604 21:55:46.078612       1 dynamic_cafile_content.go:167] Starting request-header::/data/server/tls/request-header-ca.crt
I0604 21:55:46.078630       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/data/server/tls/client-ca.crt
I0604 21:55:46.079004       1 dynamic_serving_content.go:130] Starting serving-cert::/data/server/tls/serving-kube-apiserver.crt::/data/server/tls/serving-kube-apiserver.key
I0604 21:55:46.079611       1 secure_serving.go:197] Serving securely on 127.0.0.1:6444
I0604 21:55:46.079631       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0604 21:55:46.079690       1 apf_controller.go:261] Starting API Priority and Fairness config controller
I0604 21:55:46.079693       1 available_controller.go:475] Starting AvailableConditionController
I0604 21:55:46.079736       1 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/data/server/tls/client-auth-proxy.crt::/data/server/tls/client-auth-proxy.key
I0604 21:55:46.079740       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0604 21:55:46.079770       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0604 21:55:46.079791       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0604 21:55:46.079739       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0604 21:55:46.079824       1 controller.go:86] Starting OpenAPI controller
I0604 21:55:46.079851       1 naming_controller.go:291] Starting NamingConditionController
I0604 21:55:46.079872       1 establishing_controller.go:76] Starting EstablishingController
I0604 21:55:46.079894       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0604 21:55:46.079914       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0604 21:55:46.079943       1 crd_finalizer.go:266] Starting CRDFinalizer
I0604 21:55:46.079987       1 autoregister_controller.go:141] Starting autoregister controller
I0604 21:55:46.079999       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0604 21:55:46.080028       1 controller.go:83] Starting OpenAPI AggregationController
I0604 21:55:46.080062       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0604 21:55:46.080071       1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I0604 21:55:46.080574       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0604 21:55:46.080585       1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0604 21:55:46.080625       1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/data/server/tls/client-ca.crt
I0604 21:55:46.081106       1 dynamic_cafile_content.go:167] Starting request-header::/data/server/tls/request-header-ca.crt
I0604 21:55:46.142405       1 shared_informer.go:247] Caches are synced for node_authorizer 
I0604 21:55:46.179817       1 apf_controller.go:266] Running API Priority and Fairness config worker
I0604 21:55:46.179817       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0604 21:55:46.179843       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0604 21:55:46.180041       1 cache.go:39] Caches are synced for autoregister controller
I0604 21:55:46.180113       1 shared_informer.go:247] Caches are synced for crd-autoregister 
I0604 21:55:46.180618       1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller 
I0604 21:55:46.186770       1 controller.go:609] quota admission added evaluator for: namespaces
E0604 21:55:46.217452       1 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocated ip:10.43.0.1 with error:cannot allocate resources of type serviceipallocations at this time
E0604 21:55:46.218258       1 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/10.42.0.8, ResourceVersion: 0, AdditionalErrorMsg: 
I0604 21:55:47.078685       1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0604 21:55:47.078732       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0604 21:55:47.089007       1 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
I0604 21:55:47.094908       1 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0604 21:55:47.094951       1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0604 21:55:47.654599       1 controller.go:609] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0604 21:55:47.693050       1 controller.go:609] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0604 21:55:47.749626       1 lease.go:233] Resetting endpoints for master service "kubernetes" to [10.42.0.8]
I0604 21:55:47.750552       1 controller.go:609] quota admission added evaluator for: endpoints
I0604 21:55:47.754111       1 controller.go:609] quota admission added evaluator for: endpointslices.discovery.k8s.io
time="2021-06-04T21:55:48.095659943Z" level=info msg="Kube API server is now running"
time="2021-06-04T21:55:48.095718982Z" level=info msg="k3s is up and running"
Flag --address has been deprecated, see --bind-address instead.
I0604 21:55:48.100911       1 controllermanager.go:176] Version: v1.20.4+k3s1
I0604 21:55:48.101263       1 deprecated_insecure_serving.go:53] Serving insecurely on 127.0.0.1:10252
time="2021-06-04T21:55:48.102840712Z" level=info msg="Creating CRD addons.k3s.cattle.io"
time="2021-06-04T21:55:48.107067601Z" level=info msg="Creating CRD helmcharts.helm.cattle.io"
time="2021-06-04T21:55:48.111963597Z" level=info msg="Creating CRD helmchartconfigs.helm.cattle.io"
time="2021-06-04T21:55:48.119053686Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
time="2021-06-04T21:55:48.621757509Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
time="2021-06-04T21:55:48.621786065Z" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
time="2021-06-04T21:55:49.124524060Z" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
time="2021-06-04T21:55:49.124549908Z" level=info msg="Waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2021-06-04T21:55:49.627166966Z" level=info msg="Done waiting for CRD helmchartconfigs.helm.cattle.io to become available"
time="2021-06-04T21:55:49.638156596Z" level=info msg="Writing static file: /data/server/static/charts/traefik-1.81.0.tgz"
time="2021-06-04T21:55:49.638442987Z" level=info msg="Writing manifest: /data/server/manifests/rolebindings.yaml"
time="2021-06-04T21:55:49.638594232Z" level=info msg="Writing manifest: /data/server/manifests/coredns.yaml"
time="2021-06-04T21:55:49.739233614Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2021-06-04T21:55:49.749741984Z" level=info msg="Cluster dns configmap has been set successfully"
I0604 21:55:49.751620       1 controller.go:609] quota admission added evaluator for: serviceaccounts
I0604 21:55:49.789560       1 controller.go:609] quota admission added evaluator for: deployments.apps
I0604 21:55:49.805183       1 controller.go:609] quota admission added evaluator for: addons.k3s.cattle.io
I0604 21:55:49.821433       1 request.go:655] Throttling request took 1.04843254s, request: GET:https://127.0.0.1:6444/apis/coordination.k8s.io/v1?timeout=32s
time="2021-06-04T21:55:49.840479939Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
time="2021-06-04T21:55:49.840494475Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2021-06-04T21:55:49.840502643Z" level=info msg="Starting /v1, Kind=Service controller"
time="2021-06-04T21:55:49.840507482Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2021-06-04T21:55:49.840509119Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2021-06-04T21:55:49.840515344Z" level=info msg="Starting /v1, Kind=Node controller"
time="2021-06-04T21:55:49.840518828Z" level=info msg="Starting /v1, Kind=Endpoints controller"
time="2021-06-04T21:55:50.216429888Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2021-06-04T21:55:50.221620904Z" level=info msg="Active TLS secret k3s-serving (ver=237) (count 7): map[listener.cattle.io/cn-10.42.0.8:10.42.0.8 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=BE9B09BF2487E6F5BD28D40EF63D5F11C05D60B6]"
I0604 21:55:50.727343       1 shared_informer.go:240] Waiting for caches to sync for tokens
I0604 21:55:50.744902       1 controllermanager.go:554] Started "endpointslice"
I0604 21:55:50.744976       1 endpointslice_controller.go:237] Starting endpoint slice controller
I0604 21:55:50.744994       1 shared_informer.go:240] Waiting for caches to sync for endpoint_slice
I0604 21:55:50.754798       1 controllermanager.go:554] Started "endpointslicemirroring"
W0604 21:55:50.754823       1 controllermanager.go:533] "persistentvolume-expander" is disabled
I0604 21:55:50.754826       1 endpointslicemirroring_controller.go:211] Starting EndpointSliceMirroring controller
I0604 21:55:50.754842       1 shared_informer.go:240] Waiting for caches to sync for endpoint_slice_mirroring
I0604 21:55:50.764512       1 controllermanager.go:554] Started "podgc"
W0604 21:55:50.764528       1 controllermanager.go:533] "bootstrapsigner" is disabled
W0604 21:55:50.764534       1 controllermanager.go:533] "nodeipam" is disabled
I0604 21:55:50.764622       1 gc_controller.go:89] Starting GC controller
I0604 21:55:50.764640       1 shared_informer.go:240] Waiting for caches to sync for GC
E0604 21:55:50.772476       1 core.go:92] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
W0604 21:55:50.772498       1 controllermanager.go:546] Skipping "service"
I0604 21:55:50.780317       1 controllermanager.go:554] Started "clusterrole-aggregation"
I0604 21:55:50.780393       1 clusterroleaggregation_controller.go:149] Starting ClusterRoleAggregator
I0604 21:55:50.780407       1 shared_informer.go:240] Waiting for caches to sync for ClusterRoleAggregator
I0604 21:55:50.787585       1 controllermanager.go:554] Started "pv-protection"
W0604 21:55:50.787603       1 controllermanager.go:546] Skipping "ttl-after-finished"
W0604 21:55:50.787611       1 controllermanager.go:546] Skipping "ephemeral-volume"
I0604 21:55:50.787682       1 pv_protection_controller.go:83] Starting PV protection controller
I0604 21:55:50.787694       1 shared_informer.go:240] Waiting for caches to sync for PV protection
I0604 21:55:50.794696       1 controllermanager.go:554] Started "replicaset"
I0604 21:55:50.794718       1 replica_set.go:182] Starting replicaset controller
I0604 21:55:50.794729       1 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
I0604 21:55:50.827498       1 shared_informer.go:247] Caches are synced for tokens 
I0604 21:55:50.832928       1 controllermanager.go:554] Started "ttl"
I0604 21:55:50.833063       1 ttl_controller.go:121] Starting TTL controller
I0604 21:55:50.833076       1 shared_informer.go:240] Waiting for caches to sync for TTL
I0604 21:55:50.843520       1 controllermanager.go:554] Started "root-ca-cert-publisher"
I0604 21:55:50.843540       1 publisher.go:98] Starting root CA certificate configmap publisher
I0604 21:55:50.843692       1 shared_informer.go:240] Waiting for caches to sync for crt configmap
I0604 21:55:50.853044       1 controllermanager.go:554] Started "endpoint"
I0604 21:55:50.853069       1 endpoints_controller.go:184] Starting endpoint controller
I0604 21:55:50.853081       1 shared_informer.go:240] Waiting for caches to sync for endpoint
I0604 21:55:50.881429       1 controllermanager.go:554] Started "serviceaccount"
I0604 21:55:50.881491       1 serviceaccounts_controller.go:117] Starting service account controller
I0604 21:55:50.881506       1 shared_informer.go:240] Waiting for caches to sync for service account
I0604 21:55:51.031809       1 controllermanager.go:554] Started "job"
I0604 21:55:51.031831       1 job_controller.go:148] Starting job controller
I0604 21:55:51.031842       1 shared_informer.go:240] Waiting for caches to sync for job
W0604 21:55:51.031832       1 core.go:246] configure-cloud-routes is set, but no cloud provider specified. Will not configure cloud provider routes.
W0604 21:55:51.031859       1 controllermanager.go:546] Skipping "route"
W0604 21:55:51.031866       1 controllermanager.go:533] "attachdetach" is disabled
I0604 21:55:51.181710       1 controllermanager.go:554] Started "replicationcontroller"
I0604 21:55:51.181758       1 replica_set.go:182] Starting replicationcontroller controller
I0604 21:55:51.181766       1 shared_informer.go:240] Waiting for caches to sync for ReplicationController
I0604 21:55:51.946539       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0604 21:55:51.946628       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for deployments.apps
I0604 21:55:51.946683       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for cronjobs.batch
I0604 21:55:51.946789       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.extensions
I0604 21:55:51.946857       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
I0604 21:55:51.946939       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for podtemplates
I0604 21:55:51.947000       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
W0604 21:55:51.947069       1 shared_informer.go:494] resyncPeriod 15h23m41.631502455s is smaller than resyncCheckPeriod 20h44m24.485449763s and the informer has already started. Changing it to 20h44m24.485449763s
I0604 21:55:51.947192       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
I0604 21:55:51.947275       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io
I0604 21:55:51.947337       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0604 21:55:51.947405       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for addons.k3s.cattle.io
I0604 21:55:51.947456       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpoints
I0604 21:55:51.947529       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for daemonsets.apps
I0604 21:55:51.947590       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0604 21:55:51.947649       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0604 21:55:51.947709       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for limitranges
W0604 21:55:51.947742       1 shared_informer.go:494] resyncPeriod 16h32m22.14377626s is smaller than resyncCheckPeriod 20h44m24.485449763s and the informer has already started. Changing it to 20h44m24.485449763s
I0604 21:55:51.947828       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for serviceaccounts
I0604 21:55:51.947888       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for events.events.k8s.io
I0604 21:55:51.947951       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io
I0604 21:55:51.948017       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0604 21:55:51.948154       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for replicasets.apps
I0604 21:55:51.948231       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for statefulsets.apps
I0604 21:55:51.948283       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for jobs.batch
I0604 21:55:51.948342       1 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
I0604 21:55:51.948381       1 controllermanager.go:554] Started "resourcequota"
I0604 21:55:51.948415       1 resource_quota_controller.go:273] Starting resource quota controller
I0604 21:55:51.948446       1 shared_informer.go:240] Waiting for caches to sync for resource quota
I0604 21:55:51.948508       1 resource_quota_monitor.go:304] QuotaMonitor running
I0604 21:55:51.968160       1 controllermanager.go:554] Started "disruption"
W0604 21:55:51.968197       1 controllermanager.go:533] "persistentvolume-binder" is disabled
I0604 21:55:51.968242       1 disruption.go:331] Starting disruption controller
I0604 21:55:51.968268       1 shared_informer.go:240] Waiting for caches to sync for disruption
I0604 21:55:51.983563       1 controllermanager.go:554] Started "pvc-protection"
I0604 21:55:51.983732       1 pvc_protection_controller.go:110] Starting PVC protection controller
I0604 21:55:51.983758       1 shared_informer.go:240] Waiting for caches to sync for PVC protection
I0604 21:55:52.082201       1 controllermanager.go:554] Started "horizontalpodautoscaling"
I0604 21:55:52.082307       1 horizontal.go:169] Starting HPA controller
I0604 21:55:52.082331       1 shared_informer.go:240] Waiting for caches to sync for HPA
I0604 21:55:52.235168       1 controllermanager.go:554] Started "daemonset"
W0604 21:55:52.235203       1 controllermanager.go:533] "tokencleaner" is disabled
I0604 21:55:52.235204       1 daemon_controller.go:285] Starting daemon sets controller
W0604 21:55:52.235215       1 controllermanager.go:533] "nodelifecycle" is disabled
I0604 21:55:52.235224       1 shared_informer.go:240] Waiting for caches to sync for daemon sets
W0604 21:55:52.235226       1 controllermanager.go:533] "cloud-node-lifecycle" is disabled
I0604 21:55:52.490067       1 controllermanager.go:554] Started "namespace"
I0604 21:55:52.490117       1 namespace_controller.go:200] Starting namespace controller
I0604 21:55:52.490133       1 shared_informer.go:240] Waiting for caches to sync for namespace
I0604 21:55:52.682358       1 garbagecollector.go:142] Starting garbage collector controller
I0604 21:55:52.682397       1 shared_informer.go:240] Waiting for caches to sync for garbage collector
I0604 21:55:52.682443       1 graph_builder.go:289] GraphBuilder running
I0604 21:55:52.682690       1 controllermanager.go:554] Started "garbagecollector"
I0604 21:55:52.931843       1 controllermanager.go:554] Started "deployment"
I0604 21:55:52.931896       1 deployment_controller.go:153] Starting deployment controller
I0604 21:55:52.931905       1 shared_informer.go:240] Waiting for caches to sync for deployment
I0604 21:55:53.084603       1 controllermanager.go:554] Started "statefulset"
I0604 21:55:53.084701       1 stateful_set.go:146] Starting stateful set controller
I0604 21:55:53.084724       1 shared_informer.go:240] Waiting for caches to sync for stateful set
I0604 21:55:53.235830       1 controllermanager.go:554] Started "cronjob"
I0604 21:55:53.235894       1 cronjob_controller.go:96] Starting CronJob Manager
I0604 21:55:53.485138       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-serving"
I0604 21:55:53.485181       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-serving
I0604 21:55:53.485188       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.485869       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-client"
I0604 21:55:53.485897       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-client
I0604 21:55:53.485939       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.486545       1 certificate_controller.go:118] Starting certificate controller "csrsigning-kube-apiserver-client"
I0604 21:55:53.486579       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client
I0604 21:55:53.486580       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.487192       1 controllermanager.go:554] Started "csrsigning"
I0604 21:55:53.487290       1 certificate_controller.go:118] Starting certificate controller "csrsigning-legacy-unknown"
I0604 21:55:53.487318       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-legacy-unknown
I0604 21:55:53.487350       1 dynamic_serving_content.go:130] Starting csr-controller::/data/server/tls/client-ca.crt::/data/server/tls/client-ca.key
I0604 21:55:53.534160       1 controllermanager.go:554] Started "csrapproving"
I0604 21:55:53.534259       1 certificate_controller.go:118] Starting certificate controller "csrapproving"
I0604 21:55:53.534286       1 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving
I0604 21:55:53.583704       1 controllermanager.go:554] Started "csrcleaner"
I0604 21:55:53.584604       1 cleaner.go:82] Starting CSR cleaner controller
I0604 21:55:53.632039       1 shared_informer.go:247] Caches are synced for deployment 
I0604 21:55:53.632089       1 shared_informer.go:247] Caches are synced for job 
I0604 21:55:53.633147       1 shared_informer.go:247] Caches are synced for TTL 
I0604 21:55:53.634413       1 shared_informer.go:247] Caches are synced for certificate-csrapproving 
I0604 21:55:53.635365       1 shared_informer.go:247] Caches are synced for daemon sets 
I0604 21:55:53.643906       1 shared_informer.go:247] Caches are synced for crt configmap 
I0604 21:55:53.645150       1 shared_informer.go:247] Caches are synced for endpoint_slice 
I0604 21:55:53.648666       1 shared_informer.go:247] Caches are synced for resource quota 
I0604 21:55:53.653291       1 shared_informer.go:247] Caches are synced for endpoint 
I0604 21:55:53.655024       1 shared_informer.go:247] Caches are synced for endpoint_slice_mirroring 
I0604 21:55:53.664875       1 shared_informer.go:247] Caches are synced for GC 
I0604 21:55:53.668492       1 shared_informer.go:247] Caches are synced for disruption 
I0604 21:55:53.668525       1 disruption.go:339] Sending events to api server.
I0604 21:55:53.680481       1 shared_informer.go:247] Caches are synced for ClusterRoleAggregator 
I0604 21:55:53.681645       1 shared_informer.go:247] Caches are synced for service account 
I0604 21:55:53.682001       1 shared_informer.go:247] Caches are synced for ReplicationController 
I0604 21:55:53.683151       1 shared_informer.go:247] Caches are synced for HPA 
I0604 21:55:53.684464       1 shared_informer.go:247] Caches are synced for PVC protection 
I0604 21:55:53.685604       1 shared_informer.go:247] Caches are synced for stateful set 
I0604 21:55:53.685664       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-serving 
I0604 21:55:53.686842       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-client 
I0604 21:55:53.686907       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kube-apiserver-client 
I0604 21:55:53.688163       1 shared_informer.go:247] Caches are synced for PV protection 
I0604 21:55:53.688185       1 shared_informer.go:247] Caches are synced for certificate-csrsigning-legacy-unknown 
I0604 21:55:53.690290       1 shared_informer.go:247] Caches are synced for namespace 
I0604 21:55:53.694425       1 controller.go:609] quota admission added evaluator for: replicasets.apps
I0604 21:55:53.694868       1 shared_informer.go:247] Caches are synced for ReplicaSet 
I0604 21:55:53.698059       1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set coredns-854c77959c to 1"
I0604 21:55:53.996138       1 shared_informer.go:240] Waiting for caches to sync for resource quota
I0604 21:55:53.996202       1 shared_informer.go:247] Caches are synced for resource quota 
I0604 21:55:54.142095       1 shared_informer.go:240] Waiting for caches to sync for garbage collector
E0604 21:55:54.190863       1 clusterroleaggregation_controller.go:181] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
E0604 21:55:54.191008       1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
E0604 21:55:54.195596       1 clusterroleaggregation_controller.go:181] edit failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "edit": the object has been modified; please apply your changes to the latest version and try again
E0604 21:55:54.197994       1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
I0604 21:55:54.291427       1 event.go:291] "Event occurred" object="kube-system/coredns-854c77959c" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-854c77959c-qtr54"
I0604 21:55:54.342313       1 shared_informer.go:247] Caches are synced for garbage collector 
I0604 21:55:54.382631       1 shared_informer.go:247] Caches are synced for garbage collector 
I0604 21:55:54.382665       1 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
time="2021-06-04T21:55:54.455175576Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.460399269Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.474040009Z" level=info msg="Updated coredns node hosts entry [192.107.92.226 t320.xxx]"
time="2021-06-04T21:55:54.506725940Z" level=warning msg="Unable to remove node password: secrets \"t320.xxx.node-password.k3s\" not found"
time="2021-06-04T21:55:54.510141597Z" level=info msg="Removed coredns node hosts entry [192.107.92.226 t320.xxx]"
time="2021-06-04T21:55:54.521277608Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.525072518Z" level=error msg="No InternalIP found for node t320.xxx"
time="2021-06-04T21:55:54.552516661Z" level=info msg="Updated coredns node hosts entry [192.104.140.162 t320.xxx]"
W0604 21:55:57.752695       1 lease.go:233] Resetting endpoints for master service "kubernetes" to [10.42.0.8]

The host cluster is a free RKE cluster:

k get nodes -o wide
NAME                             STATUS   ROLES               AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION         CONTAINER-RUNTIME
rke-master-01.xxx   Ready    controlplane,etcd   15d   v1.20.6   192.168.140.194   <none>        Debian GNU/Linux 10 (buster)   4.19.0-16-amd64        docker://20.10.6
t320.xxx            Ready    worker              15d   v1.20.6   192.168.140.189   <none>        Debian GNU/Linux 10 (buster)   5.10.0-0.bpo.5-amd64   docker://20.10.6

Any Idea what I can do to connect to the vcluster?

Allow namespace kubeconfig secret is written to configurable.

At present it looks like when using --out-kube-config-secret that the secret lands in the current namespace if --target-namespace is not used, and in the target namespace when --target-namespace is used.

For use case was trying to implement the latter caused problems as there was a deployment that needed to mount the secret but it was being created back in the same namespace where the vcluster control plain was running. So having the secret be created in the target namespace meant that couldn't work.

Things would be more flexible if the namespace the secret for the kubeconfig is created in can be specified, overriding the defaults. This should allow any namespace (expecting that RBAC would be set up by user deploying vcluster to allow creation if a distinct namespace) to be used, giving easy access through mounting of the secret to any in cluster application no matter where it lives.

Add leader election for the syncer container

Keep Labels on ingress sync

The labels seem to be blown away on the synced ingress object, these should probably be kept so that things like cert-manager can still be configured.

vCluster elements stuck in terminate state

After firing the vcluster delete command, i observe the pods are in terminating state.

**root@PoCVM-F5:~# kubectl get all -n host-namespace-1
NAME READY STATUS RESTARTS AGE
pod/coredns-854c77959c-79tqj-x-kube-system-x-vcluster-1 0/1 Terminating 1 6d3h
pod/vcluster-1-0 0/2 Terminating 2 6d3h

root@PoCVM-F5:~# kubectl get all -n host-namespace-2
NAME READY STATUS RESTARTS AGE
pod/coredns-854c77959c-rn8q8-x-kube-system-x-vcluster-1 0/1 Terminating 0 3h12m
pod/vcluster-1-0 0/2 Terminating 0 3h12m
root@PoCVM-F5:~#**

Although the pvc show a different status in different ns

root@PoCVM-F5:~# kubectl get pvc -n host-namespace-1
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-vcluster-1-0 Terminating pvc-36006bd9-e74b-41e7-a09f-f17f2d10c9e6 5Gi RWO standard 6d3h
root@PoCVM-F5:~#
root@PoCVM-F5:~#
root@PoCVM-F5:~#
root@PoCVM-F5:~# kubectl get pvc -n host-namespace-2
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-vcluster-1-0 Bound pvc-2c26ad14-1727-45db-8928-ce6b9f984ac3 5Gi RWO standard 3h12m

Workloads Recreated automatically on deleting and creating the vCluster

After creating a vCluster, as per the getting started documentation, I deployed an nginx workload. Everything worked as expected. After which, I deleted the vCluster and things worked well. Host cluster namespace had no resources. Then, when I created a new vCluster with the same name as the one I deleted before, the workload I had created before also came up and its pods were running too.

Seems to me as if it's a bug.

Another observation, if I delete the host cluster namespace where I create the vCluster and then create the namespace again with the same name and create the vCluster with the same name, the problem is not observed. Seems to be some cache issue or a syncer issue with scheduler.

#Snap1 - Here, I have couple of deployments within the vcluster and its pods are running in the host cluster namespace.

root@PoCVM-F5:~# kubectl get all -n host-namespace-1
NAME READY STATUS RESTARTS AGE
pod/coredns-854c77959c-dcmww-x-kube-system-x-vcluster-1 1/1 Running 0 39h
pod/nginx-deployment-84cd76b964-9k7ts-x-demo-nginx-x-vcluster-1 1/1 Running 0 39h
pod/rook-nfs-operator-5fd99c7c8-dcnnj-x-rook-nfs-system--b1cadd30c5 1/1 Running 0 39h
pod/vcluster-1-0 2/2 Running 0 39h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns-x-kube-system-x-vcluster-1 ClusterIP 10.108.62.254 53/UDP,53/TCP,9153/TCP 39h
service/vcluster-1 ClusterIP 10.99.106.19 443/TCP 39h
service/vcluster-1-headless ClusterIP None 443/TCP 39h

NAME READY AGE
statefulset.apps/vcluster-1 1/1 39h

#Snap2 - Here, I delete the vcluster-1 which was already running within the host-namespace-1 ns in host cluster

root@PoCVM-F5:# vCluster delete vcluster-1 -n host-namespace-1
[info] Delete helm chart with helm delete vcluster-1 --namespace host-namespace-1 --kubeconfig /tmp/910487953 --repository-config=''
[done] √ Successfully deleted virtual cluster vcluster-1 in namespace host-namespace-1
root@PoCVM-F5:# kubectl get all -n host-namespace-1
No resources found in host-namespace-1 namespace.

#Snap3 - Here, I have created ( supposedly ) a new vCluster but with the same name and I see the workloads from the deleted vCluster running on the host-namespace-1

root@PoCVM-F5:# vcluster create vcluster-1 -n host-namespace-1
[info] execute command: helm upgrade vcluster-1 vCluster --repo https://charts.loft.sh --kubeconfig /tmp/000012067 --namespace host-namespace-1 --install --repository-config='' --values /tmp/293892134
[done] √ Successfully created virtual cluster vcluster-1 in namespace host-namespace-1. Use 'vcluster connect vcluster-1 --namespace host-namespace-1' to access the virtual cluster
root@PoCVM-F5:# kubectl get all -n host-namespace-1
NAME READY STATUS RESTARTS AGE
pod/coredns-854c77959c-rnmnx-x-kube-system-x-vcluster-1 1/1 Running 0 33s
pod/nginx-deployment-84cd76b964-9k7ts-x-demo-nginx-x-vcluster-1 1/1 Running 0 36s
pod/rook-nfs-operator-5fd99c7c8-dcnnj-x-rook-nfs-system--b1cadd30c5 1/1 Running 0 36s
pod/vcluster-1-0 2/2 Running 0 41s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns-x-kube-system-x-vcluster-1 ClusterIP 10.103.121.6 53/UDP,53/TCP,9153/TCP 36s
service/vcluster-1 ClusterIP 10.99.6.18 443/TCP 41s
service/vcluster-1-headless ClusterIP None 443/TCP 41s

NAME READY AGE
statefulset.apps/vcluster-1 1/1 41s

Problem with coredns pod's while creating cluster

Hello,

when I install a vcluster vlc in my microk8s cluster, the coredns pods are not started.
This happens every time no matter how many times I try or wait. Do you have any tips on how I can fix it?

Events:
Type     Reason     Age                   From               Message
----     ------     ----                  ----               -------
Normal   Scheduled  2m43s                 default-scheduler  Successfully assigned vcl/coredns-7448499f4d-grq5j-x-kube-system-x-vcl to xxxx
Warning  Unhealthy  2m42s                 kubelet            Readiness probe failed: Get "http://10.1.161.71:8181/ready": dial tcp 10.1.161.71:8181: connect: connection refused
Normal   Pulled     2m1s (x4 over 2m43s)  kubelet            Container image "rancher/coredns-coredns:1.8.3" already present on machine
Normal   Created    2m1s (x4 over 2m43s)  kubelet            Created container coredns
Normal   Started    2m1s (x4 over 2m43s)  kubelet            Started container coredns
Warning  BackOff    88s (x12 over 2m40s)  kubelet            Back-off restarting failed container

In microk8s cluster are following plugins enabled

microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
dns                  # CoreDNS
ha-cluster           # Configure high availability on the current node
helm3                # Helm 3 - Kubernetes package manager
ingress              # Ingress controller for external access
rbac                 # Role-Based Access Control for authorisation
storage              # Storage class; allocates storage from host directory

Thanks for any ideas.

Panic with latest alpha release

The latest 0.3.0-alpha.0 release of the vcluster CLI will panic with the following stack when run (even when requesting for help, e.g. vcluster help):

panic: create flag redefined: release-values

goroutine 1 [running]:
github.com/spf13/pflag.(*FlagSet).AddFlag(0xc00028a400, 0xc000428d20)
        /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/pflag/flag.go:848 +0x825
github.com/spf13/pflag.(*FlagSet).VarPF(0xc00028a400, 0x1835300, 0xc0004289a8, 0x1663e03, 0xe, 0x0, 0x0, 0x16a89dd, 0x3f, 0xc000428c80)
        /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/pflag/flag.go:831 +0x10b
github.com/spf13/pflag.(*FlagSet).VarP(...)
        /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/pflag/flag.go:837
github.com/spf13/pflag.(*FlagSet).StringVar(0xc00028a400, 0xc0004289a8, 0x1663e03, 0xe, 0x0, 0x0, 0x16a89dd, 0x3f)
        /Users/runner/work/vcluster/vcluster/vendor/github.com/spf13/pflag/string.go:37 +0x9b
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.NewCreateCmd(0xc0003cdb48, 0xc000189ef0)
        /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/create.go:108 +0x3c7
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.BuildRoot(0x185ab58, 0x2106ec0, 0xe)
        /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/root.go:57 +0xfe
github.com/loft-sh/vcluster/cmd/vclusterctl/cmd.Execute()
        /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/cmd/root.go:33 +0x47
main.main()
        /Users/runner/work/vcluster/vcluster/cmd/vclusterctl/main.go:16 +0x3e

List vclusters?

Hi,

I'm not sure how hard this would be to implement but when I was playing around with vcluster I found myself at one point wishing there was a "vcluster list" subcommand or something similar, so I could see which vclusters existed in my host cluster. I think the scenario was that I had created a vcluster but lost connection to it, and went to connect again later but I didn't remember what I had named it.

Set port to expose `vcluster create --expose`

In my cluster, using --expose does not work because the Service does not come up:

Error syncing load balancer: failed to ensure load balancer: Service cannot use 80 and 443 port

But if I manually create the load balancer service with port 8443, then it works. So it would come very in handy if I could set the port to be used in the vcluster create command.

Default kind admin clusterrole is not enough to deploy a vcluster

I've already mentioned it in #42 but I thought I'd create a proper issue for it.

When trying to deploy a vcluster using a service account that has a namespaced rolebinding to the admin clusterrole the helm command fails because it tries to adds permissions that the admin clusterrole doesn't have. Below is the error output of vcluster create.

~ $ vcluster create vcluster-1
[info]   execute command: helm upgrade vcluster-1 vcluster --repo https://charts.loft.sh --version 0.3.0 --kubeconfig /tmp/247832214 --namespace vcluster --install --repository-config='' --values /tmp/285100285
[fatal]  error executing helm upgrade vcluster-1 vcluster --repo https://charts.loft.sh --version 0.3.0 --kubeconfig /tmp/247832214 --namespace vcluster --install --repository-config='' --values /tmp/285100285: Error: UPGRADE FAILED: failed to create resource: roles.rbac.authorization.k8s.io "vcluster-1" is forbidden: user "system:serviceaccount:vcluster:default" (groups=["system:serviceaccounts" "system:serviceaccounts:vcluster" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["configmaps"], Verbs:["*"]}
{APIGroups:[""], Resources:["endpoints"], Verbs:["*"]}
{APIGroups:[""], Resources:["events"], Verbs:["*"]}
{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["*"]}
{APIGroups:[""], Resources:["pods"], Verbs:["*"]}
{APIGroups:[""], Resources:["pods/attach"], Verbs:["*"]}
{APIGroups:[""], Resources:["pods/exec"], Verbs:["*"]}
{APIGroups:[""], Resources:["pods/log"], Verbs:["*"]}
{APIGroups:[""], Resources:["pods/portforward"], Verbs:["*"]}
{APIGroups:[""], Resources:["pods/proxy"], Verbs:["*"]}
{APIGroups:[""], Resources:["secrets"], Verbs:["*"]}
{APIGroups:[""], Resources:["services"], Verbs:["*"]}
{APIGroups:[""], Resources:["services/proxy"], Verbs:["*"]}
{APIGroups:["networking.k8s.io"], Resources:["ingresses"], Verbs:["*"]}

The admin clusterrole for kind v0.11.1 contains these permissions:

Name:         admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  rolebindings.rbac.authorization.k8s.io          []                 []              [create delete deletecollection get list patch update watch]
  roles.rbac.authorization.k8s.io                 []                 []              [create delete deletecollection get list patch update watch]
  configmaps                                      []                 []              [create delete deletecollection patch update get list watch]
  endpoints                                       []                 []              [create delete deletecollection patch update get list watch]
  persistentvolumeclaims                          []                 []              [create delete deletecollection patch update get list watch]
  pods                                            []                 []              [create delete deletecollection patch update get list watch]
  replicationcontrollers/scale                    []                 []              [create delete deletecollection patch update get list watch]
  replicationcontrollers                          []                 []              [create delete deletecollection patch update get list watch]
  services                                        []                 []              [create delete deletecollection patch update get list watch]
  daemonsets.apps                                 []                 []              [create delete deletecollection patch update get list watch]
  deployments.apps/scale                          []                 []              [create delete deletecollection patch update get list watch]
  deployments.apps                                []                 []              [create delete deletecollection patch update get list watch]
  replicasets.apps/scale                          []                 []              [create delete deletecollection patch update get list watch]
  replicasets.apps                                []                 []              [create delete deletecollection patch update get list watch]
  statefulsets.apps/scale                         []                 []              [create delete deletecollection patch update get list watch]
  statefulsets.apps                               []                 []              [create delete deletecollection patch update get list watch]
  horizontalpodautoscalers.autoscaling            []                 []              [create delete deletecollection patch update get list watch]
  cronjobs.batch                                  []                 []              [create delete deletecollection patch update get list watch]
  jobs.batch                                      []                 []              [create delete deletecollection patch update get list watch]
  daemonsets.extensions                           []                 []              [create delete deletecollection patch update get list watch]
  deployments.extensions/scale                    []                 []              [create delete deletecollection patch update get list watch]
  deployments.extensions                          []                 []              [create delete deletecollection patch update get list watch]
  ingresses.extensions                            []                 []              [create delete deletecollection patch update get list watch]
  networkpolicies.extensions                      []                 []              [create delete deletecollection patch update get list watch]
  replicasets.extensions/scale                    []                 []              [create delete deletecollection patch update get list watch]
  replicasets.extensions                          []                 []              [create delete deletecollection patch update get list watch]
  replicationcontrollers.extensions/scale         []                 []              [create delete deletecollection patch update get list watch]
  ingresses.networking.k8s.io                     []                 []              [create delete deletecollection patch update get list watch]
  networkpolicies.networking.k8s.io               []                 []              [create delete deletecollection patch update get list watch]
  poddisruptionbudgets.policy                     []                 []              [create delete deletecollection patch update get list watch]
  deployments.apps/rollback                       []                 []              [create delete deletecollection patch update]
  deployments.extensions/rollback                 []                 []              [create delete deletecollection patch update]
  localsubjectaccessreviews.authorization.k8s.io  []                 []              [create]
  pods/attach                                     []                 []              [get list watch create delete deletecollection patch update]
  pods/exec                                       []                 []              [get list watch create delete deletecollection patch update]
  pods/portforward                                []                 []              [get list watch create delete deletecollection patch update]
  pods/proxy                                      []                 []              [get list watch create delete deletecollection patch update]
  secrets                                         []                 []              [get list watch create delete deletecollection patch update]
  services/proxy                                  []                 []              [get list watch create delete deletecollection patch update]
  bindings                                        []                 []              [get list watch]
  events                                          []                 []              [get list watch]
  limitranges                                     []                 []              [get list watch]
  namespaces/status                               []                 []              [get list watch]
  namespaces                                      []                 []              [get list watch]
  persistentvolumeclaims/status                   []                 []              [get list watch]
  pods/log                                        []                 []              [get list watch]
  pods/status                                     []                 []              [get list watch]
  replicationcontrollers/status                   []                 []              [get list watch]
  resourcequotas/status                           []                 []              [get list watch]
  resourcequotas                                  []                 []              [get list watch]
  services/status                                 []                 []              [get list watch]
  controllerrevisions.apps                        []                 []              [get list watch]
  daemonsets.apps/status                          []                 []              [get list watch]
  deployments.apps/status                         []                 []              [get list watch]
  replicasets.apps/status                         []                 []              [get list watch]
  statefulsets.apps/status                        []                 []              [get list watch]
  horizontalpodautoscalers.autoscaling/status     []                 []              [get list watch]
  cronjobs.batch/status                           []                 []              [get list watch]
  jobs.batch/status                               []                 []              [get list watch]
  daemonsets.extensions/status                    []                 []              [get list watch]
  deployments.extensions/status                   []                 []              [get list watch]
  ingresses.extensions/status                     []                 []              [get list watch]
  replicasets.extensions/status                   []                 []              [get list watch]
  ingresses.networking.k8s.io/status              []                 []              [get list watch]
  poddisruptionbudgets.policy/status              []                 []              [get list watch]
  serviceaccounts                                 []                 []              [impersonate create delete deletecollection patch update get list watch]

You can reproduce this issue using the script below. You only require kind and kubectl in your path. Just write it to a file, give it the executable flag and execute it. It is idempotent so you can change any step and it wil just re run it if required.

#!/bin/bash

set -eux

if [ ! $(kind get clusters) ]; then
        kind create cluster --wait 1m
fi

# create a rolebinding to the default service account in the default namespace so our test pod
# has the admin role.
if [ ! $(kubectl get rolebinding namespace-admin) ]; then
        kubectl create rolebinding namespace-admin --serviceaccount default:default --clusterrole admin
fi

# start an ubuntu container that we can use as a base.
if [ ! $(kubectl get pod test-container) ]; then
        kubectl run test-container --image docker.io/library/ubuntu:20.04 -- sleep infinity
fi

kubectl wait --for condition=Ready pod test-container

# exec inside our test contaienr and install kubectl, helm and vcluster
kubectl exec -i test-container -- sh <<EOF
        set -eux

        apt-get update -qq
        apt-get install -qqy wget

        if [ ! -f /usr/local/bin/kubectl ]; then
                wget -qO /usr/local/bin/kubectl https://dl.k8s.io/release/v1.21.0/bin/linux/amd64/kubectl
                chmod +x /usr/local/bin/kubectl
        fi

        if [ ! -f /usr/local/bin/vcluster ]; then
                wget -qO- "https://github.com/loft-sh/vcluster/releases/latest" | sed -nE 's!.*"([^"]*vcluster-linux-amd64)".*!https://github.com\1!p' | xargs -n 1 wget -qO /usr/local/bin/vcluster
                chmod +x /usr/local/bin/vcluster
        fi

        if [ ! -f /usr/local/bin/helm ]; then
                wget -qO- https://get.helm.sh/helm-v3.6.1-linux-amd64.tar.gz | gzip -d | tar -O -x -f - linux-amd64/helm > /usr/local/bin/helm
                chmod +x /usr/local/bin/helm
        fi

        # test if the container can execute kubectl commands against the kind cluster
        kubectl get pod

        # print the privileges of the service account for debugging
        kubectl auth can-i --list

        # finally try to create a vcluster, this will fail!
        vcluster create my-failing-cluster
EOF

core-dns failing in civo cluster

Issue
Unable to create vcluster in Civo kubernetes cluster which internally uses k3s. Able to connect to api-server but unable to deploy anything as coredns is failing for vcluster

Steps

Created namespace; kubectl create namespace vcluster-1
Used vcluster cli: vcluster create vcluster-1 -n vcluster-1
Connect to cluster: vcluster connect vcluster-1 -n vcluster-1
Tried deploying nginx, it stays in pending state inside vcluster but running state on host.
coredns container is in Pending state for vcluster and ContainerCreating state for host

Logs
Events from describe command of core-dns

Unable to attach or mount volumes: unmounted volumes=[config-volume], unattached volumes=[config-volume coredns-token-mbwt2]: timed out waiting for the condition
MountVolume.SetUp failed for volume "config-volume" : configmap references non-existent config key: NodeHosts

Self fix or debug
Tried changing default storageclass in cluster from civo-volume from provisioner csi.civo.com to local-path from provisioner rancher.io/local-path but same result.

Cluster Type: Managed Cluster based on k3s by Civo.com

Describe how you compare with Kubernetes WG-Multi Tenancy implementation of Virtual Clusters

How does your implementation compare with https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster?

Are you aware of, or working with, this team?

Rewriting secrets in the azureFile volume

I'm having an issue with the applications deployed in vCluster, whose secret is defined in k8s inline mount. When trying to deploy the application, the secret is looked up in the wrong path.

Example of application deployment:
Host cluster -> vcluster (namespace) -> application (namespace) -> secret

Instead of looking for the secret in the application namespace (inside vCluster), the deployment looks it up in the vcluster/secret namespace. I would expect deployment to search the secret in vCluster's namespace, where the mount and resource are defined.

Here is the k8s resource for the inline mount sample:

...
    volumeMounts:
    - mountPath: /some/path/file
    name: secret
volumes:
- azureFile:
    secretName: secret-name
    shareName: sharexyz
    name: secret
...

Would it be possible to have vCluster rewriting the secrets in the azureFile volume?