Comments (6)
Related issue in Kubernetes Provider repo: crossplane-contrib/provider-kubernetes#105
from provider-helm.
I'm encountering the same issue when trying to connect to an AKS cluster with authentication based on Azure AD Service Principals and Managed Identities. The error message indicates that the kubelogin
executable is not found.
Error Details:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning CannotConnectToProvider 23m (x10 over 27m) managed/release.helm.crossplane.io cannot create new Kubernetes client: cannot create Kubernetes client: Get "https://test123-aks-23c35881-9a9e-4ee3-aef9-2b4cc02d7d00-rq3nt806.hcp.westus.azmk8s.io:443/api?timeout=32s": getting credentials: exec: executable kubelogin not found
It looks like you are trying to use a client-go credential plugin that is not installed.
To learn more about this feature, consult the documentation available at:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
kubelogin is not installed which is required to connect to AAD enabled cluster.
To learn more, please go to https://aka.ms/aks/kubelogin
Provided Kubeconfig:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xyz==
server: https://test123-aks-23c35881-9a9e-4ee3-aef9-2b4cc02d7d00-rq3nt806.hcp.westus.azmk8s.io:443
name: test123-aks-j5wsb-f2b4q
contexts:
- context:
cluster: test123-aks-j5wsb-f2b4q
user: clusterUser_start123-rg_test123-aks-j5wsb-f2b4q
name: test123-aks-j5wsb-f2b4q
current-context: test123-aks-j5wsb-f2b4q
kind: Config
preferences: {}
users:
- name: clusterUser_start123-rg_test123-aks-j5wsb-f2b4q
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- get-token
- --environment
- AzurePublicCloud
- --server-id
- 6dae42f8-1234-4678-1234-3960e28e3630
- --client-id
- 80faf920-1234-4b52-1234-a8e7bedfc67a
- --tenant-id
- b9925bc4-1234-4c37-1234-fa456d1bb1c7
- --login
- devicecode
command: kubelogin
env: null
installHint: |2
kubelogin is not installed which is required to connect to AAD enabled cluster.
To learn more, please go to
https://aka.ms/aks/kubelogin
from provider-helm.
using the following manifest for kubernetesCluster:
apiVersion: containerservice.azure.upbound.io/v1beta1
kind: KubernetesCluster
metadata:
name: test123-aks-j5wsb-wdh8r
spec:
deletionPolicy: Delete
forProvider:
apiServerAccessProfile:
- vnetIntegrationEnabled: true
autoScalerProfile:
- emptyBulkDeleteMax: "10"
expander: random
maxGracefulTerminationSec: "600"
maxNodeProvisioningTime: 15m
maxUnreadyNodes: 3
maxUnreadyPercentage: 45
newPodScaleUpDelay: 0s
scaleDownDelayAfterAdd: 10m
scaleDownDelayAfterDelete: 10s
scaleDownDelayAfterFailure: 3m
scaleDownUnneeded: 10m
scaleDownUnready: 20m
scaleDownUtilizationThreshold: "0.5"
scanInterval: 10s
skipNodesWithSystemPods: true
azureActiveDirectoryRoleBasedAccessControl:
- adminGroupObjectIds:
- 6f49479a-1234-480b-1234-741b51a973a1
azureRbacEnabled: true
managed: true
tenantId: b9925bc4-1234-4c37-1234-fa456d1bb1c7
azurePolicyEnabled: true
defaultNodePool:
- customCaTrustEnabled: false
enableAutoScaling: true
enableHostEncryption: false
enableNodePublicIp: false
fipsEnabled: false
kubeletDiskType: OS
maxCount: 10
maxPods: 30
minCount: 1
name: nodepool1
nodeCount: 1
onlyCriticalAddonsEnabled: false
orchestratorVersion: 1.27.3
osDiskSizeGb: 128
osDiskType: Ephemeral
osSku: Ubuntu
scaleDownMode: Delete
type: VirtualMachineScaleSets
ultraSsdEnabled: false
vmSize: Standard_DS3_v2
workloadRuntime: OCIContainer
dnsPrefix: test123-aks-d679da55-b462-415e-8b04-7487ac5bbd3d
identity:
- identityIds:
- /subscriptions/038f2b7c-1234-43b8-1234-c9ad5da610a8/resourceGroups/start123-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test123-aks-j5wsb-q46cg
type: UserAssigned
imageCleanerIntervalHours: 48
kubernetesVersion: 1.27.3
location: westus
networkProfile:
- networkPlugin: azure
nodeResourceGroup: MC_start123-rg_test123-aks-j5wsb-wdh8r_westus
oidcIssuerEnabled: true
omsAgent:
- logAnalyticsWorkspaceId: /subscriptions/038f2b7c-1234-43b8-1234-c9ad5da610a8/resourceGroups/start123-rg/providers/Microsoft.OperationalInsights/workspaces/test123-aks-j5wsb-clzf4
msiAuthForMonitoringEnabled: true
publicNetworkAccessEnabled: true
resourceGroupName: start123-rg
roleBasedAccessControlEnabled: true
runCommandEnabled: true
skuTier: Standard
tags:
app: test123
env: dev
namespace: start123
windowsProfile:
- adminUsername: azureuser
workloadIdentityEnabled: true
initProvider: {}
managementPolicies:
- '*'
providerConfigRef:
name: default
writeConnectionSecretToRef:
name: 7867facd-54cd-42ed-8402-bff9af903ba0-akscluster
namespace: upbound-system
from provider-helm.
I was playing around with the missing kubelogin - my first idea was to ship the kubelogin simply with the Dockerfile. The result is the following:
kubectl logs -n upbound-system crossplane-contrib-provider-helm-1e553bfdff7a-5f968cb5d8-49kvm
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
Error: failed to get token: expected an empty error but received: AzureCLICredential: fork/exec /bin/sh: no such file or directory
The next thing I tried was to add the Azure CLI, which relies on a lot of Python. For that reason, I switched the base image and needed to install many other tools. But the result is still as follows:
kubectl logs -n upbound-system crossplane-contrib-provider-helm-4a269aae5ec5-6cd44f9464-sj8h7
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
Error: failed to get token: expected an empty error but received: AzureCLICredential: ERROR: Please run 'az login' to set up the account.
What I tried then was to exec into the provider and did an az login
with device authentication. After that, the Helm release was synced, and ready true.
I think adding the right credentials could do the trick, but overall, this way is not the best because the Docker image is more than 650MB, and we rely on some CLI and binary tools. Therefore, we should try to use the suggested kubelogin Go library for better integration.
for reference:
# Use Alpine as the base image
FROM alpine
ARG TARGETOS
ARG TARGETARCH
ENV HELM_CACHE_HOME /tmp/helm-cache
ENV HELM_CONFIG_HOME /tmp/helm-config
ENV HELM_DATA_HOME /tmp/helm-data
# Add necessary binaries
ADD bin/$TARGETOS\_$TARGETARCH/provider /usr/local/bin/provider-helm
# Download and unzip kubelogin binary
ADD https://github.com/Azure/kubelogin/releases/download/v0.0.33/kubelogin-${TARGETOS}-${TARGETARCH}.zip /tmp/kubelogin.zip
RUN unzip /tmp/kubelogin.zip -d /tmp && \
mv /tmp/bin/$TARGETOS\_$TARGETARCH/kubelogin /usr/local/bin/ && \
rm -rf /tmp/kubelogin.zip /tmp/bin
# Install Azure CLI
RUN apk add --no-cache --update python3 py3-pip && \
apk add --no-cache --update --virtual=build gcc musl-dev python3-dev libffi-dev openssl-dev cargo make && \
pip3 install --no-cache-dir --prefer-binary azure-cli && \
apk del build
# Create .azure directory
RUN mkdir -p /home/65532/.azure && chown -R 65532:65532 /home/65532/.azure
USER 65532
ENTRYPOINT ["provider-helm"]
from provider-helm.
crossplane-contrib/provider-upjet-azure#459
from provider-helm.
Thanks a lot @haarchri for your investigations. I agree that we should better fix this in the go code rather than introducing binaries and other dependencies as you've find out.
I believe we can come up with a similar API and solution as #109 here, and started looking the code at https://github.com/Azure/kubelogin. Based on my observations so far, the code there is not friendly to be imported as a library with private methods and inflexible options in the exposed methods. I will dig further.
from provider-helm.
Related Issues (20)
- Ability to store secrets in vault
- On Helm release with the specified option pullSecretRef does not find the Secret in the namespace HOT 1
- Support aws s3 bucket as a helm repository HOT 8
- Establish Ownership and Visibility of Helm Resources in ArgoCD via OwnerReferences HOT 2
- Passing args to configure the Helm provider package to configure the controller and be more verbose HOT 2
- make: *** No rule to make target `local.up', needed by `local-dev'. Stop. HOT 3
- Retry should be enabled by default HOT 1
- Add Proxy Support
- howto debugging reconciling loop HOT 1
- Installation of provider-helm fails with "resource name may not be empty" in air-gapped environment. HOT 1
- Unable to pull Charts from Private Registry HOT 4
- Add x-kubernetes-map-type: granular to release values to support serve side apply
- identity section for AzureAD auth in ProviderConfig should not be processed when the supplied kubeconfig does not require AzureAD auth
- Enable support for Management Policies in Helm Provider HOT 1
- What is the purpose of putting an `*` as an option in an enum?? HOT 2
- Drop v1alpha1 or add conversion webhooks HOT 13
- Release Ready status should be aggregation of readines of object deployed within a release
- Observe data from resources created by Release object
- ProviderConfig does not support "InjectedIdentity" as `source.identity`
- Rollback feature doesn't work as expected.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from provider-helm.