zalando-incubator/kube-aws-iam-controller

Distribute different AWS IAM credentials to different pods in Kubernetes via secrets.

kube-aws-iam-controller's Issues

Deprecate pod discovery

Deprecate and later remove the feature of automatically discovering IAM roles by looking at pod with a secret mount named aws-iam-*

The feature is very limited and has been replaced by #13

Obsolete with new AWS feature ("Fine-Grained IAM Roles for Service Accounts")?

See https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

Twitter thread: https://twitter.com/mhausenblas/status/1169292857108324353

OSS repo: https://github.com/aws/amazon-eks-pod-identity-webhook

.NET SDK support?

We are a .NET Core shop and looking at migrating workloads from static json creds files on GCP? GKE to AWS/EKS. Working on possible solutions for credential handling for EKS. This project looks great but the lack of specific documentation and support for the AWS .NET SDK has me wondering if this can work. Any guidance or knowledge on the topic? Thanks!

Support for all AWS partitions

Currently the controller hard codes the arnPrefix making it not possible to use it within environments such as AWS GovCloud where the ARN prefix differs: e.g. arn:aws-us-gov:iam::.

I am happy to create a patch but wondered how you would want this implemented as I see two directions:

• User provided via a flag making it the operators duty to toggle the configuration in alternative partitions
• Parsed from the user provided or autodiscovered base role ARN

Let me know if you have any thoughts or ideas regarding the implementation details.

Thanks for the great project!

Role which use of paths GetCredentialsFailed

I tried CRD method

apiVersion: amazonaws.com/v1
kind: AWSIAMRole
metadata:
  name: alb-ingress-controller-role
spec:
  # The roleReference allows specifying an AWS IAM role name or arn
  # Possible values:
  #   "aws-iam-role-name"
  #   "arn:aws:iam::<account-id>:role/aws-iam-role-name"
  roleReference: "arn:aws:iam::XXXX:role/k8s/kube-system/alb-ingress-controller_role"

Error occurred GetCredentialsFailed in kube-aws-iam-controller.

time="2019-05-09T11:55:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"AWSIAMRole\", Namespace:\"kube-system\", Name:\"alb-ingress-controller-role\", UID:\"ef0dadf2-724c-11e9-85ed-0a61e5f811aa\", APIVersion:\"amazonaws.com/v1\", ResourceVersion:\"20473920\", FieldPath:\"\"}): type: 'Warning' reason: 'GetCredentialsFailed' Failed to get creedentials for role 'arn:aws:iam::XXXX:role/k8s/kube-system/alb-ingress-controller_role': invalid roleARN: arn:aws:iam::XXXX:role/k8s/kube-system/alb-ingress-controller_role"

When I tried on a Role that does not include path, no error occurred.

kube-aws-iam-controller deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "6"
  creationTimestamp: 2019-05-09T11:08:42Z
  generation: 6
  labels:
    app: kube-aws-iam-controller
    chart: kube-aws-iam-controller-0.0.1
    heritage: Tiller
    release: kube-aws-iam-controller
  name: kube-aws-iam-controller
  namespace: operator
  resourceVersion: "20474580"
  selfLink: /apis/extensions/v1beta1/namespaces/operator/deployments/kube-aws-iam-controller
  uid: ce0bd319-724a-11e9-85ed-0a61e5f811aa
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kube-aws-iam-controller
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kube-aws-iam-controller
    spec:
      containers:
      - args:
        - --interval=10s
        - --refresh-limit=10m
        command:
        - /kube-aws-iam-controller
        image: mikkeloscar/kube-aws-iam-controller:v0.0.6
        imagePullPolicy: IfNotPresent
        name: kube-aws-iam-controller
        resources:
          limits:
            cpu: 100m
            memory: 64Mi
          requests:
            cpu: 25m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        node-role.kubernetes.io/master: ""
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-aws-iam-controller
      serviceAccountName: kube-aws-iam-controller
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-05-09T11:11:30Z
    lastUpdateTime: 2019-05-09T11:11:30Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2019-05-09T11:08:42Z
    lastUpdateTime: 2019-05-09T11:48:43Z
    message: ReplicaSet "kube-aws-iam-controller-7c8b58d7df" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 6
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

In this document,

    // Use the role session name to uniquely identify a session when the same role
    // is assumed by different principals or for different reasons. In cross-account
    // scenarios, the role session name is visible to, and can be logged by the
    // account that owns the role. The role session name is also used in the ARN
    // of the assumed role principal.

I don't care much about it, butI tried to patch code and then no error with role include path.

cw-sakamoto@451af42
cw-sakamoto@43c035c

Emit events to pods

The controller should emit events to pods when it's unable to setup the IAM credentials requested.

Connect ServiceAccounts and AWS IAM roles

Currently the controller expects the AWS IAM roles to be created separately and available when pods are created. It would be nice to connect the AWS IAM role with a Kubernetes serviceAccounts and use the native service accounts as the only identity needed to be specified by a pod.

The way it would work is that users would specify a serviceAccount like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-application
  annotations:
    iam.amazonaws.com/role-policy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Action": "route53:*",
            "Resource": "*",
            "Effect": "Allow"
          }
        ]
      }

The iam.amazonaws.com/role-policy would include an AWS IAM role policy for which the controller would automatically provision a AWS IAM role and provide it to the pod who specified the given serviceAccount in the pod spec.

This would not only keep the AWS identity and the Kubernetes identity connected it would also make things simpler for the user, e.g. they would not need to specify a trust relationship on the AWS IAM roles they are creating, as it would be done transparently.

In cases where an AWS IAM role already exists, it could just connect it to a serviceAccount like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-application
  annotations:
    iam.amazonaws.com/role: my-role-name

Taking control of seed credential

When running in a non-AWS environment, we have to seed a credential in the system. Ideally, it would be great for that credential to then be managed by the controller so it gets auto-rotated. These are the steps I took to make it work, but wanted to check with you all to see if this is the anticipated route (or talk about another route).

1. I first created the seed credential named kube-aws-iam-controller

2. I created an AWSIAMRole with the same name in the same namespace (in my case, kube-aws-iam-controller)

3. When the controller sees the role, it tries to create credentials, but can't because:

   reason: 'CreateSecretFailed' Failed to create secret kube-system/kube-aws-iam-controller-iam-role with credentials: secrets \"kube-aws-iam-controller-iam-role\" already exists"
   

4. Looking at the source, I found that you use a label named heritage with a value of kube-aws-iam-controller to know the secret is being managed by the controller. I added this label to the seed secret.

5. The controller then sees the secret doesn't have an expire key, so auto-rotates the credential.

Since then, I seem to be in a stable state. But, I fully recognize labels aren't normally part of public API, so are subject to change. Is there another approach I should take or any recommendations on a route to "adopt an existing secret"?

Are credentials persistent?

I need a mechanism where credentials obtained by Pod are persistent. Are they persistent based on IAM role or they are valid for certain amount of time and must be obtained again?

Monitoring IAM Controller

Any plans of adding monitoring and metrics related to functioning of iam controller? Any ideas around how it can be implemented?

Update CRD apiVersion to v1

v1beta1 is deprecated as of Kubernetes 1.19

Embedding role identifier in secret name precludes use of paths

At present, kube-aws-iam-controller allows pod authors to nominate an IAM role by way of appending that role's name to a sentinel prefix, forming a Kubernetes Secret name. kube-aws-iam-controller forms the role's complete ARN by concatenating either a designated or discovered base ARN with the suffix of the Secret name. Since Kubernetes Secret names can't contain forward slash characters, users can't designate IAM roles with paths that contain such slashes.

It's possible for a user to specify an IAM role ARN prefix—including a path—by using the --base-role-arn command-line flag. If all the roles nominated by pods happen to all live under that same path, the current concatenation scheme does allow forming role ARNs with paths. However, if pod authors will want to nominate IAM roles under different paths, they can't embed those paths in Secret names.

kiam doesn't have this problem because the annotation value it reads for nominating an IAM role ARN has no syntactic constraints. Secret names—and all Kubernetes object names—are too limited for this purpose.

Consider an IAM role with an ARN like arn:aws:iam::123456789012:role/division/department/team/ReadAssets. We can discover the base role ARN as arn:aws:iam::123456789012:role, and we can include the role name "ReadAssets" in a Secret name ("aws-iam-ReadAssets"), but concatenating the two parts gives us the wrong ARN: arn:aws:iam::123456789012:role/ReadAssets. Against this, we can't name our Secret "aws-iam-role/division/department/team/ReadAssets."

If we specified a --base-role-arn flag value of "arn:aws:iam::123456789012:role/division/department/team/" then we could name our Secret "aws-iam-ReadAssets" and get the right ARN for that role, but another pod author couldn't then nominate a role with an ARN under a different path like arn:aws:iam::123456789012:role/division/project/mint/CreateMoney.

The project documentation currently bears this promise or threat:

This way of specifying the role on pod specs are subject to change.

I read the rest of the paragraph as a concern for the pod authors' convenience, but there's also this gap in capability. I do like the current design, so perhaps we could augment it just enough by accepting an optional annotation on the pods to designate an ARN path. In the example above, the Secret name would be "aws-iam-ReadAssets," and the yet-to-be-named annotation value would be "division/department/team." IAM does have this odd identifier model, where the object names are global within the containing AWS account, yet the ARNs can include this path, and I don't think that you can omit a path from an ARN.

Roles also have unique IDs that a Secret name could accommodate syntactically. Unfortunately, the STS AssumeRoleInput type doesn't accept such a unique ID in lieu of a role ARN.

Have you considered this problem before? What do you think about the proposed optional annotation for specifying an ARN path?

Is support for JS AWS SDK available?

As seen in this PR: aws/aws-sdk-js#1923

It does seem they merged the support for credential_process, but in https://github.com/zalando-incubator/kube-aws-iam-controller/blob/master/docs/sdk-configuration.md does not seem to be yet available the version in which they put it

Can someone with knowledge on this specific SDK clarify this?

Thanks!

Incomplete setup documentation

I have tried to seutp kube-aws-iam-controller as documented but have not been able to do so even after several attempts. At first instance I got error when applying the deployment that kube-aws-iam-controller service account not found. I explicitly created service-account and granted clusterwide role to get, list and watch secrets (which is undocumented). After this the deployment succeeds but I get the following error from the pod:

level=error msg="secrets is forbidden: User \"system:serviceaccount:kube-system:kube-aws-iam-controller\" cannot list resource \"secrets\" in API group \"\" at the cluster scope"

Controller is vulnerable to being a confused deputy (without an external ID)

In the current design, the controller creates a Kubernetes Secret object when it notices one or more pods that wish to mount a Secret with the right name. Any pods within that namespace can mount that same Secret object, regardless of governing RBAC permissions, and any pods in any namespace can assume whichever IAM roles trust the controller's initial role. That's a broad entitlement for role delegation.

In kiam, there's a way to limit which IAM roles pod authors can assume within a namespace, but even that doesn't go far enough: It allows the Kubernetes administrator some control over IAM role assumption, but it does not help the IAM roles' owners limit their trust scope. The IAM roles' owners must trust kiam or kube-aws-aim-controller for all the pods in an entire Kubernetes cluster; essentially, the role owners must trust an entire cluster. That's too broad of a scope with a multi-tenant cluster, where an IAM role owner might trust only a namespace, or only particular pods within the namespace.

By forcing the IAM role owners to trust the entire cluster, kube-aws-iam-controller becomes a confused deputy. Pod authors in one namespace may be able to learn the names of Secret objects in another namespace—as some multi-tenant clusters allow inter-namespace observation, if not mutation—and once they have that role name, they too can use it, even if the given IAM role owner only intended pods for one tenant to be able to assume it.

One way around this problem as prescribed by AWS is to use an External ID. When assuming the IAM role nominated by the Secret name suffix, the assuming STS client will include an external ID specific to the requesting party—in this case, a Kubernetes pod—that the IAM role owner can assess in his trust policy for the role being assumed.

Here, kube-aws-iam-controller could supply three pieces of information to the IAM role owner:

• pod name
• containing namespace
• cluster ID (optional)

With these three pieces, an IAM role owner could limit the scope of his trust, down from many Kubernetes clusters—in the case where kube-aws-iam-controller's initial assumed role is shared among clusters—to a single cluster, further down to a single namespace within a cluster, and even further down to a prefix name of the pods within a namespace.

The external ID syntax would need to preclude false aliasing. We know that neither namespace names nor pod names can contain a forward slash, but we have no idea what constitutes a cluster ID, so we can construct the external ID as

namespace name / pod name [ / cluster ID ]

treating the suffix as optional, only included if kube-aws-iam-controller is configured with a nonempty cluster ID. It may be useful to forbid a cluster ID to include a forward slash character ('/').

With this external ID include in its calls to sts:AssumeRole, kube-aws-iam-controller would allow the IAM role owner to include conditions in its trust policy, using the string condition operators to match expected values:

Constraint	Condition
Within namespace	{"StringLike": {"sts:ExternalId": "trusted-ns/*"}}
Pods with generated name	{"StringLike": {"sts:ExternalId": "trusted-ns/server-*"}}
Within cluster	{"StringLike": {"sts:ExternalId": "*/*/cluster-1"}}
Within specific pod	{"StringLike": {"sts:ExternalId": "trusted-ns/server-0/cluster-1"}}

Note that due to the limited capability of the StringLike operator, the "Within cluster" example above is vulnerable to false aliasing if a cluster ID can contain forward slashes, assuming that the StringLike operator uses greedy matching for wildcards. Likewise, it is difficult to match a pod name prefix of, say, "server-" when another set of pods could use the prefix "server-impostor-."

Some refinement is possible for the external ID format, but I hope this makes the idea clear enough: Without an external ID, kube-aws-iam-controller's trust scope is too broad for some environments, and AWS already has a way of narrowing such a scope.

The main problem I see here is that Kubernetes Secret names are scoped to the containing namespace, so kube-aws-iam-controller can't easily include a pod name in an external ID; any number of pods in the same namespace could mount the Secret, so long as the first pod's projected external ID satisfies the assumed IAM role's trust criteria. Trying to create a Secret that only some pods within a namespace could make use of is complicated, if not impossible. (Perhaps a validating admission control plugin could handle that, though an IAM role's trust criteria could change after the pod is created.) Given that, omitting the pod name and using just the namespace name and optional cluster ID (with the format reduced to namespace name / [ cluster ID ]) would still be an improvement.

I'd like to hear your thoughts on the proposal. In the meantime, thank you for your work on the project!

the "awsiamroles" crd is broken

I've tried to apply the docs/aws_iam_role_crd.yaml file and got the following error:

$ kubectl apply -f aws_iam_role_crd.yaml

The CustomResourceDefinition "awsiamroles.zalando.org" is invalid: 
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].properties[AssumeRolePolicyDocument].properties[Statement].items.properties[Principal].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].properties[AssumeRolePolicyDocument].properties[Statement].items.type: Required value: must not be empty for specified array items
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].properties[AssumeRolePolicyDocument].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].properties[Policies].properties[PolicyDocument].properties[Statement].items.type: Required value: must not be empty for specified array items
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].properties[Policies].properties[PolicyDocument].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].properties[Policies].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.properties[spec].properties[roleDefinition].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.properties[status].type: Required value: must not be empty for specified object fields
* spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root

Am I missing something here?

Allow controller to assume an intermediate IAM role at startup

The vaguely similar kiam controller allows use of what it calls (confusingly) a "Server role"—an IAM role that the controller assumes when it starts, and which other application-scoped IAM roles must mention in a trust relationship in order for the controller to be able to assume them on behalf of requesting pods.

By offering this feature, application IAM role authors don't have to trust the IAM role initially held by the controller pod, which is likely to be the IAM role of the EC2 instance on which the controller pod is running. These instance-level roles are often established per cluster, if not per machine, and are neither sufficient abstract nor stable for application authors to reference in their trust relationships.

Consider emulating this feature here, introducing an optional command-line flag that allows specifying this intermediate "pod delegate" IAM role. If specified, the controller would immediately attempt to assume this role when it starts.

Adding this would require augmenting the documentation to show a more complete example of the following:

• The initial (machine-level) IAM role must allow assumption of the intermediate "pod delegate" role.
• The intermediate "pod delegate" role must trust the initial (machine-level) role.
• The intermediate "pod delegate" role must allow assumption of either all roles, or some narrower subset that encompasses what the cluster operators will allow.
• An application author's IAM role must trust the intermediate "pod delegate" role.

With this in place, application authors never need to know about these initial (machine-level) roles.

ValidationError session-name

When I tried to check CRD merged in #13 , I got the following error.

time="2019-05-07T02:31:53Z" level=error msg="Failed to get credentials for role kube-fluentd-operator-role: ValidationError: 1 validation error detected: Value 'arn_aws_iam__XXXXXX_role.kube-fluentd-operator-role-session' at 'roleSessionName' failed to satisfy constraint: Member must have length less than or equal to 64\n\tstatus code: 400, request id: 4690bf63-7070-11e9-bc91-XXXXXX"

Secret content places specific requirements on pods

The Kubernetes Secrets created by the controller include two files: credentials.json, which is refreshed regularly, and credentials.process, which remains fixed over time. The content of the latter file is as follows:

[default]
credential_process = cat /meta/aws-iam/credentials.json

This content forces pods that wish to refresh their credentials regularly to accommodate the following:

• cat must be present in the container image.
  The documentation for the Python and Go SDKs mentions this requirement today.
• sh must be present in the container image.
  Using the aws.credentials.processcreds.ProcessProvider type invokes sh -c as the prefix to the specified command.
• The pod must mount the Secret at /meta/aws-iam.
  The examples in the documentation show this, but don't mention that it's mandatory when using the credentials.process file.

The last one surprised me, but as I started thinking of how to work around it, I discovered that the AWS SDKs make no promises about the current working directory of the command invoked by way of the credential_process directive. Presumably it inherits the working directory of the parent process.

It may not seem like a significant imposition on pod authors, but it warrants documenting the constraint if we can't find a workaround. One approach is to use an init container to write the equivalent of the credentials.process into a mounted "emptyDir" volume, which would allow the pod author to point at a credentials.json file mounted elsewhere. Doing that probably requires another fifteen lines of configuration.

The first two impositions make it difficult to use the scratch base image, or something similarly slim like gcr.io/distroless/base, both popular with programs written in Go. Again, it may be possible to use an init container to copy cat and sh into a mounted volume, in order to allow use of an existing container image that lacks these tools.

I had thought that it would be worth mentioning in the documentation that applications that only need to read and use credentials very early in their initialization stage (such as reading an S3 object when starting) could safely read the credentials.json file, since it wouldn't matter if the credentials expired afterward. However, the credentials.json file is refreshed on a schedule that's not synchronized with any pod's lifecycle. It's possible that a pod starts running and mounts the credentials.json file just before those credentials are about to expire. The controller does offer the --refresh-limit command-line flag to adjust this tolerance period, but it would be disingenuous to claim that every pod is guaranteed to start with a credentials.json file that will be valid for that period.

This is partly a request for augmenting the documentation, and partly a request for ideas for how we could reduce these requirements on containers that consume the Secrets published by this controller. Don't get me started on AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.