aws/eks-anywhere-packages

Amazon EKS Anywhere Curated Packages

Build status:

---

The Amazon EKS Anywhere Curated Packages are only available to customers with the Amazon EKS Anywhere Enterprise Subscription. To request a free trial, talk to your Amazon representative or connect with one here.

---

EKS Anywhere Curated Packages is a management system for installation, configuration and maintenance of additional components for your Kubernetes cluster. Examples of these components may include Container Registry, Ingress, and LoadBalancer, etc.

Here are the steps for getting started with EKS Anywhere Curated Packages.

Development

EKS Anywhere Curated Packages is tested using Prow, the Kubernetes CI system. EKS operates an installation of Prow, which is visible at https://prow.eks.amazonaws.com/. Please read our CONTRIBUTING guide before making a pull request.

The dependencies which make up EKS Anywhere Curated Packages are defined and built via the build-tooling repo.

Local Development

Local development can be done using tilt.

Setup

• install tilt binary on your machine following instructions
• install and configure amazon-ecr-credential-helper
• set REGISTRY and KUBERNETES_CONTEXTS env var:

export REGISTRY='public.ecr.aws/<your-public-ecr-registry-id>'
export KUBERNETES_CONTEXTS=$(kubectl config current-context)

If running tilt on a remote host, you can port-forward tilt's web UI by forwarding over ssh:

ssh -v -L 10350:localhost:10350 <remote-host>

After running tilt up, tilt's UI should now be available at localhost:10350 on your local machine.

Security

If you discover a potential security issue in this project, or think you may have discovered a security issue, we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public GitHub issue.

License

This project is licensed under the Apache-2.0 License.

eks-anywhere-packages's Issues

Don't match errors on strings

In general, matching errors based on strings like this is a no-no. After the release, let's fix this up.

Originally posted by @ewollesen in #178 (comment)

Use Code Build build number instead of hash for dev builds

Helm is very particular about versions and it would be better if our dev builds used the code build build number (if it is available) instead of the git hash. That way our dev builds would be more useful because we could do upgrades just with another build rather than updating GIT_TAG. Also, the current format of the version in the Chart.yaml and image tag are technically not correct.

Disclaimer on Controller Install

The CLI should have this disclaimer when controller install:

Preview and pricing disclaimer
The EKS Anywhere package controller and the EKS Anywhere Curated Packages (referred to as “features”) are provided as “preview features” subject to the AWS Service Terms, (including Section 2 (Betas and Previews)) of the same. During the EKS Anywhere Curated Packages Public Preview, the AWS Service Terms are extended to provide customers access to these features free of charge. These features will be subject to a service charge and fee structure at ”General Availability“ of the features.

Don't expose passwords via the command-line

We're exposing passwords to the process table here.

Originally posted by @ewollesen in #184 (comment)

There are lots of places in our release files where we do this.

Package installation sometimes takes too much memory

1. Install a large chart (harbor)
2. Track memory usage

Expected: Low memory usage, successful installation
Actual: High memory usage, sometimes OOMKilled.

Fix helm 3.8 digest validation issues

Our current code does not use the manifest digest when retrieving charts. The digest is not used, and so the clients are falling back to grabbing the highest semver tag.

Helm 3.8 does not support pulling by sha256 as of 3.8.

We need to do two things:

1. Change our bundle generation to use the version tags in addition to the digest.
2. Once we pull a chart, verify the sha256 in its manifest against the digest from the bundle.

Package bundles are listed in ascending order

1. kubectl get packagebundle -A

Expected: The newest bundle should be at the top sorted in descending order.
Actual: The newest bundle is at the bottom, sorted in ascending order.

Install package controller on cluster creation

Install the package controller on cluster creation using the helm chart.

• Add a state in the CLI state machine to helm install package controller
• May need pkg/executable helm for helm install - use shasum
• Add package controller helm chart to EKS Anywhere release bundle
• After install of package controller, add state to install any configured packages

Harbor does not support target namespace

1. Create a Package resource for harbor with targetNamespace: my-ns

Expected: Harbor is installed in my-ns

Actual: Harbor is installed in eksa-packages

Create SNS topic to publish availability of new bundles

When a bundle goes live in production, we should automatically notify users using an SNS topic. The text of the message can be very plain text. Something like

1-21-1003 bundle available

We may want to support json for automation

• create SNS topic
• send announcement on production bundle promotion

Increase test coverage of pkg/driver

Test code coverage of pkg/drvier is less than 25%.

Allow registry credentials to be passed to the helm driver

The helm driver currently only works on public registries. Add a configuration option that allows to use a secret with credentials for pulling artifacts with the helm registry client.

Create e2e tests for hello-eks-anywhere application

Create automated end-to-end tests for hello-eks-anywhere application. This test should focus on making sure the various actions of the controller work.

• installation

• configuration

• update

• upgrade

• delete

• Phase 1: Just the minimum

  • installation
  • validate functionality

• Phase 2: Expanded lifecycle (currently in progress)

  Each step is assumed to include validation

  • configuration
  • update
  • upgrade
  • delete

• Phase 3: Expanded environments

  Each combination is assumed to include validation

  Provider	OS	Kubernetes Version
  vSphere	Bottlerocket	1.22
  		1.21
  	Ubuntu	1.22
  		1.21
  		1.20
  Docker	Bottlerocket	1.22
  		1.21
  	Ubuntu	1.22
  		1.21
  		1.20

Provide better help messages

Here's an example usage:

morph:~$ eksctl-anywhere generate packages
Error: required flag(s) "directory", "source" not set

I think it would be great to show the usage in this case, as if the user had run eksctl-anywhere generate packages --help. The output (at time of writing) would look something like this:

morph:~$ eksctl-anywhere generate packages
Error: required flag(s) "directory", "source" not set
This command is used to generate curated packages

Usage:
  anywhere generate packages [flags]

Aliases:
  packages, package, packages

Flags:
  -d, --directory string      Directory to save generated packages
  -h, --help                  help for packages
      --kubeversion string    Kubernetes Version of the cluster to be used. Format <major>.<minor>
      --source BundleSource   Location to find curated packages: (cluster, registry)

Global Flags:
  -v, --verbosity int   Set the log level verbosity

My reasoning is that if they've left off required parameters, they probably don't know they're required, so I think it makes sense to go ahead and show them the help info.

For bonus points, we could probably check if stdin is open, and if not, we can skip the above (stdin being closed implies that eksctl-anywhere is being run from a script, so there's really no need to output help text).

Add requires.yaml contents to bundle in generatebundlefile

The CRD supports the contents, generatebundlefile just needs to be modified to

• Download the helm chart
• Unzip it
• extract the requires.yaml
• Parse requires.yaml
• images to bundle
• config name, required, and default to bundle

Packages and Package Controller should not show up in helm list

This may or may not be possible, but have it so helm charts for packages and the package controller do not show up with helm list

Increase code coverage of pkg/bundle

Our test code coverage is 50% for pkg/bundle which is not great for some pretty key code. It should be closer to 80%

Disclaimer on EKS-A currated docs pages

EKS-A docs should include this disclaimer:

Preview and pricing disclaimer
The EKS Anywhere package controller and the EKS Anywhere Curated Packages (referred to as “features”) are provided as “preview features” subject to the AWS Service Terms, (including Section 2 (Betas and Previews)) of the same. During the EKS Anywhere Curated Packages Public Preview, the AWS Service Terms are extended to provide customers access to these features free of charge. These features will be subject to a service charge and fee structure at ”General Availability“ of the features.

Harbor Helm Chart Support `sourceRegistry`

Change the Harbor helm chart to use sourceRegistry in the values.yaml file so the customer can easily install it from a private registry.

Need method to download any bundle by tag

If someone deletes the active bundle, we should download it again.

• We need this method https://github.com/aws/eks-anywhere-packages/blob/main/pkg/bundle/manager.go#L145 the LatestBundle method, but the download part needs to be pulled out and a public DownloadBundle method created.
• probably need a method to get the name of the active bundle
• If the bundle controller detects the active bundle was deleted https://github.com/aws/eks-anywhere-packages/blob/main/controllers/packagebundle_controller.go#L106 currently it is ignored, but it should attempt to download the bundle again
• Use the k8s client interface to recreate the bundle

Add Disclaimer on package install

The CLI should have this disclaimer when installing a package:

Preview and pricing disclaimer
The EKS Anywhere package controller and the EKS Anywhere Curated Packages (referred to as “features”) are provided as “preview features” subject to the AWS Service Terms, (including Section 2 (Betas and Previews)) of the same. During the EKS Anywhere Curated Packages Public Preview, the AWS Service Terms are extended to provide customers access to these features free of charge. These features will be subject to a service charge and fee structure at ”General Availability“ of the features.

Add Disclaimer to README

The README should have this disclaimer:

Preview and pricing disclaimer
The EKS Anywhere package controller and the EKS Anywhere Curated Packages (referred to as “features”) are provided as “preview features” subject to the AWS Service Terms, (including Section 2 (Betas and Previews)) of the same. During the EKS Anywhere Curated Packages Public Preview, the AWS Service Terms are extended to provide customers access to these features free of charge. These features will be subject to a service charge and fee structure at ”General Availability“ of the features.

GenerateBundle Improvements

• Skopeo SDK
• Stub AWS calls in unit tests
• Implement Interface for ECR and ECRPublic instead of having functions for both
• Move strings.contains() errors checks to Helper functions.
• Change Functions iterating, and comparing elements (getLastestImageSha, imageTagFilter) to use sort, and filter for speed increase.
• Look into Helm SDK if it's possible to pass an Auth file or use an Auth ENV VAR.

Bundle controller should ignore any bundle with wrong k8s version

Bundle controller should ignore any bundle with wrong k8s version

• add method to api/v1alpha1/bundle.go something like matchesKubeVersion(kubeVersion) bool the returns true of the version of the bundle matches the specified kubernetes version
• Add some new state to the bundle state like WRONGK8S, that is a terrible name, come up with something better
• the bundler controller reconciler should mark the bundle as WRONGK8S if the current cluster kubeVersion doesn't match the bundle

Create e2e Tests for Harbor

When the end-to-end tests are up, create a set of test cases to automate testing Harbor. The bug bash doc could be used an example and all of that might not be possible, but as much as possible.

Set Bundle Status.State to invalid if GetMajorMinorBuild returns any zero value

If any value of the tuple returned by GetMajorMinorBuild is zero, set the sate of the bundle to invalid.

Create EKS Anywhere documentation

Create EKS Anywhere Curated Packages documentation

• Tasks: Create cluster install
• Tasks: CLI install
• Tasks: kubectl
• Package spec
• PackageController spec
• PackageBundle spec
• PackageBundleController spec

E2E Tests Support Various K8s Rleases

Run end-to-end tests on various k8s releases

• Ubuntu
  • 20
  • 21
  • 22
• Bottlerocket
  • 21
  • 22

Package bundle controller controller getting too long

The packagebundlecontroller_controller.go is getting too long and some of the logic should be moved into the manager. A real state machine like we have for the package controller would be nice. More use of the bundle/client.go as well would help.

We should review it and see if it follows our design intents and/or how we can logically break it into smaller, more bite sized chunks.

Originally posted by @ewollesen in #162 (comment)

Integrate Oras with Docker ECR Cred Helper

It would be useful to figure out the most effective way to integrate ORAS login into the cred-helper.

Let's see if we move our command to something more secure without directly

${BASE_DIRECTORY}/bin/oras push -u AWS -p "${ECR_PASSWORD}"

Into

aws ecr-public get-login-password --region us-east-1 \
    | oras login --username AWS --password-stdin "${IMAGE_REGISTRY}"
oras push "${IMAGE_REGISTRY}/eks-anywhere-packages-bundles:v1" bundle-1.20.yaml

Generate bundle tests are broken

Generate bundle tests seem require a connection to ECR to work, they are broken for me. Ideally, they would use a mock, but maybe some special registry to go into test mode. We should probably have the tests run as a presubmit.

Packages deleted while the controller was down are not reconciled

1. Install a package
2. Shut down the controller
3. Delete the package
4. Start the controller

Expected: The controller removes the installation part of the deleted package
Actual: The installation remains

Promotion Prod Tasks

• Add the Controller Helm chart to EKS-A release Bundle
• Infra to push Packages to Prod
• Infra to build Bundle in prod

Label all resources created by the Package Controller

If it is possible, label everything we create. It may be that kustomize label transformer would help. By everything we create, this means what the helm chart creates. If a helm chart creates a secret for example, the secret would have a label like eks-anywhere=package or something like that.

Harbor needs targetNamespace in values.yaml for namespace supprot

The package controller relies on a known value in the values.yaml to override targetNamespace and that value does not exist in the harbor helm chart. It should be similar to sourceRegistry.

Generate documentation for bug bash

Generate a couple page doc for bug bash. We don't need full high quality docs, but enough for people to try it out. Probably generate as pdf since we won't have anywhere to publish it publicly.

• Install add-on controller
• Install an add-on
• Update bundle
• Delete add-on

Investigate adding a basic `helm test` for Harbor

It would be nice to have a basic helm test for harbor. Upstream it maybe.

Harbor Build Pipeline Broken

It looks like the harbor build pipeline is broken in the promotion stage. It appears it is because it is use a "v" in the version tag instead of proper semantic versioning. The eks-anywhere-test and eksa-packages repos could be used as examples for how it should be done. I'm not sure the exact problem

Allow bundle signing key to be configured via PackageBundleController

Allow bundle signing key to be configured via custom resource aka PackageBundleController

• Add signing key to bundle controller CRD
• Modify bundle validation webhook to read bundle controller CR and get key
• Modify helm chart to take signing key in values.yaml and populate bundle controller CR
• Leave env of some sort in values.yaml for other purposes
• Remove signing key from env in values.yaml so it doesn't confuse users, use new value to configure that

Updating package config doesn't trigger an upgrade

1. Create a package
2. After deployment, modify the package config
3. Apply the modified package

Expected: The package should be upgraded using the new config.
Actual: The package remains untouched.

Create defaulting/validating webhook for configuration for Packages

There may be two webhooks for packages, but ideally the config should be validated and defaulted.

Reassess SortBundlesDescending Logic

Currently SortBundlesDescending method perform descending sort based on following orders: K8s major version, K8s minor version, build version (per implementation here). This may not be ideal since there may be cases where we should choose a lower K8s version with a higher build version. The actual revision and implementation needs further discussion by the team.

Better secret handling

Harbor package for example takes a secret for harborAdminPassword: "Harbor12345" which is unencrypted in the Harbor package custom resource. Only administrators have access to this CR, but this is not great. Ideally, we could populate these values as a secret ref from a k8s secret (and potentially other sources in the future).

Failure to download bundle should update add-on status

If the bundle controller fails to download the bundle, it should update the status of the add-ons to show that they may not be up to date.

Switch from time-based sorting to numeric

Not for this PR, but this is probably another place we can switch from time-based sorting to numeric, at least in the name. That is rename to something like SortBundlesDescending or something more like that.

Originally posted by @ewollesen in #154 (comment)

Package configuration modifications are not immediately applied

1. Install a package
2. Modify the package config field
3. Apply the modified package

Expected: The package is updated to reflect the config
Actual: Nothing happens, waits for an actual version upgrade.

Change the name of eks-anywhere-test to hello-eks-anywhere

• create the repository
• mirror the hello-eks-anywhere repo maybe if we need to
• rename the folder in build tooling
• upstream our changes to hello-eks-anywhere
• add the helm chart to hello-eks-anywhere
• create new prow job
• Delete the old prow job
• Delete the old project
• Update all-projects in the build tooling Makefile to add the hello app

Harbor install should fail if no secretKey is provided

1. Create a harbor package file without config
2. Apply

Expected: The controller should fail installing because secretKey was not provided (and defaulted to not-a-secure-key)
Actual: Harbor is deployed with not-a-secret-key as secretKey

This requires implementing the concept of required config, either in the controller or helm chart itself (probably the latter).

Prepare README for public preview

The README needs to be prepared for public preview

• move the current contents to somewhere like docs/developer.md
• follow this example https://github.com/aws/eks-anywhere
• add a high level description of the project
• badges, we need some stinking badges
• development pointing to the former contents and high level description
• security
• license

Support Bundle: Add appropriate collectors for add-ons

The EKS Anywhere support bundle should include resources, logs and other information to support add-ons. Examples include:

• CRDs and associated custom resources
• Logs for the add-on controller
• Potentially logs and resources for installed add-ons
• Redact as needed