googlecloudplatform/cloud-sql-proxy-operator

A Kubernetes Operator to automatically configure secure connections to Cloud SQL

Cloud SQL Proxy Operator

Cloud SQL Proxy Operator is an open-source Kubernetes operator that automates most of the intricate steps needed to connect a workload in a kubernetes cluster to Cloud SQL databases.

The operator introduces a custom resource AuthProxyWorkload, which specifies the Cloud SQL Auth Proxy configuration for a workload. The operator reads this resource and adds a properly configured Cloud SQL Auth Proxy container to the matching workload pods.

Installation

Check for the latest version on the releases page and use the following instructions.

Confirm that kubectl can connect to your kubernetes cluster.

kubectl cluster-info

Install cert-manager using helm. Note that you need to use this particular version with these specific cli arguments to make cert-manager work on your GKE cluster.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version "v1.9.1" \
  --create-namespace \
  --set global.leaderElection.namespace=cert-manager \
  --set installCRDs=true

Run the following command to install the cloud sql proxy operator into your kubernetes cluster:

kubectl apply -f https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy-operator/v1.4.5/cloud-sql-proxy-operator.yaml

Confirm that the operator is installed and running by listing its pods:

kubectl get pods -n cloud-sql-proxy-operator-system

Usage

See the Quick Start Guide for a description of basic usage. Additional usage may be found in the Examples.

Frequently Asked Questions

Why would I use the Cloud SQL Auth Proxy Operator?

The Cloud SQL Auth Proxy Operator gives you an easy way to add a proxy container to your kubernetes workloads, configured correctly for production use.

Writing the kubernetes configuration for a proxy to the production level requires a great deal of deep kubernetes and proxy knowledge. The Cloud SQL Proxy team has worked to encapsulate that knowledge in this operator. This saves you from having to know all the details to configure your proxy.

Reference Documentation

• Quick Start Guide
• API Documentation
• Cloud SQL Proxy
• Developer Getting Started
• Developing End-to-End tests
• Contributing
• Code of Conduct
• Examples

Support policy

Major version lifecycle

This project uses semantic versioning, and uses the following lifecycle regarding support for a major version:

Active - Active versions get all new features and security fixes (that wouldn’t otherwise introduce a breaking change). New major versions are guaranteed to be "active" for a minimum of 1 year. Deprecated - Deprecated versions continue to receive security and critical bug fixes, but do not receive new features. Deprecated versions will be publicly supported for 1 year. Unsupported - Any major version that has been deprecated for >=1 year is considered publicly unsupported.

Contributing

Contributions are welcome. Please, see the Contributing document for details.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Contributor Code of Conduct for more information.

cloud-sql-proxy-operator's Issues

feat: Finish implementing operator reconcile loop

GIVEN a k8s cluster with operator installed
WHEN a user requests to create, update, or delete an AuthProxyWorkload resource
THEN the operator will respond to the change in the AuthProxyWorkload resource and update the state of the cluster workloads accordingly.

We need to land the implementation of the operator Reconcile() method.

Add support for better status reporting for AuthProxyWorkload

Roll up the status for the individual containers and report it in status.conditions for the AuthProxyWorkload resource.

Installer does not work with GKE Autopilot Clusters

Expected Behavior

When you run the installer following the instructions in Quick Start, it should work.

Actual Behavior

It doesn't work. There is a problem installing cert-manager related to permissions for
webhooks in the autopilot clusters. This causes the webhooks in the operator to fail,
thereby rendering the operator inoperable.

Steps to Reproduce the Problem

1. Follow the steps in the quickstart guide, creating an Autopilot cluster
2. Run the install script. Notice strange errors when the cert-manager installer runs and
   when the operator install script runs.
3. Try to create an AuthProxyWorkload resource. Notice that it fails to create.

Specifications

• Version: 0.1.0
• Platform: n/a

Enable proxy telemetry settings through the CRD

implement: spec.telemetry.httpPort
implement: spec.telemetry.telemetryProject, telemetryPrefix, telemetrySampleRate
implement: spec.telemetry.disableTraces, disableMetrics
implement: spec.telemetry.prometheusNamespace

Create a test harness application for e2e tests

Update E2E tests so that they succeed when they actually connect to the database and run a basic query.

• Expose Kubernetes health checks to demonstrate that it can connect to the database
• Read the proxy settings from environment variables or cli arguments as needed
• Connect to SQL Server, Postgres and MySQL as requested by configuration

This will be complete when:

• Modify existing e2e test cases to use Postgres public client images to run a query
• Modify existing e2e test cases to use MySQL public client images to run a query
• Modify existing e2e test cases to use SQL Server public client images to run a query
• Modify existing e2e test cases to use the status from this container to validate the test cases.

Customize renovatebot to update all dependency versions

Customize the renovatebot configuration to do the following:

• Update explicit tool versions in the Makefile
• Update latest version of the Proxy docker image in source code

Update the Makefile to hold specific versions for each tool.
Add the necessary Renovate configuration to automatically find and update the tool versions.
Eliminate latest versions from the Makefile wherever possible.

For Example: CONTROLLER_TOOLS_VERSION and KUSTOMIZE_VERSION should be pinned to a specific version for now (one is currently on latest on one is on a version).

E2E test connection: private ip, tcp, workload identity, mysql, db-user, Deployment

E2E A Deployment uses a tcp socket to connect to a mysql db with a private ip using db-user database credentials and workload identity gcloud credentials

Switch from cli flags to envvars when configuring the proxy

In release v2 preview.3 the proxy will support env vars for all the configuration. Switch the operator to use this mechanism for configuring the proxy because it will be more stable.

E2E test connection: public ip, tcp, workload identity, mysql, db-user, Cronjob

E2E A Cronjob uses a tcp socket to connect to a mysql db with a public ip using db-user database credentials and workload identity gcloud credentials

Implement TCP port proxy connections

implement: spec.instances.portEnvName, hostEnvName
implement: spec.instances.socketType=TCP, port

Upgrade the running proxy container images when the published `latest` proxy changes.

The default proxy image used by the operator will be the latest published version of the proxy when the operator is released.

GIVEN a k8s cluster with AuthProxyWorkload that uses the default operator image
WHEN the customer upgrades the operator version running in that cluster
THEN the operator will update the proxy image used by the proxy container, initiating a rolling upgrade of the workload containers.

Improve release process to push to global repos

Improve the release process script so that it pushes the operator image to all 4 regional repos used by Cloud SQL Proxy:

us.gcr.io
eu.gcr.io
asia.gcr.io

User can install the Cloud SQL Proxy Operator with a simple bash command

The user should be able to run a command like this to install the cloud sql proxy operator into the GKE cluster:

curl https://raw.githubusercontent.com/GoogleCloudPlatform/cloud-sql-proxy-operator/v0.0.1-alpha1/install.sh | bash

This script will

• Read the current KUBECONFIG variable
• Download configuration files appropriate for this release version this github repo
• Apply the operator configuration to the cluster

This depends on hosting pre-build docker images of the operator in a well known location.

This will be complete when:

• The script works as described above
• There is an integration test that calls this script to ensure that it works on a fake EnvTest kubernetes api

E2E test connection: public ip, tcp, vm identity, mysql, db-user, Deployment

E2E A Deployment uses a tcp socket to connect to a mysql db with a public ip using db-user database credentials and vm identity gcloud credentials

Implement authentication using credential file secrets

GIVEN a customer with a Google Cloud credential JSON file stored in a ConfigMap
WHEN the customer creates an auth proxy workload configured with Credential File set
THEN the proxy will connect to the Google Cloud API using the credential file

This configures the --credential-file flag on the proxy container and sets up a volume mount to the Secret.

E2E test connection: public ip, fuse, file in k8s secret, mysql, db-user, Deployment

E2E A Deployment uses a fuse socket to connect to a mysql db with a public ip using db-user database credentials and file in k8s secret gcloud credentials

E2E periodic test with the latest released proxy image and the HEAD of branch main for proxy

Run an E2E happy path test with both the latest released proxy image and the head of the proxy main branch to make sure that the proxy is always working with the operator.

Workload webhook should update the pod spec on the pod, not the workload definition

When applying the proxy container to an existing workload, the container should be applied to the PodSpec on the Pod owned by a workload (Deployment, StatefulSet, etc.) As implemented now, the container gets applied to the PodSpec of the workload.

This feature will be complete when

• the workload webhook updates the pod for a pod owned by a workload
• unit tests properly check the webhook method
• e2e tests properly check the pod
• integration tests checking the current behavior are modified

Update hardcoded proxy version before public preview

The operator has a hardcoded url to the default proxy image. After the next release of the proxy, we need to update to the
latest proxy version.

The version is here: internal/workload/podspec_updates.go:1077

E2E test connection: public ip, tcp, workload identity, postgres, iam-authn, Deployment

E2E A Deployment uses a tcp socket to connect to a postgres db with a public ip using iam-authn database credentials and workload identity gcloud credentials

Bug: HostEnvVar should be 127.0.0.1, not localhost

Expected Behavior

When the HostEnvName field is set on a InstanceSpec, it should set an environment variable on the workload pods
with the value 127.0.0.1.

Actual Behavior

Instead it sets it to localhost which is problematic for the mysql command line tool. Mysql assumes localhost means "a local socket connection" while 127.0.0.1 is unambiguously a tcp connection.

Use default proxy image

GIVEN An AuthProxyWorkload resource with spec.proxyContainer.image not set
WHEN the proxy container is created
THEN it uses the latest released image for the proxy container

Always use the latest proxy image as of the release date of the operator: gcr.io/cloud-sql-connectors/cloud-sql-proxy:$version

E2E test connection: Deployment, tcp, mysql, workload identity, db-user, workload identity

E2E A Deployment uses a tcp socket to connect to a mysql db with a public ip using db-user database credentials and workload identity gcloud credentials

Make operator work on GKE AutoPilot Clusters

As of v0.0.3 the operator will not install correctly on GKE Autopilot clusters.

GKE Autopilot reconfigures the kubernetes cluster, restricting some behavior. This includes webhooks and certificate management.

The default installation of cert-manager is broken on GKE with autopilot: see cert-manager/cert-manager#3717. Apparently there are now workarounds, but this will require some work.

Current plan:

• Make an installer that works without cert-manager
• Test on clusters with Autopilot enabled and disabled
• Document the tradeoffs regarding use of cert-manager
• Investigate using Google Cloud Managed Certificates instead

Build and test operator on arm64 architecture kubernetes nodes

Build the operator multiarch image and test it on an arm64 GKE cluster.

Implement validation on the AuthProxyWorkload resource

GIVEN a kubernetes cluster with the operator installed
WHEN a user submits a requests to create or modify an AuthProxyWorkload resource
THEN the resource is validated according the documented validation rules

Restructure the AuthProxyWorkloadSpec so that the data structure enforces (or encourages) users to create a valid configuration. (For example, you can't set a unix socket path and a tcp port on the same instance, so the data structure should these settings mutually exclusive.)

Throughout the definition of the AuthProxyWorkload and its children, we describe the rules for the valid state of the object. For example from api/v1alpha1/authproxyworkload_types.go:92:

// WorkloadSelectorSpec describes which workloads should be configured with this
// proxy configuration. To be valid, WorkloadSelectorSpec must specify Kind
// and either Name or Selector.
type WorkloadSelectorSpec struct {

	// Kind specifies what kind of workload
	// Supported kinds: Deployment, StatefulSet, Pod, DaemonSet, Job, CronJob
	// Example: "Deployment" "Deployment.v1" or "Deployment.v1.apps".
	//+kubebuilder:validation:Required
	//+kubebuilder:validation:Pattern=\w+(\.\w+)*
	Kind string `json:"kind"`
// ...

This feature request will be complete when:

• AuthProxyWorkload.Validate() will validate the structure according to documented rules.
• Unit tests are written demonstrating covering most valid and invalid states for an AuthProxyWorkload
• Integration tests are written demonstrating that the validation webhook is enabled and can invalidate create
  and update requests.

Validation Rules

Create Validation Rules

• WorkloadSelectorSpec
• AuthProxyContainer
  • Container valid
  • Resources valid
  • MaxConnections >= 0
  • MaxSigTermDelay >= 0
  • SQLAdminAPIEndpoint is a URL
  • Image is a valid Docker image url
  • RolloutStrategy is {"Workload","None"}
  • TelemetrySpec
• InstanceSpec
  • ConnectionString is valid, well-formed connection
  • Port > 0
  • PortEnvName valid EnvVar name
  • HostEnvName valid EnvVar name
  • UnixSocketPath is an absolute path
  • UnixSocketPathEnvName valid EnvVar
  • Either Unix or TCP config set, not both
• Proxy can start - run the configuration through the default Proxy public API to check if any errors occur.

Update Validation

• WorkloadSelectorSpec
  • No changes allowed on update
• AuthProxyContainer
  • RolloutStrategy No changes allowed
• InstanceSpec

chore: Update to latest cert-manager

Update the cert-manager version to the latest version according to
https://cert-manager.io/docs/installation/.

Make necessary changes to our configuration YAML so that it works.

E2E test connection: public ip, unix, file in k8s secret, mysql, db-user, Deployment

E2E A Deployment uses a unix socket to connect to a mysql db with a public ip using db-user database credentials and file in k8s secret gcloud credentials

Use hardcoded default proxy image for public preview

Hard code the current latest proxyv2 image for public preview, which is "gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.0.0-preview.2"

Behavior of the operator when choosing the default proxy image will need to be updated before GA. This is tracked in issue #49

Move code into internal package

All the go code should be in the internal/ package. We are not exporting any code for use as a library in another go project.

chore: Move away from LDFlags embedding for version and git sha, use go embed instead

Currently we use LDFLAGS to embed the contents of version.txt and the head SHA from our git repo into the built artifact.
Instead we want to use go embed. Investigate how to do this, and make sure it works properly.

• Embed the version from version.txt
• Embed the git sha

Upgrade to go 1.19, when compatible versions of kubebuilder/controller runtime are available

Currently we are using Go 1.18 because that is the version supported by controller-runtime and kubebuilder. Soon these frameworks will support Go 1.19. When that happens we need to update our project accordingly.

Configure proxy container resources and other details

Provide these fields on the CRD: spec.proxyContainer.resources, image, container, sqlAdminApiEndpoint

Add cloudsql-proxy user agent env variable for tracking

Add an environment variable CLOUD_SQL_PROXY_OPERATOR_VERSION to the proxy container with the operator version and build information

Enhance the proxy to read CLOUD_SQL_PROXY_OPERATOR_VERSION and add this value to its request User-Agent header when making requests to the google cloud API

E2E test connection: public ip, tcp, file in k8s secret, postgres, db-user, Pod

E2E A Pod uses a tcp socket to connect to a postgres db with a public ip using db-user database credentials and file in k8s secret gcloud credentials

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Found renovate config warnings
• WARN: Use matchDepPatterns instead of matchPackagePatterns

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update gcr.io/distroless/static:nonroot docker digest to e9ac71e
• chore(deps): update build tools (cert-manager/cert-manager, hashicorp/terraform)
• chore(deps): update gcr.io/cloud-sql-connectors/cloud-sql-proxy docker tag to v2.11.1
• chore(deps): update github action dependencies (actions/setup-go, github/codeql-action, ossf/scorecard-action)
• chore(deps): update kubernetes packages to v0.30.1 (k8s.io/api, k8s.io/apimachinery)
• chore(deps): update kubernetes runtime dependencies (k8s.io/client-go, kubernetes/kubernetes, sigs.k8s.io/controller-runtime, sigs.k8s.io/controller-tools)
• chore(deps): update module github.com/golangci/golangci-lint/cmd/golangci-lint to v1.58.1
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

dockerfile

Dockerfile

• gcr.io/distroless/static nonroot@sha256:f41b84cda410b05cc690c2e33d1973a31c6165a2721e2b5343aab50fecb63441

Dockerfile-operator

• gcr.io/distroless/static nonroot@sha256:f41b84cda410b05cc690c2e33d1973a31c6165a2721e2b5343aab50fecb63441

github-actions

.github/workflows/codeql.yml

• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• actions/setup-go v5.0.0@0c52d547c9bc32b1aa3301fd7a9cb496313a4491
• github/codeql-action v2.25.1@e949a1676c32f4c215780f7429eb9f00ff18b225
• github/codeql-action v2.25.1@e949a1676c32f4c215780f7429eb9f00ff18b225
• github/codeql-action v2.25.1@e949a1676c32f4c215780f7429eb9f00ff18b225

.github/workflows/labels.yaml

• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• micnncim/action-label-syncer v1.3.0@3abd5ab72fda571e69fffd97bd4e0033dd5f495c

.github/workflows/scorecard.yml

• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• ossf/scorecard-action v2.3.1@0864cf19026789058feabb7e87baa5f140aac736
• actions/upload-artifact v3.1.3@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
• github/codeql-action v2.25.1@e949a1676c32f4c215780f7429eb9f00ff18b225

.github/workflows/tests-main.yaml

• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• google-github-actions/auth v1.3.0@3a3c4c57d294ef65efaaee4ff17b22fa88dd3c69
• actions/setup-go v5.0.0@0c52d547c9bc32b1aa3301fd7a9cb496313a4491
• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• google-github-actions/auth v1.3.0@3a3c4c57d294ef65efaaee4ff17b22fa88dd3c69
• google-github-actions/setup-gcloud v1.1.1@e30db14379863a8c79331b04a9969f4c1e225e0b
• actions/setup-go v5.0.0@0c52d547c9bc32b1aa3301fd7a9cb496313a4491
• docker/setup-qemu-action v2.2.0@2b82ce82d56a2a04d2637cd93a637ae1b359c0a7
• docker/setup-buildx-action v2.10.0@885d1462b80bc1c1c7f0b00334ad271f09369c55

.github/workflows/tests.yaml

• actions/github-script v6.4.1@d7906e4ad0b1822421a7e6a35d5ca353c962f410
• actions/setup-go v5.0.0@0c52d547c9bc32b1aa3301fd7a9cb496313a4491
• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744
• google-github-actions/auth v1.3.0@3a3c4c57d294ef65efaaee4ff17b22fa88dd3c69
• google-github-actions/setup-gcloud v1.1.1@e30db14379863a8c79331b04a9969f4c1e225e0b
• actions/setup-go v5.0.0@0c52d547c9bc32b1aa3301fd7a9cb496313a4491
• docker/setup-qemu-action v2.2.0@2b82ce82d56a2a04d2637cd93a637ae1b359c0a7
• docker/setup-buildx-action v2.10.0@885d1462b80bc1c1c7f0b00334ad271f09369c55

gomod

go.mod

• go 1.22
• github.com/go-logr/logr v1.4.1
• go.uber.org/zap v1.27.0
• k8s.io/api v0.29.3
• k8s.io/apimachinery v0.29.3
• k8s.io/client-go v0.29.3
• sigs.k8s.io/controller-runtime v0.17.2
• sigs.k8s.io/yaml v1.4.0

kustomize

config/manager/kustomization.yaml

regex

Makefile

• cert-manager/cert-manager v1.14.4
• kubernetes/kubernetes v1.29.3
• hashicorp/terraform v1.8.1
• sigs.k8s.io/controller-tools v0.14.0
• github.com/elastic/crd-ref-docs v0.0.12
• github.com/golangci/golangci-lint/cmd/golangci-lint v1.57.2
• github.com/google/go-licenses v1.6.0

internal/workload/podspec_updates.go

• gcr.io/cloud-sql-connectors/cloud-sql-proxy 2.11.0

terraform

infra/permissions/main.tf

• google 4.84.0

infra/resources/main.tf

• google 4.84.0
• google-beta 4.84.0

---

• Check this box to trigger a request for Renovate to run again on this repository

E2E test connection: public ip, tcp, workload identity, mysql, db-user, DaemonSet

E2E A DaemonSet uses a tcp socket to connect to a mysql db with a public ip using db-user database credentials and workload identity gcloud credentials

chore: Move away from LDFlags embedding for version and git sha, use go embed instead

Currently we use LDFLAGS to embed the contents of version.txt and the head SHA from our git repo into the built artifact.
Instead we want to use go embed. Investigate how to do this, and make sure it works properly.

• Embed the version from version.txt
• Embed the git sha

test: Implement basic end-to-end tests for the operator on GKE

GIVEN A developer who can build this codebase, a Google Cloud account with an empty Google Cloud Project
WHEN the developer configures and runs end-to-end tests
THEN the end-to-end tests will provision resources in the project and run end-to-end test defined in this codebase.

This will only cover very basic CRUD tests for the AuthProxyWorkload resource.

Show status on a AuthProxyWorkload for each container in the workload

Periodically make the reconcile loop check all AuthProxyWorkload resources and update the status to reflect the current state of the workload's proxy containers.

UnixSocketEnvVar should contain the whole path to the database's socket

Expected Behavior

When you use this configuration:

apiVersion: cloudsql.cloud.google.com/v1alpha1
kind: AuthProxyWorkload
metadata:
  name: authproxyworkload-sample
spec:
  authProxyContainer:
    image: "gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.0.0-preview.2"
  workloadSelector:
    kind: "Deployment"
    name: "gke-cloud-sql-app"
  instances:
    - connectionString: "my-project:us-central1:my-instance"
      unixSocketPathEnvName: "DB_SOCKET_PATH"
      socketType: "unix"
      unixSocketPath: "/csql/pg"

Then the value of DB_SOCKET_PATH should be /csql/pg/my-project:us-central1:my-instance/.s.PGSQL.5432 the full path to the postgres unix socket file.

For other database types, the DB_SOCKET_PATH should be the full path to the unix socket file as well.

The operator should make sure that this is true, regardless of database type or particular implementation in the proxy.

Actual Behavior

Then the value of DB_SOCKET_PATH is set to /csql/pg

doc: Generate API documentation for the AuthProxyWorkload CRD

Add a new make target that creates generated API documentation for AuthProxyWorkload. This should
be run as part of make generate

Create Cloud Build job to release the operator

On release:

• Publish the operator container image to a well known public location.
• Publish the install script to a well known public location
• Update "getting started" to point at the correct image and install script

Public Location: We will publish this to the cloud-sql-connectors registry as cloud-sql-proxy-operator

E2E test connection: public ip, tcp, workload identity, sql server, db-user, StatefulSet

E2E A StatefulSet uses a tcp socket to connect to a sql server db with a public ip using db-user database credentials and workload identity gcloud credentials

Support FUSE directory for proxy connections

implement: spec.proxy.fuseDirectory
also implement: spec.instances.unixSocketPathEnvName

Add support for unix sockets

implement: spec.instances.socketType=UNIX, unixSocketDirectory

implement: spec.instances.unixSocketPathEnvName

E2E test connection: public ip, tcp, workload identity, mysql, db-user, Deployment

E2E A Deployment uses a tcp socket to connect to a mysql db with a public ip using db-user database credentials and workload identity gcloud credentials

Install Flakeybot

Build and test operator on windows

Release the operator image as a multi-architecture container image supporting windows. Create end-to-end test on a GKE cluster with windows nodes.

E2E test connection: public ip, tcp, workload identity, mysql, db-user, Job

E2E A Job uses a tcp socket to connect to a mysql db with a public ip using db-user database credentials and workload identity gcloud credentials