kubernetes-sigs/container-object-storage-interface-provisioner-sidecar

Container Object Storage Interface (COSI) provisioner responsible to interface with COSI drivers.

Container Object Storage Provisioner Sidecar

Container Object Storage Interface (COSI) provisioner sidecar is responsible to manage lifecycle of COSI objects and is deployed as a sidecar to a provisioner. Specifically, the sidecar monitors the lifecycle of the CRDs generated by the COSI Controller:

• Bucket - Represents a Bucket or its equivalent in the storage backend
• BucketAccess - Represents a access token or service account in the storage backend

and makes gRPC calls to the associated provisioner.

In this repo you will also find a sample provisioner based on Minio.

Developer Guide

Before diving into the code of this repo, we suggest that you familiarize yourself with:

• The Spec of CRDs in objectstorage.k8s.io/v1alpha1/types.go
• The Spec of the COSI objects sigs.k8s.io/container-object-storage-interface-spec

A good starting point towards understanding the functionality of this repo would be to study the tests:

• Bucket Controller Test
• Bucket Access Controller Test

Build and Test

In order to build and generate the Docker images execute:

make container

Notice that this command will also generate the image for the sample-driver:

objectstorage-sidecar:latest
sample-driver:latest

In order to run the tests execute:

make test

Running the Sidecar

To see an example of running the sidecar, take a look at the COSI Driver Minio sub-project.

References

• Documentation
• Deployment Guide
• Weekly Meetings
• Roadmap

Community, discussion, contribution, and support

You can reach the maintainers of this project at:

• #sig-storage-cosi slack channel
• container-object-storage-interface mailing list

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

container-object-storage-interface-provisioner-sidecar's Issues

Secret namespace incorrect

Bug Report

What happened:

When the sidecar creates a secret, it is always creating it in a static namespace.

What you expected to happen:
It should be making secrets in the same namespace as the sidecar itself.

How to reproduce this bug (as minimally and precisely as possible):
Run the kustomize manifests, add a BR -> Bucket -> BA. The secret it not created.

Anything else relevant for this bug report?:

Environment:
N/A

Update Bucket API in Sidecar

Unit tests for GRPC client/server

Add CI for this project

Add healthz endpoint to cosi driver framework

Sidecar sending multiple create bucket requests for a single bucket

Bug Report

What happened:
When the sidecar starts creating a bucket, it sends multiple different backing create requests. Although the driver is assumed to be idempotent, the name is also different in these requests.

What you expected to happen:
It's okay if it sends multiple requests, but the name should be consistent across these requests.

How to reproduce this bug (as minimally and precisely as possible):

Anything else relevant for this bug report?:

Environment:
N/A

Move code from temp repo to official repo

Implement GrantAccess for Minio Provisioner

The current implementation of the sample Minio provisioner needs also the GrantAccess implementation

Add Deployment documentation

Now that kustomize files have been added to the root of the project, we need a getting started guide which explains how to deploy and concerns around it. The deployment step itself is a single command now:

  kubectl create -k github.com/kubernetes-sigs/container-object-storage-provisioner-sidecar

cosi sidecar shouldn't crash if driver is not ready

Whenever the provisioner starts up and waits for the driver to come up, it fails if the driver is not already ready and listening on the socket. This can lead to a loop where the provisioner dies too soon, and prevents the driver from coming up leading to endless crashloopbackoff

During initial connection, try 5 times, with a connect timrout of 30s each. If all 5 attempts fail, then exit.

Review sidecar deployment labels

container-object-storage-interface/cosi-controller-manager#9 (comment)

Unit tests for Sample provisioner

[01/13/2023] - Fail to make container for amd64 architecture

Bug Report

What happened:
When build sidecar running make container, find the command is defined in container-object-storage-interface-spec/release-tool/build.make_, but it fail to build out image with GOARCH=amd64, the objectstorage-sidecar-arm64 cannot run in arm64 environment which I tried and verified.

What you expected to happen:
In current build command:
build-%: check-go-version-go mkdir -p bin CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$* if [ "$(ARCH)" = "amd64" ]; then \ CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \ CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* ; \ CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-arm64 ./cmd/$* ; \ fi
should also append:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-amd64 ./cmd/$* ; to it, so that it can work for amd64 cpu.

How to reproduce this bug (as minimally and precisely as possible):
Running command make container in repo:
https://github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar

Environment:

• Kubernetes version: 1.24
• OS: SLES15
• Kernel: Linux hostname1 5.3.18-150300.59.54-default #1 SMP Sat Mar 5 10:00:50 UTC 2022 (1d0fa95) x86_64 x86_64 x86_64 GNU/Linux

Unit Tests for Bucket and BucketAccess controllers

In addition to the tests present, add tests for verifying controller logic for add and delete scenarios

Sample Provisioner

Implement sample provisioner for e2e tests

Create deployment yaml

Create deployment yamls that will deploy the sidecar and a driver that can be used to deploy just those components, or be part of a global deploy that would deploy all of object storage.

Other components of object storage have been using kustomize to do this, so this needs to fit into that scheme.

Inconsistency on the images name and the deployment yamls

Enhancement

Is your feature request related to a problem?/Why is this needed
Not really an important problem. Just an effort to make the whole development process more consistent.
When building the images (by invoking make container) the produced images are:

objectstorage-sidecar
sample-driver

While on the yamls the images used are:

objectstorage-sidecar
objectstorage-sample-driver

Describe the solution you'd like in detail

Just need to implement the convention to rename every component with the objectstorage- prefix

When Bucket is created in the backend, a finalizer should be added to the bucket object

Process the bucket access policy on the bucket

The sidecar should process the bucket access policy on the bucket and not tell the provisioner to delete the backend bucket unless the retention policy is delete.

Update BucketAccess API in Sidecar

Update README.md

The current readme.md is out of date, and does not give guidelines for new persons interested in the project to easily get started and contribute.

Endpoint and region are not being retrieved from credentials.

container-object-storage-interface-provisioner-sidecar/pkg/bucketaccess/bucketaccess_controller.go

Line 207 in 655cc76

Endpoint: "https://s3.amazonaws.com",

Could we change endpoint and region to be retrieved from credentials just like accessKeys and secretKeys ?

Reimplement sample driver [minio]

The current driver was intended as a prototype, and requires some rework before it can be used for testing/prod purposes

Add/Remove finalizers on bucket

The sidecar should create a finalizer on a bucket for each bucket access created by the provisioner that uses that bucket. It should remove the finalizer when the bucket access is deleted. The bucket access create needs to get the latest version of the bucket access before processing to avoid a race condition between the add and delete operations.

[Apr-29] - Missing kustomization.yaml and errors when deploying provisioner sidecar

Bug Report

What happened:
Hi, I am running into errors when deploying using quick start linked below:
https://github.com/kubernetes-sigs/container-object-storage-interface-api/blob/master/docs/deployment-guide.md#quick-start

In particular, the CRD and the controller deployment went fine, however, provisioner sidecar deployment had following errors:

└─ $ ▶ kubectl create -k github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar
error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/private/var/folders/5g/2hsd6d41441dqz17w1yy5llh0000gn/T/kustomize-688061678'

What you expected to happen:
Expecting to deploy various COSI components without errors

How to reproduce this bug (as minimally and precisely as possible):
Deploy using quick steps and it should error out as mentioned above due to missing kustomization.yaml file

Anything else relevant for this bug report?:

Environment:

• Kubernetes version (use kubectl version), please list client and server:

└─ $ ▶ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.16-gke.502", GitCommit:"a2a88ab32201dca596d0cdb116bbba3f765ebd36", GitTreeState:"clean", BuildDate:"2021-03-08T22:06:24Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.18) exceeds the supported minor version skew of +/-1

• Sidecar version (provide the release tag or commit hash):
  e6139cd
• Provisoner name and version (provide the release tag or commit hash):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

Add/Remove finalizers on the BucketAccess

The sidecar needs to manage a finalizer on the BucketAccess to ensure that the credential in the backend is deleted before the BucketAccess object is deleted from kubernetes. The sidecar will need to add a finalizer to the BucketAccess once a credential has been created in the backend. The central controller will set an annotation on the BucketAccess if the controller is deleting the BucketAccess. The sidecar needs to check in the update event whether the annotation is set, and if set ensure a safe deletion. If the annotation is not set then the sidecar needs to do a forced removal of the credential.

Safe delete:

• Verify annotation is set on the BucketAccess
• Wait for all pod finalizers to be removed
• Delete the backend credential
• Once the deletion is successful remove the finalizer on the BucketAccess

Forced delete:

• Verify the annotation on the BucketAccess is not set
• Remove all pod finalizers on the BucketAccess
• Delete the backend credential
• Once the deletion is successful remove the finalizer on the BucketAccess

Build should be able to build sidecar and driver binaries and images

make - should build both binaries // this works correctly
make sidecar, make sample-driver // these step should be added to build individual binaries
make container // should build sidecar and sample-driver images
make container-sidecar // should build sidecar image
make container-sample-driver // should build sample-driver image