kubernetes-sigs / azurefile-csi-driver Goto Github PK
View Code? Open in Web Editor NEWAzure File CSI Driver
License: Apache License 2.0
Azure File CSI Driver
License: Apache License 2.0
What happened:
There could be little possibility that mount operation would hang some times, need to add timeout(10m) for mount operation:
What you expected to happen:
How to reproduce it:
Anything else we need to know?:
Environment:
kubectl version
):uname -a
):What happened:
Summarizing 5 Failures:
[Fail] Controller Service CreateVolume [It] should not fail when requesting to create a volume with already existing name and same capacity.
/home/priyanshu/work/src/github.com/csi-driver/azurefile-csi-driver/vendor/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:371
[Fail] Controller Service CreateVolume [It] should fail when requesting to create a volume with already existing name and different capacity.
/home/priyanshu/work/src/github.com/csi-driver/azurefile-csi-driver/vendor/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:442
[Fail] Controller Service ValidateVolumeCapabilities [It] should fail when the requested volume does not exist
/home/priyanshu/work/src/github.com/csi-driver/azurefile-csi-driver/vendor/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:705
[Fail] Node Service [It] should work
/home/priyanshu/work/src/github.com/csi-driver/azurefile-csi-driver/vendor/github.com/kubernetes-csi/csi-test/pkg/sanity/node.go:464
Ran 26 of 57 Specs in 77.639 seconds
FAIL! -- 21 Passed | 5 Failed | 0 Pending | 31 Skipped
following issue has been fixed:
[Fail] Controller Service DeleteVolume [It] should succeed when an invalid volume id is used
/home/priyanshu/work/src/github.com/csi-driver/azurefile-csi-driver/vendor/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:543
What you expected to happen:
How to reproduce it:
run sanity test: https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/master/test/sanity
Anything else we need to know?:
Environment:
kubectl version
):uname -a
):Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Since this driver is not working no windows node now, need to add node selector for the following deployment file:
https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/deploy/csi-azurefile-controller.yaml
https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/deploy/csi-azurefile-node.yaml
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
What happened:
If create an azure file on a new k8s cluster without a storage account, create the first azure file would cost a lot more time since it would create an azure storage account first and then create azure file, below are the error logs:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 8s (x3 over 73s) file.csi.azure.com_csi-azurefile-controller-6b4775bfd-xkmfm_7e7076e3-a705-11e9-9437-ae978b7a0a75 External provisioner is provisioning volume for claim "azurefile-1803/pvc-tzf5s"
Warning ProvisioningFailed 8s (x3 over 53s) file.csi.azure.com_csi-azurefile-controller-6b4775bfd-xkmfm_7e7076e3-a705-11e9-9437-ae978b7a0a75 failed to provision volume with StorageClass "azurefile-1803-file.csi.azure.com-dynamic-sc-zrlw5": rpc error: code = Unknown desc = failed to create file share(pvc-7065a3e1-a705-11e9-9ad7-da3c35cee84f) on account() type(Standard_LRS) rg() location() size(10), error: could not get storage key for storage account : could not get storage key for storage account f7ebe9d24a70511e9acf3ae: storage.AccountsClient#ListKeys: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="StorageAccountIsNotProvisioned" Message="The storage account provisioning state must be 'Succeeded' before executing the operation."
Normal ExternalProvisioning 5s (x7 over 95s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "file.csi.azure.com" or manually created by system administrator
Mounted By: <none>
What you expected to happen:
How to reproduce it:
Anything else we need to know?:
Environment:
kubectl version
):uname -a
):/assign
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html
https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md
Describe alternatives you've considered
Additional context
The minimum supported k8s version is 1.15
What happened:
Currently secret value would be exposed in logs, while it should not
I0421 02:40:19.501807 1 utils.go:112] GRPC response: capabilities:<rpc:<> >
I0421 02:40:19.506657 1 utils.go:106] GRPC call: /csi.v1.Node/NodePublishVolume
I0421 02:40:19.506674 1 utils.go:107] GRPC request: volume_id:"arbitrary-volumeid" target_path:"/var/lib/kubelet/pods/cc9552fd-63de-11e9-8b22-000d3a004ffb/volumes/kubernetes.io~csi/pv-blobfuse/mount" volume_capability:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > secrets:<key:"accountkey" value:"xxx" > secrets:<key:"accountname" value:"andyacidiag" > volume_context:<key:"containerName" value:"test" > volume_context:<key:"resourceGroup" value:"andy-aci" > volume_context:<key:"storageAccount" value:"andyacidiag" >
We could disable this logging easily, while I would like to filter out only the secret
value, for the other request info, it's useful for debugging:
azurefile-csi-driver/pkg/csi-common/utils.go
Line 107 in 224a7d3
What you expected to happen:
How to reproduce it:
Anything else we need to know?:
Environment:
kubectl version
):uname -a
):Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Currently driver depends on azure.json to get storage account key, need to remove this dependency
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
resize feature is not completed, there are a few work items:
external-resizer
in deploymentDescribe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
CSI support is in alpha in 1.14 release and requires enabling ExpandCSIVolumes
feature gate.
kubernetes/website#12928
kubernetes/enhancements#556 (comment)
https://github.com/container-storage-interface/spec/blob/master/spec.md#nodeexpandvolume
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like in detail
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
could refer to:
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/cloud-provider-azure/cloud-provider-azure-config.yaml
https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-sigs
WIP PR: kubernetes/test-infra#12386
Describe alternatives you've considered
Additional context
/assign @ashishranjan738
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
add support for Pod Identity to enable finer grain scope of identity as an alternative to using the cluster’s identity. We do see cases where Azure resources are closely associated with the cluster. But we also see customers partitioning AKS clusters using namespaces, and in those cases it’s more likely that RBAC grants to resources like storage/key vault would be to Managed Identities that were scoped to a Kubernetes namespace.
Describe alternatives you've considered
Additional context
As a warm up, could you(@Masquerade0097 ) write a few unit tests first in this project:
getFileShareInfo
, getStorageAccount
under https://github.com/andyzhangx/azurefile-csi-driver/blob/master/pkg/azurefile/azurefile.go
You could refer to https://github.com/andyzhangx/azurefile-csi-driver/blob/master/pkg/azurefile/azurefile_test.go#L25-L67
Here is the development guide:
https://github.com/andyzhangx/azurefile-csi-driver/blob/master/docs/csi-dev.md
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
The volumeid in azurefile is composed of storageAccount#filesharename, it looks strange, need to figure out how to use a normal format.
I0417 05:17:09.259451 1 mount_linux.go:212] Unmounting /var/lib/kubelet/pods/d1536df6-60cd-11e9-9ad7-000d3a04b60e/volumes/kubernetes.io~csi/pvc-c700881f-60cd-11e9-9ad7-000d3a04b60e/mount
I0417 05:17:09.304526 1 nodeserver.go:180] azurefile: volume /var/lib/kubelet/pods/d1536df6-60cd-11e9-9ad7-000d3a04b60e/volumes/kubernetes.io~csi/pvc-c700881f-60cd-11e9-9ad7-000d3a04b60e/mount/#f6613a6b24ed011e9812400#pvc-file-dynamic-cab15d3f-60cd-11e9-ae8a-000d3a0e181c has been unmounted.
Describe alternatives you've considered
Additional context
/assign
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Currently the integration test is using bash, we should switch to use golang
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
/assign @ashishranjan738
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
/assign
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
The deployment scripts under https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/master/charts/azurefile-csi-driver/templates is outdated, now all deployment are using master/agent mode, need to update
If you have already installed Helm, you can also use it to install azurefile CSI driver. Please see [Installation with Helm](../charts/README.md).
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Currently azure file don't have a restore from snapshot SDK, need to implement that when it's available.
als see ref: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/issues/13#issuecomment-528861659
https://docs.microsoft.com/en-us/azure/backup/restore-afs
https://docs.microsoft.com/en-us/azure/backup/restore-azure-file-share-rest-api
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
add UT for createDisk func, at least negative cases
azurefile-csi-driver/pkg/azurefile/azurefile.go
Lines 350 to 373 in ea75bfe
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
This driver requires k8s v1.13.x, so need to make sure OpenShift supports v1.13.x first.
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
Currently CSI driver is linux based, cannot run on Windows. The main difficulty is design difference between Linux and Windows, all devices are files in Linux while it's not true for Windows.
Here is a issue link in moby talking about this issue: moby/moby#38479
Describe the solution you'd like in detail
We could make the attacher and azure file csi driver running as a standalone process on Windows, that could work. (could refer to kube-proxy and kubelet implementation in aks-engine: https://github.com/Azure/aks-engine/blob/master/parts/k8s/kuberneteswindowssetup.ps1#L129-L130)
For the provision part, it could still run on Linux(on master node?). That would make it a little easier to implement.
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
including
negative tests are done:
azurefile-csi-driver/test/e2e/dynamic_provisioning_test.go
Lines 268 to 296 in f0a7d58
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
could refer to kubernetes-sigs/cloud-provider-azure#106
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
add following e2e cases:
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
add priorityClassName: system-cluster-critical and priorityClassName: system-node-critical to controller and node YAMLs respectively
Make the driver high priority and less likely to be evicted
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Currently this driver code only uses https://github.com/kubernetes-sigs/azuredisk-csi-driver/tree/master/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure and some k8s util lib, while the vendor
directory has whole k8s library, need to update go dep config to reduce vendor size
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Currently there is only one e2e test on https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/master/test/e2e, need to write more e2e tests into https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/master/test/e2e/testsuites
Describe alternatives you've considered
Additional context
/assign @ZeroMagic
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
check kubernetes/kubernetes#85233 for details
Describe alternatives you've considered
Additional context
Original built-in storage class has mountOptions
, we also need to support that
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=1000
- gid=1000
- mfsymlinks
- nobrl
- cache=none
parameters:
skuName: Standard_LRS
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
Add windows build test
make azurefile-windows
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
/assign @ashishranjan738
Is your feature request related to a problem?/Why is this needed
Setting environment variables from repository settings
$ export aadClientId=[secure]
$ export aadClientSecret=[secure]
$ export tenantId=[secure]
$ export subscriptionId=[secure]
$ export resourceGroup=[secure]
$ export nodeid=[secure]
Describe the solution you'd like in detail
suppose the testing environment is already ready
go get -u github.com/onsi/ginkgo/ginkgo
export KUBECONFIG=$HOME/.kube/config
ginkgo -p -nodes=$NODES -v --focus="$FOCUS" tests/e2e
TEST_PASS=$?
Describe alternatives you've considered
Additional context
/assign @ashishranjan738
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
tests to add:
note:
since azure disk csi driver and in-tree driver use the same azure cloud provider lib, unit tests could be applied to both csi and in-tree drivers.
Currently there are no e2e tests on in-tree azure disk & file drivers, we could leverage existing csi e2e tests and enable those e2e tests, run against on both csi driver and in-tree driver.
current test cases:
more cases to add:
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
design doc: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-migration.md
depends on cs-attacher v3.0.0: #430
depends on csi-provisioner fix: kubernetes/kubernetes#94853
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
could refer to kubernetes-sigs/cloud-provider-azure#106
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
Reference: https://kubernetes-csi.github.io/docs/volume-cloning.html
Need to check whether azure file supports create an azure file from existing azure file(cloning)
azure disk PR for reference:
kubernetes-sigs/azuredisk-csi-driver#196
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
add support for Pod Identity to enable finer grain scope of identity as an alternative to using the cluster’s identity. We do see cases where Azure resources are closely associated with the cluster. But we also see customers partitioning AKS clusters using namespaces, and in those cases it’s more likely that RBAC grants to resources like storage/key vault would be to Managed Identities that were scoped to a Kubernetes namespace.
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
scenario is like this:
https://github.com/csi-driver/blobfuse-csi-driver#option1-use-existing-credentials-in-k8s-cluster
Need to add a new parameter(shareName
) in azure file storage class, and create an azure file if it does not exist:
Describe alternatives you've considered
Additional context
/assign
What happened:
We should handle below ShareBeingDeleted
in DeleteVolume function, need to retry:
I0912 23:15:06.139324 12105 utils.go:111] GRPC call: /csi.v1.Controller/DeleteVolume
I0912 23:15:06.139355 12105 utils.go:112] GRPC request: volume_id:"#blobfuseci#sanity-node-full-8b4579a6-63d6317e"
E0912 23:15:06.298687 12105 utils.go:116] GRPC error: rpc error: code = Internal desc = DeleteFileShare sanity-node-full-8b4579a6-63d6317e under blobfuseci failed with error: -> github.com/Azure/azure-storage-file-go/azfile.newStorageError, /home/travis/gopath/pkg/mod/github.com/!azure/[email protected]/azfile/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=ShareBeingDeleted) =====
Description=The specified share is being deleted. Try operation later.
RequestId:58c3b74a-f01a-0185-5fbf-69ba2e000000
Time:2019-09-12T23:15:06.2888953Z, Details:
Code: ShareBeingDeleted
DELETE https://blobfuseci.file.core.windows.net/sanity-node-full-8b4579a6-63d6317e?restype=share&timeout=61
Authorization: REDACTED
User-Agent: [Azure-Storage/0.5.0 (go1.12.1; linux)]
X-Ms-Client-Request-Id: [e1117938-be6a-44eb-4ea5-290f123e9650]
X-Ms-Date: [Thu, 12 Sep 2019 23:15:06 GMT]
X-Ms-Delete-Snapshots: [include]
X-Ms-Version: [2018-03-28]
--------------------------------------------------------------------------------
RESPONSE Status: 409 The specified share is being deleted. Try operation later.
Content-Length: [244]
Content-Type: [application/xml]
Date: [Thu, 12 Sep 2019 23:15:06 GMT]
Server: [Windows-Azure-File/1.0 Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [ShareBeingDeleted]
X-Ms-Request-Id: [58c3b74a-f01a-0185-5fbf-69ba2e000000]
X-Ms-Version: [2018-03-28]
cleanup: error: DeleteVolume: rpc error: code = Internal desc = DeleteFileShare sanity-node-full-8b4579a6-63d6317e under blobfuseci failed with error: -> github.com/Azure/azure-storage-file-go/azfile.newStorageError, /home/travis/gopath/pkg/mod/github.com/!azure/[email protected]/azfile/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=ShareBeingDeleted) =====
Description=The specified share is being deleted. Try operation later.
RequestId:58c3b74a-f01a-0185-5fbf-69ba2e000000
Time:2019-09-12T23:15:06.2888953Z, Details:
Code: ShareBeingDeleted
DELETE https://blobfuseci.file.core.windows.net/sanity-node-full-8b4579a6-63d6317e?restype=share&timeout=61
Authorization: REDACTED
User-Agent: [Azure-Storage/0.5.0 (go1.12.1; linux)]
X-Ms-Client-Request-Id: [e1117938-be6a-44eb-4ea5-290f123e9650]
X-Ms-Date: [Thu, 12 Sep 2019 23:15:06 GMT]
X-Ms-Delete-Snapshots: [include]
X-Ms-Version: [2018-03-28]
--------------------------------------------------------------------------------
RESPONSE Status: 409 The specified share is being deleted. Try operation later.
Content-Length: [244]
Content-Type: [application/xml]
Date: [Thu, 12 Sep 2019 23:15:06 GMT]
Server: [Windows-Azure-File/1.0 Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [ShareBeingDeleted]
X-Ms-Request-Id: [58c3b74a-f01a-0185-5fbf-69ba2e000000]
X-Ms-Version: [2018-03-28]
What you expected to happen:
How to reproduce it:
Anything else we need to know?:
Environment:
kubectl version
):uname -a
):/assign @ZeroMagic
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
Is your feature request related to a problem?/Why is this needed
Describe the solution you'd like in detail
could refer to kubernetes-sigs/cloud-provider-azure#142
Describe alternatives you've considered
Additional context
Hello,
I mainly followed this link:
https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-file
In order to create a azure-file storage class. We wanted to mention a specific storageaccount in order to prevent AKS from creating a new one.
In attachment (filesc - Copy.txt) , you will find the definition of the storage class that I used (I removed the values of our storage account name)
filesc - Copy.txt
as you can notice, I'm requesting to use secret a specific secret and which is located in namespace technical (and which contains the storage account key to access the storage account). The purpose being that whenever a developer would like to dynamycally create a file share, he can just refer to this storage class and the file share will be created in the right storage account.
Then I tried to create the private volume claim (pvc) (please check the pvc definition in the attachment)
Problem: pvc stuck in pending
and I see an error
'invalid option 'secretName' for volume plugin kubernetes.io/azure-file'
Is this a bug or is the official documentation of kubernetes wrong?
thank you very much for your help
Is your feature request related to a problem? Please describe.
Describe the solution you'd like in detail
Describe alternatives you've considered
Additional context
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.