Giter Site home page Giter Site logo

kubernetes-sigs / nfs-ganesha-server-and-external-provisioner Goto Github PK

View Code? Open in Web Editor NEW
399.0 16.0 136.0 5.9 MB

NFS Ganesha Server and Volume Provisioner.

License: Apache License 2.0

Makefile 7.13% Go 43.52% Dockerfile 1.97% Shell 43.77% Mustache 0.65% Python 2.97%
k8s-sig-storage

nfs-ganesha-server-and-external-provisioner's Introduction

NFS Ganesha server and external provisioner

nfs-ganesha-server-and-external-provisioner is an out-of-tree dynamic provisioner for Kubernetes 1.14+. You can use it to quickly & easily deploy shared storage that works almost anywhere. Or it can help you write your own out-of-tree dynamic provisioner by serving as an example implementation of the requirements detailed in the proposal.

It works just like in-tree dynamic provisioners: a StorageClass object can specify an instance of nfs-ganesha-server-and-external-provisioner to be its provisioner like it specifies in-tree provisioners such as GCE or AWS. Then, the instance of nfs-ganesha-server-and-external-provisioner will watch for PersistentVolumeClaims that ask for the StorageClass and automatically create NFS-backed PersistentVolumes for them. For more information on how dynamic provisioning works, see the docs or this blog post.

Note: This repository was migrated from https://github.com/kubernetes-incubator/external-storage/tree/HEAD/nfs. Some of the following instructions will be updated once the build and release automtion is setup. To test container image built from this repository, you will have to build and push the nfs-provisioner image using the following instructions.

make build
make container
# `nfs-provisioner:latest` will be created. 
# To upload this to your customer registry, say `gcr.io/myorg`, you can use
# docker tag nfs-provisioner:latest gcr.io/myorg/nfs-provisioner:latest
# docker push gcr.io/myorg/nfs-provisioner:latest

Quickstart

Choose some volume for your nfs-ganesha-server-and-external-provisioner instance to store its state & data in and mount the volume at /export in deploy/kubernetes/deployment.yaml. It doesn't have to be a hostPath volume, it can e.g. be a PVC. Note that the volume must have a supported file system on it: any local filesystem on Linux is supported & NFS is not supported.

...
  volumeMounts:
    - name: export-volume
      mountPath: /export
volumes:
  - name: export-volume
    hostPath:
      path: /tmp/nfs-provisioner
...

Choose a provisioner name for a StorageClass to specify and set it in deploy/kubernetes/deployment.yaml

...
args:
  - "-provisioner=example.com/nfs"
...

Create the deployment.

$ kubectl create -f deploy/kubernetes/deployment.yaml
serviceaccount/nfs-provisioner created
service "nfs-provisioner" created
deployment "nfs-provisioner" created

Create ClusterRole, ClusterRoleBinding, Role and RoleBinding (this is necessary if you use RBAC authorization on your cluster, which is the default for newer kubernetes versions).

$ kubectl create -f deploy/kubernetes/rbac.yaml
clusterrole.rbac.authorization.k8s.io/nfs-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-provisioner created

Create a StorageClass named "example-nfs" with provisioner: example.com/nfs.

$ kubectl create -f deploy/kubernetes/class.yaml
storageclass "example-nfs" created

Create a PersistentVolumeClaim with storageClassName: example-nfs.

$ kubectl create -f deploy/kubernetes/claim.yaml
persistentvolumeclaim "nfs" created

A PersistentVolume is provisioned for the PersistentVolumeClaim. Now the claim can be consumed by some pod(s) and the backing NFS storage read from or written to.

$ kubectl get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM         REASON    AGE
pvc-dce84888-7a9d-11e6-b1ee-5254001e0c1b   1Mi        RWX           Delete          Bound       default/nfs             23s

Deleting the PersistentVolumeClaim will cause the provisioner to delete the PersistentVolume and its data.

Deleting the provisioner deployment will cause any outstanding PersistentVolumes to become unusable for as long as the provisioner is gone.

Running

To deploy nfs-ganesha-server-and-external-provisioner on a Kubernetes cluster see Deployment.

To use nfs-ganesha-server-and-external-provisioner once it is deployed see Usage.

Releases done here in external-storage will not have corresponding git tags (external-storage's git tags are reserved for versioning the library), so to keep track of releases check this README, the changelog, or GCR

Writing your own

Go here for an example of how to write your own out-of-tree dynamic provisioner.

Roadmap

The source code in this repository was migrated from kubernetes-incubator/external-storage. We are yet to complete the following migration tasks.

  • Update e2e tests
  • Automate building container images to the new registry
  • Update helm chart

This is still alpha/experimental and will change to reflect the out-of-tree dynamic provisioner proposal

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this project at:

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

nfs-ganesha-server-and-external-provisioner's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nfs-ganesha-server-and-external-provisioner's Issues

image quay.io/external_storage/nfs-ganesha-server-and-provisioner does not exist

Hello!

We're playing around with nfs-ganesha-server-and-external-provisioner, but our server/provisioner pod is going into ImagePullBackOff:

  Warning  Failed     9m48s (x4 over 11m)  kubelet, ip-10-100-89-6.us-west-2.compute.internal  Failed to pull image "quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest": rpc error: code = Unknown desc = Error response from daemon: unauthorized: access to the requested resource is not authorized
  Warning  Failed     9m48s (x4 over 11m)  kubelet, ip-10-100-89-6.us-west-2.compute.internal  Error: ErrImagePull
  Warning  Failed     9m36s (x6 over 11m)  kubelet, ip-10-100-89-6.us-west-2.compute.internal  Error: ImagePullBackOff
  Normal   BackOff    76s (x41 over 11m)   kubelet, ip-10-100-89-6.us-west-2.compute.internal  Back-off pulling image "quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest"

pulling the image from the command line doesn't seem to work either, it looks like the image just isn't there on quay.io at all:

$ docker pull quay.io/external_storage/nfs-ganesha-server-and-external-provisioner:latest
Error response from daemon: unauthorized: access to the requested resource is not authorized

I don't mind building the image ourselves, but the docs should probably be updated to state that it's not available on quay just yet :)

Cheers!

Timeout when Pods try to mount volumes referencing PVCs of storageClass `nfs` on K3D cluster

Problem:
I am trying to get nfs-server-provisioner to work in K3D. PVCs and PVs are handled correctly, but Pods cant mount the volume referenced by a PVC. I created an example to make it easy to reproduce the issue in a GitRepo: https://github.com/tricky42/k3d-nfs with a more detailed description.

In this setup the local-path provider from K3D is used to provide the PV for the NFS server automatically through the PVC issued by the nfs-server-provisioner. PVCs of storage class nfs can be provisioned (status: bound), but still pods trying to mount volumes which reference a PVC of storageClass nfs(in this example the PVC nfs-for-pods) are not starting up due to:

MountVolume.SetUp failed for volume "pvc-52c0d57a-b285-45b6-b04c-41175604b0d0" : mount failed: exit status 255
Mounting command: mount Mounting arguments: -t nfs -o vers=3 10.43.192.199:/export/pvc-52c0d57a-b285-45b6-b04c-
41175604b0d0 /var/lib/kubelet/pods/c9e21798-76ad-404d-b089-c9fff88e0a86/volumes/kubernetes.io~nfs/pvc-
52c0d57a-b285-45b6-b04c-41175604b0d0 Output: mount: mounting 10.43.192.199:/export/pvc-52c0d57a-b285-45b6-
b04c-41175604b0d0 on /var/lib/kubelet/pods/c9e21798-76ad-404d-b089-c9fff88e0a86/volumes/kubernetes.io~nfs/pvc-
52c0d57a-b285-45b6-b04c-41175604b0d0 failed: Connection refused

Unable to attach or mount volumes: unmounted volumes=[nfs-mount], unattached volumes=[nfs-mount default-token
-t7qxd]: timed out waiting for the condition

To quickly reproduce this, just enter:

$ make all

So far I couldnt find out, why the Pod cant mount the volume managed by the nfs server :(

Stable helm chart deprecated

The documented helm install command:

helm install stable/nfs-server-provisioner

gives a deprecated warning as the helm stable repo has been deprecated. Is there a plan to move this to a different helm repo?

Local export perms

Hi there,
I would like to know which perms you recommend about the directory that will be exported by the NFS Storage Class.
Many thanks

Regenerate vfs.conf on initialization

Consider regenerating ganesha exports config file /export/vfs.conf every time at startup.

the export block already can be found in allocated PVs:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    EXPORT_block: "\nEXPORT\n{\n\tExport_Id = 1;\n\tPath = /export/pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d;\n\tPseudo
      = /export/pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d;\n\tAccess_Type = RW;\n\tSquash
      = no_root_squash;\n\tSecType = sys;\n\tFilesystem_id = 1.1;\n\tFSAL {\n\t\tName
      = VFS;\n\t}\n}\n"
    Export_Id: "1"
    Project_Id: "0"
    Provisioner_Id: 82f81614-14e4-474a-b852-facc2906d3ab
    kubernetes.io/createdby: nfs-dynamic-provisioner
    pv.kubernetes.io/provisioned-by: cluster.local/nextcloud-nfs-server-provisioner
  name: pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: nextcloud-nfs
    namespace: nextcloud
    resourceVersion: "331724805"
    uid: 40a3866c-83d6-47e9-ac9f-6d0462a4ab1d
  mountOptions:
  - vers=3
  nfs:
    path: /export/pvc-40a3866c-83d6-47e9-ac9f-6d0462a4ab1d
    server: 10.96.15.81
  persistentVolumeReclaimPolicy: Delete
  storageClassName: nextcloud-nfs

have the nfs provisioner support hight availability

I have been testing the helm chart in order to deploy the nfs provisioner because we would like to use it for production for many dynamic temporary PVs.
All the default values works fine but it creates only one replica of the provisionner (and the server).
For my use case I need to have more relyability and may be more throughput and was hoping to be able to increase the number of replicas.

1/ So I tried to set the replicaCount=2. Both replicas started fine. But then we I try to claim a persistent volume it failed.
This is the error from the replica-0

controller.go:908] error syncing claim "36caa83a-31f9-4875-bc1d-b4656e06895e": failed to provision volume with StorageClass "nfs":

here is the error in the of the pvc

  Warning  ProvisioningFailed    38s (x4 over 2m23s)  cluster.local/nfs-nfs-server-provisioner_nfs-nfs-server-provisioner-0_09128ba9-3d58-46eb-bcc2-6318389bc26e  failed to provision volume with StorageClass "nfs": error getting NFS server IP for volume: service SERVICE_NAME=nfs-nfs-server-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.223.67.159

I figured that because it us using a single service in front of a 2 pods then the loadbalancing cannot make this setup work. ( maybe headless service would be more appropriate).

2/ So I have decided to alter the helm chart to disable the SERVICE_NAME environment variable by commenting this. I have seen in the code that the provisioner would use the pod IP so why not give it a try.
So this works better as the PV managed to be created. but...
But it seems that only the first replica is managing the PV and the second replica is simply ignored.

Is there a way to make this work ? I am not familiar with the provider mechanism but is it possible that multiple instances may manage different PVs ?

Sorry for this long post, I hope that my questions are clean enough.

How to use with a specified namespace?

Right now we are trying to figure out how to install this chart in a dedicated namespace?
But it looks like it's not possible when I look at the chart. Is there any reason for this, so that one can specify one?

Our helmfile looks like following:

environments:
  production:
  staging:

repositories:
  - name: kvaps
    url: https://kvaps.github.io/charts

releases:
  - name: {{ .Environment.Name }}
    chart: charts/company
    values:
      - charts/company/configs/{{ .Environment.Name }}.yaml
    set:
      - name: railsmasterkey
        file: config/master.key
  - name: nfs-server-provisioner
    version: 1.2.0
    chart: kvaps/nfs-server-provisioner
    namespace: {{ .Environment.Name }} <--- this is not working?

Image not found, from the deployment

The yaml doesn't match the text.

https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner#quickstart

$ docker pull quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest
Error response from daemon: unauthorized: access to the requested resource is not authorized

https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/blob/master/docs/deployment.md

kubectl get deploy nfs-provisioner -o yaml | sed "s|quay.io/external_storage/nfs-ganesha-server-and-provisioner:latest|k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0|" | kubectl replace -f -

nfs-provisioner arm64 support

Failed to pull image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0": rpc error: code = Unknown desc = Error choosing image instance: no image found in manifest list for architecture arm64, variant "v8", OS linux

Kubernetes YAML manifests not maintained in sync with Helm chart

I note that there is a set of yaml files that i assume to mirror the Helm chart, but they are not updated as work is done on the Helm chart.

Should they be maintained at all, and if so, how should they be maintained? By using helm template to generate a rendered set of resources from the Helm charts default values perhaps?

Read-only file system - cannot write to pod

Hey,

I have installed the provider through helm:

helm install data-nfs-server-provisioner stable/nfs-server-provisioner -f nvalues.yaml  --namespace clearml

Then I have added a Persistent Volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-nfs-server-provisioner-0
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  gcePersistentDisk:
    fsType: "ext4"
    pdName: "clearml-cache"
  claimRef:
    namespace: clearml
    name: data-data-nfs-server-provisioner-0

While clearml-cache was created with:

$ gcloud compute disks create --size=100GB --zone=$ZONE --project "$PROJECT_ID" clearml-cache

Then I have added a PV and PVC:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: clearml-agent-cachingt-claim
  namespace: clearml
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100G
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: clearml-agent-cachingt
  namespace: clearml
spec:
  capacity:
    storage: 100G
  storageClassName: "nfs"
  accessModes:
    - ReadWriteMany
  nfs:
    path: /
    server: data-nfs-server-provisioner.clearml.svc.cluster.local
  claimRef:
    namespace: clearml
    name: clearml-agent-cachingt-claim
EOF

and a pod:

apiVersion: v1
kind: Pod
metadata:
  name: nfs-shaked-test
spec:
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    fsGroup: 0
  containers:
  - name: nfs-shaked-test
    image: d3fk/nfs-client
    command: [ "sleep" ]
    args: [ "infinity" ]
    securityContext:
      privileged: true
    resources: {}
    env:
      - name: SHARE
        value: "/shakedsnfs"
      - name: SERVER
        value: "data-nfs-server-provisioner.clearml.svc.cluster.local"
    volumeMounts:
      - mountPath: /shakedsnfs
        name: caching-nfs
        readOnly: false
  volumes:
    - name: caching-nfs
      persistentVolumeClaim:
        claimName: clearml-agent-cachingt-claim

I tried to set this pod with and without some of the above configuration, for example with and without privileged, runAs, fsGroup and readOnly: false. I have also tried to use the write-pod example but I keep seeing the following error:

/ # touch shakedsnfs/1
touch: shakedsnfs/1: Read-only file system

I have found nothing in the logs:

ganesha.logs

14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
14/05/2021 19:09:08 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
14/05/2021 19:10:38 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
16/05/2021 17:20:18 : epoch 609ecad4 : data-nfs-server-provisioner-0 : nfs-ganesha-22[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 5962, old: 5961; deq new: 5961, old: 5961
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
16/05/2021 21:00:56 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 21:02:26 : epoch 60a18808 : data-nfs-server-provisioner-0 : nfs-ganesha-21[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
16/05/2021 22:22:43 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
16/05/2021 22:24:13 : epoch 60a19b33 : data-nfs-server-provisioner-0 : nfs-ganesha-21[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE

kubectl logs -f data-nfs-server-provisioner-0:

I0516 22:22:43.183805       1 main.go:64] Provisioner cluster.local/data-nfs-server-provisioner specified
I0516 22:22:43.183891       1 main.go:88] Setting up NFS server!
I0516 22:22:43.401571       1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0516 22:22:43.401613       1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0516 22:22:43.414202       1 server.go:134] Running NFS server!

I assume I am missing something, but honestly I can't seem to find it.

Any chance you could help?
Thank you
Shaked

Three Dockerfiles in repo could be a single

In #56 @kvaps suggests to let two dockerfiles be the same Dockerfile. Excellent!

I'm not sure about the status of the arm Dockerfile, but trialing to build the x86_64 arm dockerfile on my arm64 based Raspberry PI 4, it seem to work fine as well.

Perhaps there should simply be one single Dockerfile instead of three or two?

nfs-provisioner Apple Silicon M1 ARM support

Current image doesn't run on newer Macbooks:

Failed to pull image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0": rpc error: code = N โ”‚
โ”‚ otFound desc = failed to pull and unpack image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0": no match for platform in manifest: not found

Any chance ARM variant of this image can be added to the build?

Thanks!

[Proposal] Multi-server storage class and provisioner

Volume resizing and splitting servers by subpaths has two problems in my usage:

  • Volume resizing doesn't happen automatically (or at least I haven't been able to figure that out), so the underlying storage is either overprovisioned, or may block PVCs using this storage class. While this isn't technically wrong, it breaks the contracts many cloud providers have with their storage classes (roughly unlimited storage), and reduces the utility of dynamic volume provisioning.
  • Volume resizing is much more difficult to do (both manually and automatically) if the resizing exceeds the underlying storage for PVCs deployed with this storage class (since both the provisioner's volume and the the volume the PVC is bound to must be resized)
  • Even if resizing was made automatic, volumes can't be resized down, and so the underlying storage class would be dramatically overprovisioned.

One solution might be a different Provisioner that provisions NFS servers and the storage that back them according to parameters of the storage class, the PVC that triggered provisioning, or the pod that bound the triggering PVC (if volumeBindingMode: WaitForFIrstConsumer)


Mode 1: If the user wants to be able to provision PVCs ahead of time.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-fast
  labels:
       storageClass: 'fast'
provisioner: cluster.local/multi-nfs-provisioner
parameters:
  storageClass: 'fast'
  resources:
    requests:
        cpu: 100/1Ti
        memory: 250/1Ti
  topology:
    ...
  affinities:
    ...
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: Immediate

Mode2: User is ok with slower provisioning in exchange for better utilization, and keeping the server near its consumers.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-fast-dynamic
  labels: 
       storageClass: 'fast'
parameters:
  storageClass: 'fast'
  resources:
    requests:
        cpu: 100m/1Ti
        memory: 250m/1Ti
provisioner: cluster.local/multi-nfs-provisioner
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

When a new volume is requested with storage class nfs-fast A nfs server backed by a dynamically provisioned volume of storageClass with topology/affinity constraints either from parameters or the bound pod, and size from the requesting PVC.
Additionally resource requests could be specified in terms of the storage size requested, since for most volume types I/O performance scales with size (here, for every terabyte of requested storage server would request 100m CPU and 250m memory)

The statefulset for the provisioner, instead of maintaining a mapping from PVs to subpaths, would maintain a mapping from PVs to servers.

Summary of advantages over current approach:

  • Additional storage can be dynamically requested and released from underlying storage by PVCs as expected.
  • Better isolation of PVC requests (PVCs can't spill above the storage of the underlying volume and take away from "guaranteed storage" for other PVCs).
  • NFS Server requests can be scaled according to underlying disk size, allowing users of cloud providers where I/O scales with drive size to better saturate their drives
  • For multi-zone clusters allows for colocation of server and PVC when bound just-in-time.
  • Easier for users to add a new "kind" of NFS storage based on another storage class (rather than deploying a second helm chart, just create a new parameterized storage class)

I'm relatively new to the CSI ecosystem but I'm wondering if something like this is feasible/has been thought about.

PriorityClass name is set to nil even when none is created.

The Priority Class section is causing issues in version 1.4.0.
If priorityClass.create is false and priorityClass.name is "", priorityClassName is set to nil (see below)
Error: template: nfs-server-provisioner/templates/statefulset.yaml:120:24: executing "nfs-server-provisioner/templates/statefulset.yaml" at <.Values.priorityClass.name>: nil pointer evaluating interface {}.name

If priorityClass.create is set to true and priorityclass.name is set to "default", the error is as shown below.
Error: error validating "": error validating data: ValidationError(PriorityClass): missing required field "value" in io.k8s.api.scheduling.v1.PriorityClass

docker container(external) can't start after being stopped

reproduce:
docker stop nfs-ganesha (or restart host os)
docker start nfs-ganesha

Container goes into infinite restart loop with error message:

Error setting up NFS server: dbus-daemon failed with error: exit status 1, output: dbus-daemon[18]: Failed to start message bus: The pid
file "/var/run/dbus/pid" exists, if the message bus is not running, remove this file

Issues when using more replicas!

I'm using the helm chart with the following values:

replicaCount: 3

persistence:
  enabled: true

  ## Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "do-block-storage"

  accessMode: ReadWriteOnce
  size: 10Gi

## For creating the StorageClass automatically:
storageClass:
  create: true

  ## Set a provisioner name. If unset, a name will be generated.
  # provisionerName:

  ## Set StorageClass as the default StorageClass
  ## Ignored if storageClass.create is false
  defaultClass: false

  ## Set a StorageClass name
  ## Ignored if storageClass.create is false
  name: nfs

  # set to null to prevent expansion
  allowVolumeExpansion: true
  ## StorageClass parameters
  parameters: {}

  mountOptions:
    - vers=3

  ## ReclaimPolicy field of the class, which can be either Delete or Retain
  reclaimPolicy: Delete

When I try to create pvc it stay in the pending state with the following events:

Events:
  Type     Reason                Age               From                                                                                                                Message
  ----     ------                ----              ----                                                                                                                -------
  Normal   Provisioning          7s (x2 over 22s)  cluster.local/pro-nfs-nfs-server-provisioner_pro-nfs-nfs-server-provisioner-0_9522a528-c992-48c2-bdc3-6dec4370104d  External provisioner is provisioning volume for claim "default/test-rwx"
  Warning  ProvisioningFailed    7s (x2 over 22s)  cluster.local/pro-nfs-nfs-server-provisioner_pro-nfs-nfs-server-provisioner-0_9522a528-c992-48c2-bdc3-6dec4370104d  failed to provision volume with StorageClass "nfs": error getting NFS server IP for volume: service SERVICE_NAME=pro-nfs-nfs-server-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.244.1.183
  Normal   ExternalProvisioning  2s (x3 over 22s)  persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "cluster.local/pro-nfs-nfs-server-provisioner" or manually created by system administrator

If I change the replicaCount to 1 everything goes fine!

Resizing support?

It might still relate to kubernetes-retired/external-storage#1142.

and when I tried to resize the volume it shows this message on the describe command:

Events:
  Type     Reason             Age                From           Message
  ----     ------             ----               ----           -------
  Warning  ExternalExpanding  42s (x3 over 12h)  volume_expand  Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.

Is there any plan to support this feature?

MountVolume.SetUp failed for volume "kube-api-access-b9tt8" : object "default"/"kube-root-ca.crt" not registered

$ k version 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
$ k applay -f deploy/kubernetes/write-pod.yaml                                                                                                              
3m15s       Normal    Scheduled                 pod/write-pod                                           Successfully assigned default/write-pod to k8s-worker04.kubernetes.lan
3m14s       Normal    Pulled                    pod/write-pod                                           Container image "gcr.io/google_containers/busybox:1.24" already present on machine
3m14s       Normal    Created                   pod/write-pod                                           Created container write-pod
3m14s       Normal    Started                   pod/write-pod                                           Started container write-pod
3m12s       Warning   FailedMount               pod/write-pod                                           MountVolume.SetUp failed for volume "kube-api-access-b9tt8" : object "default"/"kube-root-ca.crt" not registered
$ k get pod
NAME                               READY   STATUS      RESTARTS   AGE
nfs-provisioner-5b485c6b65-96t5r   1/1     Running     0          14m
read-pod                           0/1     Completed   0          8m41s
write-pod                          0/1     Completed   0          8m48s

feature request: helm chart pod annotations

Please add a field to allow us to specify pod annotations.

Example use case: disallowing pod eviction by Cluster Autoscaler:

cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Cannot get NFS to create PV

Running ganesha in k8s.

Getting the following error when trying to create PV:

I0406 00:26:56.616749       1 main.go:65] Provisioner example.com/nfs specified
I0406 00:26:56.617163       1 main.go:89] Setting up NFS server!
I0406 00:26:57.573661       1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0406 00:26:57.573762       1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0406 00:26:57.575869       1 server.go:134] Running NFS server!
E0406 00:27:02.889823       1 controller.go:908] error syncing claim "2937d4b7-2215-4d84-be2d-5552790e7cd4": failed to provision volume with StorageClass "example-nfs": error getting NFS server IP for volume: service SERVICE_NAME=nfs-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.42.2.26

Here is the sum of my config:

kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: ganesha
  name: nfs-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  replicas: 1
  strategy:
    type: Recreate 
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      nodeSelector:
        purpose: nas
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-provisioner
          image: k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.0
          ports:
            - name: nfs
              containerPort: 2049
            - name: nfs-udp
              containerPort: 2049
              protocol: UDP
            - name: nlockmgr
              containerPort: 32803
            - name: nlockmgr-udp
              containerPort: 32803
              protocol: UDP
            - name: mountd
              containerPort: 20048
            - name: mountd-udp
              containerPort: 20048
              protocol: UDP
            - name: rquotad
              containerPort: 875
            - name: rquotad-udp
              containerPort: 875
              protocol: UDP
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: statd
              containerPort: 662
            - name: statd-udp
              containerPort: 662
              protocol: UDP
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
          args:
            - "-provisioner=example.com/nfs"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: export-volume
              mountPath: /export
      volumes:
        - name: export-volume
          hostPath:
            path: /mnt/data
---
apiVersion: v1
kind: Namespace
metadata:
  name: ganesha
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  namespace: ganesha
  name: nfs
spec:
  storageClassName: example-nfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: ganesha
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: ganesha
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
     # replace with namespace where provisioner is deployed
    namespace: ganesha
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: ganesha
roleRef:
  kind: Role
  name: leader-locking-nfs-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ganesha
  name: nfs-provisioner
kind: Service
apiVersion: v1
metadata:
  namespace: ganesha
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  ports:
    - name: nfs
      port: 2049
      targetPort: 2049
      protocol: TCP
    - name: nfs-udp
      port: 2049
      targetPort: 2049
      protocol: UDP
    - name: nlockmgr
      port: 32803
      targetPort: 32803
      protocol: TCP
    - name: nlockmgr-udp
      port: 32803
      targetPort: 32803
      protocol: UDP
    - name: mountd
      port: 20048
      targetPort: 20048
      protocol: TCP
    - name: mountd-udp
      port: 20048
      targetPort: 20048
      protocol: UDP
    - name: rquotad
      port: 875
      targetPort: 875
      protocol: TCP
    - name: rquotad-udp
      port: 875
      targetPort: 875
      protocol: UDP
    - name: rpcbind
      port: 111
      targetPort: 111
      protocol: TCP
    - name: rpcbind-udp
      port: 111
      targetPort: 111
      protocol: UDP
    - name: statd
      port: 662
      targetPort: 662
      protocol: TCP
    - name: statd-udp
      port: 663
      targetPort: 663
      protocol: UDP
  selector:
    app: nfs-provisioner
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  namespace: ganesha
  name: example-nfs
provisioner: example.com/nfs
mountOptions:
  - vers=4.1
---
apiVersion: v1
kind: Pod
metadata:
  namespace: ganesha
  name: write-pod
spec:
  containers:
  - name: write-pod
    image: gcr.io/google_containers/busybox:1.24
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: nfs

Clustering and high availability

It's my understanding that the project currently doesn't support creating a highly-available NFS server cluster given the lack of examples on the subject. However, I do also see the leader-elect flag on the binary with the description Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability. Default false..

I'm wondering if anyone has used the leader-elect in a highly available NFS cluster configuration and what the setup for that looked like.

From an old presentation on NFS-Ganesha here, it seems that NFS-Ganesha requires some coordination to handle failover well. I suspect this would be set up manually by any user wanting to use a highly available NFS cluster setup with this provisioner, but I'm not certain.

Going off of the assumption that this project currently can't set up a multiple-NFS-Server cluster that is highly available today, is it within the purview of this project to support such a thing and accept contributions to design and implement the ability to set up a highly-available NFS cluster inside Kubernetes?

@wongma7 @kvaps @kmova

Sharing NFS folder between two namespaces

Hi,
Using this provisioner works fine when pods share files from storage obtained using the same namespace/PVC as the shared folder name is dynamically created based on namespace/PVC/PV.
How can I manage to share files between two namespaces ?
Regards
Mic

Remove or relocate deploy/docker/config

I'm not sure what this file does, but I see no mention of it in docs and it lacks comments etc. I believe it is a Kubernetes config on how to access a k8s cluster, and if it is that, it wouldn't make sense to locate in the deploy/docker folder.

apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: http://0.0.0.0:8080
name: local
contexts:
- context:
cluster: local
user: ""
name: local
current-context: local
kind: Config
preferences: {}
users: []

Suggested action points

  1. Remove the file
  2. Relocate the file
  3. Add inline comments about the file's purpose

Upgrade NFS - Ganesha to version 3.5

Hello

The version of NFS - Ganesha used in this provisionner is the version 2.8.2 which is old (Jully 2019 according to its repository.
Could you please upgrade the version to 3.5, latest GA available ?

Best regards

Security: add Network policies

Is there the possibility to add Network Policies to avoid other pods in a multi-tenant cluster from accessing Ganesha server without impacting volume mounts ?

Pods fail when are allocated in a different node to the nfs-provisioner pod is running.

Hi! I tried to use nfs-provisioner with openshift.
But pods are failing when are allocated in a different node where the nfs-provisioner pod is running. I suppose that it is because CLUSTERIP service is not accessible from pods in different nodes.
Is there another way to expose the provisioner rather than using clusterIP?

NAME                                 READY   STATUS              RESTARTS   AGE     IP            NODE                               NOMINATED NODE   READINESS GATES
item-onboard-test                    0/1     ContainerCreating   0          2m18s   <none>        jorge-thks6-worker-eastus1-pc27r   <none>           <none>
item-onboard-test-85c9549869-khfnn   1/1     Running             0          106s    10.129.2.17   jorge-thks6-worker-eastus2-kgcf8   <none>           <none>
item-onboard-test-x7gz2              0/1     ContainerCreating   0          100s    <none>        jorge-thks6-worker-eastus3-xjn78   <none>           <none>
nfs-provisioner-7d898f4fdd-tn6ws     1/1     Running             0          8m37s   10.129.2.14   jorge-thks6-worker-eastus2-kgcf8   <none>           <none>
test                                 1/1     Running             0          2m45s   10.129.2.16   jorge-thks6-worker-eastus2-kgcf8   <none>           <none>
test-no-pvc                          1/1     Running             0          8m36s   10.129.2.13   jorge-thks6-worker-eastus2-kgcf8   <none>           <none>

Many thanks,
Jorge

High CPU makes instance unusbale

When deploying using

git clone https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner
/tmp/nfs-ganesha-server-and-external-provisioner/deploy/helm
git checkout V3.0.0
helm install nfs-server-provisioner .  \
  --namespace nfs-server-provisioner --create-namespace \
  --set persistence.storageClass="openebs-hostpath" \
  --set persistence.size="10Gi" \
  --set persistence.enabled=true

The k8s instance become unresponsible with a high cpu load.

Not even sure where to start diagnosing this issue.

Any ideas?

top - 07:15:51 up 25 min,  3 users,  load average: 28.98, 45.59, 28.49
Tasks: 247 total,   2 running, 245 sleeping,   0 stopped,   0 zombie
%Cpu(s): 38.0 us, 15.5 sy,  0.0 ni, 28.7 id, 17.3 wa,  0.3 hi,  0.2 si,  0.0 st
MiB Mem :  31980.4 total,   2275.2 free,  28775.4 used,    929.8 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2822.9 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  33770 root      20   0 1101992 376880  75920 S 105.6   1.2   0:04.78 kube-apiserver
  32624 root      20   0   52.0g  27.0g   3392 R  99.3  86.6   0:09.76 rpcinfo

k8s version: 1.21
os: fedora coreos

NFS Server Provisioner pod keeps restarting

Hello,

I believe there is something probably off with the latest nfs server provisioner chart.

My kubernetes version:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Steps I did:

$ helm repo add nfs-ganesha-server-and-external-provisioner https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner/
$ helm install my-release nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner

Results:
image

Why the pod keeps restarting? Looking at the logs, it exits with return code 137

kl my-release-nfs-server-provisioner-0
I1022 01:44:14.624283       1 main.go:65] Provisioner cluster.local/my-release-nfs-server-provisioner specified
I1022 01:44:14.631565       1 main.go:89] Setting up NFS server!
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Fri, 22 Oct 2021 12:41:43 +1100
      Finished:     Fri, 22 Oct 2021 12:42:48 +1100

I was digging into the github action that tests the chart, I am wondering if the workflow did what it was supposed to do:
image

Some steps takes 0s ? Is that expected ?

Thank you.

Cheers.

Problematical use of Filesystem_Id in nfs-ganesha exports

In the nfs-ganesha project, we have this issue raised when Export_Id 152 is provisioned:

nfs-ganesha/nfs-ganesha#615

I am wondering why the provisioning is specifying Filesystem_Id? It is not recommended to use this export configuration option for nfs-ganesha and instead allow the underlying filesystem to provide the filesystem ID via a UUID or by the device ID the filesystem is mounted on. There are situations where a stable Filesystem_Id is not available and the option must be specified. One such place is FSAL_PSEUDO exports, in particular the automatically configured FSAL_PSEUDO export for the NFSv4 PseudoFS root. This auto-configured export (Export_Id 0) uses Filesystem_Id 152.152. This is a long standing configuration predating the ability to use UUID or Device ID from the underlying filesystem. The use of Export_Id.Export_Id for the provisioned exports Filesystem_Id is a problem if Export_Id 152 is used.

If you could either change the Filesystem_Id or eliminate the option, that would be ideal. If not, the best solution would then be to consider Export_Id 152 to also be reserved.

Another option would be for the provisioner to explicitly configure the PseduoFS root and specify a Filesystem_Id that doesn't conflict with the Export_Id.Export_Id ideal. I'm not sure if 0.0 is ok to use, but any X.Y where X and Y are not the same would work.

container restarts when api server is failing

Issue was reported by @albanm on the older repo at: kubernetes-retired/external-storage#1184

A proposed solution by @linki for this is available at: kubernetes-retired/external-storage#1352

Copy past of the issue description.

Describe the bug

The container restarts frequently when the api server is not responding but it should not have to do it.

What happened:

I have a cluster with latency and sometimes failure issues of the api server. I have submitted a bug to my cloud provider that manages the control plane.
My issue with nfs-server-provisioner is that during periods of failures of the api server I seel logs like this and the container restarts :

I0619 07:05:11.965197       1 leaderelection.go:231] failed to renew lease nfs/cluster.local-nfs-data-nfs-server-provisioner: failed to tryAcquireOrRenew context deadline exceeded
F0619 07:05:11.966172       1 controller.go:646] leaderelection lost

I have lots of problems with the PVCs based on this provisionner: binding timeouts, lost file writes, etc.
I am not certain but I think it is related to these frequent restarts of the nfs server.

What you expected to happen:

The nfs server should tolerate api server failures.

How to reproduce it (as minimally and precisely as possible):

I do not manage the control plane of my cluster. But I suppose that manually stopping the api server should recreate the issue.

Anything else we need to know:

I understand restarting the container in this case is probably the expected behavior, but I think it is wrong.

replication is not relevant in the case of nfs-server-provisioner and I don't see the point of a leader election
stability issues of the api server should not propagate to a storage provisioner and then to a bunch of volumes used by a bunch of services

Increase the timeout for the prow jobs

With addition of non-amd64 platforms, now prow job is taking more time to finish building the container images because of the cross-build(qemu simulation mode) and times out for default timeout(2hrs)

Sample job run:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-nfs-ganesha-server-and-external-provisioner-push-images/1459900631392194560

something like this is needed to fix the timeout issue:
https://github.com/kubernetes/test-infra/blob/9299b9708f6c5c15e8a41c2f2b989033f6d239b0/config/jobs/kubernetes/sig-network/sig-network-misc.yaml#L13:L15,

No Releases or Updated Changelog

There are no releases for this project. It looks some pull requests in the prior read-only project were not pulled across into this version. It is hard to tell what version this corresponds to as there are no tags or releases. The helm chart included in this repo is not versioned either (typically there's a branch that communicates the releases to third party helm chart hubs).

The Quay repository referenced in the helm chart shows version 2.3.0, but then nowhere is a tag or release that corresponds to it.

Update Helm chart with current best practices - PRs wanted?

As Helm has developed over time, various best practices has been updated. I'd be happy to contribute PRs that would make this Helm chart up to date with such practices. But, I'd like some input with regards to if that would be wanted, and if it could be okay to make some breaking changes at this time.

I figure since this Helm chart isn't being published right now, it would be suitable to do breaking changes now before it becomes published again, for example at https://artifacthub.io/ which is the new Helm Hub location.

Breaking changes examplified

  • Use modern labels, so instead of app it would be app.kubernetes.io/name (like discussed for this Helm chart: jupyterhub/zero-to-jupyterhub-k8s#1867).
    This is tricky to update with helm3 upgrade even though it was quite easy with helm2 upgrade --force, so if it is ever to be done, it's good to do sooner rather than later in my mind.
  • _Apply
  • Update Chart.yaml to apiVersion: 2.
    A change that will make Helm 2 no longer be supported, but Helm 2 reach end of life November 13.

Non breaking changes examplified

  • Apply systematic whitespace chomping practices (always chomp left, almost never chomp right).
  • Use {{- with ... }} instead of {{- if }} when it makes sense
  • Use include instead of template.
  • Use nindent with whitespace chomping left instead of indent when using toYaml
  • Follow indentation practices for readability

Feedback wanted

Would you be open to such breaking changes and/or such non breaking changes to the Helm chart?

About me

I'm maintaining the JupyterHub Helm chart and have spent a lot of time working with k8s and Helm chart the last ~3 years. I'm motivated to contribute to this project as I think it serves a very important piece in the infrastructure I want to help many others make use of, for example universities that want to provide a compute platform for their researchers and students with personal home directories.

Mount PVC as specific user

Hi, Is it possible to use this to mount the pvc as a specific user? I see we can mount with a certain gid using that parameter, but then the uid is some random number. There is no parameter option for uid as I've tried, so is there some other way to mount an nfs pvc with a specific uid?

NFS Server not working

Hi there,

apologies if I am missing out something obvious.
I am using minikube to test out the NFS server and following the instructions and using the deployment:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
---
kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  ports:
    - name: nfs
      port: 2049
    - name: nfs-udp
      port: 2049
      protocol: UDP
    - name: nlockmgr
      port: 32803
    - name: nlockmgr-udp
      port: 32803
      protocol: UDP
    - name: mountd
      port: 20048
    - name: mountd-udp
      port: 20048
      protocol: UDP
    - name: rquotad
      port: 875
    - name: rquotad-udp
      port: 875
      protocol: UDP
    - name: rpcbind
      port: 111
    - name: rpcbind-udp
      port: 111
      protocol: UDP
    - name: statd
      port: 662
    - name: statd-udp
      port: 662
      protocol: UDP
  selector:
    app: nfs-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccountName: nfs-provisioner
      containers:
        - name: nfs-provisioner
          image: quay.io/kubernetes_incubator/nfs-provisioner:latest
          ports:
            - name: nfs
              containerPort: 2049
            - name: nfs-udp
              containerPort: 2049
              protocol: UDP
            - name: nlockmgr
              containerPort: 32803
            - name: nlockmgr-udp
              containerPort: 32803
              protocol: UDP
            - name: mountd
              containerPort: 20048
            - name: mountd-udp
              containerPort: 20048
              protocol: UDP
            - name: rquotad
              containerPort: 875
            - name: rquotad-udp
              containerPort: 875
              protocol: UDP
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: statd
              containerPort: 662
            - name: statd-udp
              containerPort: 662
              protocol: UDP
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
            privileged: true
          args:
            - "-provisioner=example.com/nfs"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: "Always"
          volumeMounts:
            - name: export-volume
              mountPath: /export
      volumes:
        - name: export-volume
          hostPath:
            path: /tmp/export

and rbac:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-provisioner
  apiGroup: rbac.authorization.k8s.io

In ganesha.log I am getting:

04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] main :MAIN :EVENT :nfs-ganesha Starting: Ganesha Version 2.8.2
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] claim_posix_filesystems :FSAL :CRIT :Could not stat directory for path /nonexistent
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] vfs_create_export :FSAL :CRIT :resolve_posix_filesystem(/nonexistent) returned No such file or directory (2)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL VFS
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/nonexistent) to (/nonexistent)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] export_commit_common :CONFIG :CRIT :Export id 0 can only export "/" not (/nonexistent)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): 1 validation errors in block FSAL
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:28): Errors processing block (FSAL)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): 1 validation errors in block EXPORT
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] config_errs_to_log :CONFIG :CRIT :Config File (/export/vfs.conf:12): Errors processing block (EXPORT)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/usr/local/var/run/ganesha) already exists
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22)
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
04/12/2020 17:46:48 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
04/12/2020 17:48:18 : epoch 5fca7608 : nfs-provisioner-7b87c6b986-j77q9 : nfs-ganesha-22[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE

Would appreciate any pointers

error while mounting - 1 validation errors in block EXPORT

PVC is not getting provisioned on some systems. It works on some and not on others. This is the error in the PVC

  Type     Reason              Age                    From                                                                                                                                           Message
  ----     ------              ----                   ----                                                                                                                                           -------
  Warning  ProvisioningFailed  6m4s                   mediation.apps.nfs.mycompany.nfs.provisioner.io_mediation-apps-nfs-mycompany-nfs-provisioner-66fd6b784-lp6dq_d33c2841-0caf-4b29-b087-61e3fbe9a24e  failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error getting NFS server IP for volume: service SERVICE_NAME=mediation-apps-nfs-mycompany-nfs-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.42.0.151
  Normal   Provisioning        2m19s (x5 over 6m4s)   mediation.apps.nfs.mycompany.nfs.provisioner.io_mediation-apps-nfs-mycompany-nfs-provisioner-66fd6b784-lp6dq_d33c2841-0caf-4b29-b087-61e3fbe9a24e  External provisioner is provisioning volume for claim "mycompany-cloud/mediation-apps-nfs"
  Warning  ProvisioningFailed  2m19s (x4 over 5m49s)  mediation.apps.nfs.mycompany.nfs.provisioner.io_mediation-apps-nfs-mycompany-nfs-provisioner-66fd6b784-lp6dq_d33c2841-0caf-4b29-b087-61e3fbe9a24e  failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error creating export for volume: error exporting export block
EXPORT
{
  Export_Id = 1;
  Path = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
  Pseudo = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
  Access_Type = RW;
  Squash = no_root_squash;
  SecType = sys;
  Filesystem_id = 1.1;
  FSAL {
    Name = VFS;
  }
}
: error calling org.ganesha.nfsd.exportmgr.AddExport: 0 export entries in /export/vfs.conf added because (export create, block validation) errors. Details:
Config File (/export/vfs.conf:54): 1 validation errors in block FSAL
Config File (/export/vfs.conf:54): Errors processing block (FSAL)
Config File (/export/vfs.conf:45): 1 validation errors in block EXPORT
Config File (/export/vfs.conf:45): Errors found in configuration block EXPORT
  Normal  ExternalProvisioning  6s (x26 over 6m18s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "mediation.apps.nfs.mycompany.nfs.provisioner.io" or manually created by system administrator

Abd this is the error in the nfs pod

I0408 10:20:25.839702       1 main.go:65] Provisioner mediation.apps.nfs.mycompany.nfs.provisioner.io specified
I0408 10:20:25.839822       1 main.go:89] Setting up NFS server!
I0408 10:20:26.019674       1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0408 10:20:26.019702       1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0408 10:20:26.023483       1 server.go:134] Running NFS server!
E0408 10:20:31.139572       1 controller.go:908] error syncing claim "32c863f3-a086-4cc1-9e19-f809f41a8bec": failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error getting NFS server IP for volume: service SERVICE_NAME=mediation-apps-nfs-mycompany-nfs-provisioner is not valid; check that it has for ports map[{111 TCP}:true {111 UDP}:true {662 TCP}:true {662 UDP}:true {875 TCP}:true {875 UDP}:true {2049 TCP}:true {2049 UDP}:true {20048 TCP}:true {20048 UDP}:true {32803 TCP}:true {32803 UDP}:true] exactly one endpoint, this pod's IP POD_IP=10.42.0.151
I0408 10:20:46.146203       1 provision.go:450] using service SERVICE_NAME=mediation-apps-nfs-mycompany-nfs-provisioner cluster IP 10.43.203.168 as NFS server IP
E0408 10:20:46.149486       1 controller.go:908] error syncing claim "32c863f3-a086-4cc1-9e19-f809f41a8bec": failed to provision volume with StorageClass "mediation-apps-nfs-mycompany-nfs-provisioner": error creating export for volume: error exporting export block
EXPORT
{
 Export_Id = 1;
 Path = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
 Pseudo = /export/pvc-32c863f3-a086-4cc1-9e19-f809f41a8bec;
 Access_Type = RW;
 Squash = no_root_squash;
 SecType = sys;
 Filesystem_id = 1.1;
 FSAL {
  Name = VFS;
 }
}
: error calling org.ganesha.nfsd.exportmgr.AddExport: 0 export entries in /export/vfs.conf added because (export create, block validation) errors. Details:
Config File (/export/vfs.conf:54): 1 validation errors in block FSAL
Config File (/export/vfs.conf:54): Errors processing block (FSAL)
Config File (/export/vfs.conf:45): 1 validation errors in block EXPORT
Config File (/export/vfs.conf:45): Errors found in configuration block EXPORT

v3.0.0

NFS share is not created when PVC and PV are not created via app/deploymentset.

I have deployed the NFS service and it works when volumes are automatically created with the volumeClaimTemplate. An NFS share is generated and all is fine. However, when I create a PVC and PV, it does not seem to generate the relative NFS share.

Storage class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mynfs-nfs
mountOptions:
- vers=4.1
provisioner: mynfs/nfs
reclaimPolicy: Delete
volumeBindingMode: Immediate

PV (create with and without provisioner_id, no change)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: storage
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 30Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc-storage
    namespace: shares
  nfs:
    path: /export/storage
    server: 10.158.1.1
  persistentVolumeReclaimPolicy: Retain
  storageClassName: mynfs-nfs
  volumeMode: Filesystem

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-storage
  namespace: shares
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: mynfs-nfs
  volumeMode: Filesystem
  volumeName: storage

I did notice that the EXPORT section is not present when getting the yaml details of the volume, and also vfs.conf is not updated.

*** EDIT ***

The path mentioned above, /export/storage is also not created. It seems the whole folder, and export initialization is skipped.

Use the nfs service DNS name instead of IP to specify the PV

Hello,

We've had a disaster recovery on our cluster and we had to re-provision the nfs server provisioner.

When we reprovisioned the server, it created a new service which got a new IP address, which made all the PVs invalid.
We restored the functionality by adding a new ad-hoc service in which we specified the service IP to correspond to the PVs ones. We now need this wart in the cluster.

I was wondering if we could use the nfs service name instead of the IP when provisioning the PVs ?

Right now, when the PV is created, the IP is placed:

Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    10.222.45.230
    Path:      /export/pvc-fd579ce4-6203-4eb8-a5eb-30436d8cf52f
    ReadOnly:  false

See here:

return service.Spec.ClusterIP, nil
where we return the service IP instead of the name.

I'm quite sure there's a good reason for doing it via IP, but this is limiting when there's a disaster recoveery. Any idea/comment ?

Support for xattr

I've been doing some testing and with NFSv4.2 pvs provisioned do not support xattr.

But they should by my understanding.

lgetxattr("/data/upper/.", "system.posix_acl_access", 0x196d920, 4096) = -1 EOPNOTSUPP (Operation not supported)
writev(6, [{iov_base="\20\0\0\0\241\377\377\377\22\0\0\0\0\0\0\0", iov_len=16}], 1) = 16
splice(6, NULL, 8, NULL, 135168, 0)     = 72

Any ideas?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.