openebs/mayastor

Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.

Welcome to OpenEBS

OpenEBS is a modern Block-Mode storage platform, a Hyper-Converged software Storage System and virtual NVMe-oF SAN (vSAN) Fabric that is natively integrates into the core of Kubernetes.

	Try our Slack channel If you have questions about using OpenEBS, please use the CNCF Kubernetes OpenEBS slack channel, it is open for anyone to ask a question

Important

OpenEBS provides...

• Stateful persistent Dynamically provisioned storage volumes for Kubernetes
• High Performance NVMe-oF & NVMe/RDMA storage transport optimized for All-Flash Solid State storage media
• Block devices, LVM, ZFS, ext2/ext3/ext4, XFS, BTRFS...and more
• 100% Cloud-Native K8s declarative storage platform
• A cluster-wide vSAN block-mode fabric that provides containers/Pods with HA resilient access to storage across the entire cluster.
• Node local K8s PV's and n-way Replciated K8s PV's
• Deployable On-premise & in-cloud: (AWS EC2/EKS, Google GCP/GKE, Azure VM/AKS, Oracle OCI, IBM/RedHat OpenShift, Civo Cloud, Hetzner Cloud... and more)
• Enterprise Grade data management capabilities such as snapshots, clones, replicated volumes, DiskGroups, Volume Groups, Aggregates, RAID
  

Type	Storage Engine	Type of data services	Status	In OSS ver
Replicated_PV		Replicated data volumes (in a Cluster wide vSAN block mode fabric)
	Mayastor	for High Availability deploymemnts distributing & replicating volumes across the cluster	Stable, deployable in PROD	v4.0.1
Local PV		Non-replicated node local data volumes	(Local-PV has multiple variants. See below)	v4.0.1
	Local PV HostPath	for integration with local node hostpath (e.g. /mnt/fs1)	Stable, deployable in PROD	v4.0.1
	Local PV ZFS	for integration with local ZFS storage deployments	Stable, deployable in PROD	v4.0.1
	Local PV LVM	for integration with local LVM2 storage deployments	Stable, deployable in PROD	v4.0.1
	Local PV Rawfile	for integration with Loop mounted Raw device-file filesystem	Stable, deployable in PROD, undergoing evaluation & integration release: v0.70	v4.0.1

STANDARD is optimized for NVMe and SSD Flash storage media, and integrates ultra modern cutting-edge high performance storage technologies at its core...

☑️   It uses the High performance SPDK storage stack - (SPDK is an open-source NVMe project initiated by INTEL)
☑️   The hyper modern IO_Uring Linux Kernel Async polling-mode I/O Interface - (fastest kernel I/O mode possible)
☑️   Native abilities for RDMA and Zero-Copy I/O
☑️   NVMe-oF TCP Block storage Hyper-converged data fabric
☑️   Block layer volume replication
☑️   Logical volumes and Diskpool based data managment
☑️   a Native high performance Blobstore
☑️   Native Block layer Thin provisioning
☑️   Native Block layer Snapshots and Clones

Get in touch with our team.

Vishnu Attur	@avishnu		Admin, Maintainer
Abhinandan Purkait	😎 @Abhinandan-Purkait		Maintainer
Niladri Halder	🚀 @niladrih		Maintainer
Ed Robinson	🐶 @edrob999		CNCF Primary Liason Special Maintainer
Tiago Castro	⚡ @tiagolobocastro		Admin, Maintainer
David Brace	⭐ @orville-wright		Admin, Maintainer

---

Activity dashbaord

Current status

Release	Support	Twitter/X	Contrib	License statue	CI Staus

---

Read this in 🇩🇪 🇷🇺 🇹🇷 🇺🇦 🇨🇳 🇫🇷 🇧🇷 🇪🇸 🇵🇱 🇰🇷 other languages.

Deployment

• In-cloud: (AWS EC2/EKS, Google GCP/GKE, Azure VM/AKS, Oracle OCI, IBM/RedHat OpenShift, Civo Cloud, Hetzner Cloud... and more)
• On-Premise: Bare Metal, Virtualzied Hypervisor infra using VMWare ESXi, KVM/QEMU (K8s KubeVirt), Proxmox
• Deployed as native K8s elemets: Deployments, Containers, Servcies, Stateful sets, CRD's, Sidecars, Jobs and Binaries all on K8s worker nodes.
• Runs 100% in K8s userspace. So it's highly portable and run across many OS's & platforms.

Roadmap (as of June 2024)

• Replciated PV Mayastor Roadmap

---

QUICKSTART : Installation

NOTE: Depending on which of the 5 storage engines you choose to deploy, pre-requests that must be met. See detailed quickstart docs...

1. Setup helm repository.

# helm repo add openebs https://openebs.github.io/openebs
# helm repo update

2a. Install the Full OpenEBS helm chart with default values.

• This installs ALL OpenEBS Storage Engines* in the openebs namespace and chart name as openebs:
  Local PV Hostpath, Local PV LVM, Local PV ZFS, Replicated Mayastor

# helm install openebs --namespace openebs openebs/openebs --create-namespace

2b. To Install just the OpenEBS Replicated Mayastor Storage Engine, use the following command:

# helm install openebs --namespace openebs openebs/openebs --set engines.replicated.mayastor.enabled=false --create-namespace

3. To view the chart

# helm ls -n openebs

Output:
NAME     NAMESPACE   REVISION  UPDATED                                   STATUS     CHART           APP VERSION
openebs  openebs     1         2024-03-25 09:13:00.903321318 +0000 UTC   deployed   openebs-4.0.1   4.0.1

4. Verify installation
   • List the pods in namespace
   • Verify StorageClasses

# kubectl get pods -n openebs

Example Ouput:
NAME                                              READY   STATUS    RESTARTS   AGE
openebs-agent-core-674f784df5-7szbm               2/2     Running   0          11m
openebs-agent-ha-node-nnkmv                       1/1     Running   0          11m
openebs-agent-ha-node-pvcrr                       1/1     Running   0          11m
openebs-agent-ha-node-rqkkk                       1/1     Running   0          11m
openebs-api-rest-79556897c8-b824j                 1/1     Running   0          11m
openebs-csi-controller-b5c47d49-5t5zd             6/6     Running   0          11m
openebs-csi-node-flq49                            2/2     Running   0          11m
openebs-csi-node-k8d7h                            2/2     Running   0          11m
openebs-csi-node-v7jfh                            2/2     Running   0          11m
openebs-etcd-0                                    1/1     Running   0          11m
openebs-etcd-1                                    1/1     Running   0          11m
openebs-etcd-2                                    1/1     Running   0          11m
...

# kubectl get sc

Example Output:
NAME                                              READY   STATUS    RESTARTS   AGE
openebs-localpv-provisioner-6ddf7c7978-jsstg      1/1     Running   0          3m9s
openebs-lvm-localpv-controller-7b6d6b4665-wfw64   5/5     Running   0          3m9s
openebs-lvm-localpv-node-62lnq                    2/2     Running   0          3m9s
openebs-lvm-localpv-node-lhndx                    2/2     Running   0          3m9s
openebs-lvm-localpv-node-tlcqv                    2/2     Running   0          3m9s
openebs-zfs-localpv-controller-f78f7467c-k7ldb    5/5     Running   0          3m9s
...

For more details, please refer to OpenEBS Documentation.

OpenEBS is a CNCF project and DataCore, Inc is a CNCF Silver member. DataCore support's CNCF extensively and has funded OpenEBS participating in every KubeCon event since 2020. Our project team is managed under the CNCF Storage Landscape and we contribute to the CNCF CSI and TAG Storage project initiatives. We proudly support CNCF Cloud Native Community Groups initiatives.

Project updates, subscribe to OpenEBS Announcements
Interacting with other OpenEBS users, subscribe to OpenEBS Users

Commercial Offerings

Commerically supported deployments of openEBS are avaialble via key companies. (Some provide services, funding, technology, infra, rescourced to the openEBS proejct).

• DataCore Software, Inc.
• Clouds Sky GmbH
• CodeWave
• Gridworkz Cloud Services

(openEBS OSS is a CNCF project. CNCF does not endorse any specific company).

mayastor's Issues

Assertion failure when unpublishing or destroying a nexus in faulted state

Describe the bug
Mayastor fails an assertion when attempting to destroy a nexus that is in the faulted state:

mayastor: subsystem.c:465: nvmf_subsystem_set_state: Assertion `actual_old_state == expected_old_state' failed.

To Reproduce
Create a nexus with 1 child being a remote replica served by an nvmf target, i.e.:

$ mayastor-client nexus list -c
NAME                                 PATH                                                                                        SIZE STATE  REBUILDS CHILDREN                                                                         
5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 nvmf://127.0.0.1:8430/nqn.2019-05.io.openebs:nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 60000000 online        0 nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391

Publish the nexus over nvmf, connect to it with the kernel NVMe initiator and send IO e.g. with fio.
Make the nvmf target serving the replica inaccessible. In this case, sending SIGSTOP to that mayastor instance.

The mayastor instance serving the nexus detects a timeout:

[2020-12-01T17:54:24.942056536+00:00  WARN mayastor::spdk:bdev_nvme.c:1149] Warning: Detected a timeout. ctrlr=0x557ec85b6d70 qpair=(nil) cid=2 

and eventually notices that a reset is required:

[2020-12-01T17:56:29.874237303+00:00 ERROR mayastor::spdk:bdev_nvme.c:1153] Controller Fatal Status, reset required   
[2020-12-01T17:56:34.872024397+00:00  WARN mayastor::spdk:bdev_nvme.c:1149] Warning: Detected a timeout. ctrlr=0x557ec85b6d70 qpair=(nil) cid=27   

At this point, send SIGCONT to the mayastor instance serving the replica.
Back at the other mayastor instance, the (only) child is faulted:

[2020-12-01T17:57:50.724579130+00:00 ERROR mayastor::bdev::nexus::nexus_io:nexus_io.rs:319] name: 127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391n1, driver: nvme, product: NVMe disk, num_blocks: 122880, block_len: 512
[2020-12-01T17:57:50.724733805+00:00  WARN mayastor::bdev::nexus::nexus_io:nexus_io.rs:329] core 0 thread Some(Mthread(0x557ec7c83090)), faulting child nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391: Faulted(IoError), blk_cnt: 122880, blk_size: 512

Network errors are logged:

[2020-12-01T17:57:50.825560183+00:00 ERROR mayastor::spdk:posix.c:171] getpeername() failed (errno=107)   
[2020-12-01T17:57:50.825668020+00:00 ERROR mayastor::spdk:tcp.c:958] spdk_sock_getaddr() failed of tqpair=0x557ec85d5fa0   

and the nexus is reconfigured to fault the child and remove it from the nexus:

[2020-12-01T17:57:50.844321020+00:00  INFO mayastor::bdev::nexus::nexus_bdev:nexus_bdev.rs:454] nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391: Dynamic reconfiguration event: ChildFault started
[2020-12-01T17:57:50.844403241+00:00  INFO mayastor::bdev::nexus::nexus_channel:nexus_channel.rs:102] nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391(thread:"mayastor_nvmf_tcp_pg_core_0"), refreshing IO channels
[2020-12-01T17:57:50.845618145+00:00  INFO mayastor::bdev::nexus::nexus_channel:nexus_channel.rs:244] nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391: Reconfigure completed
[2020-12-01T17:57:50.846884423+00:00  INFO mayastor::bdev::nexus::nexus_bdev:nexus_bdev.rs:468] nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391: Dynamic reconfiguration event: ChildFault completed 0

The child removal is complete:

[2020-12-01T17:57:50.847500441+00:00 ERROR mayastor::bdev::nexus::nexus_io:nexus_io.rs:344] :nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391: state: Mutex { data: Open } blk_cnt: 112607, blk_size: 512
        nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391: Faulted(IoError), blk_cnt: 122880, blk_size: 512
 has no children left... 
[2020-12-01T17:57:50.847578102+00:00  INFO mayastor::core::bdev:bdev.rs:168] Received remove event for bdev 127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391n1
[2020-12-01T17:57:50.847634549+00:00  INFO mayastor::bdev::nexus::nexus_child:nexus_child.rs:367] Removing child nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391
[2020-12-01T17:57:50.847766549+00:00  INFO mayastor::bdev::nexus::nexus_child:nexus_child.rs:405] Child nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 removed

However, the replica is still listed as a child in the nexus:

$ mayastor-client nexus list -c
NAME                                 PATH                                                                                        SIZE STATE   REBUILDS CHILDREN                                                                         
5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 nvmf://127.0.0.1:8430/nqn.2019-05.io.openebs:nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 60000000 faulted        0 nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391

but when you try to destroy the nexus mayastor fails an assertion:

mayastor: subsystem.c:465: nvmf_subsystem_set_state: Assertion `actual_old_state == expected_old_state' failed.

where the assertion is here.

Expected behavior
The nexus should be destroyed.

Screenshots
** OS info:**

• Distro: Ubuntu 20.04
• Kernel 5.4.0-56-generic
• Mayastor 0.6

Additional context
Developer debug build.

Be more explicit when we are running a CPUs older then 10 years

Describe the bug
Due to the instructions needed by DPDK, we only support icore7 and upwards. When we receive SIGILL kubectl logs is silent about it (as we crash hard) During startup, it would be nice if we were to detect this and crash more friendly in away.

To Reproduce
start mayastor on non supported CPU

Expected behavior
Nice log message describing the problem right now we need to look at exit code from k8s

set limit on core size

The new environment does not specifically set the core size. We should consider setting one to avoid getting no core dump when the size is set to zero.

Ability to auto-create Mayastor pools

Is your feature request related to a problem? Please describe.
I need a better way to automate the bringing up of Kubernetes Clusters with Mayastor configured. I use Cluster API to setup the Kubernetes cluster that takes care of setting up the nodes with the prerequisites to run Mayastor. As part of the Mayastor installation, labelling the nodes, I should be able to specify what devices should be used to create Mayastor pools automatically, instead of me having to add a node, label a node, fetch devices and create a MSP. I am looking to avoid the steps => fetch devices and create MSP.

Describe the solution you'd like
It should be as simple as labeling the node with devices to be used to create MSP.

Describe alternatives you've considered
Still exploring

Additional context
Creating this issue based on the discussion started in the Kubernetes Slack thread

Block Device On Minio

Is your feature request related to a problem? Please describe.
I currently am using local pvs for a few stateful containers and it is working well. two of which are cyrus imap and minio.

Describe the solution you'd like
Use a block device on top of minio for storing maildir email messages. I saw in the project description that this is a possible feature in the future.

Describe alternatives you've considered
I can use the clustering built into cyrus imap using local pvs as I am doing for other applications that provide clustering.

Additional context
This is more out of curiousity than anything else as I really like minio and would like to leverage it for email if possible.

Bringing down a node with nexus brings down the whole volume

It is a kind of known issue in mayastor that we plan to solve in long term by providing multiple instances of nexus for each volume running on different nodes. Then when one nexus goes down, nvme initiator in Linux kernel can switch to another nexus that is healthy. Though there is something more we could do in the meantime to recover a volume and although not as fast it could work for all protocols.

What happens if a node crashes or is willingly removed from the cluster is that k8s CSI calls unpublish on the volume and then it publishes the volume on a different node. We could recreate the nexus in control plane if we get a publish request for a volume with missing nexus (currently we fail such request) and that would make the volume available again. That is relatively a small change. We could take that even further and create nexus only if the volume is published. That has been proposed in the past and we should evaluate how much work that would be to implement and what are the cons (if any).

Note: This is a pure control plane issue that does not affect the data plane (mayastor).

container security scanner marks the mayastor images as unsupported.

Describe the bug

Both trivy based container scanning and quay image scanner report the mayastor container images as unsupported.

To Reproduce

The quay.io status can be seen here: https://quay.io/repository/openebs/moac?tab=tags

To test with trivy, you can use the following steps.

• Download Trivy

• Download the mayastor container image

• Scan the image using Trivy as follows:
  $ ./trivy --exit-code 1 --severity CRITICAL --no-progress openebs/moac:0.1.0

  The output looks like:

  2020-05-16T02:20:51.566Z        FATAL   error in image scan: scan failed: failed to apply layers: unknown OS
  

Expected behavior
Scanning should complete without errors.

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: [e.g. NixOS]
• Kernel version
• MayaStor revision or container image: 0.1.0 version for all mayastor container images.

Additional context
Though nix images are secure, some platforms like AWS Marketplace, RH expect the containers to pass their certification test. Raising this issue to help verify if images can be uploaded to AWS Marketplace for running with EKS platforms. ( cc: @muratkars )

Also may need to investigate if an issue needs to raised against https://github.com/aquasecurity/trivy to allow trivy to pass the nix based container images.

[Doc Source]: namespace yaml is in JSON format and incorrect value of "minimum" in mayastorpoolcrd.yaml

Are you reporting an issue with existing content?

Basically, namespace.yaml is in JSON format in YAML tab
Link of the affected page: namespace

In mayastorpoolcrd.yaml , at line number 78, expected value of minimum is int 0 but mention string are in YAML tab and giving error when user apply mayastorpoolcrd.yaml.
Link of the affected page: mayastorpoolcrd

In Control Plane section, .yaml is missing from kubectl apply -f moac-deployment in Command (local repo) tab
Link of the affected page: moac-deployment

Are you proposing new content, or a change to the existing documentation layout or structure?
Please describe your proposal.

Create mayastor-client docker image

It would be helpful to have an image on DockerHub with mayastor-client in it. Typical use case is to be able to quickly run in within existing k8s cluster with something like

kubectl run debian --rm -i --tty --image mayastor-client -- mayastor-client

Reuse pool with faulty replica

MOAC will try to create a new replica in case of one replica failing. It needs to have sufficient pool available. This may not be the case, when number of replicas corresponds with number of storage nodes. In such cases, it would make sense to remove the faulted replica from the nexus and join it again to kick off the rebuild (while staying on the same pool).

[introduction/quickstart/preparing-the-cluster]: [Further detail for hugepage allocation]

Are you reporting an issue with existing content?

A warning should be given to users that 512x2MB hugepages may not be enough, as it appears that other software will use hugepages if they are available but don't need them. This can be surprising for users where hugepages are not used, as they naturally assume that adding 512 will satisfy the 512 that are required by mayastor. When other processes "steal" hugepages, this means that upon reboot, there will not be enough hugepages for the MSN, and MSN resources won't be created by the daemonset nodes.

https://mayastor.gitbook.io/introduction/quickstart/preparing-the-cluster

Are you proposing new content, or a change to the existing documentation layout or structure?

Add a warning that users should make sure to verify that the hugepages are not being used by other, opportunistic processes and, if possible, to provision more than 512 pages, or at least to check that after a reboot there continues to be at least 512 hugepages that are still available in HugePages_Free.

Basically the examples should NOT be with:

vm.nr_hugepages = 512

They should be something like

vm.nr_hugepages = 1024

Or possibly

vm.nr_hugepages = 512 # PLUS whatever is taken after a reboot, PLUS a margin for other applications

Another option could be to have the daemonset pods error out, though I have no idea whether there is a usecase for these if the node doesn't have hugepages.

Nexus statics could be combined into one struct

When creating the nexus driver we setup up function tables etc such we can create instances of a nexus. We use lazy_static!() to create global shared values for the function table but also for the module itself. We should combine this into one single struct

Create a unit test for resetting of lvol content

When a new lvol is created we should make sure that any previous content is gone by writing zeros to it (could be unmap on devices which support it). Not doing so can result in issues with using stale filesystem which existed on a previous lvol and is a security problem in general. We should have a unit test which verifies that.

[Doc Source]: Typo in "Deploy the FIO Test Pod" section

Are you reporting an issue with existing content?
There is typo in kubectl command for Deploy the FIO Test Pod section, kubect is present in place of kubectl at two occurrences.

Existing : kubect apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/fio.yam
Expected: kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/fio.yam

Existing: kubect get pod fio
Expected: kubectl get pod fio

Link of the affected page(s): deploy-the-fio-test-pod

Are you proposing new content, or a change to the existing documentation layout or structure?
Please describe your proposal.

Colour escape sequences in mayastor k8s log files

After we changed mayastor logger module, colour escape sequences started to appear in k8s log files. Instead, the logger should be smart enough to not generate these when writing to anything else than terminal output as it was before.

Failed to stage volume: attach failed: Invalid argument (os error 22)

Describe the bug
Was trying the mayastor and run dbench on a mayastor nvme volume. The dbench pod can not start due to pvc mount error. The k8s event is MountVolume.MountDevice failed for volume "pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d" : rpc error: code = Internal desc = Failed to stage volume ac2b83cd-3489-43fe-80c0-697694a7a72d: attach failed: Invalid argument (os error 22).

root@testr01n01:~/stone# kubectl get pod
NAME                         READY   STATUS              RESTARTS   AGE
dbench-fbrnv                 0/1     Completed           0          15h
dbench-mayastor-nvmf-xdjx5   0/1     ContainerCreating   0          23s
root@testr01n01:~/stone# kubectl describe pod dbench-mayastor-nvmf-xdjx5
Name:           dbench-mayastor-nvmf-xdjx5
Namespace:      default
Priority:       0
Node:           testr01n01/192.168.101.174
Start Time:     Tue, 20 Oct 2020 09:47:28 +0800
Labels:         controller-uid=6d363037-5b85-4bef-b046-0d1457708d88
                job-name=dbench-mayastor-nvmf
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  Job/dbench-mayastor-nvmf
Containers:
  dbench:
    Container ID:
    Image:          fbuchmeier/dbench:latest
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      DBENCH_MOUNTPOINT:  /data
    Mounts:
      /data from dbench-pv (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gjxb5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  dbench-pv:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  dbench-pv-claim-mayastor-nvmf
    ReadOnly:   false
  default-token-gjxb5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-gjxb5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Warning  FailedScheduling        37s               default-scheduler        error while running "VolumeBinding" filter plugin for pod "dbench-mayastor-nvmf-xdjx5": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               35s               default-scheduler        Successfully assigned default/dbench-mayastor-nvmf-xdjx5 to testr01n01
  Normal   SuccessfulAttachVolume  35s               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d"
  Warning  FailedMount             3s (x6 over 19s)  kubelet, testr01n01      MountVolume.MountDevice failed for volume "pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d" : rpc error: code = Internal desc = Failed to stage volume ac2b83cd-3489-43fe-80c0-697694a7a72d: attach failed: Invalid argument (os error 22)

To Reproduce
Deploy mayastor following https://mayastor.gitbook.io/introduction/, create a mayastor pool, create storage class mayastor-nvmf, specify mayastor-nvmf in dbench.yaml, then kubectl apply -f dbench.yaml.

Expected behavior
Volume can be attached/mounted to dbench pod.

Screenshots
If applicable, add screenshots to help explain your problem.

** OS info (please complete the following information):**

• Distro: [e.g. NixOS]

root@testr01n01:~/stone# kubectl get node -o wide
NAME         STATUS   ROLES           AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
testr01n01   Ready    master,worker   9d    v1.17.6   192.168.101.174   <none>        Ubuntu 16.04.5 LTS   4.15.0-58-generic   docker://18.9.3
testr01n02   Ready    master,worker   9d    v1.17.6   192.168.101.175   <none>        Ubuntu 16.04.5 LTS   4.15.0-58-generic   docker://19.3.8
testr01n03   Ready    master,worker   9d    v1.17.6   192.168.101.176   <none>        Ubuntu 16.04.5 LTS   4.15.0-58-generic   docker://19.3.8

• Kernel version
  4.15.0-58-generic

• MayaStor revision or container image
  Tried both below on master branch and version v0.5.0. Same error.

root@testr01n01:~/stone/Mayastor# git log HEAD
commit 6fb4926fae0704791c33f9a685b2a9ab507a8647
Merge: 724ff0e 994acff
Author: mayastor-bors <[email protected]>
Date:   Fri Oct 16 17:16:55 2020 +0000

Additional context

root@testr01n01:~# kubectl get msp -n mayastor
NAME             NODE         STATE    AGE
pool-on-node-1   testr01n01   online   19h
pool-on-node-2   testr01n02   online   19h
pool-on-node-3   testr01n03   online   19h
root@testr01n01:~# kubectl -n mayastor get msp pool-on-node-1 -o yaml
apiVersion: openebs.io/v1alpha1
kind: MayastorPool
metadata:
  creationTimestamp: "2020-10-19T06:44:21Z"
  finalizers:
  - finalizer.mayastor.openebs.io
  generation: 1
  name: pool-on-node-1
  namespace: mayastor
  resourceVersion: "3971074"
  selfLink: /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorpools/pool-on-node-1
  uid: fef9a86e-c05d-4315-848d-867b7015ecd0
spec:
  disks:
  - /dev/nvme3n1
  node: testr01n01
status:
  capacity: 3199238930432
  disks:
  - aio:///dev/nvme3n1?uuid=1157d4e0-cb62-4538-b85a-2ab9dc2c99ae
  reason: ""
  state: online
  used: 1075889307648
root@testr01n01:~# kubectl -n mayastor get msp pool-on-node-2 -o yaml
apiVersion: openebs.io/v1alpha1
kind: MayastorPool
metadata:
  creationTimestamp: "2020-10-19T06:44:55Z"
  finalizers:
  - finalizer.mayastor.openebs.io
  generation: 1
  name: pool-on-node-2
  namespace: mayastor
  resourceVersion: "3971005"
  selfLink: /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorpools/pool-on-node-2
  uid: c8cb69f9-e706-4556-89fb-8ad441cceca9
spec:
  disks:
  - /dev/nvme3n1
  node: testr01n02
status:
  capacity: 3199238930432
  disks:
  - aio:///dev/nvme3n1?uuid=32841a70-d690-4c7f-8585-3913fa17b198
  reason: ""
  state: online
  used: 1074815565824
root@testr01n01:~# kubectl -n mayastor get msp pool-on-node-3 -o yaml
apiVersion: openebs.io/v1alpha1
kind: MayastorPool
metadata:
  creationTimestamp: "2020-10-19T06:45:09Z"
  finalizers:
  - finalizer.mayastor.openebs.io
  generation: 1
  name: pool-on-node-3
  namespace: mayastor
  resourceVersion: "3970820"
  selfLink: /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorpools/pool-on-node-3
  uid: 29600bd5-e303-43e9-8854-e5658b622401
spec:
  disks:
  - /dev/nvme3n1
  node: testr01n03
status:
  capacity: 3199238930432
  disks:
  - aio:///dev/nvme3n1?uuid=5d2753f7-bc3b-4b3a-88cc-62846e9a2679
  reason: ""
  state: online
  used: 1074815565824
root@testr01n01:~# kubectl get msv -n mayastor
NAME                                   NODE         SIZE            STATE     AGE
361fde22-1d67-4b36-bce9-84ffb902517e   testr01n01   1073741824      healthy   18h
8f36f936-b87e-47af-af80-eb16d1369196   testr01n01   1073741824      healthy   19h
ac2b83cd-3489-43fe-80c0-697694a7a72d   testr01n02   1073741824000   healthy   33m
root@testr01n01:~# kubectl get sc mayastor-nvmf -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubesphere.io/support-snapshot: "false"
  creationTimestamp: "2020-10-19T06:45:52Z"
  name: mayastor-nvmf
  resourceVersion: "3552972"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/mayastor-nvmf
  uid: 1b62348b-435a-4100-af25-dbbcf220d2c1
parameters:
  protocol: nvmf
  repl: "3"
provisioner: io.openebs.csi-mayastor
reclaimPolicy: Delete
volumeBindingMode: Immediate
root@testr01n01:~# kubectl get pvc
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS             AGE
dbench-pv-claim                 Bound    pvc-46ea9c4f-50d4-457a-8c37-583b4940d525   1000Gi     RWO            openebs-sc-statefulset   16h
dbench-pv-claim-mayastor-nvmf   Bound    pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d   1000Gi     RWO            mayastor-nvmf            38m
ms-volume-claim-iscsi           Bound    pvc-361fde22-1d67-4b36-bce9-84ffb902517e   1Gi        RWO            mayastor-iscsi           18h
root@testr01n01:~# kubectl get pvc dbench-pv-claim-mayastor-nvmf -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"dbench-pv-claim-mayastor-nvmf","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1000Gi"}},"storageClassName":"mayastor-nvmf"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: io.openebs.csi-mayastor
  creationTimestamp: "2020-10-20T01:47:26Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: dbench-pv-claim-mayastor-nvmf
  namespace: default
  resourceVersion: "3970747"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/dbench-pv-claim-mayastor-nvmf
  uid: ac2b83cd-3489-43fe-80c0-697694a7a72d
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1000Gi
  storageClassName: mayastor-nvmf
  volumeMode: Filesystem
  volumeName: pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1000Gi
  phase: Bound
root@testr01n01:~# kubectl get pv pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: io.openebs.csi-mayastor
  creationTimestamp: "2020-10-20T01:47:27Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/io-openebs-csi-mayastor
  name: pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d
  resourceVersion: "3970758"
  selfLink: /api/v1/persistentvolumes/pvc-ac2b83cd-3489-43fe-80c0-697694a7a72d
  uid: ffbd7090-7fb0-482d-8038-7d6e33d09143
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1000Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: dbench-pv-claim-mayastor-nvmf
    namespace: default
    resourceVersion: "3970721"
    uid: ac2b83cd-3489-43fe-80c0-697694a7a72d
  csi:
    driver: io.openebs.csi-mayastor
    fsType: ext4
    volumeAttributes:
      protocol: nvmf
      repl: "3"
      storage.kubernetes.io/csiProvisionerIdentity: 1603088996571-8081-io.openebs.csi-mayastor
    volumeHandle: ac2b83cd-3489-43fe-80c0-697694a7a72d
  persistentVolumeReclaimPolicy: Delete
  storageClassName: mayastor-nvmf
  volumeMode: Filesystem
status:
  phase: Bound

nexus child remains degraded even though that the node is gone

This happens with 0.6 images. We have nvmf-shared volume with 3 replicas. A node with one of the replicas is taken offline (most likely not a proper shutdown but rather stops responding) and the volume does not maintain the desired redundancy. We still see a volume that has 2 replicas in the online state and one of them degraded (the one that is gone). The control plane (moac) thinks that there is a rebuild in progress and doesn't do anything in that case (we can argue if that is the best thing to do because it also knows that the replica is offline). In any case, it should be the nexus that eventually should change the state of the child to faulted and that would trigger the replacement of the faulted replica. But that does never happen. Mayastor logfile from the node with nexus is full of errors (goes on and on ...):

[2020-12-08T14:43:13.134045710+00:00 ERROR mayastor::spdk:nvme_tcp.c:1742] sock connection error of tqpair=0x55efcdd5c7c0 with addr=10.12.0.131, port=8420   
[2020-12-08T14:43:13.134050038+00:00 ERROR mayastor::spdk:nvme_ctrlr.c:1409] Controller reinitialization failed.   
[2020-12-08T14:43:13.134053836+00:00 ERROR mayastor::spdk:nvme_ctrlr.c:924] ctrlr 10.12.0.131 in failed state.   
[2020-12-08T14:43:13.134057352+00:00 ERROR mayastor::spdk:bdev_nvme.c:336] Resetting controller failed.   
[2020-12-08T14:43:13.134112377+00:00 ERROR mayastor::spdk:posix.c:536] connect() failed, errno = 111   

If this error repeats a million times as in this case that also signifies that something went wrong. Log files of mayastor and moac are attached.

dmesg.txt
moac.txt
mayastor_logs.txt

Implement hold on channels

When we online a child we need a reference to the IO channels. Right now this is implemented as:

let ch = get_io_channel()
... code
put_io_channel(ch)

It would be nice if we can do a simple self.channel.hold() or something like that

Potential shadow file exposed

Hello Team,

While checking I came across a potential shadow file that is exposed to this repository. Please check on the below path and restrict the file according to its criticality.

Path to file: https://github.com/openebs/Mayastor/blob/master/nix/pkgs/ms-buildenv/root/etc/shadow

Attaching screenshot of the same.

set logfn in bindings to write to rust logging infrastructure

We currently write our logs to spdk_logger which, we want to flip that i.e spdk_logger should write to a rust logger.

This can be done by overriding the (global) variable g_log.

Environment init thread should lazy_static

The init thread (mm_thread) is currently thread-local. This works fine in the general case; however, when implementing tests cases, threads are spawned by the test framework, which results in the local variable being absent.

Bringing down one of the replicas renders the volume unusable

It is a combination of control & data plane issues that results in a sad moment of volume being unavailable although that just one replica out of N is missing. Test environment is a 3 node cluster with 3 replicas of the same volume. We deliberately terminate one of the nodes that runs the application and one of the replicas (not a nexus - that's a different issue #508). So what happens then:

Following message appears cca 20x in mayastor log file (instance with the nexus):

2020-11-12T15:46:05.443326042+00:00  WARN mayastor::logger: Warning: Detected a timeout. ctrlr=0x55621430eca0 qpair=(nil) cid=2

then

2020-11-12T15:47:05.445682886+00:00 ERROR mayastor::logger: Controller Fatal Status, reset required
2020-11-12T16:01:40.388109878+00:00 ERROR mayastor::logger: CQ error, abort requests after transport retry counter exceeded
2020-11-12T16:01:40.388162053+00:00 ERROR mayastor::logger: ctrlr 172.20.52.199 in failed state.
2020-11-12T16:01:42.388601297+00:00 ERROR mayastor::logger: Failed to construct the tqpair=0x5562143105e0 via correct icresp
2020-11-12T16:01:42.388646242+00:00 ERROR mayastor::logger: Unable to connect the tqpair
2020-11-12T16:01:42.388710242+00:00 ERROR mayastor::logger: Controller reinitialization failed.
2020-11-12T16:01:42.388721983+00:00 ERROR mayastor::logger: ctrlr 172.20.37.187 in failed state.
2020-11-12T16:01:42.388729381+00:00 ERROR mayastor::logger: Resetting controller failed.
2020-11-12T16:01:42.395028594+00:00 ERROR mayastor::logger: getpeername() failed (errno=107)
2020-11-12T16:01:42.395066223+00:00 ERROR mayastor::logger: spdk_sock_getaddr() failed
2020-11-12T16:01:42.395088161+00:00 ERROR mayastor::logger: spdk_iscsi_connection_construct() failed
2020-11-12T16:01:42.395120389+00:00 ERROR mayastor::logger: Submitting Keep Alive failed
2020-11-12T16:01:43.959954068+00:00 ERROR mayastor::logger: Connect command failed
2020-11-12T16:01:43.960000578+00:00 ERROR mayastor::logger: Failed to send an NVMe-oF Fabric CONNECT command

repeated many times. And then a panic at the end:

2020-11-12T16:03:16.119653376+00:00 ERROR mayastor::logger: Failed to send an NVMe-oF Fabric CONNECT command
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: GetIoChannel { name: "172.20.37.187:8420/nqn.2019-05.io.openebs:c8059146-35e1-480e-a983-2d69a0ad1d79n1" }', mayastor/src/bdev/nexus/nexus_channel.rs:155:71
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2020-11-12T16:03:16.119742718+00:00 ERROR mayastor::logger: Controller reinitialization failed.
2020-11-12T16:03:16.119793063+00:00 ERROR mayastor::logger: ctrlr 172.20.37.187 in failed state.
2020-11-12T16:03:16.119814482+00:00 ERROR mayastor::logger: Resetting controller failed.
2020-11-12T16:03:16.120056430+00:00 ERROR mayastor::logger: get_cc failed

What does control plane in the meantime?

Nov 12 15:56:45.166 debug [csi]: Request to unpublish volume "fc60eaf5-b5c7-4031-815f-094241595536"
Nov 12 15:56:45.171 debug [nexus]: Unpublishing nexus "fc60eaf5-b5c7-4031-815f-094241595536@ip-172-20-37-
187.eu-central-1.compute.internal" ...

k8s tries to unpublish the volume so that it can move the application to a different node. But mayastor node fails to reply and moac waits forever (that is what #506 is about). And because this fails the application cannot be scheduled and is stuck. Nothing else can use the volume. Control plane side of the things will be fixed on behalf of #506. This ticket exists to track the problem in data plane (mayastor) and its inadequate reaction to the situation when one of the replicas becomes unreachable (should not crash).

Verify why we are missing some constants (defines) in the sys crate

For some reason we are missing a #define for max spdk_sys::NVMF_TGT_NAME_MAX_LENGTH

MSP replicas are not cleared as expected

Describe the bug

When deleting a replica the data is not removed as expected.

To Reproduce

1. create a replica, format it with your favourite FS, delete the replica.
2. create a new replica, observe the old replica is returned

Expected behaviour

Replicas should be zeroed out

Nexus driver should write out an MBR when labeling

When we write the GPT labels on a new device, we do not write out an MBR. Although we don't need it, we should write it out to avoid warnings by tools that expect it to be there.

Do not expose CSI grpc services on TCP port

CSI methods are called from k8s sidecars using unix domain socket file. Exposing CSI services on TCP port is useless and not safe without implementing authentication. It is believed that it is required for running the mayastor tests. Remove the CSI services from TCP port and make necessary changes to the test suite (if needed).

Different block size of nexus and child dev should not fail the nexus create request

We tripped over the following problem:

nexus_child.rs: 103:: *ERROR*: nexus-32b154c1-6bf9-4021-a3e1-bb2d09df407e: Invalid block size for bdev:///32b154c1-6bf9-4021-a3e1-bb2d09df407e, wanted: 4096 got 512
jsonrpc.rs: 185:: *ERROR*: failed to create nexus

it is suboptimal if the block sizes are different but perhaps not fatal. Explore and test if the check can be removed and replaced by a log message.

Some threads are polled too often

When polling threads, the last argument can be as a timestamp. This value, however, is always set to 0 leading excessive calls to get_ticks(). For each loop, we should determine the TSC once and use it to poll all threads.

SIGBUS access to undefined memory when running rust tests

Jenkins CI/CD nightly job reported a new problem (https://mayastor-ci.mayadata.io/job/Mayastor/job/develop/23/):

Caused by:
  process didn't exit successfully: `sudo -E /var/lib/jenkins/workspace/Mayastor_staging/target/debug/deps/add_child-58a119cbacf304d5 --test-threads=1` (signal: 7, SIGBUS: access to undefined memory)

It was reported by one more colleague when running the tests manually. So it is a real issue that we should try to reproduce, get coredump and see if we can get any kind of interesting info out of it.

It could be related to recent change of removing nvmf_event library from SPDK, but could be any other change that went into the repo recently as well.

Reserve space for metadata

The nexus driver should, implicitly, reserve disk space for metadata. The metadata is used during the rebuild process.

Openshift / OKD support (RHCOS/FCOS)

Feature request
Additional docs for deployment on Openshift / OKD, CoreOS machines.

Describe the solution
If you create scc's for moac/moac-csi service accounts, mayastor can run in openshift; example - http://paste.openstack.org/show/798355/

P.S. It is worth to add in the docs that if you run k8s cluster on virtual machines; those need to support "+ssse3,+sse4.1,+sse4.2" cpu modules.

Segfault after a remote replica comes back online after being offline

Describe the bug
This is similar to #549 except that the NVMf target serving the remote replica is brought back online before the NVMf initiator in the nexus decides to reset the connection and this results in the latter process terminating after a segfault.

To Reproduce
Create a nexus with 1 child being a remote replica served by an nvmf target, i.e.:

$ mayastor-client nexus list -c
NAME                                 PATH                                                                                        SIZE STATE  REBUILDS CHILDREN                                                                         
5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 nvmf://127.0.0.1:8430/nqn.2019-05.io.openebs:nexus-5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391 60000000 online        0 nvmf://127.0.0.1:8420/nqn.2019-05.io.openebs:5b5b04ea-c1e3-11ea-bd82-a7d5cb04b391

Publish the nexus over nvmf, connect to it with the kernel NVMe initiator and send IO e.g. with fio.
Make the nvmf target serving the replica inaccessible. In this case, sending SIGSTOP to that mayastor instance.

The mayastor instance serving the nexus detects a timeout:

[2020-12-03T15:19:30.851447209+00:00  WARN mayastor::spdk:bdev_nvme.c:1149] Warning: Detected a timeout. ctrlr=0x555556765550 qpair=(nil) cid=0   
[2020-12-03T15:19:35.848760271+00:00  WARN mayastor::spdk:bdev_nvme.c:1149] Warning: Detected a timeout. ctrlr=0x555556765550 qpair=(nil) cid=2   

Before it logs an error with Controller Fatal Status, reset required (typically after cid reaches 26 or so), send SIGCONT to the mayastor instance serving the replica.
Back at the other mayastor instance, what happens depends on how mayastor was built. In a release build, it simply segfaults:

Segmentation fault

and terminates. Running that through gdb:

Thread 1 "mayastor" received signal SIGSEGV, Segmentation fault.
0x00007ffff649b212 in nvme_request_check_timeout (req=0x0, cid=9, 
    active_proc=active_proc@entry=0x200024c0d0c0, 
    now_tick=now_tick@entry=66222996158570) at nvme.c:479
479             if (req->timed_out || req->submit_tick == 0) {
(gdb) bt
#0  0x00007ffff649b212 in nvme_request_check_timeout (req=0x0, cid=9, 
    active_proc=active_proc@entry=0x200024c0d0c0, 
    now_tick=now_tick@entry=66222996158570) at nvme.c:479
#1  0x00007ffff64a1d51 in nvme_tcp_qpair_check_timeout (qpair=0x555556766e90)
    at nvme_tcp.c:1554
#2  nvme_tcp_qpair_process_completions (qpair=0x555556766e90, 
    max_completions=32) at nvme_tcp.c:1597
#3  0x00007ffff649a401 in spdk_nvme_qpair_process_completions (
    qpair=0x555556766e90, max_completions=max_completions@entry=0)
    at nvme_qpair.c:714
#4  0x00007ffff648bfe2 in spdk_nvme_ctrlr_process_admin_completions (
    ctrlr=0x555556765550) at nvme_ctrlr.c:3425
#5  0x00007ffff643a671 in bdev_nvme_poll_adminq (arg=0x5555563ba4c0)
    at bdev_nvme.c:275
#6  0x00007ffff6471629 in thread_poll (now=66222996154691, 
    max_msgs=<optimized out>, thread=0x555555e31240) at thread.c:644
#7  spdk_thread_poll (thread=0x555555e31240, max_msgs=<optimized out>, 
    now=66222996154691) at thread.c:732
#8  0x0000555555913d8b in <&mayastor::core::reactor::Reactor as core::future::future::Future>::poll ()
#9  0x000055555561476c in <futures_util::future::try_join_all::TryJoinAll<F> as core::future::future::Future>::poll ()
#10 0x000055555573643d in mayastor::main ()
#11 0x0000555555670693 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#12 0x0000555555738e46 in main ()

In a debug build, this assertion fails:

mayastor: nvme_tcp.c:1552: nvme_tcp_qpair_check_timeout: Assertion `tcp_req->req != NULL' failed.

which is here.

Expected behavior
Mayastor should continue running.

OS info

• Distro: Ubuntu 20.04
• Kernel 5.4.0-56-generic
• Mayastor 0.6

User-created Mayastor Pool CRDs have missing "STATE" information

Describe the bug
When Mayastor Pool CRDs are created by the user, the corresponding pools are not actually created by Mayastor.

The output of kubectl -n mayastor get msp doesn't report a value for the STATE field of the affected pool(s)
e.g.

NAME         NODE                                STATE   AGE
diskpool-1   aks-agentpool-41030923-vmss000003           46m
diskpool-2   aks-agentpool-41030923-vmss000004           46m
diskpool-3   aks-agentpool-41030923-vmss000005           46m

The status section of any affected MSP CRD object is missing:

kubectl -n mayastor describe msp diskpool-1

Name:         diskpool-1
Namespace:    mayastor
Labels:       <none>
Annotations:  <none>
API Version:  openebs.io/v1alpha1
Kind:         MayastorPool
Metadata:
  Creation Timestamp:  2020-09-16T15:35:55Z
  Generation:          1
  Resource Version:    13783
  Self Link:           /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorpools/diskpool-1
  UID:                 ba8adfd2-58a7-4fef-bd7f-0a24510157b1
Spec:
  Disks:
    /dev/sdc
  Node:  aks-agentpool-41030923-vmss000003

When this occurs, examination of the logs from the moac container and the associated mayastor container(s) will show that no messages regarding pool operations are logged - it appears as if MOAC has simply not seen, or ignored, the new pool CRD(s).

If the MOAC container is restarted (e.g. scale the moac deployment to 0 replicas and back to 1), the affected pool(s) are created and the corresponding MSP CRDs then present as expected (with pool State=online and their status section present).

To Reproduce

This behaviour is intermittent When it does manifest, it is necessary only to kubectl create a valid MSP CRD is order to observe the aberrant response (i.e. pool not created and MSP CRD lacking the Status section).

Expected behavior

When a valid MSP CRD is created, the corresponding pool should be created by MOAC/Mayastor and report its state as online.

OS info:

• Distro: Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-1092-azure x86_64)
• Kernel version: 4.15.0-1092-azure
• MayaStor revision or container image: Mayastor v0.3.0, Mayastor RC v0.4.0

Additional context
Add any other context about the problem here.

LOGS
moac_container_log.txt
mayastor_container_log.txt

New volume creation stuck on a lost reply for unrelated previous request

There was a case when for unknown reason mayastor failed to send a reply for nexus unpublish request. moac is strictly serializing grpc requests to a storage node so all future requests get stuck as a result of that.

We don't want to overwhelm storage node with a number of requests that in the worse case could be related. So keep the current behavior but would be nice to have some kind of a timeout after which we continue with other requests.

Serialize volume creation requests in control plane

When investigating issue #474 I have found out that moac tries to create multiple volumes at once. That raises at least one concern: how it can keep accounting accurate if the free space on a pool changes while it tries to create a different volume. Instead of trying to determine what everything can go wrong while creating volumes in parallel, I suggest we stay on the safe side and serialize all requests for creating new volumes. That will also put us in a better position when detecting retransmits of the same create-volume request that happens frequently with k8s.

If it makes sense to detect retransmits for other operations as well (namely publish, unpublish and destroy) and avoid retransmitting those to mayastor storage nodes remains to be answered. It is an optional extension of a fix for this ticket.

Incoming queue polled to often

While running performance benchmarks, it observed that for each loop, we poll the reactor(s) for incoming threads. The queue on which we insert new threads is atomic which is rather expensive and could be polled less frequently.

Instead of doing while let Ok(i) = self.incomming.pop() every iteration it is better to set a flag "incoming" to true when we schedule a new thread. When this flag is set, we should while loop over the queue.

Destroying a pool created via yaml config results in "thread 'main' panicked"

Describe the bug
Destroying a pool created via yaml config results in "thread 'main' panicked" though the Mayastor process remains running and usable.

To Reproduce
Steps to reproduce the behavior:

1. Create a file as a backing store for the pool:

truncate -s 128M /tmp/pool0.img

2. Create a Mayastor config and save it as pool.yaml:

---
pools:
  - name: pool0
    disks:
      - /tmp/pool0.img
    blk_size: 512
    io_if: 0
    replicas:
      - name: 00000000-1234-abcd-cdef-012345678901
        share: ~
implicit_share_base: true
sync_disable: true

3. Start Mayastor:

# mayastor -g 127.0.0.1:10124 -y pool.yaml

4. Verify the pool exists, then delete it:

$ mayastor-client pool list
NAME  STATE   CAPACITY USED DISKS
pool0 online 130023424    0      

$ mayastor-client pool destroy pool0
Error: Status { code: Unknown, message: "transport error: http2 error: protocol error: stream no longer needed" }

5. Note the panic in the Mayastor log:

2020-10-02T16:33:54.457261484+01:00  INFO mayastor::subsys::config: creating pool pool0
2020-10-02T16:33:54.457429219+01:00  INFO mayastor::pool: aio bdev /tmp/pool0.img was created
2020-10-02T16:33:54.459128346+01:00  INFO mayastor::pool: The pool pool0 has been created
2020-10-02T16:33:54.459237551+01:00  INFO mayastor::subsys::config: creating nexus devices
2020-10-02T16:33:54.459563656+01:00  INFO mayastor::grpc::server: gRPC server configured at address 127.0.0.1:10124
2020-10-02T16:33:54.459726049+01:00  INFO mayastor::core::reactor: core 0 set to developer delayed poll mode
2020-10-02T16:33:54.460933198+01:00  INFO mayastor: Mayastor started 🚀 ...
2020-10-02T16:33:59.027100971+01:00  INFO mayastor::lvs::lvs_pool: pool pool0 destroyed successfully
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', mayastor/src/lvs/lvs_pool.rs:491:44
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

At this point Mayastor is still usable and you can still create and delete pools.

Expected behavior
Not panic when deleting a pool.

OS info

• Distro: Ubuntu 20.04
• Kernel version 5.4.0-42-generic
• MayaStor develop branch commit ce9246e

Additional context
If the pool was created was creating using gRPC, the panic does not occur as that uses the new pool API at mayastor::lvs::lvs_pool instead of mayastor::pool used by the yaml config import code.

Replica destruction may take a while depending on load and scheme

While investigating issue #474 we have found that destruction of a relatively small replica (~5G) can take a minute or two. If we interpolate that to larger replicas then the destruction could take tens of minutes, which seems to be unacceptable. I suspect that what takes a long time is clearing the old data from the volume. Depending on the configuration that can be UNMAP or write zeroes operation. The latter is very time-consuming for sure. A couple of tasks here:

1. find a minimal test case how to reproduce the problem
2. find out what k8s mandates from storage driver in terms of clearing old data from a volume
3. if clearing old data is required, based on limits of SPDK figure out if that could be done asynchronously

Mayastor restarting: out of hugepages

Describe the bug
Mayastor process restarting due to being out of hugepages.

To Reproduce
Leave it running with periodic traffic running to it (percona mysql with sysbench should do)

Expected behavior
Mayastor should not run out of hugepages.

Screenshots

bmath@pop-os:~/maya/kops/test392.bob.bmath.nyc$ kubectl logs -n mayastor mayastor-5kkcr --previous
2020-10-21T17:41:45.363813184+00:00  INFO mayastor: Starting Mayastor ..
2020-10-21T17:41:45.363886495+00:00  INFO mayastor: kernel io_uring support: yes
2020-10-21T17:41:45.363896115+00:00  INFO mayastor: free_pages: 512 nr_pages: 512
2020-10-21T17:41:45.366074310+00:00  INFO mayastor::subsys::mbus::mbus_nats: Connecting to the nats server nats:4222...
2020-10-21T17:41:46.105049658+00:00  INFO mayastor::subsys::mbus::mbus_nats: Successfully connected to the nats server nats:4222
2020-10-21T17:41:46.105362783+00:00  INFO mayastor::subsys::config: Applying Mayastor configuration settings
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized

** OS info (please complete the following information):**

• Distro: Ubuntu 20
• Kernel version 5.9
• MayaStor revision or container image v0.5

Additional context
Add any other context about the problem here.

Improve lifecycle of the deployment configuration

We have a problem that we have to maintain a duplicate set of yamls for deploying mayastor in our e2e tests. Also there is a separate repository for OpenEBS Helm Charts. Instead we would like to have template yamls that we can configure for whatever use case we have (user, developer, ci/cd, …). Sounds like a perfect job for Helm, but we also use Terraform for deployments, so need to find a good integration setup.

Possible approaches

• Use original YAMLs - change images via sed - deploy using kubectl/terraform
  • One copy of kubectl yamls
• Use original YAMLs - use kubectl patch - deploy via terraform
  • One copy of kubectl yamls
  • Not very nice as patch will result in redeploys within infrastructure (which costs time, but also changes state)
• Use original YAMLs - apply kustomize.yaml - deploy using kubectl/terraform
  • One copy of kubectl yamls
• Create helm chart - apply variables (via helm) - deploy using helm/terraform
  • Two copies - yamls + helm | drop original yamls and force users to use helm
  • Needs maintenance - importing original yaml changes
• Create native terraform templates (HCL autogenerate) - deploy using terraform
  • Needs maintenance - regeneration on original yaml change
• (CHOSEN SOLUTION) Generate YAMLs from common template (terraform/jinja/helm/sed) - generate deploy/* + generate terraform (or e2e yamls) and anything that comes in our way - deploy however
  • Needs to be run in CI and committed to git to develop and to master and openebs/charts
  • E2e tests will run the “engine” and generate deploy yamls for itself
• Generate yamls from TF code
  • Strange - not supported workflow - might be possible
  • Not sure it can be done nicely with TF - probably above (common source and generate two is better)
• Use NIX

Selected approach

As marked, we have chosen the pretty standard approach of template -> render -> artifact approach. While there is plenty of possible templating engines out there, we have chose helm as that will give us help charts for free and it is a well known tool for the k8s comunity. Quick prototype shows that this should be pretty feasible way.

RWX support via NFS-provisioner

Is your feature request related to a problem? Please describe.
Is it possible to use mayastor with nfs-provisioner for rwx pvc's?

Describe the solution you'd like

helm install nfs stable/nfs-server-provisioner --namespace=mayastor --set=persistence.enabled=true,persistence.storageClass=mayastor-iscsi,persistence.size=120Gi,storageClass.name=nfs-sc,storageClass.provisionerName=openebs.io/nfs

Currently, it is failing to create a new pvc:

failed to provision volume with StorageClass "mayastor-iscsi": rpc error: code = ResourceExhausted desc = Cannot find suitable storage pool(s) for the volume

RUST_LOG is not taken into account for log messages

Describe the bug
Cannot customise the log level using RUST_LOG

To Reproduce
sudo RUST_LOG=debug mayastor -g 127.0.0.1

Expected behavior
Should see debug messages

Additional context
Seems like we're missing the "env-filter" on the subscriber init.
I've added .with_env_filter(EnvFilter::from_default_env()) to the subscriber and it seems to resolve the issue.

conf file parsing for the nexus should be removed

Describe the bug
When using legacy .conf files, nexus::open() does not get spawned correctly.

To Reproduce
create a config file with a nexus and start mayastor with -c conf.conf

Expected behavior
The nexus should be opened but remains in the Init state

Additional context
The legacy .conf file handling should be removed as we don't use it anymore instead, we process json/yaml during start.

Unable to attach or mount volumes on AWS EKS

Describe the bug
This affects both v0.1.0 and v0.2.0. Following the steps outlined in the deploy README, I'm able to create the necessary Mayastor services, but volumes fail to mount with the following error:

MountVolume.MountDevice failed for volume "pvc-fb3242d3-4c69-4ddd-a429-21a50e2ade76" : rpc error: code = Internal desc = Failed to mount ext4 fs on /dev/sda with opts "" to /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb3242d3-4c69-4ddd-a429-21a50e2ade76/globalmount: No such device (os error 19)

Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[ms-volume default-token-6qqk4]: timed out waiting for the condition

To Reproduce

• AWS EKS
• K8S version v1.16.8-eks-e16311
• Amazon Linux 2
• m5ad.large (also tried several other types)
• 1 ephemeral device (not formatted or mounted): /dev/nvme0n1
• iscsi installed / enabled:

sudo yum install -y iscsi-initiator-utils
sudo systemctl enable iscsid
sudo systemctl start iscsid
echo 512 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
echo 'vm.nr_hugepages = 512' | sudo tee -a /etc/sysctl.conf
sudo service kubelet restart

• pool and storage class defined identical to deploy steps in v0.1.0/v0.2.0 (relevant node name and device name in storage pool):

NODE_NAME=$(kubectl get nodes -o=jsonpath='{.items[0].metadata.name}')
DISK_NAME='/dev/nvme0n1'

cat <<EOF | kubectl apply -f -
apiVersion: "openebs.io/v1alpha1"
kind: MayastorPool
metadata:
  name: pool-on-node-1
  namespace: mayastor
spec:
  node: $NODE_NAME
  disks: ["$DISK_NAME"]
EOF

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: mayastor-iscsi
parameters:
  repl: '1'
  protocol: 'iscsi'
provisioner: io.openebs.csi-mayastor

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ms-volume-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: mayastor-iscsi

kind: Pod
apiVersion: v1
metadata:
 name: fio
spec:
 volumes:
   - name: ms-volume
     persistentVolumeClaim:
      claimName: ms-volume-claim
 containers:
   - name: fio
     image: dmonakhov/alpine-fio
     args:
       - sleep
       - "1000000"
     volumeMounts:
       - mountPath: "/volume"
         name: ms-volume

Expected behavior
Expect volume to mount, container to create, Pod to start.

** OS info (please complete the following information):**

• Distro: Amazon Linux 2
• Kernel 4.14
• Mayastor v0.1.0 / v.0.2.0

two fio pods are stuck in terminating state

This happens on a cluster in aws set up by kops:

[Macbook:~]$ kubectl get pod
NAME   READY   STATUS        RESTARTS   AGE
fio    1/1     Terminating   0          17h
fio2   1/1     Terminating   0          17h

Looking at the log file there seems to be some kind of a problem with destroying the volumes used in the pods. More information about the resources in the cluster:

[Macbook:~]$ kubectl get pod -n mayastor
NAME                    READY   STATUS    RESTARTS   AGE
mayastor-5kkcr          1/1     Running   29         40h
mayastor-9drjm          1/1     Running   29         40h
mayastor-csi-5wx9x      2/2     Running   0          42h
mayastor-csi-8zjvg      2/2     Running   0          42h
mayastor-csi-lqsfg      2/2     Running   0          42h
mayastor-csi-mpx7t      2/2     Running   4          42h
mayastor-k8cpm          1/1     Running   28         40h
mayastor-w4smq          1/1     Running   0          38h
moac-5b4969ddd7-hkhzq   3/3     Running   0          19h
nats-5fc4d79d66-vjq5s   1/1     Running   0          42h

[Macbook:~]$ kubectl -n mayastor get msn
NAME                                          STATE    AGE
ip-172-20-34-25.us-east-2.compute.internal    online   19h
ip-172-20-52-180.us-east-2.compute.internal   online   19h
ip-172-20-56-216.us-east-2.compute.internal   online   19h
ip-172-20-63-224.us-east-2.compute.internal   online   19h

[Macbook:~]$ kubectl -n mayastor get msp
NAME    NODE                                          STATE    AGE
pool1   ip-172-20-34-25.us-east-2.compute.internal    online   20h
pool2   ip-172-20-52-180.us-east-2.compute.internal   online   20h
pool3   ip-172-20-56-216.us-east-2.compute.internal   online   20h
pool4   ip-172-20-63-224.us-east-2.compute.internal   online   20h

[Macbook:~]$ kubectl -n mayastor get msv
NAME                                   NODE                                          SIZE         STATE     AGE
83627c0a-20a2-4125-a698-0a4ba754a80f   ip-172-20-34-25.us-east-2.compute.internal    6442450944   healthy   17h
be70f20c-c4d7-415f-87d0-06f2e10430a6   ip-172-20-52-180.us-east-2.compute.internal   6442450944   healthy   17h

Stripped output of describe msv:

[Macbook:~]$ kubectl -n mayastor describe msv
Name:         83627c0a-20a2-4125-a698-0a4ba754a80f
Namespace:    mayastor
Labels:       <none>
Annotations:  <none>
API Version:  openebs.io/v1alpha1
Kind:         MayastorVolume
Metadata:
  Creation Timestamp:  2020-10-22T14:11:08Z
  Generation:          1
  Resource Version:  293911
  Self Link:         /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorvolumes/83627c0a-20a2-4125-a698-0a4ba754a80f
  UID:               fcb7f83f-5109-4915-9ac5-638efcd1ccab
Spec:
  Limit Bytes:  0
  Preferred Nodes:
  Protocol:        nvmf
  Replica Count:   2
  Required Bytes:  6442450944
  Required Nodes:
Status:
  Nexus:
    Children:
      State:     CHILD_ONLINE
      Uri:       bdev:///83627c0a-20a2-4125-a698-0a4ba754a80f
      State:     CHILD_ONLINE
      Uri:       nvmf://172.20.63.224:8420/nqn.2019-05.io.openebs:83627c0a-20a2-4125-a698-0a4ba754a80f
    Device Uri:  nvmf://172.20.34.25:8420/nqn.2019-05.io.openebs:nexus-83627c0a-20a2-4125-a698-0a4ba754a80f
    State:       NEXUS_ONLINE
  Node:          ip-172-20-34-25.us-east-2.compute.internal
  Reason:
  Replicas:
    Node:     ip-172-20-34-25.us-east-2.compute.internal
    Offline:  false
    Pool:     pool1
    Uri:      bdev:///83627c0a-20a2-4125-a698-0a4ba754a80f
    Node:     ip-172-20-63-224.us-east-2.compute.internal
    Offline:  false
    Pool:     pool4
    Uri:      nvmf://172.20.63.224:8420/nqn.2019-05.io.openebs:83627c0a-20a2-4125-a698-0a4ba754a80f
  Size:       6442450944
  State:      healthy
Events:       <none>

Name:         be70f20c-c4d7-415f-87d0-06f2e10430a6
Namespace:    mayastor
Labels:       <none>
Annotations:  <none>
API Version:  openebs.io/v1alpha1
Kind:         MayastorVolume
Metadata:
  Creation Timestamp:  2020-10-22T14:21:18Z
  Generation:          1
  Resource Version:  295919
  Self Link:         /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorvolumes/be70f20c-c4d7-415f-87d0-06f2e10430a6
  UID:               477a9aae-9152-42ef-afbf-15fb1e9bff3a
Spec:
  Limit Bytes:  0
  Preferred Nodes:
  Protocol:        nvmf
  Replica Count:   2
  Required Bytes:  6442450944
  Required Nodes:
Status:
  Nexus:
    Children:
      State:     CHILD_ONLINE
      Uri:       bdev:///be70f20c-c4d7-415f-87d0-06f2e10430a6
      State:     CHILD_ONLINE
      Uri:       nvmf://172.20.56.216:8420/nqn.2019-05.io.openebs:be70f20c-c4d7-415f-87d0-06f2e10430a6
    Device Uri:  nvmf://172.20.52.180:8420/nqn.2019-05.io.openebs:nexus-be70f20c-c4d7-415f-87d0-06f2e10430a6
    State:       NEXUS_ONLINE
  Node:          ip-172-20-52-180.us-east-2.compute.internal
  Reason:
  Replicas:
    Node:     ip-172-20-52-180.us-east-2.compute.internal
    Offline:  false
    Pool:     pool2
    Uri:      bdev:///be70f20c-c4d7-415f-87d0-06f2e10430a6
    Node:     ip-172-20-56-216.us-east-2.compute.internal
    Offline:  false
    Pool:     pool3
    Uri:      nvmf://172.20.56.216:8420/nqn.2019-05.io.openebs:be70f20c-c4d7-415f-87d0-06f2e10430a6
  Size:       6442450944
  State:      healthy
Events:       <none>

[Macbook:~]$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS    REASON   AGE
pvc-83627c0a-20a2-4125-a698-0a4ba754a80f   6Gi        RWO            Delete           Bound    default/ms-volume-claim    mayastor-nvmf            17h
pvc-be70f20c-c4d7-415f-87d0-06f2e10430a6   6Gi        RWO            Delete           Bound    default/ms-volume-claim2   mayastor-nvmf            17h

[Macbook:~]$ kubectl get pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
ms-volume-claim    Bound    pvc-83627c0a-20a2-4125-a698-0a4ba754a80f   6Gi        RWO            mayastor-nvmf   17h
ms-volume-claim2   Bound    pvc-be70f20c-c4d7-415f-87d0-06f2e10430a6   6Gi        RWO            mayastor-nvmf   17h

Log files will be attached.

[introduction/quickstart/deploy-mayastor]: [Add mitigation details for potential error case]

Are you reporting an issue with existing content?
Please describe the error or omission, or your suggestion for improvement.
Please include a link the affected page(s)

https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor

Are you proposing new content, or a change to the existing documentation layout or structure?

Add a warning that if you receive the error:

$ kubectl -n mayastor get msn
error: the server doesn't have a resource type "msn"

Then the problem may well be that no nodes have sufficient HugePages. That is entirely non-obvious given the message. The problem being that the daemonset nodes run with no errors and no suggestion that there is any problem but they do not do what the documentation suggests they will, and they do it silently.

Try to move to stable when rust 1.39 releases

As we make heavy use of async -- we have been on a nightly compiler for some time. Fortunately, async is getting stabilized and so it would be nice if could move to a stable rust version.

UDS listner does not handle existing sockets properly

When starting mayastor-agent, with a preexisting CSI socket, we fail the bind.

let mut tcp = tokio::net::UnixListener::bind(std::path::Path::new(path))?.incoming();

note this code is in the fork of tonic

The above line creates a new listener if and only if path does not exist. We should bind to the socket even when it exists or, clean up any stale/existing sockets prior to starting the server.

It takes a very long time to reboot a node with app using nvmf mayastor volume

I have reproduced this twice when testing nexus failover and it seems to be always the case when rebooting a node with an application using a mayastor volume. We suspect that nvmf connection from the initiator to nexus target is preventing the node from getting rebooted. It eventually reboots but after a very long time like 15 minutes or so. I'm not sure if we can do something about it but at least we should understand why that happens.

Improve lifecycle of the deployment configuration

We have a problem that we have to maintain a duplicate set of yamls for deploying mayastor in our e2e tests. Also there is a separate repository for [OpenEBS Helm Charts|https://github.com/openebs/charts/]. Instead we would like to have template yamls that we can configure for whatever use case we have (user, developer, ci/cd, …). Sounds like a perfect job for Helm, but we also use Terraform for deployments, so need to find a good integration setup.

h2. Possible approaches

• Use original YAMLs - change images via sed - deploy using kubectl/terraform
  ** One copy of kubectl yamls
• Use original YAMLs - use kubectl patch - deploy via terraform
  ** One copy of kubectl yamls
  ** Not very nice as patch will result in redeploys within infrastructure (which costs time, but also changes state)
• Use original YAMLs - apply kustomize.yaml - deploy using kubectl/terraform
  ** One copy of kubectl yamls
• Create helm chart - apply variables (via helm) - deploy using helm/terraform
  ** Two copies - yamls + helm | drop original yamls and force users to use helm
  ** Needs maintenance - importing original yaml changes
• Create native terraform templates (HCL autogenerate) - deploy using terraform
  ** Needs maintenance - regeneration on original yaml change
• (CHOSEN SOLUTION) Generate YAMLs from common template (terraform/jinja/helm/sed) - generate deploy/ + generate terraform (or e2e yamls) and anything that comes in our way - deploy however
  ** Needs to be run in CI and committed to git to develop and to master and openebs/charts
  ** E2e tests will run the “engine” and generate deploy yamls for itself
• Generate yamls from TF code
  ** Strange - not supported workflow - might be possible
  ** Not sure it can be done nicely with TF - probably above (common source and generate two is better)
• Use NIX

h2. Selected approach

As marked, we have chosen the pretty standard approach of template -> render -> artifact approach. While there is plenty of possible templating engines out there, we have chose helm as that will give us help charts for free and it is a well known tool for the k8s comunity. Quick prototype shows that this should be pretty feasible way.