How hwameistor specify raw disk data volume?

[AmazingPangWei]

How hwameistor specify raw disk data volume?

[SSmallMonster]

@AmazingPangWei Thanks for your feedback.

Question 1: I want to know whether it is correct that localdisknodes only show node pw3? (my other nodes such as pw1, pw2 only own root disk /dev/sda)

A1: All pw1,pw2 nodes (related to ds/localdiskmanager) should be displayed. If a problem is displayed, check whether the hwameistor-local-disk-manager-xxxx POD has error logs on the corresponding node.

Question 2: Why pw3 FREECAPACITY、TOTALCAPACITY、TOTALDISK are empty?

A2: From the information provided, there is no disk allocated to the localdisknode resource. By specifying .spec.owner for the localdiskclaim resource and .spec.owner for the localdisk resource, respectively.

You can view Disk Storage Node to expand the disk storage pool.

Question 3: It seems like hwameistor always select my block device to extend lvm pool, so my node pw3 /dev/sdb、/dev/sde are managed by lvm. But how I specify node pw3 /dev/sdb、/dev/sde as raw disk data volume like this doc?

A3: Yes, raw disk can be provisioned directly to application.

[buffalo1024]

That's OK. Is there another way to work around this case now? Or, when the next operator version is published?

There is a way at this moment.Pull down the operator helm chart and find the hwameistor-cluster cr in the templates and add configs like this

[AmazingPangWei]

@sun7927 In my scenario, whole raw block device and LVM volume are both needed. In other words, a vm can use a whole raw block device to get a better performance, or a vm can use a normal LVM volume for a ordinary use case.

Hwameistor is able to manage them both, which is fine for me.

They are different. My understanding is raw block volume which can be LVM volume or others but should be sent into the pod as a block device (no filesystem on it).

You are right. My vm pods use exactly block device.

[AmazingPangWei]

@SSmallMonster Thanks for your reply.

A1: All pw1,pw2 nodes (related to ds/localdiskmanager) should be displayed. If a problem is displayed, check whether the hwameistor-local-disk-manager-xxxx POD has error logs on the corresponding node.
A2: From the information provided, there is no disk allocated to the localdisknode resource. By specifying .spec.owner for the localdiskclaim resource and .spec.owner for the localdisk resource, respectively.
You can view Disk Storage Node to expand the disk storage pool.

You help me to solve Q1&Q2 perfectly. After checking my cluster, pw1 and pw2 were added taints by my workmate. All localdisknodes show successfully after removing taints. Besides, I add a disk to pw2 and put it to LocalDiskNode by following your suggestion, and now localdisknodes as follows:

NAME   FREECAPACITY   TOTALCAPACITY   TOTALDISK   STATUS   AGE
pw1                                               Ready    71m
pw2                   214748364800    1           Ready    73m
pw3                                               Ready    3d18h

But I'm still confused to Question 3. After I install hwameistor, /dev/sdb、/dev/sde in pw3 has been managed by lvm automatically. Here are my questions:

1. what is the strategy of hwameistor to choose disk/lvm storage? Why /dev/sdb、/dev/sde are lvm storage? Can I specify it when or after I install hwameistor?
2. Is it possible to turn lvm storage to disk storage?
3. Is it possible to coexist between lvm storage and disk storage in one node?

I would greatly appreciate it if you could answer these questions.

[SSmallMonster]

1. what is the strategy of hwameistor to choose disk/lvm storage? Why /dev/sdb、/dev/sde are lvm storage? Can I specify it when or after I install hwameistor?

A1: By default, when the owner of LocalDiskClaim is local-storage, the empty disks(with no partition and no filesystem on it) will be used for lvm storage.

If you want specify it after installing hwameistor, there are 2 options as below:

• Remove empty disks on the host before installation.(may be suitable on virtual machine)
• Reserve the disks you want use later manually.

2. Is it possible to turn lvm storage to disk storage?

A2: For disks, the answer is No.

3. Is it possible to coexist between lvm storage and disk storage in one node?

A3: Sure. We support lvm storage and disk storage at the same time.

@AmazingPangWei If you have any other questions, feel free to talk.

[SSmallMonster]

The method to reserve disk manually as bellow:

$ kubectl edit ld <local_disk_name>
apiVersion: hwameistor.io/v1alpha1
kind: LocalDisk
spec:
  capacity: 10485760
-reserved: false
+reserved: true

After doing that, this empty disk won't be assigned to an incoming LocalDiskClaim.

[AmazingPangWei]

I install hwameistor by following install docs as below:

helm repo add hwameistor-operator https://hwameistor.io/hwameistor-operator
helm repo update hwameistor-operator

helm install hwameistor-operator hwameistor-operator/hwameistor-operator -n hwameistor --create-namespace \
--set global.hwameistorImageRegistry=ghcr.m.daocloud.io \
--set global.k8sImageRegistry=m.daocloud.io/registry.k8s.io

But according to this install docs as below:

Operator will claim disk automatically while installing, the available disks will be added into pool of LocalStorageNode.

After I execute these commands, my disks will be claimed by hwameistor-operator (lvm storage by default) automatically. When and how should I edit ld in order to reserve disk manually?

[SSmallMonster]

@AmazingPangWei You can reserve disks before helm install ....

In addition, the disk reserve function by hwameistor-operator will be published at next operator version~

If it is convenient, can you describe the scenarios in which you use hwameistor? This will help us to better improve our function

[AmazingPangWei]

@AmazingPangWei You can reserve disks before helm install ....

Sorry, I don't understand. There are no LocalDisk crd and object in k8s cluster before helm install ..... How I reserve disks?

If it is convenient, can you describe the scenarios in which you use hwameistor? This will help us to better improve our function

Well, I'm looking for a local storage system to support kubevirt, which can support vm use local disk. And I think hwameistor matches perfectly, so I wanna go deeper with it.

[SSmallMonster]

@AmazingPangWei Sorry, I may have misdescribed it. Currently, disk reservation is implemented in the back-end storage system, but hwameistor-operator has not yet exposed this interface.

[AmazingPangWei]

That's OK. Is there another way to work around this case now? Or, when the next operator version is published?

[SSmallMonster]

That's OK. Is there another way to work around this case now? Or, when the next operator version is published?

@buffalo1024 cc, please have a look~

[sun7927]

@AmazingPangWei does kubevirt need raw block volume or disk volume? They are different. My understanding is raw block volume which can be LVM volume or others but should be sent into the pod as a block device (no filesystem on it).

[AmazingPangWei]

@buffalo1024 Thank you so much. I'll try this manner.

[sun7927]

@sun7927 In my scenario, whole raw block device and LVM volume are both needed. In other words, a vm can use a whole raw block device to get a better performance, or a vm can use a normal LVM volume for a ordinary use case.

Hwameistor is able to manage them both, which is fine for me.

They are different. My understanding is raw block volume which can be LVM volume or others but should be sent into the pod as a block device (no filesystem on it).

You are right. My vm pods use exactly block device.

Got it. For your case, both LVM and disk volumes are required to support kubevirt for different purposes. Same as your understanding, HwameiStor manages both volume types, and each has the special features. You can deep into the details from official docs.

You can list your requirements on the storage (HwameiStor) if you like, we will try best to work out the best solutions with you.

[AmazingPangWei]

@sun7927 Thank you for your kindness. Here are my another requirements:

1.DiskReserveConfigurations in cluster spec are pretty tricky if I want to reserve most disks for raw disk storage, because I have to specify most disks in most nodes one by one. So can operator just stop claiming disk automatically while installing?

In this way, user like me could batch specifying(e.g. using bash scripts) lvm/disk storage localdiskclaim base on localdisk attributes like capacity, nodeName, disk protocol, etc.

2.Localdisk is a very good crd. What I need to do is getting localdisk from control layer, and I can know all disks info. But currently localDisk exposes few attributes, which is not enough in some certain scenarios.

For example, I need to input block device path when I install rook:

storage:
  nodes:
  - devices:
    - name: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-ua-datadisk-0
    name: pw1
  - devices:
    - name: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-ua-datadisk-0
    name: pw3
  - devices:
    - name: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-ua-datadisk-0
    name: pw2

I can't get block device path(DEVLINKS in udevadm) from Localdisk, although it's very easy to get for hwameistor.

[SSmallMonster]

@AmazingPangWei Thank you for your patient feedback.

DiskReserveConfigurations in cluster spec are pretty tricky if I want to reserve most disks for raw disk storage, because I have to specify most disks in most nodes one by one. So can operator just stop claiming disk automatically while installing?

Sure, although the one of initial purposes of the Operator design was to make Day1's operation simpler, I still believe that your requirement is reasonable. We should control the behavior of automatically claiming disks, leaving more operating space for managers and compatible with more scenarios.

I can't get block device path(DEVLINKS in udevadm) from Localdisk, although it's very easy to get for hwameistor.

Sure, we can add this attribute DEVLINKS . Can you describe why this attribute is needed and in what scenarios cannot devname or devpath be satisfied? Thanks!!

[sun7927]

@sun7927 Thank you for your kindness. Here are my another requirements:

1.DiskReserveConfigurations in cluster spec are pretty tricky if I want to reserve most disks for raw disk storage, because I have to specify most disks in most nodes one by one. So can operator just stop claiming disk automatically while installing?

We are still thinking about the better way for user to configure the disk. This could be one of cases.

In this way, user like me could batch specifying(e.g. using bash scripts) lvm/disk storage localdiskclaim base on localdisk attributes like capacity, nodeName, disk protocol, etc.

2.Localdisk is a very good crd. What I need to do is getting localdisk from control layer, and I can know all disks info. But currently localDisk exposes few attributes, which is not enough in some certain scenarios.

For example, I need to input block device path when I install rook:

storage:
  nodes:
  - devices:
    - name: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-ua-datadisk-0
    name: pw1
  - devices:
    - name: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-ua-datadisk-0
    name: pw3
  - devices:
    - name: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-ua-datadisk-0
    name: pw2

I can't get block device path(DEVLINKS in udevadm) from Localdisk, although it's very easy to get for hwameistor.

Currently, HwameiStor manages the real local disk only. The external volume disk as above should be attached from an external SAN storage system, which are not managed by HwameiStor. If this is your case, we can have a further discussion on it.

In addition, what's the additional attributes of the localdisk you want? In fact, LocalDisk has lots of disk details as below:

apiVersion: hwameistor.io/v1alpha1
kind: LocalDisk
metadata:
  creationTimestamp: "2023-07-11T10:57:12Z"
  generation: 3
  name: k8s-node1-sdb
  resourceVersion: "17464"
  uid: 3ae14a02-581f-4cea-b5ff-e569864462e6
spec:
  capacity: 107374182400
  claimRef:
    apiVersion: hwameistor.io/v1alpha1
    kind: LocalDiskClaim
    name: k8s-node1-hdd-claim
    resourceVersion: "17379"
    uid: bea694cc-989f-4ae6-81fc-b715ea4b2244
  devicePath: /dev/sdb
  diskAttributes:
    devType: disk
    modelName: LVM PV SpQCW1-eBUe-wQCp-Sr4n-oN7c-LJxI-S9Ms08 on /dev/sdb
    protocol: scsi
    type: HDD
    vendor: VMware
  nodeName: k8s-node1
  owner: local-storage
  partitionInfo:
  - filesystem:
      fsType: LVM2_member
    hasFileSystem: true
    path: ""
  partitioned: true
  raidInfo: {}
  smartInfo:
    overallHealth: Passed
  state: Active
  uuid: 6b0a24d5eb3ce140a6c7401ba17f775e
status:
  claimState: Bound

[AmazingPangWei]

Can you describe why this attribute is needed and in what scenarios cannot devname or devpath be satisfied? Thanks!!

Well, devname or devpath are not immutable actually. For example, say I have a disk, and its device name is /dev/sdc. But device name may be changed to /dev/sdb after reboot, which may make some trouble. A good way to solve it is using DEVLINKS, because it's stable. So some applications which use block device directly(e.g. rook) often specify device by /dev/disk/by-id/xxx.

By the way, can hwameistor handles the condition that device name changes after reboot?

[AmazingPangWei]

Currently, HwameiStor manages the real local disk only. The external volume disk as above should be attached from an external SAN storage system, which are not managed by HwameiStor. If this is your case, we can have a further discussion on it.

Maybe we could have a further discussion on the Slack

[sun7927]

Can you describe why this attribute is needed and in what scenarios cannot devname or devpath be satisfied? Thanks!!

Well, devname or devpath are not immutable actually. For example, say I have a disk, and its device name is /dev/sdc. But device name may be changed to /dev/sdb after reboot, which may make some trouble. A good way to solve it is using DEVLINKS, because it's stable. So some applications which use block device directly(e.g. rook) often specify device by /dev/disk/by-id/xxx.

By the way, can hwameistor handles the condition that device name changes after reboot?

hwameistor identifies the disk by it's serial number which is unique and stable, and is able to handle name change

[AmazingPangWei]

I've distributed our discussion to #982 #983 #984. Thank you all for your kindness.