hwameistor / hwameistor Goto Github PK
View Code? Open in Web Editor NEWHwameistor is an HA local storage system for cloud-native stateful workloads.
Home Page: https://hwameistor.io/
License: Apache License 2.0
Hwameistor is an HA local storage system for cloud-native stateful workloads.
Home Page: https://hwameistor.io/
License: Apache License 2.0
Output of lsblk
:
# paste output here
Output of blkid
:
# paste output here
Output of kubectl version
:
# paste output here
Output of kubectl get lsn -o yaml
:
# paste output here
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
# paste output here
I could not find the uninstall processes in ReadMe, not sure if there is or not?
It is necessary sometimes to remove all the hwamei components completely for testing purpose.
Change scheduler-name in deployment yaml is not easy to use.
If user want to install wordpress from bitnami, he cannot run
helm install wp -n wp5 community/wordpress --create-namespace
He should run:
helm install wp -n wp5 community/wordpress --set global.imageRegistry=docker.m.daocloud.io --set schedulerName=hwameistor-scheduler --set mariadb.schedulerName=hwameistor-scheduler --create-namespace
The config is very diffcult for user.
During HwameiStor deployment, first we need to create HwameiStor system namespace. With Helm Charts deployment the namespace created is 'hwameistor':
https://github.com/hwameistor/helm-charts/blob/main/README.md#step-1-install-hwameistor
While in the code https://github.com/hwameistor/local-storage/blob/main/deploy/01_namespace.yaml, the namespace is 'local-storage-system'.
Suggest to make these 2 in consistent.
After repeatedly installing and uninstalling hwameistor,Incorrect volume data remains in lsn, resulting in usedVolumeCount inconsistent with reality
apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
creationTimestamp: "2022-06-22T09:26:35Z"
generation: 1
managedFields:
kubectl get lv
NAME POOL REPLICAS CAPACITY ACCESSIBILITY STATE RESOURCE PUBLISHED AGE
pvc-2a570cad-a440-470b-a124-df46e4622151 LocalStorage_PoolHDD 3 1073741824 Ready 1 demo-dev-worker-01 19h
pvc-6f972b09-1394-4a4e-b9ff-4ed4b5b11e9a LocalStorage_PoolHDD 1 1073741824 Ready -1 demo-dev-worker-01 43h
pvc-b0cf7e2f-9e0a-4210-b39a-a96c96d89265 LocalStorage_PoolHDD 2 1073741824 Ready 3 demo-dev-master-03 15h
According to the docs : https://kubernetes-csi.github.io/docs/node-driver-registrar.html, csi-node-driver-registrar is from k8s.gcr.io. But the is from the https://github.com/hwameistor/helm-charts/blob/main/charts/hwameistor/values.yaml#L1, It's from the Quay.io .
When we install the hwamei stor, the user must to use "node label" to select the node enabled
Should we change it to the config file , like https://github.com/rancher/local-path-provisioner#configuration
Benifit:
Much inconsistency is found in docs. It's time to add a glossary list, so everyone can follow the glossary and reduce confusion. Starting from four modules of HwameiStor:
Hope we add more day by day.
Hi:
I use helm to deploy hwameister
Warning Failed 3m11s (x5 over 4m36s) kubelet Error: failed to start container "local-disk-manager": Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/local-disk-manager": stat /local-disk-manager: no such file or directory: unknown
HwameiStor documentation has three different topics currently. It is recommended to provide a simple style guide if more than 20 topics in a docsite.
References:
There is a problem with the namespace of CRD in helm chart
A new module of "admission control" is added into the HwameiStor, it's request to keep the document up to date.
ldc 删除后,ld 还显示为Claimed状态,是否类似pv的处理显示未一个特殊状态Released之类的?
Output of kubectl version
:
v1.23.1
Output of kubectl get lsn -o yaml
:
apiVersion: v1
items:
- apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
creationTimestamp: "2022-04-17T08:14:05Z"
generation: 1
name: kind-k8s-1-18-worker
resourceVersion: "279502"
selfLink: /apis/hwameistor.io/v1alpha1/localstoragenodes/kind-k8s-1-18-worker
uid: 0a68078b-8a7c-4621-bf14-fc21d5c60c20
spec:
hostname: kind-k8s-1-18-worker
storageIP: 172.18.0.9
topogoly:
region: default
zone: default
status:
pools:
LocalStorage_PoolHDD:
class: HDD
freeCapacityBytes: 1178960134144
freeVolumeCount: 998
name: LocalStorage_PoolHDD
totalCapacityBytes: 1181107617792
totalVolumeCount: 1000
type: REGULAR
usedCapacityBytes: 2147483648
usedVolumeCount: 2
volumeCapacityBytesLimit: 1181107617792
volumes:
- pvc-33dbf764-168f-43df-abcd-88f2c8bac963
- pvc-675e03fa-ec15-4f38-8f07-e984f61085c8
LocalStorage_PoolSSD:
class: SSD
disks:
- capacityBytes: 1073737629696
devPath: /dev/loop1
state: Available
type: SSD
freeCapacityBytes: 1073737629696
freeVolumeCount: 1000
name: LocalStorage_PoolSSD
totalCapacityBytes: 1073737629696
totalVolumeCount: 1000
type: REGULAR
usedCapacityBytes: 0
usedVolumeCount: 0
volumeCapacityBytesLimit: 1073737629696
state: Ready
- apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
creationTimestamp: "2022-04-17T08:14:05Z"
generation: 1
name: kind-k8s-1-18-worker2
resourceVersion: "3743"
selfLink: /apis/hwameistor.io/v1alpha1/localstoragenodes/kind-k8s-1-18-worker2
uid: 58f151a9-b4de-4d9f-85e5-1fbfa0b6090d
spec:
hostname: kind-k8s-1-18-worker2
storageIP: 172.18.0.7
topogoly:
region: default
zone: default
status:
state: Ready
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
kind
Currently, there are two ways to configure LDC
During installation, by helm chart's storageNodes[]
After installation, by yaml's kind: LocalDiskClaim
Both have drawbacks:
Method 1 is a batch process, but cannot be modified after installation because helm upgrade
ignores all CRDs and their instances. Therefore it is not suitable for node expansion and reduction.
Mothod 2 can be configured anytime but is not a batch process, therefore each node needs to be configured with a separate yaml.
We need a new method to configure LDC in batch both during installation and after it.
@SSmallMonster please comment
Output of lsblk
:
# paste output here
Output of blkid
:
# paste output here
Output of kubectl version
:
# paste output here
Output of kubectl get lsn -o yaml
:
# paste output here
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
# paste output here
sc yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-disk-hdd-disk
provisioner: disk.hwameistor.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
reclaimPolicy: Retain
parameters:
diskType: "HDD"
pvc yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-disk-pvc
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
storageClassName: local-disk-hdd-disk
resources:
requests:
storage: 5Gi
I created a sc whose reclaimPolicy is Retain, and created the corresponding pvc mount deploy.
Then I deleted the deploy, pvc, but because the reclaimPolicy is Retain, the pv is not deleted.
At this time, how can I delete the used storage and release the corresponding local disk?
The content of "What is HwameiStor" should be refined as below:
When we use helm to install hwameistor. It shows:
The user cannot know whether the install is success or fail.
Show we add Notes in helm like bitnami (https://github.com/bitnami/charts/blob/master/bitnami/nginx/templates/NOTES.txt)
Give the use some suggestion to check the install.
作为insight tester,我在按照README.md文档### STEP 3: Claim Disk By Type On Node创建LDC时,没有将spec.nodeName设置为node的hostname以至于LDC未到Bound状态;请明确此处设置
Suggested the new structure of document as below:
There are multiple components in HwameiStor, including LDM, LS, Scheduler, Admission Controller. Had better to build up a matrix to describe the installation options for supporting multiple scenarios. For example:
hwameistor-webhook uses image "docker.io/jackmmzhou/self-signed:v1" which needs to be mirrored to other repositories for better access:
error:
Failed to pull image "jackmmzhou/self-signed:v1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/jackmmzhou/self-signed:v1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/jackmmzhou/self-signed/manifests/sha256:c1f6eea3c47ed1fdc560530235d2abcf3e27d17664ff7a8b18e9641ff3032982: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
The node-role.kubernetes.io/master taint is deprecated and will be removed in kubeadm version 1.25.
It would be replace by node-role.kubernetes.io/control-plane. like:
https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/manifests/vanilla/vsphere-csi-driver.yaml#L220
Hi, My go version is go version go1.18rc1 darwin/amd64
,when I exec 'go mod tidy'(After I clone github.com/hwameistor/local-disk-manager),I got this:
go: downloading github.com/onsi/ginkgo v1.12.1
go: downloading github.com/onsi/gomega v1.10.1
go: downloading k8s.io/apiextensions-apiserver v0.18.6
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: downloading go.etcd.io/bbolt v1.3.3
go: downloading github.com/gorilla/websocket v1.4.0
go: downloading golang.org/x/lint v0.0.0-20190930215403-16217165b5de
go: downloading golang.org/x/tools v0.1.1
go: downloading golang.org/x/mod v0.4.2
go: downloading sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e
go: downloading github.com/docker/docker v1.4.2-0.20200203170920-46ec8731fbce
go: downloading google.golang.org/genproto v0.0.0-20200117163144-32f20d992d24
go: downloading go.etcd.io/bbolt v1.3.4
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.12.1
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.1.0
github.com/hwameistor/local-storage/cmd/scheduler imports
k8s.io/kubernetes/cmd/kube-scheduler/app imports
github.com/spf13/cobra loaded from github.com/spf13/[email protected],
but go 1.16 would select v1.0.0
github.com/hwameistor/local-storage/cmd/scheduler imports
k8s.io/kubernetes/cmd/kube-scheduler/app imports
k8s.io/apiserver/pkg/util/term imports
github.com/docker/docker/pkg/term loaded from github.com/docker/[email protected],
but go 1.16 would select v1.4.2-0.20200203170920-46ec8731fbce
github.com/hwameistor/local-storage/pkg/member/csi imports
github.com/container-storage-interface/spec/lib/go/csi imports
google.golang.org/grpc/status imports
google.golang.org/genproto/googleapis/rpc/status loaded from google.golang.org/[email protected],
but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/component-base/metrics imports
github.com/blang/semver loaded from github.com/blang/[email protected]+incompatible,
but go 1.16 would select v3.5.1+incompatible
github.com/hwameistor/local-storage/cmd/scheduler imports
k8s.io/kubernetes/cmd/kube-scheduler/app imports
k8s.io/apiserver/pkg/server/routes imports
k8s.io/kube-openapi/pkg/handler imports
github.com/NYTimes/gziphandler loaded from github.com/NYTimes/[email protected],
but go 1.16 would select v1.1.1
github.com/hwameistor/local-storage/cmd/scheduler imports
k8s.io/kubernetes/cmd/kube-scheduler/app imports
k8s.io/apiserver/pkg/util/term imports
github.com/docker/docker/pkg/term imports
github.com/docker/docker/pkg/term/windows loaded from github.com/docker/[email protected],
but go 1.16 would select v1.4.2-0.20200203170920-46ec8731fbce
github.com/hwameistor/local-storage/pkg/member/csi imports
github.com/container-storage-interface/spec/lib/go/csi imports
google.golang.org/grpc/codes tested by
google.golang.org/grpc/codes.test imports
google.golang.org/genproto/googleapis/rpc/code loaded from google.golang.org/[email protected],
but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/member/csi imports
github.com/container-storage-interface/spec/lib/go/csi imports
google.golang.org/grpc/status tested by
google.golang.org/grpc/status.test imports
google.golang.org/genproto/googleapis/rpc/errdetails loaded from google.golang.org/[email protected],
but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
github.com/coreos/pkg/capnslog loaded from github.com/coreos/[email protected],
but go 1.16 would select v0.0.0-20180928190104-399ea9e2e55f
github.com/hwameistor/local-storage/cmd/alerter imports
sigs.k8s.io/controller-runtime/pkg/client/config tested by
sigs.k8s.io/controller-runtime/pkg/client/config.test imports
sigs.k8s.io/controller-runtime/pkg/log/zap imports
go.uber.org/zap tested by
go.uber.org/zap.test imports
golang.org/x/lint/golint loaded from golang.org/x/[email protected],
but go 1.16 would select v0.0.0-20191125180803-fdd1cda4f05f
github.com/hwameistor/local-storage/pkg/apis imports
k8s.io/apimachinery/pkg/runtime imports
sigs.k8s.io/structured-merge-diff/v3/value imports
gopkg.in/yaml.v2 tested by
gopkg.in/yaml.v2.test imports
gopkg.in/check.v1 imports
github.com/kr/pretty loaded from github.com/kr/[email protected],
but go 1.16 would select v0.2.1
github.com/hwameistor/local-storage/cmd/alerter imports
sigs.k8s.io/controller-runtime/pkg/client/config tested by
sigs.k8s.io/controller-runtime/pkg/client/config.test imports
sigs.k8s.io/controller-runtime/pkg/log/zap imports
go.uber.org/zap tested by
go.uber.org/zap.test imports
golang.org/x/lint/golint imports
golang.org/x/lint loaded from golang.org/x/[email protected],
but go 1.16 would select v0.0.0-20191125180803-fdd1cda4f05f
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
github.com/grpc-ecosystem/grpc-gateway/runtime loaded from github.com/grpc-ecosystem/[email protected],
but go 1.16 would select v1.12.1
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
github.com/tmc/grpc-websocket-proxy/wsproxy loaded from github.com/tmc/[email protected],
but go 1.16 would select v0.0.0-20190109142713-0ad062ec5ee5
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
go.etcd.io/bbolt loaded from go.etcd.io/[email protected],
but go 1.16 would select v1.3.4
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/etcdserver/api/v3rpc imports
github.com/grpc-ecosystem/go-grpc-middleware loaded from github.com/grpc-ecosystem/[email protected],
but go 1.16 would select v1.1.0
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
github.com/grpc-ecosystem/grpc-gateway/runtime imports
github.com/grpc-ecosystem/grpc-gateway/internal loaded from github.com/grpc-ecosystem/[email protected],
but go 1.16 would select v1.12.1
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
github.com/grpc-ecosystem/grpc-gateway/runtime imports
github.com/grpc-ecosystem/grpc-gateway/utilities loaded from github.com/grpc-ecosystem/[email protected],
but go 1.16 would select v1.12.1
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
github.com/grpc-ecosystem/grpc-gateway/runtime imports
google.golang.org/genproto/googleapis/api/httpbody loaded from google.golang.org/[email protected],
but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/scheduler imports
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
k8s.io/kubernetes/pkg/controller/volume/scheduling imports
k8s.io/apiserver/pkg/storage/etcd3 tested by
k8s.io/apiserver/pkg/storage/etcd3.test imports
go.etcd.io/etcd/integration imports
go.etcd.io/etcd/embed imports
github.com/grpc-ecosystem/grpc-gateway/runtime imports
google.golang.org/genproto/protobuf/field_mask loaded from google.golang.org/[email protected],
but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
To upgrade to the versions selected by go 1.16:
go mod tidy -go=1.16 && go mod tidy -go=1.17
If reproducibility with go 1.16 is not needed:
go mod tidy -compat=1.17
For other options, see:
https://golang.org/doc/modules/pruning
It seems right?
What's go version support exactly?
time="2022-08-09T03:27:30Z" level=debug msg="Finished running command" func=RunCommand file="executor.go:92" args="[--mount=/proc/1/ns/mnt --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net --uts=/proc/1/ns/uts -- lvcreate --size 1073741824B --stripes 1 LocalStorage_PoolHDD -n pvc-4deca8ca-0258-49a6-85b9-3efb5670f7f8 -y]" command=nsenter error="exit status 5" stderr=" Logical Volume "pvc-4deca8ca-0258-49a6-85b9-3efb5670f7f8" already exists in volume group "LocalStorage_PoolHDD"" stdout= timeout=30
hello,
Which versions of kubernetes are supported。
Because we watch DRBD events use nsenter
use -t
, so host ${PATH}
is not working in container:
https://github.com/hwameistor/local-storage/blob/6668a3af1b80a5c2d5c0c499820f74161819adc4/pkg/member/node/configer/drbd.go#L507
Besides we didn't providedrbdsetup
in Dockerfile
:
https://github.com/hwameistor/local-storage/blob/6668a3af1b80a5c2d5c0c499820f74161819adc4/build/Dockerfile#L1-L7
So we got no DRBD events to fill up m.resourceCache
map.
https://github.com/hwameistor/local-storage/blob/6668a3af1b80a5c2d5c0c499820f74161819adc4/pkg/member/node/configer/drbd.go#L507-L508
Kubernetes version: 1.24.3
After deploying hwameistor with storageNodes
left empty, set storageNodes
by helm -n hwameistor upgrade hwameistor ./helm/hwameistor --set "storageNodes={k8s-worker-1,k8s-worker-2,k8s-worker-3}"
.
The command succeeds but then all pods under hwameistor namespace get terminated.
Need to test and clarify if HwameiStor can work with OpenShift and Ranch besides the Kubernetes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.