emqx / emqx-operator Goto Github PK
View Code? Open in Web Editor NEWA Kubernetes Operator for EMQX
Home Page: https://www.emqx.com
License: Apache License 2.0
A Kubernetes Operator for EMQX
Home Page: https://www.emqx.com
License: Apache License 2.0
Hey guys,
I am trying to run your operator on a local Kubernetes cluster, but get an error in the pod's log:
kubectl logs emqx-0
2022-01-21T22:58:52.565992+00:00 [error] Error creating /opt/emqx/data/configs: permission denied
Do you have any idea how I could fix this?
BR, Christian
Describe the bug
A clear and concise description of what the bug is.
kubectl logs -f emqx-2
Defaulted container "emqx" out of: emqx, reloader
!!!!!!
WARNING: Default (insecure) Erlang cookie is in use.
Please set node.cookie in /opt/emqx/etc/emqx.conf or override from environment variable EMQX_NODE_COOKIE
NOTE: Use the same config value for all nodes in the cluster.
!!!!!!
Starting emqx on node [email protected]
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 8081 successfully.
Start http:dashboard listener on 18083 successfully.
EMQ X Broker 4.4.14 is running now!
Stop http:management listener on 0.0.0.0:8081 successfully.
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
rpc error: code = NotFound desc = an error occurred when try to find container "40ca01dce6a94173b829ba3655f0bdc8f896ccc5f3992ca4c80efc0d1aa
To Reproduce
Steps to reproduce the behavior:
emqx.yaml
apiVersion: apps.emqx.io/v1beta4
kind: EmqxBroker
metadata:
name: emqx
labels:
"apps.emqx.io/instance": "emqx"
spec:
replicas: 3
persistent:
spec:
storageClassName: local-path
resources:
requests:
storage: 100Mi
accessModes:
- ReadWriteOnce
template:
metadata:
labels:
"apps.emqx.io/instance": "emqx"
spec:
emqxContainer:
image:
repository: emqx/emqx
version: 4.4.14
emqxConfig:
name: emqx
cluster.discovery: dns
cluster.dns.type: srv
cluster.dns.app: emqx
cluster.dns.name: emqx-headless.default.svc.cluster.local
listener.tcp.external: "1883"
node.process_limit: "2097152"
node.max_ports: "1048576"
emqxACL:
- '{allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}.'
- '{allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}.'
- '{allow, all, subscribe, ["$SYS/brokers/+/clients/#"]}.'
- '{allow, all}.'
ports:
- containerPort: 8081
- containerPort: 18083
- containerPort: 1883
- containerPort: 11883
- containerPort: 8083
- containerPort: 8084
volumeMounts:
- name: emqx-http-vol
mountPath: /opt/emqx/etc/plugins/emqx_auth_http.conf
subPath: emqx_auth_http.conf
volumes:
- name: emqx-http-vol
configMap:
name: emqx-http
items:
- key: emqx_auth_http.conf
path: emqx_auth_http.conf
serviceTemplate:
metadata:
name: emqx
namespace: default
labels:
"apps.emqx.io/instance": "emqx"
spec:
type: LoadBalancer
selector:
"apps.emqx.io/instance": "emqx"
ports:
- name: "http-management-8081"
port: 8081
protocol: "TCP"
targetPort: 8081
- name: "http-dashboard-18083"
port: 18083
protocol: "TCP"
targetPort: 18083
- name: "mqtt-tcp-1883"
protocol: "TCP"
port: 1883
targetPort: 1883
Expected behavior
A clear and concise description of what you expected to happen.
$ kubectl logs -f emqx-2
Defaulted container "emqx" out of: emqx, reloader
!!!!!!
WARNING: Default (insecure) Erlang cookie is in use.
Please set node.cookie in /opt/emqx/etc/emqx.conf or override from environment variable EMQX_NODE_COOKIE
NOTE: Use the same config value for all nodes in the cluster.
!!!!!!
Starting emqx on node [email protected]
Start mqtt:tcp:external listener on 0.0.0.0:1883 successfully.
Start mqtt:ws:external listener on 0.0.0.0:8083 successfully.
Start mqtt:ssl:external listener on 0.0.0.0:8883 successfully.
Start mqtt:wss:external listener on 0.0.0.0:8084 successfully.
Start http:management listener on 8081 successfully.
Start http:dashboard listener on 18083 successfully.
EMQ X Broker 4.4.14 is running now!
Stop http:management listener on 0.0.0.0:8081 successfully.
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
rpc error: code = NotFound desc = an error occurred when try to find container "40ca01dce6a94173b829ba3655f0bdc8f896ccc5f3992ca4c80efc0d1aafdb22": not found
$ kubectl get EmqxBroker emqx -o json | jq '.status'
{
"conditions": [
{
"lastTransitionTime": "2023-02-16T06:55:40Z",
"lastUpdateTime": "2023-02-16T07:02:58Z",
"message": "All resources are ready",
"reason": "ClusterReady",
"status": "True",
"type": "Running"
}
],
"currentStatefulSetVersion": "emqx-974f94d58",
"emqxNodes": [
{
"node": "[email protected]",
"node_status": "Running",
"otp_release": "24.3.4.2/12.3.2.2",
"version": "4.4.14"
},
{
"node": "[email protected]",
"node_status": "Running",
"otp_release": "24.3.4.2/12.3.2.2",
"version": "4.4.14"
},
{
"node": "[email protected]",
"node_status": "Running",
"otp_release": "24.3.4.2/12.3.2.2",
"version": "4.4.14"
}
],
"readyReplicas": 3,
"replicas": 3
}
Anything else we need to know?:
Environment details::
Kubernetes version:
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9+k3s1", GitCommit:"4dcf0a33bc05159688ad4358aa6d5d4321a77277", GitTreeState:"clean", BuildDate:"2022-12-21T00:52:22Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9+k3s1", GitCommit:"4dcf0a33bc05159688ad4358aa6d5d4321a77277", GitTreeState:"clean", BuildDate:"2022-12-21T00:52:22Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"}
Cloud-provider/provisioner: none
emqx-operator version: apps.emqx.io/v1beta4
Install method: e.g. helm/static manifests
helm repo add emqx https://repos.emqx.io/charts
helm repo update
helm install emqx-operator emqx/emqx-operator --namespace emqx-operator-system --create-namespace
in v1beta sample:
spec:
serviceAccountName: emqx
image: emqx/emqx:4.3.11
replicas: 3
labels:
cluster: emqx
storage:
volumeClaimTemplate:
spec:
storageClassName: standard
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
in v1beta2 sample:
spec:
serviceAccountName: "emqx"
image: emqx/emqx:4.3.11
replicas: 3
labels:
cluster: emqx
storage:
storageClassName: standard
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteMany
Originally posted by burhanshaikh April 14, 2022
At the moment I am not able to specify Liveness and Readiness Probes in EmqxBroker Custom Resource.
I want to specify custom initialDelaySeconds and other fields under the probe.
It is not defined in the CRD, or am I missing something? If so can someone please share a sample CR with those fields?
Using chart version: emqx-operator-1.0.3 and app version: 1.1.5
After acl.conf is mounted to /mounted/acl/acl.conf by the operator, is the native ACL HTTP API unavailable?
At the same time, after changing the acl configuration by modifying the acl field under the spec, you need to reload the acl module before the corresponding rules take effect. is that so?
Originally posted by alexa22013 February 27, 2023
When I downscaled core nodes from 3 to 2 the repicant nodes were crashing
replicant node logs showed this error
[emqx-replicant-6d5577f77d-lx4: 2023-02-26T19:08:47.518993+00:00 [error] line: 182, mfa: mria_lb:list_core_nodes/1, msg: mria_lb_core_discovery divergent cluster, node: '[email protected]', previous_cores: [], returned_cores: ['[email protected]','[email protected]'], unknown_nodes: ['[email protected]']]
how to avoid this?
Also is there any practice on when to scale core node and when to scale replicants?
Is your feature request related to a problem? Please describe.
We are operating emqx broker together with a companion container for cloudsql-proxy.
Describe the solution you'd like
Because of this, it would be very useful to be able to specify additional containers in the EmqxBroker crd.
Describe alternatives you've considered
Can't think of an alternative atm.
Is your feature request related to a problem? Please describe.
There does not seem to be a way to use pre-existing certificates in kind: EmqxBroker
Describe the solution you'd like
Have an option to define an existing secret containing a TLS certificate for the secure ports
Describe alternatives you've considered
I can't find one right now in the information available on the kind: EmqxBroker. An other way of doing this would be to be able to specifiy an existing cluster issuer and dns names for automatic generation of the needed certs.
Related issue emqx/emqx#6337
Describe the bug
Following the deployment guide I got a mounting error by creating the pod:
Normal Scheduled 16s default-scheduler Successfully assigned infra/emqx-core-0 to aks-nodepool-36253939-vmss000001
Warning FailedMount 8s (x5 over 16s) kubelet MountVolume.MountDevice failed for volume "pvc-e529276e-79e8-4af7-8138-e40fe495adbd" : rpc error: code = Internal desc = volume(dev-rg#fb884d6e96a1f417587d381#pvcn-e529276e-79e8-4af7-8138-e40fe495adbd###infra) mount fb884d6e96a1f417587d381.file.core.windows.net:/fb884d6e96a1f417587d381/pvcn-e523276e-79e8-4af7-8138-e40fe495adbd on /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/c1362300e0665788580430e44151e8a12846b896d45d3f93c0847c17a67b5eee/globalmount failed with mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o nconnect=8,vers=4,minorversion=1,sec=sys fb884d6e96a1f417587d381.file.core.windows.net:/fb884d6e96a1f417587d381/pvcn-e523276e-79e8-4af7-8138-e40fe495adbd /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/c1362300e0665788580430e44151e8a12846b896d45d3f93c0847c17a67b5eee/globalmount
Output: mount.nfs: access denied by server while mounting fb884d6e96a1f417587d381.file.core.windows.net:/fb884d6e96a1f417587d381/pvcn-e523276e-79e8-4af7-8138-e40fe495adbd
Warning FailedMount 73s kubelet Unable to attach or mount volumes: unmounted volumes=[emqx-core-data], unattached volumes=[bootstrap-user bootstrap-config kube-api-access-9r6hs emqx-core-data]: timed out waiting for the condition
Environment details::
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: azurefile-csi-nfs
provisioner: file.csi.azure.com
allowVolumeExpansion: true
parameters:
protocol: nfs
mountOptions:
- nconnect=8
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
name: emqx
namespace: infra
spec:
image: "emqx/emqx:5.0.19"
imagePullPolicy: IfNotPresent
coreTemplate:
spec:
volumeClaimTemplates:
storageClassName: azurefile-csi-nfs
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
replicas: 1
replicantTemplate:
spec:
replicas: 3
dashboardServiceTemplate:
metadata:
name: emqx-dashboard
spec:
type: ClusterIP
selector:
apps.emqx.io/db-role: core
ports:
- name: "dashboard-listeners-http-bind"
protocol: TCP
port: 18083
targetPort: 18083
listenersServiceTemplate:
metadata:
name: emqx-listeners
spec:
type: ClusterIP
ports:
- name: "tcp-default"
protocol: TCP
port: 1883
targetPort: 1883
Describe the bug
The emqx operator reload the emqx_auth_mysql very frequently. When the message come to emqx when emqx_auth_mysql is reloading and the cache need to be refreshed, the message will failed because the hook is disappearing. Since the result will cache, so the client will failed until cache refreshed.
To Reproduce
Deploy the emqx by emqx operator , and auth by mysql.
Expected behavior
The message will not dropped.
Anything else we need to know?:
Broker EMQX with v4.4.3
EMQX operator with
Environment details::
Describe the bug
The installation of emqx-operator worked fine, but on try install emqx cluster the follow error happened:
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "mutating.broker.emqx.io": failed to call webhook: Post "https://emqx-operator-webhook-service.emqx-operator-system.svc:443/mutate-apps-emqx-io-v1beta4-emqxbroker?timeout=10s": context deadline exceeded
Output if emqx-operator is running:
$ kubectl get pods -l "control-plane=controller-manager" -n emqx-operator-system
NAME READY STATUS RESTARTS AGE
emqx-operator-controller-manager-6496bd4c9d-rg8hx 1/1 Running 0 158m
Below the codes tried to get emqx cluster running as described here: https://github.com/emqx/emqx-operator/blob/main/docs/en_US/deployment/on-gcp-gke.md
cat << "EOF" | kubectl apply -f -
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
name: emqx
spec:
image: "emqx:5.0"
coreTemplate:
spec:
volumeClaimTemplates:
storageClassName: standard
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
dashboardServiceTemplate:
spec:
type: LoadBalancer
listenersServiceTemplate:
spec:
type: LoadBalancer
EOF
cat << "EOF" | kubectl apply -f -
apiVersion: apps.emqx.io/v1beta4
kind: EmqxBroker
metadata:
name: emqx
spec:
persistent:
metadata:
name: emqx
spec:
storageClassName: standard
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
template:
spec:
emqxContainer:
image:
repository: emqx/emqx
version: 4.4.15
serviceTemplate:
spec:
type: LoadBalancer
EOF
We also tried different versions of emqx-operator.
To Reproduce
Theses documentations resulted on same error:
https://docs.emqx.com/en/emqx-operator/latest/getting-started/getting-started.html
https://github.com/emqx/emqx-operator/blob/main/docs/en_US/deployment/on-gcp-gke.md
Expected behavior
Emqx cluster ready using the emqx-operator.
Anything else we need to know?:
Environment details::
Describe the bug
When an instance of EMQX is removed, the relevant PVC is still lingering. That prevents an EMQX instance to be recreated as it runs into mounting problems.
To Reproduce
> kubectrl delete emqx emqx
> kubectrl get emqx
No resources found in infra namespace.
> kubectl get PVC
emqx-core-data-emqx-core-0 Bound pvc-03cd9d0f-010c-4fef-bdc2-edbfa4ff4f29 20Mi RWO azurefile-csi-nfs
Expected behavior
PVC should be removed together with the EMQX instance
Anything else we need to know?:
Environment details::
Describe the bug
I am trying to apply some pod annotations to the controller manager. My values contains
podAnnotations:
ad.datadoghq.com/manager.tags: |
{"env":"dev","service":"operator","version":"2.0.0"}
To Reproduce
I get an YAML parse error with version emqx-operator 2.0.0, helm v3.7.2, kubectl 1.22.0
Error: YAML parse error on emqx-operator/templates/controller-manager.yaml: error converting YAML to JSON: yaml: line 26: did not find expected key helm.go:88: [debug] error converting YAML to JSON: yaml: line 26: did not find expected key YAML parse error on emqx-operator/templates/controller-manager.yaml
It would appear that the indentation on the line https://github.com/emqx/emqx-operator/blob/main/deploy/charts/emqx-operator/templates/controller-manager.yaml#L32 is incorrect. The output produced is
template:
metadata:
annotations:
ad.datadoghq.com/haproxy.tags:
env: dev
service: operator
version: 2.0.0
I would expect
template:
metadata:
annotations:
ad.datadoghq.com/haproxy.tags:
env: dev
service: operator
version: 2.0.0
Describe the solution you'd like
We install emqx-operator by helm or yaml, but both have no method to setting the private registry for emqx-operator-reloader image.
Describe alternatives you've considered
the emqx-operator-reloader image's registry should be same to emqx-operator-controller's registry.
Describe the bug
After following the getting-started page to setup the emqx-operator I provisioned a emqx-cluster.
The pods start and are running, but the status commands return errors:
kubectl exec -n emqx -it emqx-0 -c emqx -- emqx_ctl status
Node '[email protected]' not responding to pings.
/opt/emqx/bin/emqx: line 46: die: command not found
command terminated with exit code 127
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Not to get errors on the status commands after provisioning a simple broker with no config.
Anything else we need to know?:
Environment details::
---
apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
name: emqx
labels:
app: emqx
environment: dev
spec:
persistent:
accessModes:
- ReadWriteOnce
storageClassName: ebs-gp3
resources:
requests:
storage: 1Gi
emqxTemplate:
image: emqx/emqx:4.4.6
Did I do something wrong here?
Describe the bug
I have a three node emqx-ee cluster that created by emqx-opeartor which 1.2.7 version;
When just one node running failed for some reason, for example, file direction permission, My whole emqx-ee cluster will be inaccessible, for the 1883\8883 ports will be disappear in the default service.
To Reproduce
Expected behavior
just part of node running failed, the emqx-ee cluster should be also accessible.
the service also should be like as the blow example:
Anything else we need to know?:
the latest emqx-operator version also has this same problem.
Environment details::
Describe the bug
There are two cases that failed when executing the make test
To Reproduce
Expected behavior
All cases should be passed
Anything else we need to know?:
Environment details::
The first time i run helm install everything goes smoothly, i.e. the CRD are installed and all the rest deployed.
when i run helm install on a different namespace, setting the value skipCRDs to true, i get an error of ownership in the cluster role ... how to fix?
emqx.yaml 完整配置在哪里能找到文档?
EMQX_HOST: $(POD_NAME).$(STS_HEADLESS_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
this is the operator generated
however my domain of k8s is k8s.com not cluster.local
Not sure if this is a bug but this is something I am trying to get through and not sure what I can do to get past the error.
I feel like I am close but I am not sure if I am just terminating wrong with the ingress controller and what I am trying is not possible. Or, the certificate is in the way and I need to do something to alleviate the issue.
Another idea I see is using a HAPROXY LB in front of the ingress-nginx controller.
First, I am using the ingress Nginx tcp/udp controller for Kubernetes described here with an Letsencrypt RA CA
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
This allows me through my HELM
installation to create a tcp config and mapping.
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $NAMESPACE --set controller.service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path=/healthz,controller.service.annotations.service\.beta\.kubernetes\.io/azure-dns-label-name=$DNS_LABEL --set controller.service.loadBalancerIP=$STATIC_IP --set tcp.18083=$NAMESPACE/emqx-ee:18083,tcp.8883=$NAMESPACE/emqx-ee:8883
That does allow for an mqtts://
ssl connection to go through all the way to the 8883
backend pod. The issue is when I want to terminate at the loadbalancer and send the resulting incoming traffic to terminate the TLS and go to the TCP 1883 port.
To try this I change the port tcp definition
from this:
'8883': ingress-emqx/emqx-ee:8883
to this:
'8883': ingress-emqx/emqx-ee:1883
When I connect the client mqttx
to 8883 CA I get the resulting error:
Error: Client network socket disconnected before secure TLS connection was established
What exactly does the termination for ingress-nginx? Is it only the ingress instruction rule or is it the controller TCP Proxy Protocol?
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
https://kubernetes.github.io/ingress-nginx/examples/tls-termination/
Is the only way to terminate is to utilize the ingress instruction/rewrite rule? If so, I think this is where the controller won't work because the route ingress rule only works with http and won't act upon mqtt layer-7 protocol.
Is there a way for the ingress to align with the ingress-controller service? I'm not sure it even matters to the previous issue. I simply want to take the incoming 8883 port and switch it to the resulting 1883 port and IP.
Describe the bug
When I try to deploy a v5.0.6 emqx image the resulting statefulset does not properly create emqx cluster.
It is like the operator v1.2.5 is not designed to operate emqx v5 broker images. So my question: is this a bug or it is just not designed for v5 yet ?
To Reproduce
deploy something like:
apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
name: emqx
namespace: emqx-operator-system
spec:
replicas: 3
emqxTemplate:
image: emqx/emqx:5.0.6
serviceTemplate:
metadata:
name: emqx
labels:
app: emqx
spec:
selector:
app: emqx
type: ClusterIP
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: 1883
- name: mqttssl
port: 8883
protocol: TCP
targetPort: 8883
- name: dashboard
port: 18083
protocol: TCP
targetPort: 18083
config:
prometheus.enable: "true"
modules:
- name: emqx_mod_trace
enable: true
persistent:
resources:
requests:
storage: 2Gi
accessModes:
- ReadWriteOnce
After the broker starts I have some warnings in logs and autocluster is not working. Every broker node has a cluster size of 1 -> they don't form a cluster.
emqx [warning] unknown_env_vars: ["EMQX_CLUSTER__DISCOVERY","EMQX_CLUSTER__DNS__APP",
emqx "EMQX_CLUSTER__DNS__TYPE",
emqx "EMQX_DASHBOARD__DEFAULT_USER__LOGIN",
emqx "EMQX_DASHBOARD__DEFAULT_USER__PASSWORD",
emqx "EMQX_PLUGINS__ETC_DIR","EMQX_PLUGINS__LOADED_FILE"]
emqx prometheus.enable = EMQX_PROMETHEUS__ENABLE = true
emqx rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual
emqx log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false
emqx log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true
emqx cluster.dns.name = EMQX_CLUSTER__DNS__NAME = emqx-headless.emqx-operator-system.svc.cluster.local
emqx node.name = EMQX_NODE__NAME = [email protected]
emqx 2022-08-28T10:28:07.426244+00:00 [warning] unknown_env_vars: ["EMQX_CLUSTER__DISCOVERY","EMQX_CLUSTER__DNS__APP", "EMQX_CLUSTER__DNS__TYPE", "EMQX_DASHBOARD__D
emqx Listener ssl:default on :8883 started.
emqx Listener tcp:default on :1883 started.
emqx Listener ws:default on :8083 started.
emqx Listener wss:default on :8084 started.
emqx Listener http:dashboard on :18083 started.
emqx EMQX 5.0.6 is running now!
Expected behavior
The broker starts and the number of nodes in the cluster via emqx dashboard is 3.
Anything else we need to know?:
It's like the operator v1.2.5 is not configuring proper env variables for broker v5. That's why logs show some warnings.
Environment details::
Describe the bug
when use helm uninstall the emqx-operator release, the emqxplugins.apps.emqx.io crd is not remove.
To Reproduce
Steps to reproduce the behavior:
see the picture above。
Expected behavior
all the crds removed, when use helm uninstall the emqx-operator release.
Anything else we need to know?:
Environment details::
~ % kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
% helm version version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"dirty", GoVersion:"go1.16.6"}
It seems that spec.coreTemplate.spec.extraContainers field was lost in the last update (2.1.2).
To Reproduce
Expected behavior
The extra container should be present even after the upgrade.
Anything else we need to know?:
Environment details::
Describe the bug
emqx-operator-controller-manager pod keep crashing with the below logs after deploy the emqx opensource broker to cluster. here we use kustomize.
Logs:
I1103 14:44:33.965126 1 request.go:601] Waited for 1.003207335s due to client-side throttling, not priority and fairness, request: GET:https://172.0.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s
65
2022-11-03T14:44:35Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
64
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
63
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
62
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
61
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
60
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
59
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
58
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
57
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
56
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
55
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
54
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
53
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
52
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
51
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
50
2022-11-03T14:44:35Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/validate-apps-emqx-io-v2alpha1-emqx"}
49
2022-11-03T14:44:35Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v2alpha1-emqx"}
48
2022-11-03T14:44:35Z INFO setup starting manager
47
2022-11-03T14:44:35Z INFO controller-runtime.webhook.webhooks Starting webhook server
46
2022-11-03T14:44:35Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
45
2022-11-03T14:44:35Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
44
2022-11-03T14:44:35Z INFO controller-runtime.certwatcher Updated current TLS certificate
43
I1103 14:44:35.824603 1 leaderelection.go:248] attempting to acquire leader lease emqx/19fd6fcc.emqx.io...
42
2022-11-03T14:44:35Z INFO controller-runtime.certwatcher Starting certificate watcher
41
2022-11-03T14:44:35Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
40
I1103 14:45:07.039257 1 leaderelection.go:258] successfully acquired lease emqx/19fd6fcc.emqx.io
39
2022-11-03T14:45:07Z INFO Starting EventSource {"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin", "source": "kind source: *v1beta3.EmqxPlugin"}
38
2022-11-03T14:45:07Z INFO Starting EventSource {"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "source": "kind source: *v1beta3.EmqxBroker"}
37
2022-11-03T14:45:07Z INFO Starting Controller {"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker"}
36
2022-11-03T14:45:07Z INFO Starting EventSource {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "source": "kind source: *v2alpha1.EMQX"}
35
2022-11-03T14:45:07Z INFO Starting Controller {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX"}
34
2022-11-03T14:45:07Z INFO Starting Controller {"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin"}
33
2022-11-03T14:45:07Z INFO Starting EventSource {"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise", "source": "kind source: *v1beta3.EmqxEnterprise"}
32
2022-11-03T14:45:07Z INFO Starting Controller {"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise"}
31
2022-11-03T14:45:07Z INFO Starting workers {"controller": "emqxplugin", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxPlugin", "worker count": 1}
30
2022-11-03T14:45:07Z INFO Starting workers {"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "worker count": 1}
29
2022-11-03T14:45:07Z INFO Starting workers {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "worker count": 1}
28
2022-11-03T14:45:07Z INFO Starting workers {"controller": "emqxenterprise", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxEnterprise", "worker count": 1}
27
2022-11-03T14:45:07Z INFO Observed a panic in reconciler: runtime error: index out of range [0] with length 0 {"controller": "emqx", "controllerGroup": "apps.emqx.io", "controllerKind": "EMQX", "eMQX": {"name":"mycluster1-emqx","namespace":"emqx"}, "namespace": "emqx", "name": "mycluster1-emqx", "reconcileID": "7b73e597-435a-45b0-b9e5-57fe41c34757"}
26
panic: runtime error: index out of range [0] with length 0 [recovered]
25
panic: runtime error: index out of range [0] with length 0
24
23
goroutine 492 [running]:
22
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
21
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0x1f4
20
panic({0x1847520, 0xc001c7a030})
19
/usr/local/go/src/runtime/panic.go:838 +0x207
18
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*coreUpdateStatus).nextStatus(0xc0003240f0, 0xc00015ca00, 0xc00011a800?)
17
/workspace/controllers/apps/v2alpha1/status_machine.go:163 +0x2c5
16
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*emqxStatusMachine).NextStatus(...)
15
/workspace/controllers/apps/v2alpha1/status_machine.go:111
14
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).updateStatus(0xc000601c00, 0xc00011a800)
13
/workspace/controllers/apps/v2alpha1/emqx_controller.go:160 +0x2f0
12
github.com/emqx/emqx-operator/controllers/apps/v2alpha1.(*EMQXReconciler).Reconcile(0xc000601c00, {0x1ba82c8, 0xc00086d020}, {{{0xc000404630?, 0x10?}, {0xc000404620?, 0x40d7e7?}}})
11
/workspace/controllers/apps/v2alpha1/emqx_controller.go:75 +0x134
10
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1ba8220?, {0x1ba82c8?, 0xc00086d020?}, {{{0xc000404630?, 0x1878040?}, {0xc000404620?, 0x4041f4?}}})
9
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xc8
8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001f1680, {0x1ba8220, 0xc000932580}, {0x179bae0?, 0xc000529380?})
7
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320 +0x33c
6
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001f1680, {0x1ba8220, 0xc000932580})
5
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x1d9
4
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
3
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85
2
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230 +0x325
Environment details::
Describe the bug
emqx
Service
doesn't use those nodePorts
specified in EmqxBroker
object.
To Reproduce
Steps to reproduce the behavior:
With custom object:
apiVersion: apps.emqx.io/v1beta2
kind: EmqxBroker
metadata:
name: emqx
spec:
serviceAccountName: "emqx"
image: emqx/emqx:4.3.11
replicas: 3
labels:
cluster: emqx
storage:
storageClassName: emqx
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
# env:
# - name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
# value: admin
# - name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
# value: ?
emqxTemplate:
listener:
type: NodePort
ports:
mqtt: 1883
mqtts: 8883
ws: 8083
wss: 8084
dashboard: 18083
api: 8081
nodePorts:
mqtt: 30038
mqtts: 30039
ws: 30040
wss: 30041
dashboard: 30042
api: 30043
acl:
- permission: allow
username: "dashboard"
action: subscribe
topics:
filter:
- "$SYS/#"
- "#"
- permission: allow
ipaddress: "127.0.0.1"
topics:
filter:
- "$SYS/#"
equal:
- "#"
- permission: deny
action: subscribe
topics:
filter:
- "$SYS/#"
equal:
- "#"
- permission: allow
plugins:
- name: emqx_management
enable: true
- name: emqx_recon
enable: true
- name: emqx_retainer
enable: true
- name: emqx_dashboard
enable: true
- name: emqx_telemetry
enable: true
- name: emqx_rule_engine
enable: true
- name: emqx_bridge_mqtt
enable: false
modules:
- name: emqx_mod_acl_internal
enable: true
- name: emqx_mod_presence
enable: true
Expected behavior
emqx
Service
should use nodePorts
(30038 30039 etc.) specified in EmqxBroker
object, but results in random allocated:
Anything else we need to know?:
Environment details::
Describe the bug
The original emqx-operator-webhook-service has the following config:
spec:
selector: │
control-plane: controller-manager
which is too vague. On my k8s cluster, with additional operators installed this service selector targets also other operator pods causing problems when the request goes to other operators.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The service selector for the webhook service should be more restrictive, for example:
spec:
selector: │
control-plane: controller-manager
app.kubernetes.io/name: emqx-operator
Environment details::
Hi,
How can I add extra ENV in the EmqxBroker yaml config?
So I can enable the mysql auth plugin and setup the right settings.
Should I do that with ENV or is there another way to add the configs related to the plugin?
@Rory-Z can you maybe help me out with a sample config where the emqx mysql auth plugin is added? I tried to get the env somehow in the manifest but I doesn't get it to work.
Describe the bug
Operator doesn't have correctly configured service and/or containerPort, consequently metrics endpoint doesn't work.
To Reproduce
kubectl apply -f https://github.com/emqx/emqx-operator/releases/download/1.1.5/emqx-operator-controller.yaml
emqx-operator-controller-manager-metrics-service
has one port (https/8443), this port points to targetPort https
on emqx-operator-controller-manager-...
pod. The container in the pod doesn't have any such containerPort
, because of that Prometheus metrics aren't usable.
Expected behavior
The pod should have containerPort: 8080
exposed (according to pod logs, this is where /metrics
actually is) and service should be adjusted accordingly (point to this port instead of 8443).
Environment details::
1.23
k3s
1.1.4
static manifest
Is your feature request related to a problem? Please describe.
v1/beat2 支持加载插件吗?
apiVersion: apps.emqx.io/v1beta3
kind: EmqxPlugin
metadata:
name: emqx-lwm2m
namespace: default
spec:
selector:
"foo": "bar"
pluginName: emqx_lwm2m
config:
"lwm2m.lifetime_min": "1s"
"lwm2m.lifetime_max": "86400s"
"lwm2m.bind.udp.1": "0.0.0.0:5683"
"lwm2m.bind.udp.2": "0.0.0.0:5684"
"lwm2m.bind.udp.3": "0.0.0.0:5685"
"lwm2m.xml_dir": "/opt/emqx/etc/lwm2m_xml"
if so, then which params of the CRD should be set ?
TODO:
.spec.lables
, should use meta.labels
.spec.annotations
, should use meta.annotations
.spec.license
supports base64 encoding, and support external secret.spec.serviceAccount
, when use emqx 4.4.x, serviceAccount is not requiredIf I need to change a setting inside bootstrapConfig using EMQX Operator I need to delete the cluster and create it again.
Apart from this really annoying behavior, if I enable persistence for core node, the cluster did not get created anymore, only core node is deployed and the operator stuck on replicant creation.
To start it again I need to delete the persistent volume loosing all the settings.
Change to bootstrapConfig should be allowed and StatefulSet/Deployment should be re-deployed.
Create a cluster with 1 core and 3 replicant. Core should have a persistent volume enabled.
Delete the cluster (leaving the volume), change any setting, re-deploy the cluster, only core node get deployed.
Replicant and listeners service are not re-created.
No response
$ ./bin/emqx_ctl broker
sysdescr : EMQX
version : 5.0.13
datetime : 2023-01-02T14:38:25.510136161+00:00
uptime : 6 minutes, 56 seconds
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
Linux emqx-core-0 5.4.172-90.336.amzn2.x86_64 emqx/emqx#1 SMP Wed Jan 19 23:08:01 UTC 2022 x86_64 GNU/Linux
Describe the bug
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6530916833830116e+09 ERROR controller.emqxbroker Reconciler error {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxBroker", "name": "test-emqx", "namespace": "default", "error": "S
tatefulSet.apps \"test-emqx\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Anything else we need to know?:
Environment details::
Describe the bug
Dash board reports - some error occoured
Logs show:
2022-09-12 11:41:09.622 BST2022-09-12T10:41:09.621898+00:00 [warning] exception: error, line: 116, mfa: minirest_handler:apply_callback/3, path: /listeners, reason: {badmatch,{error,{'EXIT',{{badkey,<<"listeners">>},[{erlang,map_get,[<<"listeners">>,#{}],[{error_info,#{module => erl_erts_errors}}]},{emqx_config,get_schema_mod,1,[{file,"emqx_config.erl"},{line,505}]},{emqx_listeners,do_list_raw,0,[{file,"emqx_listeners.erl"},{line,92}]},{emqx_listeners,list_raw,0,[{file,"emqx_listeners.erl"},{line,71}]},{emqx_mgmt_api_listeners,do_list_listeners,0,[{file,"emqx_mgmt_api_listeners.erl"},{line,519}]}]}}}}, stacktrace: [{emqx_mgmt_api_listeners,listener_status_by_id,2,[{file,"emqx_mgmt_api_listeners.erl"},{line,496}]},{emqx_mgmt_api_listeners,listener_status_by_id,1,[{file,"emqx_mgmt_api_listeners.erl"},{line,472}]},{emqx_mgmt_api_listeners,list_listeners,2,[{file,"emqx_mgmt_api_listeners.erl"},{line,342}]},{minirest_handler,apply_callback,3,[{file,"minirest_handler.erl"},{line,111}]},{minirest_handler,handle,2,[{file,"minirest_handler.erl"},{line,44}]},{minirest_handler,init,2,[{file,"minirest_handler.erl"},{line,27}]},{cowboy_handler,execute,2,[{file,"cowboy_handler.erl"},{line,41}]},{cowboy_stream_h,execute,3,[{file,"cowboy_stream_h.erl"},{line,318}]},{cowboy_stream_h,request_process,3,[{file,"cowboy_stream_h.erl"},{line,302}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]
To Reproduce
Followed installation steps in Google K8s - GKE
Attepted to portforward to the dashboard and login with admin / public
Expected behavior
To be able to login.
Anything else we need to know?:
Environment details::
1.22.11-gke.400
Google Cloud
V5
https://github.com/emqx/emqx-operator/blob/main/docs/en_US/getting-started/getting-started.md
Describe the bug
I installed emqx-operator
with helmfile in namespace emqx-operator
(instead of emqx-operator-system
as described in README). I configured EMQX
crd resource and while installing I got following error:
COMBINED OUTPUT:
Release "emqx" does not exist. Installing it now.
Error: release emqx failed, and has been uninstalled due to atomic being set: Internal error occurred: failed calling webhook "mutating.apps.emqx.io": failed to call webhook: Post "https://emqx-operator-webhook-service.emqx-operator-system.svc:443/mutate-apps-emqx-io-v2alpha1-emqx?timeout=10s": service "emqx-operator-webhook-service" not found
To Reproduce
Steps to reproduce the behavior:
emqx-operator
namespace
$ helm install emqx-operator emqx/emqx-operator \
--namespace emqx-operator\
--create-namespace
cat << "EOF" | kubectl apply -f -
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
name: emqx
spec:
image: emqx/emqx:5.0.6
EOF
Expected behavior
Looks like the namespace of emqx-operator
is hardcoded somewhere. Changing namespace should not break the instalation.
Anything else we need to know?:
Should get the namespace of emqx-operator within the helm chart with - .Values.Release.Namespace
Environment details::
Originally posted by mariusstaicu November 21, 2022
When using bootstrapConfig one cannot reapply the yaml and update other portions of it because of the validation error:
admission webhook "validator.apps.emqx.io" denied the request: bootstrap config cannot be updated, old bootstrap config:
How to update the EQMX resource without this validation failure?
The EMQX crd has issues with CD systems like FluxCD or ArgoCD that do server-side apply dry-runs to validate changes for reconciliation.
All potential fields on the CRD are on the schema so that server side dry-runs succeed with zero changes (or configured to ignore said changes)
Apply a broker crd:
kubectl apply --server-side -f broker.yaml
then run:
kubectl apply --server-side -f broker.yaml --dry-run=server
The response is likely to be:
Error from server: .spec.emqxTemplate.serviceTemplate.metadata.creationTimestamp: field not declared in schema
Warning ReconciliationFailed 91s (x3 over 11m) kustomize-controller EmqxBroker/mqtt/emqx dry-run failed, error: .spec.emqxTemplate.serviceTemplate.metadata.creationTimestamp: field not declared in schema
This is the message directly from Flux, but the command in the reproduction steps is all that FluxCD is doing behind the covers.
bash-5.1$ emqx_ctl broker
sysdescr : EMQ X Broker
version : 4.4.0
uptime : 17 minutes, 8 seconds
datetime : 2022-07-29 00:18:05
# On Linux:
bash-5.1$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.3
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
$ uname -a
Linux emqx-0 5.15.45-1-virt #2-Alpine SMP Mon, 06 Jun 2022 10:19:37 +0000 x86_64 Linux
Note this is just a worker node in my K3s kubernetes cluster.
Hello all,
I would like to hear your suggestion about EMQX users migration.
We are planning to upgrade emqx version and use it with operator.
What is the best way of migrating existing users from emqx v4 to v5. We are going to use emqx with operator.
I would like to hear your suggestions.
Thank you.
Describe the bug
I'm getting the following error when attempting deploy an EmqxBroker
:
1.647123650787955e+09 ERROR controller.emqxbroker Reconciler error {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxBroker", "name": "emqx2", "namespace": "home-iot", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
The service does not get created, nor does the statefulset. I do see the headless service get created, but no ports are on it.
To Reproduce
Here's the resource I am applying (note: I'm using Flux CD which replaces things like ${SECRET_CLUSTER_DOMAIN}
):
---
apiVersion: apps.emqx.io/v1beta2
kind: EmqxBroker
metadata:
name: &name emqx2
namespace: home-iot
spec:
image: emqx/emqx:4.4.0
replicas: 3
labels:
cluster: *name
env:
- name: EMQX_ALLOW_ANONYMOUS
value: 'true'
- name: EMQX_ADMIN_PASSWORD
value: "${SECRET_EMQX_ADMIN_PASSWORD}"
- name: EMQX_AUTH__MNESIA__PASSWORD_HASH
value: sha256
- name: EMQX_AUTH__USER__1__USERNAME
value: "${SECRET_MQTT_USERNAME}"
- name: EMQX_AUTH__USER__1__PASSWORD
value: "${SECRET_MQTT_PASSWORD}"
storage:
storageClassName: rook-ceph-block
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
emqxTemplate:
listener:
type: LoadBalancer
loadBalancerIP: 192.168.69.61
ports:
mqtt: 1883
mqtts: 8883
ws: 8083
wss: 8084
dashboard: 18083
api: 8081
annotations:
external-dns.alpha.kubernetes.io/hostname: "mqtt2.${SECRET_CLUSTER_DOMAIN}"
coredns.io/hostname: "mqtt2.${SECRET_CLUSTER_DOMAIN}"
acl:
- permission: allow
username: "dashboard"
action: subscribe
topics:
filter:
- "$SYS/#"
- "#"
- permission: allow
ipaddress: "127.0.0.1"
topics:
filter:
- "$SYS/#"
equal:
- "#"
- permission: deny
action: subscribe
topics:
filter:
- "$SYS/#"
equal:
- "#"
- permission: allow
plugins:
- name: emqx_management
enable: true
- name: emqx_recon
enable: true
- name: emqx_retainer
enable: true
- name: emqx_dashboard
enable: true
- name: emqx_telemetry
enable: true
- name: emqx_rule_engine
enable: true
- name: emqx_bridge_mqtt
enable: false
modules:
- name: emqx_mod_acl_internal
enable: true
- name: emqx_mod_presence
enable: true
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: apps.emqx.io/instance
operator: In
values:
- *name
topologyKey: kubernetes.io/hostname
Expected behavior
Broker and service gets created.
Anything else we need to know?:
Full logs:
1.6471236501591592e+09 INFO emqx-controller Reconciling {"Request.Namespace": "home-iot", "Request.Name": "emqx2"}
1.6471236503776915e+09 ERROR emqx-controller crate resource failed {"groupVersionKind": "/v1, Kind=Service", "namespace": "home-iot", "name": "emqx2", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
github.com/emqx/emqx-operator/controllers/apps.(*Handler).createOrUpdate
/workspace/controllers/apps/ensure.go:54
github.com/emqx/emqx-operator/controllers/apps.(*Handler).Ensure
/workspace/controllers/apps/ensure.go:25
github.com/emqx/emqx-operator/controllers/apps.(*Handler).Do
/workspace/controllers/apps/handler.go:53
github.com/emqx/emqx-operator/controllers/apps.(*Handler).Reconcile
/workspace/controllers/apps/reconcile.go:99
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6471236503984983e+09 ERROR emqx-controller Reconcile handler {"Request.Namespace": "home-iot", "Request.Name": "emqx2", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
1.6471236503985639e+09 ERROR controller.emqxbroker Reconciler error {"reconciler group": "apps.emqx.io", "reconciler kind": "EmqxBroker", "name": "emqx2", "namespace": "home-iot", "error": "Service \"emqx2\" is invalid: spec.ports: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Environment details::
Describe the bug
Emqx core node fails to start on k8s after deletion and recreation with persistent volume. The goal was to deploy a broker that preserves the settings (dashboard users, mqtt clients auth & authz) even if the EMQX respource is deleted.
To Reproduce
`1. Install emqx-operator via Helm in k8s
2. Deploy a basic emqx broker with volumeClaimTemplate
4. Wait for it to start, change login via dashboard and change password.l
5. delete EMQX resource and recreate it
One of three core nodes doesn't start and is in error.
Expected behavior
All the pods start, both core and replicants nodes.
Anything else we need to know?:
emqx yaml file:
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
name: emqx
namespace: emqx-operator-system
spec:
image: emqx/emqx:5.0.8
coreTemplate:
metadata:
name: emqx-core
labels:
apps.emqx.io/instance: emqx
apps.emqx.io/db-role: core
spec:
replicas: 3
volumeClaimTemplates:
storageClassName: longhorn-single-replica
resources:
requests:
storage: 512Mi
accessModes:
- ReadWriteOnce
livenessProbe:
httpGet:
path: /status
port: 18083
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 3
readinessProbe:
httpGet:
path: /status
port: 18083
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 12
podSecurityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
fsGroupChangePolicy: Always
containerSecurityContext:
runAsUser: 1000
runAsGroup: 1000
lifecycle:
preStop:
exec:
command: [ "/bin/sh","-c","emqx ctl cluster leave" ]
Environment details::
Logs:
dashboard.bootstrap_users_file = EMQX_DASHBOARD__BOOTSTRAP_USERS_FILE = "/opt/emqx/data/bootstrap_user"
rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual
log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false
log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true
node.db_role = EMQX_NODE__DB_ROLE = core
node.name = EMQX_NODE__NAME = emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local
2022-10-18T11:03:36.662853+00:00 [error] Mnesia('emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned
_network, 'emqx@emqx-core-1.emqx-headless.emqx-operator-system.svc.cluster.local'}
2022-10-18T11:03:36.663234+00:00 [error] Mnesia('emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned
_network, 'emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local'}
2022-10-18T11:03:36.854583+00:00 [error] Mnesia('emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'): ** ERROR ** (core dumped to file: "/opt/emqx/MnesiaCore.emqx@emqx-core-0.
emqx-headless.emqx-operator-system.svc.cluster.local_1666_91016_807092"), ** FATAL ** Failed to merge schema: Incompatible schema cookies. Please, restart from old backup.'[email protected]
eadless.emqx-operator-system.svc.cluster.local' = [{name,schema},{type,set},{ram_copies,[]},{disc_copies,['emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local','emqx@emqx-core-1
.emqx-headless.emqx-operator-system.svc.cluster.local']},{disc_only_copies,[]},{load_order,0},{access_mode,read_write},{majority,false},{index,[]},{snmp,[]},{local_content,false},{record_name,sch
ema},{attributes,[table,cstruct]},{user_properties,[{mnesia_backend_types,[{rocksdb_copies,mnesia_rocksdb},{null_copies,mria_mnesia_null_storage}]}]},{frag_properties,[]},{storage_properties,[]},
{cookie,{{1666087950880376159,-576460752303423101,1},'emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local'}},{version,{{7,0},{'[email protected]
m.svc.cluster.local',{1666,88190,221401}}}}], 'emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local' = [{name,schema},{type,set},{ram_copies,[]},{disc_copies,['[email protected]
mqx-headless.emqx-operator-system.svc.cluster.local','emqx@emqx-core-1.emqx-headless.emqx-operator-system.svc.cluster.local','emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local
']},{disc_only_copies,[]},{load_order,0},{access_mode,read_write},{majority,false},{index,[]},{snmp,[]},{local_content,false},{record_name,schema},{attributes,[table,cstruct]},{user_properties,[{
mnesia_backend_types,[{rocksdb_copies,mnesia_rocksdb},{null_copies,mria_mnesia_null_storage}]}]},{frag_properties,[]},{storage_properties,[]},{cookie,{{1666086344331374145,-576460752303423202,1},
'emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'}},{version,{{8,0},{'emqx@emqx-core-2.emqx-headless.emqx-operator-system.svc.cluster.local',{1666,86350,531401}}}}]
2022-10-18T11:03:46.809511+00:00 [error] Generic server mnesia_subscr terminating. Reason: killed. Last message: {'EXIT',<0.1830.0>,killed}. State: {state,<0.1830.0>,#Ref<0.1684304479.1029832724.
194029>}.
2022-10-18T11:03:46.809864+00:00 [error] Generic server mnesia_monitor terminating. Reason: killed. Last message: {'EXIT',<0.1830.0>,killed}. State: {state,<0.1830.0>,[],[],true,[],undefined,[],[
]}.
2022-10-18T11:03:46.809510+00:00 [error] Generic server mnesia_recover terminating. Reason: killed. Last message: {'EXIT',<0.1830.0>,killed}. State: {state,<0.1830.0>,undefined,undefined,undefine
d,0,false,true,[]}.
2022-10-18T11:03:46.811065+00:00 [error] crasher: initial call: mnesia_subscr:init/1, pid: <0.1832.0>, registered_name: mnesia_subscr, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.e
rl"},{line,481}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.1826.0>], message_queue_len: 0, messages: [], links: [], dictionar
y: [], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 6297; neighbours:
2022-10-18T11:03:46.810244+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.1825.0>, registered_name: [], exit: {{normal,{mnesia_app,start,[normal,]]}},[{application_master
,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.1824.0>], message_queue_len: 1, messages: [{'EXIT',<0.1826.
0>,normal}], links: [<0.1824.0>,<0.1685.0>], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 29, reductions: 167; neighbours:
2022-10-18T11:03:46.811386+00:00 [error] crasher: initial call: gen_event:init_it/6, pid: <0.1828.0>, registered_name: mnesia_event, exit: {killed,[{gen_event,terminate_server,4,[{file,"gen_event
.erl"},{line,405}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [mnesia_sup,<0.1826.0>], message_queue_len: 1, messages: [{notify,{mnesia_system_event,{mnesia_do
wn,'emqx@emqx-core-0.emqx-headless.emqx-operator-system.svc.cluster.local'}}}], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 10958, stack_size: 29, reductions: 27853; n
eighbours:
2022-10-18T11:03:46.813591+00:00 [error] crasher: initial call: mnesia_monitor:init/1, pid: <0.1831.0>, registered_name: mnesia_monitor, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server
.erl"},{line,481}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.1826.0>], message_queue_len: 0, messages: [], links: [<53724.183
1.0>,<53725.3553.0>,<0.1873.0>], dictionary: [], trap_exit: true, status: running, heap_size: 6772, stack_size: 29, reductions: 13705; neighbours:
2022-10-18T11:03:46.813855+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.1816.0>, registered_name: [], exit: {{bad_return,{{mria_app,start,[normal,]]}
Describe the bug
I have configured EMQX
as following
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
name: emqx
spec:
image: emqx/emqx:5.0.6
coreTemplate:
metadata:
name: emqx-core
labels:
apps.emqx.io/instance: emqx
apps.emqx.io/db-role: core
annotations:
spec:
replicas: 1
volumeClaimTemplates:
resources:
requests:
storage: 1Gi
accessModes:
- ReadWriteOnce
ports:
- containerPort: 1883
envFrom:
- prefix: EMQX_
configMapRef:
name: emqx-config
- prefix: EMQX_AUTHENTICATION__0__
secretRef:
name: emqx-pgsql-connection
- prefix: EMQX_AUTHORIZATION__SOURCES__0__
secretRef:
name: emqx-pgsql-connection
resources:
limits:
# cpu: 100m
memory: 256Mi
# requests:
# cpu: 100m
# memory: 256Mi
dashboardServiceTemplate:
metadata:
name: emqx-dashboard
spec:
selector:
apps.emqx.io/db-role: core
ports:
- name: "dashboard-listeners-http-bind"
protocol: TCP
port: 18083
targetPort: 18083
listenersServiceTemplate:
metadata:
name: emqx-listeners
spec:
ports:
- name: "tcp-default"
protocol: TCP
port: 1883
targetPort: 1883
When the emqx-core-0
starts it crashes with following error:
mkdir: cannot create directory ‘/opt/emqx/data/configs’: Permission denied │
│ Stream closed EOF for my-namespace/emqx-core-0 (emqx)
To Reproduce
Steps to reproduce the behavior:
I just uncommented volumeClaimTemplates
section under coreTemplate
[also removed hardcoded storageClassName
so that the default is selected] - removed replicantTemplate
section, as not required - removed commented code
Expected behavior
This error should not come when using "sample" config values.
Not sure, how to fix this - looks like the pod has no write permission on volume mounted - either it controller has bug or config is wrong.
Environment details::
Hello there!
Thanks for the great operator software.
I'm trying to deploy this operator and it's not possible to start reload sidecars on raspberry pi nodes as version is hardcoded to emqx-operator-reloader:0.0.1 one - which compiled without arm support.
Is there any way to define this package version somewhere in the operator config?
Hello,
I am trying to get the operator running on an openshift installation but I face an TLS issue between Kube Api Server Operator and webhooks back to the EMQX Operator (or so it seems)
I installed the operator using the provided Helmchart, but disabled cert-manager support (setting it to false in values.yaml), because Openshift comes with similar functionality.
In particular, I created a TLS-Secret for the webhook-server by annotating its Service (the one that the helm chart creates) with
annotations:
service.beta.openshift.io/serving-cert-secret-name: emqx-operator-webhook-server-cert
which triggers openshifts certificate system to create a secret of the specified name and populate it with a tls.crt (certificate chain) and a tls.key (private key).
Also on the webhook side, the ca-bundle can be injected by annotating the validating webhook resource and also the mutating webhook resource (both webhook resources that have been created by helm) with
annotations:
service.beta.openshift.io/inject-cabundle: 'true'
which populates the webhooks[].clientConfig.caBundle field with the ca cert that corresponds to the above service cert.
(I of course realize, that upon upgrading the helmchart, I will have to do that again, unless the helm chart could support an openshift mode instead of cert-manager)
However I keep getting errors on both sides:
On kube-apiserver-operator side I get many errors like
E1005 08:24:39.214279 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
E1005 08:24:40.222487 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
E1005 08:24:42.241517 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
E1005 08:24:43.248170 1 degraded_webhook.go:128] x509: certificate signed by unknown authority
and on the emqx operator I keep getting
2022/10/05 08:28:37 http: TLS handshake error from 10.130.5.86:40936: remote error: tls: bad certificate
2022/10/05 08:28:38 http: TLS handshake error from 10.130.5.86:40952: remote error: tls: bad certificate
2022/10/05 08:28:39 http: TLS handshake error from 10.130.5.86:40968: remote error: tls: bad certificate
2022/10/05 08:28:41 http: TLS handshake error from 10.130.5.86:39222: remote error: tls: bad certificate
where 10.130.5.86 is the kupe-apiserver-operator pod
I also tried getting the webhook-server certificate by using openssl tool to connect to it via the service and a port-forward. This seem to giving me the expected response.
Now upon creation of a EMQX ressource, the emqx operator pod starts crashlooping, without any apparent error in its log.
I1005 09:07:21.179399 1 request.go:601] Waited for 1.003849249s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/org.eclipse.che/v2?timeout=32s
2022-10-05T09:07:23Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxPlugin", "path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxplugin"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxBroker", "path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxbroker"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v1beta3, Kind=EmqxEnterprise", "path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v1beta3-emqxenterprise"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "apps.emqx.io/v2alpha1, Kind=EMQX", "path": "/validate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-apps-emqx-io-v2alpha1-emqx"}
2022-10-05T09:07:23Z INFO setup starting manager
2022-10-05T09:07:23Z INFO controller-runtime.webhook.webhooks Starting webhook server
2022-10-05T09:07:23Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2022-10-05T09:07:23Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
2022-10-05T09:07:23Z INFO controller-runtime.certwatcher Updated current TLS certificate
I1005 09:07:23.976315 1 leaderelection.go:248] attempting to acquire leader lease 5g-sc/19fd6fcc.emqx.io...
2022-10-05T09:07:23Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
2022-10-05T09:07:23Z INFO controller-runtime.certwatcher Starting certificate watcher
2022/10/05 09:07:32 http: TLS handshake error from 10.130.5.86:41042: remote error: tls: bad certificate
2022/10/05 09:07:33 http: TLS handshake error from 10.130.5.86:41050: remote error: tls: bad certificate
2022/10/05 09:07:35 http: TLS handshake error from 10.130.5.86:41060: remote error: tls: bad certificate
2022/10/05 09:07:38 http: TLS handshake error from 10.130.5.86:41068: remote error: tls: bad certificate
2022/10/05 09:07:39 http: TLS handshake error from 10.130.5.86:44646: remote error: tls: bad certificate
2022/10/05 09:07:41 http: TLS handshake error from 10.130.5.86:44650: remote error: tls: bad certificate
2022/10/05 09:07:54 http: TLS handshake error from 10.130.5.86:37628: remote error: tls: bad certificate
2022/10/05 09:07:55 http: TLS handshake error from 10.130.5.86:37634: remote error: tls: bad certificate
I am running out of ideas to what to check next.
Was emqx run on Openshift before?
Is there a way to skip ssl verification on the emqx ("insecure" flag)?
Any other ideas?
Kind regards
按照用户说明操作在 aws(亚马逊**宁夏区) eks 集群上跑不起来,
请问能不能提供一个在eks上运行的 demo
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
emqx-0 0/1 CrashLoopBackOff 3 112s 192.168.75.204 ip-192-168-88-157.cn-northwest-1.compute.internal <none> <none>
emqx-1 0/1 Error 4 112s 192.168.24.100 ip-192-168-21-143.cn-northwest-1.compute.internal <none> <none>
emqx-2 0/1 CrashLoopBackOff 3 112s 192.168.39.78 ip-192-168-50-143.cn-northwest-1.compute.internal <none> <none>
emqx-0 0/1 Running 4 2m1s 192.168.75.204 ip-192-168-88-157.cn-northwest-1.compute.internal <none> <none>
emqx-1 0/1 CrashLoopBackOff 4 2m4s 192.168.24.100 ip-192-168-21-143.cn-northwest-1.compute.internal <none> <none>
emqx-0 0/1 Error 4 2m5s 192.168.75.204 ip-192-168-88-157.cn-northwest-1.compute.internal <none> <none>
emqx-2 0/1 Running 4 2m6s 192.168.39.78 ip-192-168-50-143.cn-northwest-1.compute.internal <none> <none>
emqx-2 0/1 Error 4 2m11s 192.168.39.78 ip-192-168-50-143.cn-northwest-1.compute.internal <none> <none>
emqx-0 0/1 CrashLoopBackOff 4 2m16s 192.168.75.204 ip-192-168-88-157.cn-northwest-1.compute.internal <none> <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m33s default-scheduler Successfully assigned default/emqx-1 to ip-192-168-21-143.cn-northwest-1.compute.internal
Normal SuccessfulAttachVolume 7m31s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ba0d0dbc-7251-4f37-a6de-e493eafe6a26"
Normal SuccessfulAttachVolume 7m31s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-08f7c78e-e0e7-4b28-a64c-d7362c96fa95"
Normal Pulled 5m48s (x5 over 7m28s) kubelet Container image "emqx/emqx:4.4.1" already present on machine
Normal Created 5m48s (x5 over 7m28s) kubelet Created container emqx
Normal Started 5m48s (x5 over 7m28s) kubelet Started container emqx
Warning Unhealthy 5m43s (x3 over 7m23s) kubelet Readiness probe failed: Get "http://192.168.24.100:8081/status": dial tcp 192.168.24.100:8081: connect: connection refused
Warning BackOff 2m22s (x22 over 7m17s) kubelet Back-off restarting failed container
I am trying to use the operator to accomplish setting up an ingress controller so I can outbound access the mqtt broker and dashboard.
I am using Azure AKS with pretty much the default setup from the EQMX website.
https://docs.emqx.com/en/enterprise/v4.4/getting-started/install-on-k8s.html
Because ingress is setup to natively take on http traffic you have to modify the controller to handle TCP traffic such as MQTT 1883.
This stack has a pretty good overview on how you should do this.
https://stackoverflow.com/questions/61430311/exposing-multiple-tcp-udp-services-using-a-single-loadbalancer-on-k8s
The problem is, I don't really have any documentation on how I can setup and edit the custom resource. Are there specific instructions for this?
When I try to make an edit I am always getting unknown field
use validate=false. So I don't think that is the right way I am trying to edit the implied intention of what it is I want to do.
I am trying to use a configmap to map the tcp setting to the container pods. or in anyway map the ports from the ingress controller.
An example of setting up a configmap is here. https://emqx.medium.com/building-k8s-cluster-of-emq-x-mqtt-broker-starting-from-scratch-fe96ad95e563
But I don't really see a way to do that on my container installation from this operator.
When I add this to the kubectl apply I get the following error.
apiVersion: apps.emqx.io/v1beta3
kind: EmqxEnterprise
metadata:
name: emqx-ee
labels:
"foo": "bar"
spec:
emqxTemplate:
image: emqx/emqx-ee:4.4.7
envFrom:
- configMapRef:
name: emqx-ee-config
error validating data: ValidationError(EmqxEnterprise.spec.emqxTemplate): unknown field "envFrom" in io.emqx.apps.v1beta3.EmqxEnterprise.spec.emqxTemplate; if you choose to ignore these errors, turn validation off with --validate=false
Admittedly, I am new to kubernetes so if you could help me with the issue it would be greatly appreciated.
Describe the bug
A clear and concise description of what the bug is.
when i use this yaml create emqx-ee,could not create successfully
#emqx-ee.yaml
apiVersion: apps.emqx.io/v1beta2
kind: EmqxEnterprise
metadata:
name: emqx-ee
spec:
serviceAccountName: "emqx"
image: emqx/emqx-ee:4.4.0
replicas: 3
labels:
cluster: emqx
storage:
storageClassName: hostpath
resources:
requests:
storage: 20Mi
accessModes:
- ReadWriteOnce
env:
- name: "EMQX_LOG__LEVEL"
value: "debug"
emqxTemplate:
# acl: []
plugins:
- name: emqx_management
enable: true
- name: emqx_recon
enable: true
- name: emqx_retainer
enable: true
- name: emqx_dashboard
enable: true
- name: emqx_telemetry
enable: true
- name: emqx_rule_engine
enable: true
- name: emqx_bridge_mqtt
enable: false
# modules:
#
# - name: internal_acl
# enable: true
# configs:
# acl_rule_file: etc/acl.conf
# - name: presence
# enable: true
# configs:
# qos: 0
# - name: recon
# enable: true
# configs: {}
# - name: retainer
# enable: true
# configs:
# expiry_interval: 0
# max_payload_size: 1MB
# max_retained_messages: 0
# storage_type: ram
listener:
type: NodePort
ports:
mqtt: 1883
mqtts: 8883
ws: 8083
wss: 8084
dashboard: 18083
api: 8081
To Reproduce
kubeclt apply -f emqx-ee.yaml
Anything else we need to know?:
Environment details::
Is your feature request related to a problem? Please describe.
I'd like to be able to add an annotation to the Service
that the EmqxBroker
generates such as: coredns.io/hostname: "mqtt.example.com"
. This annotation specifically in my case causes coredns to add a DNS record resolving mqtt.example.com
. to the ip address of the service.
Describe the solution you'd like
I think the spec.emqxTemplate.listener
object on the EmqxBroker
should support an annotations
field which would then be used when generating the Service
around here.
Describe alternatives you've considered
Alternatives would require that my load balancer ip is static and then manually adding the DNS record to core dns.
Or manually crafting a custom Service
to run side by side with the one produced by the operator and effectively just leave the generated Service
un-used.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.